Loading 2 classes from FashionMNIST,

biscoff: <p>Hello</p>
<p>I have an assignment due and I am positively freaking out, I have spent 5 hours trying to load 2 classes in from the FashionMNIST dataset, but I simply cannot figure it out.</p>
<p>I have tried</p>
<pre><code class="lang-auto">idx = torch.as_tensor(trainset.targets) == 1
idx += torch.as_tensor(trainset.targets) == 7
dset_train = torch.utils.data.dataset.Subset(trainset, np.where(idx==1)[0])
</code></pre>
<p>this</p>
<p>but is that alone enough?</p>
<p>I use it to then load a batch of 8 images from the trouser/sneaker classes, which ends up working, but THEN I try applying StandardScaler() or PCA, and it will not work.</p>
<p>It states that ‘only one element tensors can be converted to Python scalars’ and if I try using the subset in any code, it throws attribute errors e.g. AttributeError: ‘Subset’ object has no attribute ‘numpy’</p>
<p>I am trying to apply PCA to only 2 classes (trouser/sneaker) from the FashionMNIST dataset.</p>
<p>Help please.</p>

ptrblck: <aside class="quote no-group" data-username="biscoff" data-post="1" data-topic="110518">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/7bcc69/48.png" class="avatar"> biscoff:</div>
<blockquote>
<p>but is that alone enough?</p>
</blockquote>
</aside>
<p>Yes, it should be enough to only select the samples with class1 and class7 as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">idx = torch.as_tensor(dataset.targets) == 1
idx += torch.as_tensor(dataset.targets) == 7
dset_train = torch.utils.data.dataset.Subset(dataset, np.where(idx==1)[0])
loader = torch.utils.data.DataLoader(dset_train, batch_size=64)

for data, target in loader:
    print(target)
</code></pre>
<aside class="quote no-group" data-username="biscoff" data-post="1" data-topic="110518">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/7bcc69/48.png" class="avatar"> biscoff:</div>
<blockquote>
<p>but THEN I try applying StandardScaler() or PCA, and it will not work.</p>
</blockquote>
</aside>
<p>How are you trying to apply the <code>StandardScaler</code>?<br>
As described in the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html">docs</a> you would need to <code>fit</code> it first and can then apply it on numpy arrays.<br>
Passing a <code>Dataset</code> will most likely not work.</p>

biscoff: <p>I applied it like this</p>
<pre><code class="lang-auto">scaler = preprocessing.StandardScaler()

scaleddset = scaler.fit_transform(dset_train)
</code></pre>
<p>but then it gives me the 'only one element tensors can be converted to Python scalars’ error so… I just assumed it was because of the way I was loading the data…?</p>

biscoff: <p>How would I go about fitting it first and then applying on numpy arrays?  ):</p>

biscoff: <p>Sorry about this - also, when I try to convert the dset_train to a numpy array, it gives me the ‘Subset’ object has no attribute ‘numpy’’ error ?</p>
<p>I have also tried converting the trainset to numpy array before making subset, but that did not work either. It gives me a ‘ValueError: too many values to unpack (expected 2)’ error.</p>

ptrblck: <p><code>StandardScaler</code> expects a numpy array, not a <code>torch.utils.data.Dataset</code> as its input.<br>
The <code>FashionMNIST</code> dataset stores the data as a numpy array in its internal <code>.data</code> attribute.<br>
Since you want to apply <code>scikit-learn</code> preprocessing methods to it, I would recommend to write a custom <code>Dataset</code> by deriving from <code>FashionMNIST</code> and to <code>fit_transform</code> the <code>StandardScaler</code> in the <code>__init__</code> method and <code>transform</code> the data in the <code>__getitem__</code>.<br>
If you want to use the same scaler for the training and validation (and test) datasets, you could create the object once and pass it to the custom dataset implementations.</p>

biscoff: <p>I don’t really know how to go about writing a custom Dataset with those things sdfhksjdg</p>
<p>I am only using the training data for the visualisation with the PCA</p>

ptrblck: <p>You could take a look at <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a>, which shows how to write a custom <code>Dataset</code>. Since this task is an assignment, I’m not comfortable providing the code here.</p>

biscoff: <p>Been trying but still have yet to get it working properly ): thank you though</p>

biscoff: <p>i have gotten this far:</p>
<pre><code class="lang-auto">class mydataset(Dataset):
    def __init__(self, features, labels):
        self.features = features
        self.labels = labels
        
    def __len__(self):
        return len(self.features)
    
    def __getitem__(self, index):
        image = self.features[idx]
        image = image.numpy()
        label = self.labels[idx]
        if label == 1:
            return image, label
        elif label == 7:
            return image, label
</code></pre>
<p>i’m not sure the last bit is right (the if statement)<br>
and it states that image does not have a numpy() attribute</p>