Mean and Std() for a custom dataset(olivetti)

the_coder: <p>Each of the image has got 64 * 64 dimension. The image is grayscale, so channel =1,  I converted the images to TensorDataset which gives me back size of ([1,64,64]) and then i pass this to Dataloader which gives me dimension ([1,1,64,64]) Now if i want to find the mean and std() the following code generates error:</p>
<blockquote>
<pre><code>mean = 0.
std = 0.
nb_samples = 0.
for data in loader:
    batch_samples = data.size(0)
   data = data.view(batch_samples, data.size(1), -1)
    mean += data.mean(2).sum(0)
     std += data.std(2).sum(0)
    nb_samples += batch_samples
</code></pre>
<p>mean /= nb_samples<br>
std /= nb_samples</p>
</blockquote>
<p>It gives me the following error :</p>
<blockquote>
<p>AttributeError: ‘list’ object has no attribute ‘size’</p>
</blockquote>
<p>ref: <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Now where should i run the loop ? In Dataloader object or in the image itself?</p>
<p>My second question is after i find the mean and std dev what is the code to apply the transorms operation, since it is grayscale , will the code be like the following :</p>
<blockquote>
<pre><code>train_transform = transforms.compose(transforms.Normalize ([mean,mean,mean],[std,std,std]))
</code></pre>
</blockquote>
<p>or should it be like this, as it is a grayscale dataset.</p>
<blockquote>
<pre><code>train_transform = transforms.compose(transforms.Normalize ([mean],[std]))
</code></pre>
</blockquote>
<p>And my third question is how can i apply this mean and std() to my dataset , since it is a custom dataset i will not be able to follow instruction where the datasets are imported from torchvision.</p>

ptrblck: <p>Your <code>DataLoader</code> might return a list, not a tensor, so you would have to index <code>data</code> before calling <code>size(0)</code> on it.</p>
<p>Also, as mentioned in the thread, the <code>var</code> calculation should yield a better result that the <code>std</code>, since you would now be calculating the average of the stddevs.</p>
<aside class="quote no-group" data-username="the_coder" data-post="1" data-topic="88793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_coder/48/19504_2.png" class="avatar"> the_coder:</div>
<blockquote>
<p>Now where should i run the loop ? In Dataloader object or in the image itself?</p>
</blockquote>
</aside>
<p>You should run this calculation once before your setup of the training procedure.</p>
<aside class="quote no-group" data-username="the_coder" data-post="1" data-topic="88793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_coder/48/19504_2.png" class="avatar"> the_coder:</div>
<blockquote>
<p>My second question is after i find the mean and std dev what is the code to apply the transorms operation</p>
</blockquote>
</aside>
<p>Since your images are grayscale and will thus have a single channel by default, you should use the second approach.</p>
<aside class="quote no-group" data-username="the_coder" data-post="1" data-topic="88793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_coder/48/19504_2.png" class="avatar"> the_coder:</div>
<blockquote>
<p>And my third question is how can i apply this mean and std() to my dataset , since it is a custom dataset i will not be able to follow instruction where the datasets are imported from torchvision.</p>
</blockquote>
</aside>
<p>I don’t understand this question completely.<br>
If you are using a custom <code>Dataset</code> (or e.g. <code>ImageFolder</code>), you would only have to apply the <code>Normalize</code> transformation using your calculated stats.<br>
Could you explain a bit, which instructions you won’t be able to follow?</p>

the_coder: <p>I indexed my list of Tensor as follows :<br>
The Dataloader object has 2 lists. One is the images and another list is the labels. In order to find the mean and standard deviation , i need to go through the batches and have mean and std() for each batch.<br>
Am i indexing it correctly? pls see below. i named it T in the following code. When i print T it gives me back the tensors, But i could not perform T.size on it, it gives me the error,</p>
<p>AttributeError: ‘list’ object has no attribute ‘size’</p>
<pre><code>for data in train_loader:
    T = [data[0]]
print(T)
[tensor([[[[0.3264, 0.1983, 0.2149,  ..., 0.1405, 0.1281, 0.1240],
      [0.2273, 0.2603, 0.2479,  ..., 0.1240, 0.1198, 0.0992],
      [0.2810, 0.2893, 0.2479,  ..., 0.1198, 0.1198, 0.1074],
      ...,
      [0.7975, 0.8182, 0.8058,  ..., 0.2893, 0.1983, 0.0992],
      [0.8058, 0.8347, 0.8017,  ..., 0.3140, 0.2273, 0.1033],
      [0.8223, 0.8430, 0.8099,  ..., 0.3182, 0.2562, 0.1157]]],


    [[[0.4298, 0.4587, 0.5496,  ..., 0.4959, 0.4959, 0.4959],
      [0.4421, 0.4545, 0.5331,  ..., 0.4917, 0.4917, 0.4959],
      [0.4669, 0.4587, 0.5207,  ..., 0.4835, 0.4876, 0.4917],
      ...,
      [0.4421, 0.4339, 0.4380,  ..., 0.3636, 0.3802, 0.3802],
      [0.4421, 0.4339, 0.4380,  ..., 0.3802, 0.3802, 0.3760],
      [0.4421, 0.4380, 0.4339,  ..., 0.3843, 0.3843, 0.3719]]],


    [[[0.1240, 0.0992, 0.1240,  ..., 0.1983, 0.1818, 0.1653],
      [0.1157, 0.1033, 0.1281,  ..., 0.1901, 0.1570, 0.1488],
      [0.1074, 0.1074, 0.1240,  ..., 0.1694, 0.1488, 0.1364],
      ...,
      [0.9008, 0.6901, 0.2810,  ..., 0.5620, 0.6488, 0.4959],
      [0.9132, 0.6983, 0.3099,  ..., 0.5455, 0.6198, 0.5744],
      [0.9091, 0.7562, 0.2851,  ..., 0.4793, 0.6198, 0.6157]]]
</code></pre>
<p>What i meant by the third question is that usually when we download dataset from torchvision there is a parameter called transform where we can pass in ‘transformation’ variable which includes the mean and std() values, as a result it goes through the whole dataset and normalizes it. My question is after i calculate mean and std() for my custom dataset how do i apply it in order to normalize my data so that it goes through for all the batches and normalizes it ?</p>
<pre><code>transformation = transforms.Compose([transforms.ToTensor(),
                               transforms.Normalize((0.1307,),(0.3081,))])
train_dataset = torchvision.datasets.MNIST('./data',train=True,transform= transformation, download= True)</code></pre>

the_coder: <p>I was able to apply transformation on a downloaded images(custom dataset) , But even though i applied the mean and standard deviation i am still getting test accuracy 0. While training the loss is decreasing, and after about 20 epoch it is 0. So i dont understand why i am getting 0% accuracy for the test set. Is my transformation code is correct?    i had to define the downloaded data twice (‘data’) becuase if i dont define it in the __get__item it will throw an error. If i don’t define it in <strong>init</strong> i will not be able to add the channel dimension by unsqueezing, since the dataset comes as 400<em>64</em>64 format. Pls have a look at the code and tell me if u find any bug.</p>
<pre><code>transformations = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5444,),(0.1581,))])
class dataset_transformed():

def __init__(self,transform=None):
    data = np.load("./Downloads/11928_16420_bundle_archive/olivetti_faces.npy")
    target = np.load("./Downloads/11928_16420_bundle_archive/olivetti_faces_target.npy")
    self.x = torch.from_numpy(data)
   
   
    self.x = self.x.unsqueeze(1)
   
    self.y = torch.from_numpy(target)
    self.y = self.y.type(torch.LongTensor)
   
    self.transform = transform
   
def __getitem__(self,index):
    data = np.load("./Downloads/11928_16420_bundle_archive/olivetti_faces.npy")
   
   
    if self.transform is not None:
        
        data = self.transform(data)
        
       
        
   
    return self.x[index],self.y[index]

def __len__(self):
    
    return self.n_samples

 dataset = dataset_transformed(transformations) #calling the class</code></pre>

ptrblck: <p>The code might be correct, but since I don’t have the data you could check some samples and make sure they are properly normalized.</p>
<aside class="quote no-group" data-username="the_coder" data-post="4" data-topic="88793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_coder/48/19504_2.png" class="avatar"> the_coder:</div>
<blockquote>
<p>While training the loss is decreasing, and after about 20 epoch it is 0. So i dont understand why i am getting 0% accuracy for the test set. Is my transformation code is correct?</p>
</blockquote>
</aside>
<p>A 0% accuracy during testing might not necessarily be created by a wrong transformation.<br>
Since you would expect to get at least the random accuracy, I would suggest to print some predictions as well as the targets and check, if the accuracy is really 0 or if you are calculating the accuracy in a wrong way.</p>

SuleNas: <p>Please i need a guide on how to calculate the mean and std of a dataset with images of different sizes, thanks</p>

1chimaruGin: <pre><code class="lang-auto">train_loader = DataLoader(dataset=train_set, batch_size=64, shuffle=True)

def get_mean_std(loader):
    # var[X] = E[X**2] - E[X]**2
    channels_sum, channels_sqrd_sum, num_batches = 0, 0, 0

    for data, _ in tqdm(loader):
        channels_sum += torch.mean(data, dim=[0, 2, 3])
        channels_sqrd_sum += torch.mean(data ** 2, dim=[0, 2, 3])
        num_batches += 1

    mean = channels_sum / num_batches
    std = (channels_sqrd_sum / num_batches - mean ** 2) ** 0.5

    return mean, std


mean, std = get_mean_std(train_loader)
print(mean)
print(std)
</code></pre>

tlim: <p>Just my opinion:</p>
<p>Use torch.transforms to resize to same size for all images (just like your training), then find the mean and std.</p>