I get a tensor of 600 values instead of 3 values for mean and std

Mona_Jalal: <p>I am trying to Normalize my images data and for that I need to find the mean and std for train_loader.</p>
<pre><code class="lang-auto">mean = 0.0
std = 0.0
nb_samples = 0.0
for data in train_loader:
    images, landmarks = data["image"], data["landmarks"]
    batch_samples = images.size(0)
    images_data = images.view(batch_samples, images.size(1), -1)
    mean +=  torch.Tensor.float(images_data).mean(2).sum(0)
    std += torch.Tensor.float(images_data).std(2).sum(0)
    ###mean += images_data.mean(2).sum(0)
    ###std += images_data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples
</code></pre>
<p>the mean and std here are each a torch.Size([600])</p>
<p>When I tried (almost) same code on dataloader, it worked as expected:</p>
<pre><code class="lang-auto"># code from https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560/6?u=mona_jalal
mean = 0.0
std = 0.0
nb_samples = 0.0
for data in dataloader:
    images, landmarks = data["image"], data["landmarks"]
    batch_samples = images.size(0)

    images_data = images.view(batch_samples, images.size(1), -1)
    mean += images_data.mean(2).sum(0)
    std += images_data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples
</code></pre>
<p>and I got:<br>
<code>mean is: tensor([0.4192, 0.4195, 0.4195], dtype=torch.float64), std is: tensor([0.1182, 0.1184, 0.1186], dtype=torch.float64)</code></p>
<p>So my dataloader is:</p>
<pre><code class="lang-auto">class MothLandmarksDataset(Dataset):
    """Face Landmarks dataset."""

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.landmarks_frame = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.landmarks_frame)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        img_name = os.path.join(self.root_dir, self.landmarks_frame.iloc[idx, 0])
        image = io.imread(img_name)
        landmarks = self.landmarks_frame.iloc[idx, 1:]
        landmarks = np.array([landmarks])
        landmarks = landmarks.astype('float').reshape(-1, 2)
        sample = {'image': image, 'landmarks': landmarks}

        if self.transform:
            sample = self.transform(sample)

        return sample

transformed_dataset = MothLandmarksDataset(csv_file='moth_gt.csv',
                                           root_dir='.',
                                           transform=transforms.Compose(
                                               [
                                               Rescale(256),
                                               RandomCrop(224),
                                               
                                               ToTensor()      
                                               ]
                                                                        )
                                           )



dataloader = DataLoader(transformed_dataset, batch_size=3,
                        shuffle=True, num_workers=4)
</code></pre>
<p>and train_loader is:</p>
<pre><code class="lang-auto"># Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
seed = 42
np.random.seed(seed)
torch.manual_seed(seed)

# split the dataset into validation and test sets
len_valid_set = int(0.1*len(dataset))
len_train_set = len(dataset) - len_valid_set

print("The length of Train set is {}".format(len_train_set))
print("The length of Test set is {}".format(len_valid_set))

train_dataset , valid_dataset,  = torch.utils.data.random_split(dataset , [len_train_set, len_valid_set])

# shuffle and batch the datasets
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8, shuffle=True, num_workers=4)
test_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=8, shuffle=True, num_workers=4)
</code></pre>
<p>Please let me know if more information is needed.</p>
<p>I basically need to get 3 values for mean of train_loader and 3 values for std of train_loader to use as args for Normalize.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff8513cf33f9c1e0053f7e6f7c77ee0289245364.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff8513cf33f9c1e0053f7e6f7c77ee0289245364" title="train_loader_normalization"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff8513cf33f9c1e0053f7e6f7c77ee0289245364_2_690x370.png" alt="train_loader_normalization" data-base62-sha1="AsqS3J1Fn5YCXWpPE9dVrsdZQZC" width="690" height="370" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff8513cf33f9c1e0053f7e6f7c77ee0289245364_2_690x370.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff8513cf33f9c1e0053f7e6f7c77ee0289245364_2_1035x555.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff8513cf33f9c1e0053f7e6f7c77ee0289245364_2_1380x740.png 2x" data-dominant-color="EAECEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">train_loader_normalization</span><span class="informations">1812×972 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>images_data in dataloader is torch.Size([3, 3, 50176]) inside the loop and images_data in train_loader is torch.Size([8, 600, 2400])</p>
<p>8 in torch.Size([8, 600, 2400]) for train_loader refers to the batch_size I have set for trainset</p>

Mona_Jalal: <p>Most probably this is a very dumb answer but seems to do the job. Could someone please verify if this answer makes sense?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/153d8e60c81dca8cb0d4d5e569733072f1a44fc6.png" data-download-href="https://discuss.pytorch.org/uploads/default/153d8e60c81dca8cb0d4d5e569733072f1a44fc6" title="mean_std"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/153d8e60c81dca8cb0d4d5e569733072f1a44fc6_2_690x419.png" alt="mean_std" data-base62-sha1="31TUroDU5O7fMmQHkBwXMZPo9T0" width="690" height="419" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/153d8e60c81dca8cb0d4d5e569733072f1a44fc6_2_690x419.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/153d8e60c81dca8cb0d4d5e569733072f1a44fc6_2_1035x628.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/153d8e60c81dca8cb0d4d5e569733072f1a44fc6_2_1380x838.png 2x" data-dominant-color="EFF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mean_std</span><span class="informations">2442×1486 297 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Mona_Jalal: <p>I also used a little less dumb method (the values are slightly different) - Does this approach sound reasonable?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/146ba9709a960370e18503ff093db12f7a28b843.png" data-download-href="https://discuss.pytorch.org/uploads/default/146ba9709a960370e18503ff093db12f7a28b843" title="Screenshot_2020-10-14_23-21-42"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/146ba9709a960370e18503ff093db12f7a28b843_2_690x289.png" alt="Screenshot_2020-10-14_23-21-42" data-base62-sha1="2UEdgh1nchO8VDg2A3OOyl5gUIb" width="690" height="289" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/146ba9709a960370e18503ff093db12f7a28b843_2_690x289.png, https://discuss.pytorch.org/uploads/default/original/3X/1/4/146ba9709a960370e18503ff093db12f7a28b843.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/4/146ba9709a960370e18503ff093db12f7a28b843.png 2x" data-dominant-color="E9E9E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2020-10-14_23-21-42</span><span class="informations">1016×426 41 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

fadetoblack: <p>Hi, I’m sure this can be done using pytorch as well (without the use of numpy).</p>
<p>Can you please check if the shape of <code>images_data</code> is same in both the code snippets for <code>train_loader</code> and <code>data_loader</code> i.e.,</p>
<pre><code class="lang-auto">mean = 0.0
std = 0.0
nb_samples = 0.0
for data in train_loader:
    images, landmarks = data["image"], data["landmarks"]
    batch_samples = images.size(0)
    images_data = images.view(batch_samples, images.size(1), -1)  #HERE!
    mean +=  torch.Tensor.float(images_data).mean(2).sum(0)
    std += torch.Tensor.float(images_data).std(2).sum(0)
    ###mean += images_data.mean(2).sum(0)
    ###std += images_data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples
</code></pre>
<p>and</p>
<pre><code class="lang-auto">mean = 0.0
std = 0.0
nb_samples = 0.0
for data in dataloader:
    images, landmarks = data["image"], data["landmarks"]
    batch_samples = images.size(0)

    images_data = images.view(batch_samples, images.size(1), -1)  #HERE!
    mean += images_data.mean(2).sum(0)
    std += images_data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples
</code></pre>

Mona_Jalal: <p>no it is not dataloader is the entire dataset and train_loader is just the train set. My focus is on train_loader</p>
<p>When I do that I get a tensor of 600 values for mean and same for std. I only should have 3 values for each<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d98576888e7970f201bb1da6850220932e609dc.png" data-download-href="https://discuss.pytorch.org/uploads/default/7d98576888e7970f201bb1da6850220932e609dc" title="Screenshot_2020-10-14_23-30-55"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d98576888e7970f201bb1da6850220932e609dc_2_690x407.png" alt="Screenshot_2020-10-14_23-30-55" data-base62-sha1="hV44zOARN9ysQEj0AZ1ZGcfR1cE" width="690" height="407" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d98576888e7970f201bb1da6850220932e609dc_2_690x407.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d98576888e7970f201bb1da6850220932e609dc_2_1035x610.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d98576888e7970f201bb1da6850220932e609dc_2_1380x814.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2020-10-14_23-30-55</span><span class="informations">2534×1496 380 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

fadetoblack: <p>Yes, but do both shapes look the same i.e., are both in the format such as <code>[n_images, height, width, channels]</code> where the value of <code>n_images</code> varies between data_loader and train_loader?</p>
<p>Can you please tell us the shapes of <code>images_data</code> in <code>data_loader</code> and <code>train_loader</code>? This is because we can reproduce your error by generating random data of those shapes. Thank you.</p>