How to save .npy with the same name of the image source?

nima_pw: <p>Hi,</p>
<p>I have a <code>custom Dataset</code> and <code>Dataloader</code> that work fine:<br>
My image’s names are the labels, so I must extract them as targets.<br>
train_dir contains train images:<br>
<code>train_dir = "./Train"</code><br>
<code>train_files = glob.glob(train_dir + '/*.jpg')</code></p>
<p>My dataset contains about 30,000 images.<br>
I have a function (extract_single) that extracts features from every image. I want to save them as <code>.npy</code> and feed them to my CNN model (instead of the images). Because extracting features in every training makes the training time very long</p>
<p>1- How can I save the .npy file with the same name as the original input file? So the input file is called <code>D01_nature.jpg</code> and the output file should be called <code>D01_nature.npy</code>.</p>
<p>I know the following code is not correct:</p>
<pre><code class="lang-auto">train_files = glob.glob(train_dir + '/*.jpg')
transform = transforms.Lambda(extract_single)
for i in range(0, len(train_files)):
    x = Image.open(train_files[i])
    y = transform(x)
    np.save('./train_npy/'+ str(image_name.npy), x)
</code></pre>
<p>2- When I have <code>.npy</code> files instead <code>.jpg</code>, do these data need <code>transforms.Normalize</code> yet? If yes, How do I calculate <code>mean</code> and <code>std</code>? Images are 3 channels, but the<code> .npy</code> file is 1 channel)</p>
<p>3- I want to use the <code>random crop</code> for training inputs, do I have to use it for validation data as well? for example if my inputs are 256*256, can I use <code>transforms.Resize</code><br>
to have the size of 256 for validation data or should I use the same <code>random crop</code> for better accuracy?</p>

tataganesh: <ol>
<li>The first step is to validate if your images are read correctly and the features are extracted properly (Try to validate this for a few images). Once it is verified that the code works fine, use <code>np.save</code> to save the features.<br>
For this, you need to extract the file name from each path <code>train_files[i]</code> and modify it. Here is one way to do it -</li>
</ol>
<pre><code class="lang-auto">import os
train_files = glob.glob(train_dir + '/*.jpg')
transform = transforms.Lambda(extract_single)
for i in range(0, len(train_files)):
    x = Image.open(train_files[i])
    y = transform(x)
    file_name = os.path.basename(train_files[i])[:-4] # Get file name and remove extension
    ftr_file_name = file_name + ".npy" # Add new extension
    np.save(f'./train_npy/{ftr_file_name}', y) # Use f-string to specify file path
</code></pre>
<ol start="2">
<li>
<p>Can you extract the features from the normalized images? That will help you to standardize the preprocessing step.  Otherwise, you can still use <code>transforms.Normalize</code> and specify the <code>mean</code> and <code>std</code> for just one channel instead of 3 (provided the extracted features have shape <code>H x W x 1</code>, i.e. they represent an image).</p>
</li>
<li>
<p>Usually, random crop is only applied to the training data since the validation and test data should not be augmented. The validation dataset contains unseen samples from the true data distribution, so it should not be modified using random crop. Instead, you can use random crop with padding for the training set so that you do not need to resize images in the validation set. Refer to the train and test transforms mentioned <a href="https://github.com/kuangliu/pytorch-cifar/blob/49b7aa97b0c12fe0d4054e670403a16b6b834ddd/main.py#L29-L40" rel="noopener nofollow ugc">here</a> for more details.</p>
</li>
</ol>

nima_pw: <p>Dear <a class="mention" href="/u/tataganesh">@tataganesh</a>, thank you for your great explanations.</p>
<p>1- Thanks for your code to extract the names of the images. I’ll try it.</p>
<p>2- How can I extract features from normalized images? Because transformation is something like this:</p>
<pre><code class="lang-auto">my_transform = transforms.Compose([
    transforms.Lambda(extract_single),
    transforms.ToTensor(),
    transforms.Normalize([0.4786, 0.4728, 0.4528], [0.2425, 0.2327, 0.2564])
])
</code></pre>
<p>I mean, <code>transforms.Normalize</code> is the last step. So how can it affect <code>transforms.Lambda(extract_single)</code> step? It’s after feature extraction. Changing the order of the transforms is correct?</p>
<pre><code class="lang-auto">my_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize([0.4786, 0.4728, 0.4528], [0.2425, 0.2327, 0.2564]),
    transforms.Lambda(extract_single)
])
</code></pre>
<p>How can I specify the <code>mean</code> and <code>std</code> for just one channel? There are 3 numbers for <code>mean</code> and <code>std.</code> Which number must be chosen for one channel?</p>
<p>3- My images are not the same size, so is padding applicable yet?</p>
<p>4- If I should extract features from the validation images? I mean <code>extract_single</code> function should be used for valid data?</p>

tataganesh: <ul>
<li>
<p>Changing the order of transforms is fine as long as you are okay with extracting features from the normalized image. If you want to normalize the extracted features, you need to calculate the mean and stdev of the features after performing feature extraction for all images, then normalize them.</p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/61637447/validation-set-augmentations-pytorch-example" rel="noopener nofollow ugc">This</a> is a good example of performing resize + crop if your images are not the same size.</p>
</li>
<li>
<p>Yes, you should use <code>extract_single</code> function to extract features from the validation dataset.  You should ensure that no <code>random</code> augmentations are applied to the validation set, since the randomness is only required for the training set.</p>
</li>
</ul>