IndexError: index 1 is out of bounds for axis 0 with size 1

lavender99: <p>hi everyone<br>
sorry for asking too much , i ma gitting this error and i do not seem to understand what does it mean</p>
<pre><code class="lang-auto"> for i in range(crop_num1):
        for j in range(crop_num2):
            # add image to empty_array at specific position
            empty_array[dim1_stride*i:dim1_stride*i + crop_size,
                        dim2_stride*j:dim2_stride*j + crop_size] += image[index]
            index += 1
    return empty_array
</code></pre>
<p>it happend at this part and this part</p>
<pre><code class="lang-auto">def save_prediction_image(stacked_img, im_name, epoch, save_folder_name="result_images", save_im=True):
    """save images to save_path
    Args:
        stacked_img (numpy): stacked cropped images
        save_folder_name (str): saving folder name
    """
    div_arr = division_array(512, 2, 2, 512, 512)
    img_cont = image_concatenate(stacked_img.cpu().data.numpy(), 2, 2, 512, 512)
    img_cont = polarize((img_cont)/div_arr)*255
    img_cont_np = img_cont.astype('uint8')
    img_cont = Image.fromarray(img_cont_np)
    # organize images in every epoch
    desired_path = save_folder_name + '/epoch_' + str(epoch) + '/'
    # Create the path if it does not exist
    if not os.path.exists(desired_path):
        os.makedirs(desired_path)
    # Save Image!
    export_name = str(im_name) + '.png'
    img_cont.save(desired_path + export_name)
    return img_cont_np
</code></pre>

ptrblck: <p>Which method calls the nested loop?<br>
Based on the error message, it seems dim0 of <code>empty_array</code> has only a size of 1.<br>
Could you check its shape before passing it to the nested loop?</p>

lavender99: <p>The shape is (512, 512)</p>
<pre><code class="lang-auto">def image_concatenate(image, crop_num1, crop_num2, dim1, dim2):
   """concatenate images
   Args :
       image : output images (should be square)
       crop_num2 (int) : number of crop in horizontal way (2)
       crop_num1 (int) : number of crop in vertical way (2)
       dim1(int) : vertical size of output (512)
       dim2(int) : horizontal size_of_output (512)
   Return :
       div_array : numpy arrays of numbers of 1,2,4
   """
   crop_size = image.shape[1]  # size of crop
   empty_array = np.zeros([dim1, dim2]).astype("float64")  # to make sure no overflow
   print(empty_array.shape)

   dim1_stride = stride_size(dim1, crop_num1, crop_size)  # vertical stride
   dim2_stride = stride_size(dim2, crop_num2, crop_size)  # horizontal stride
   index = 0
   for i in range(crop_num1):
       for j in range(crop_num2):
           # add image to empty_array at specific position
           empty_array[dim1_stride*i:dim1_stride*i + crop_size,
                       dim2_stride*j:dim2_stride*j + crop_size] += image[index]
           index += 1
   return empty_array
</code></pre>

ptrblck: <p>Could you check the shape of <code>image</code> then?</p>

lavender99: <p>the shape of image is (1, 512, 512)</p>

ptrblck: <p>So your code will crash in the second iteration, since you are increasing <code>index</code> by 1. Based on your code it looks like <code>image</code> should have at least <code>crop_num1*crop_num2</code> length.</p>

lavender99: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> should it be like this</p>
<pre><code class="lang-auto">def image_concatenate(image, crop_num1, crop_num2, dim1, dim2):
    """concatenate images
    Args :
        image : output images (should be square)
        crop_num2 (int) : number of crop in horizontal way (2)
        crop_num1 (int) : number of crop in vertical way (2)
        dim1(int) : vertical size of output (512)
        dim2(int) : horizontal size_of_output (512)
    Return :
        div_array : numpy arrays of numbers of 1,2,4
    """
    image= crop_num1*crop_num2
    crop_size = image # size of crop
    empty_array = np.zeros([dim1, dim2]).astype("float64")  # to make sure no overflow
    print(empty_array.shape)

    dim1_stride = stride_size(dim1, crop_num1, crop_size)  # vertical stride
    dim2_stride = stride_size(dim2, crop_num2, crop_size)  # horizontal stride
    index = 0
    for i in range(crop_num1):
        for j in range(crop_num2):
            # add image to empty_array at specific position
            empty_array[dim1_stride*i:dim1_stride*i + crop_size,
                        dim2_stride*j:dim2_stride*j + crop_size] += image[index]
            index += 1
    return empty_array
</code></pre>
<p>still crashing in the second iteration</p>
<pre><code class="lang-auto">
TypeError: 'int' object is not subscriptable
</code></pre>
<p>also, the results of the first iteration, does not make sense to me the training loss and the accuracy are above 80 is it possible?</p>
<pre><code class="lang-auto">Epoch 1 Train loss: 0.820307083427906 Train acc 0.893951416015625
Epoch 2 Train loss: 0.8146810382604599 Train acc 0.8943264484405518
Epoch 3 Train loss: 0.8138034269213676 Train acc 0.8945329189300537
Epoch 4 Train loss: 0.8148394525051117 Train acc 0.894050121307373
Epoch 5 Train loss: 0.8261850923299789 Train acc 0.8950645923614502
</code></pre>

ptrblck: <p>No, I meant the size of dim0 should be at least that large.<br>
You are passing <code>image</code> to the function, so you shouldn’t replace it with a int.<br>
How would you like to index <code>image</code> if it’s only a single grayscale image?<br>
Would you like to index in another dimension?</p>

lavender99: <p>Yes I want to index it in another array<br>
Also before the crashing it gives me the input and target shape is different</p>

ptrblck: <p>But how would you like to index the image?<br>
Currently it has a shape of <code>[1, 512, 512]</code>.<br>
If you call <code>image[0]</code>, you will get the pixel value of the one and only channel.<br>
However, inside the loop you are increasing <code>index</code> by one, so that the next call would be <code>image[1]</code>, which does not exist.</p>
<p>Could you try to explain, how the indexing should work or maybe draw a sketch?</p>

lavender99: <aside class="quote no-group" data-username="ptrblck" data-post="10" data-topic="42556">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>index the image</p>
</blockquote>
</aside>
<p>i think i am not understanding the idea of  index , anyway if i am having 3 channels i can index the image correctly ?</p>

ptrblck: <p>Not using the current code.<br>
It looks like you are trying to set some patches of en empty tensor to some input images.<br>
Could you explain your workflow a bit so that we could maybe create some (pseudo-)code for it?</p>

lavender99: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>my code is basically exploring whether the preprocessing  and postprocessing for vessels segmentation using unet  can improve the accuracy, i used ( Flip ,Gaussian_noise, Uniform_noise , Brightness ,Elastic distort {0: distort, 1:no distort} ,Crop the image<br>
,Pad the image, Sanity Check for Cropped image , Normalize the image  )<br>
also dividing the training sets to val and train , but i am not getting a high accuracy also the index error</p>

lavender99: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
this is the link for my colab drift<br>
<a href="https://colab.research.google.com/drive/19W-h5lootgxCsNi6mZBXAdqiDroGOLfN" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/19W-h5lootgxCsNi6mZBXAdqiDroGOLfN</a></p>

lavender99: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> i just want to check if you have any idea how to fix it</p>

ptrblck: <p>The code is quite long, so it would be easier if you try to explain what the nested for loops are supposed to do.</p>

lavender99: <p>ok, so the code is about data augmentation and pre-processing for vessels segmentation , the first part of the code is prepossessing and augmentation , then i train it using unet , it breaks on 5th epoch where the validation spouse to happen .</p>
<pre><code class="lang-auto">def image_concatenate(image, crop_num1, crop_num2, dim1, dim2):
    """concatenate images
    Args :
        image : output images (should be square)
        crop_num2 (int) : number of crop in horizontal way (2)
        crop_num1 (int) : number of crop in vertical way (2)
        dim1(int) : vertical size of output (512)
        dim2(int) : horizontal size_of_output (512)
    Return :
        div_array : numpy arrays of numbers of 1,2,4
    """
    crop_size = image.shape[1]  # size of crop
    empty_array = np.zeros([dim1, dim2]).astype("float64")  # to make sure no overflow
    dim1_stride = stride_size(dim1, crop_num1, crop_size)  # vertical stride
    dim2_stride = stride_size(dim2, crop_num2, crop_size)  # horizontal stride
    index = 1
    for i in range(crop_num1):
        for j in range(crop_num2):
            # add image to empty_array at specific position
            empty_array[dim1_stride*i:dim1_stride*i+ crop_size,
                        dim2_stride*j:dim2_stride*j+ crop_size] += image[index]
            index += 1
    return empty_array
</code></pre>
<p>this code is suposed to concatenate images by croping and vertical, horizontial dimentation<br>
then used to save the predaction images</p>
<pre><code class="lang-auto">
def save_prediction_image(stacked_img, im_name, epoch, save_folder_name="result_images", save_im=True):
    """save images to save_path
    Args:
        stacked_img (numpy): stacked cropped images
        save_folder_name (str): saving folder name
    """
    div_arr = division_array(256, 2, 2, 512, 512)
    img_cont = image_concatenate(stacked_img.cpu().data.numpy(), 2, 2, 512, 512)
    img_cont = polarize((img_cont)/div_arr)*255
    img_cont_np = img_cont.astype('uint8')
    img_cont = Image.fromarray(img_cont_np)
    # organize images in every epoch
    desired_path = save_folder_name + '/epoch_' + str(epoch) + '/'
    # Create the path if it does not exist
    if not os.path.exists(desired_path):
        os.makedirs(desired_path)
    # Save Image!
    export_name = str(im_name) + '.png'
    img_cont.save(desired_path + export_name)
    return img_cont_np
</code></pre>

ptrblck: <p>Thanks for the explanation.<br>
It seems you would like to create the new image by concatenating 4 images from the tensor <code>image</code>.<br>
In that case <code>image</code> should have the shape <code>[4, height, width]</code> in case you would like to use 4 different images.<br>
Currently it seems <code>image</code> only contains 1 image. Would you like to use the same image 4 times?</p>
<p>If <code>image</code> only contains this single image only “sometimes”, it might be due to a remaining last batch, which can be smaller depending on your batch size.<br>
If you are using a <code>DataLoader</code>, you could try to set <code>drop_last=True</code> and try it again.</p>

lavender99: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you so much , now it is breaks on 5 th epoch with error<br>
“RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 1, 3, 3], but got 5-dimensional input of size [1, 1, 512, 512, 3] instead”</p>

lavender99: <p>most of the answers for the question is abut using " unsqueeze(0) " with the data and thier mask i already used them</p>
<pre><code class="lang-auto">def validate_model(model, data_val, criterion, epoch, make_prediction=True, save_folder_name='prediction'):
    """
        Validation run
    """
    # calculating validation loss
    total_val_loss = 0
    total_val_acc = 0
    for batch, (images_v, masks_v, original_msk) in enumerate(data_val):
        stacked_img = torch.Tensor([]).unsqueeze(0).cuda()
        for index in range(images_v.size()[1]):
          
            with torch.no_grad():
                image_v = Variable(images_v[:, index, :, :].unsqueeze(0).float().cuda())
                mask_v = Variable(masks_v[:, index, :, :].unsqueeze(0).float().cuda())
                # print(image_v.shape, mask_v.shape)
                output_v = model(image_v)
                total_val_loss = total_val_loss + criterion(output_v, mask_v).cpu().item()
                # print('out', output_v.shape)
                output_v = torch.argmax(output_v, dim=1).float()
                stacked_img = torch.cat((stacked_img, output_v))
        if make_prediction:
            im_name = batch  # TODO: Change this to real image name so we know
            pred_msk = save_prediction_image(stacked_img, im_name, epoch, save_folder_name)
            acc_val = accuracy_check(original_msk, pred_msk)
            total_val_acc = total_val_acc + acc_val

    return total_val_acc/(batch + 1), total_val_loss/((batch + 1)*4)
</code></pre>