Running torch.nn.Sequential on CUDA breaks length of item vector

Igor_Panjan: <p>Hi there,</p>
<p>I’m working on updating the wonderful library that Nilesh Verma wrote - DeepImageSearch (<a href="https://github.com/TechyNilesh/DeepImageSearch" rel="noopener nofollow ugc">GitHUB</a>). It is a library for finding similar images. It is really accurate in super simple to use.</p>
<p>Nilesh’s library uses CPU for loading the pre-trained model and removing the last layer.</p>
<p>The problem with CPU based process is that it is very slow - 4 it/s. So for 10.000 images processing takes an hour. And I have a 1 TB of images to process.</p>
<p>I wanted to transfer this process to GPU. So I updated this code from</p>
<pre><code class="lang-auto">        print("\033[91m Please Wait Model Is Loading or Downloading From Server!")
        base_model = timm.create_model(self.model_name, pretrained=self.pretrained)
        self.model = torch.nn.Sequential(*list(base_model.children())[:-1])
</code></pre>
<p>to</p>
<pre><code class="lang-auto"> # Load the pre-trained model and remove the last layer
        print("\033[91m Please Wait Model Is Loading or Downloading From Server!")
        base_model = timm.create_model(self.model_name, pretrained=self.pretrained)
        self.model = torch.nn.Sequential(*list(base_model.children())[:-1]).cuda(0)
</code></pre>
<p>which immediately causes Exception:</p>
<pre><code class="lang-auto">Metadata and Features are already present, Do you want Extract Again? Enter yes or no
yes
  1%|          | 6/1000 [00:00&lt;00:17, 57.16it/s]/home/stoneball/DeepImageSearch_GPU_V2/lib/python3.10/site-packages/PIL/Image.py:992: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
  warnings.warn(
100%|██████████| 1000/1000 [00:22&lt;00:00, 43.62it/s]
Traceback (most recent call last):
  File "/home/stoneball/DeepImageSearch_GPU_V2/lib/python3.10/site-packages/pandas/core/indexes/range.py", line 345, in get_loc
    return self._range.index(new_key)
ValueError: 0 is not in range

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/stoneball/PycharmProjects/DeepImageSearch_GPU_V2/DeepImageSearch_GPU_V2.py", line 11, in &lt;module&gt;
    st.run_index()
  File "/home/stoneball/PycharmProjects/DeepImageSearch_GPU_V2/DeepImageSearch.py", line 168, in run_index
    self._start_indexing(data)
  File "/home/stoneball/PycharmProjects/DeepImageSearch_GPU_V2/DeepImageSearch.py", line 140, in _start_indexing
    d = len(image_data['features'][0])  # Length of item vector that will be indexed
  File "/home/stoneball/DeepImageSearch_GPU_V2/lib/python3.10/site-packages/pandas/core/series.py", line 1007, in __getitem__
    return self._get_value(key)
  File "/home/stoneball/DeepImageSearch_GPU_V2/lib/python3.10/site-packages/pandas/core/series.py", line 1116, in _get_value
    loc = self.index.get_loc(label)
  File "/home/stoneball/DeepImageSearch_GPU_V2/lib/python3.10/site-packages/pandas/core/indexes/range.py", line 347, in get_loc
    raise KeyError(key) from err
KeyError: 0
 Image Meta Information Saved: [metadata-files/vgg19/image_data_features.pkl]
</code></pre>
<p>From my understanding using CUDA changes the length of the vector. So from info I found on this forum I went for adding a Flatten layer:</p>
<pre><code class="lang-auto"> # Load the pre-trained model and remove the last layer
        print("\033[91m Please Wait Model Is Loading or Downloading From Server!")
        base_model = timm.create_model(self.model_name, pretrained=self.pretrained)
        self.model = torch.nn.Sequential(*list(base_model.children())[:-1]).cuda(0)
        self.model = torch.nn.Flatten()
        self.model.eval()
</code></pre>
<p>The code runs successfully, using CUDA speeds up the process 50x on GTX 1080 ti - <strong>BUT i break the the accuracy of the model. Similar images are all over the place.</strong></p>
<p>What am I doing wrong?</p>
<p>Thank you for any tips.</p>

ptrblck: <p>The error is raised from <code>pandas</code> and I’m unsure how it could be related to the usage of your GPU as it seems to be part of the data loading and processing. Could you check why the indexing fails with the invalid key?</p>

Igor_Panjan: <p>Thank you for input <a class="mention" href="/u/ptrblck">@ptrblck</a>. I found that pandas dataframe is empty because of this code:</p>
<pre><code class="lang-auto">    def _extract(self, img):
        # Resize and convert the image
        img = img.resize((224, 224))
        img = img.convert('RGB')

        # Preprocess the image
        preprocess = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229,0.224, 0.225]),
        ])
        x = preprocess(img)
        x = Variable(torch.unsqueeze(x, dim=0).float(), requires_grad=False)

        # Extract features
        feature = self.model(x)
        feature = feature.data.numpy().flatten()
        return feature / np.linalg.norm(feature)
</code></pre>
<p>The output of this code is input into pandas data frame.</p>
<p>So my question is: if I run this code in CPU image features are extracted properly - how can I run this code on GPU?</p>

ptrblck: <p>You would have to move the model and input data to the GPU first and should be able to execute the same code. It’s still unclear how an empty result can be returned if the same code is executed so check which object is set to <code>None</code>.</p>

Igor_Panjan: <p>I’m trying to wrap my head around <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>So I understand - model is loaded to GPU.</p>
<p>I understand I have to load tensor data also to GPU - which I tried:</p>
<pre><code class="lang-auto">    def _extract(self, img):
        # Resize and convert the image
        img = img.resize((224, 224))
        img = img.convert('RGB')

        # Preprocess the image
        preprocess = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229,0.224, 0.225]),
        ])

        x = preprocess(img)
        **x = x.cuda(0)**
        x = Variable(torch.unsqueeze(x, dim=0).float(), requires_grad=False)


        # Extract features
        feature = self.model(x)
        feature = feature.data.numpy().flatten()
        return feature / np.linalg.norm(feature)
</code></pre>
<p>and than I get Exception:</p>
<pre><code class="lang-auto">Exception: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
</code></pre>
<p>I understand I have to copy the tensor to CPU and back to GPU. How can I do that?</p>
<p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for helping out! ML is a totally different beast to tame, than anything else I did in Python.</p>

ptrblck: <p>The new error is most likely raised in:</p>
<pre><code class="lang-python">feature = feature.data.numpy().flatten()
</code></pre>
<p>and you would need to detach the tensor and move the the CPU before applying numpy operations on it:</p>
<pre><code class="lang-python">feature = feature.detach().cpu().numpy().flatten()
</code></pre>
<p>Also, don’t use the deprecated <code>Variable</code> and <code>.data</code> attribute.</p>
<p>Note however, that using numpy operations (and thus detaching the tensor) will not allow you to backpropagate through these operations anymore in case that’s needed.</p>