Unable to get repr for <class 'torch.Tensor'>

Geoffrey_Payne: <p>With this code;</p>
<pre><code>  for _, (imgs, captions) in tqdm(enumerate(coco_dataloader), total=len(coco_dataloader), leave=False):
        imgs = imgs.to(Constants.device)
        captions = captions.to(Constants.device)

        outputs = model(imgs, captions[:-1])
        outputs1 = outputs.reshape(-1, outputs.shape[2])
        captions1 = captions.reshape(-1)
        loss = criterion(outputs1, captions1)

        print(f"\nTraining loss {loss.item()}, step {step}\n")
</code></pre>
<p>I find that when I set the cuda device to captions and images and I inspect their values, I see this exception message “Unable to get repr for &lt;class ‘torch.Tensor’&gt;”<br>
One solution is not to set the device to the outputs, captions and model. but then I get this exception:</p>
<p>“Input, output and indices must be on the current device”.</p>
<p>So I can’t do that either. I appear to be in a bind when trying to fix this, what should I do?<br>
UPDATE<br>
Example of the data where the code fails;<br>
captions = tensor([[   1],[   4], [1420], [ 406], [ 139], [  35], [ 727], [1125], [ 765], [   5], [   2]])</p>
<p>imgs = tensor([[[[0.5490, 0.5373, 0.5333,  …, 0.5647, 0.5451, 0.5490],<br>
[0.5412, 0.5490, 0.5490,  …, 0.5569, 0.5725, 0.5412],<br>
[0.5176, 0.5490, 0.5137,  …, 0.5961, 0.5765, 0.5608],<br>
…,<br>
[0.8196, 0.7882, 0.8000,  …, 0.9294, 0.9333, 0.9373],<br>
[0.8000, 0.8235, 0.8039,  …, 0.9373, 0.9333, 0.9373],<br>
[0.8118, 0.7961, 0.7882,  …, 0.9333, 0.9412, 0.9412]],</p>
<pre><code>     [[0.1412, 0.1412, 0.1451,  ..., 0.5333, 0.5333, 0.5569],
      [0.1333, 0.1529, 0.1608,  ..., 0.5176, 0.5451, 0.5333],
      [0.1098, 0.1529, 0.1255,  ..., 0.5412, 0.5333, 0.5255],
      ...,
      [0.8667, 0.8353, 0.8431,  ..., 0.9843, 0.9882, 0.9961],
      [0.8471, 0.8706, 0.8471,  ..., 0.9843, 0.9882, 0.9961],
      [0.8588, 0.8431, 0.8314,  ..., 0.9804, 0.9882, 0.9882]],

     [[0.0275, 0.0235, 0.0235,  ..., 0.5216, 0.5059, 0.5059],
      [0.0196, 0.0353, 0.0392,  ..., 0.5098, 0.5137, 0.4863],
      [0.0000, 0.0353, 0.0039,  ..., 0.5373, 0.5098, 0.4902],
      ...,
      [0.8667, 0.8353, 0.8510,  ..., 0.9843, 0.9882, 0.9843],
      [0.8471, 0.8706, 0.8549,  ..., 0.9843, 0.9882, 0.9843],
      [0.8588, 0.8431, 0.8392,  ..., 0.9804, 0.9882, 0.9804]]]])
</code></pre>
<p>Sometimes data like this goes through and there is no obvious reason why it should be any different.</p>
<p>I also get lots of these error messages;</p>
<pre><code class="lang-auto">C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:605: block: [0,0,0], thread: [32,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
.
.
.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:605: block: [1,0,0], thread: [127,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\torch\_tensor_str.py", line 372, in _str
    return _str_intern(self)
  File "C:\Anaconda3\lib\site-packages\torch\_tensor_str.py", line 352, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "C:\Anaconda3\lib\site-packages\torch\_tensor_str.py", line 241, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "C:\Anaconda3\lib\site-packages\torch\_tensor_str.py", line 85, in __init__
    value_str = '{}'.format(value)
  File "C:\Anaconda3\lib\site-packages\torch\tensor.py", line 534, in __format__
    return self.item().__format__(format_spec)
RuntimeError: CUDA error: device-side assert triggered
python-BaseException
</code></pre>

ptrblck: <p>How are you trying to “inspect” these values? Are you using a debugger, which raises the error and is the code working find in the Python interpreter?</p>

Geoffrey_Payne: <p>I have updated my question with more details which hopefully will make it easier to work out what is happening.<br>
I am using Pycharm debugging tools to get the information.<br>
The exception message “Unable to get repr for &lt;class ‘torch.Tensor’&gt;” happens on this line</p>
<blockquote>
<p>imgs = imgs.to(Constants.device)</p>
</blockquote>

ptrblck: <p>Thanks for the update. I don’t know how the PyCharm debugger is working, but regarding the error you are seeing:</p>
<pre><code class="lang-python">Indexing.cu:605: block: [1,0,0], thread: [127,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
</code></pre>
<p>an indexing operation is failing.<br>
You could either execute the code on the CPU to get a better error message or run the code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code>, which should point to the line of code, which is raising this error.</p>

Geoffrey_Payne: <p>I checked again by removing the CUDA and y using the CPU as you suggested. I got a different error compared to last time. My new error message is "index out of range in self ". I found the problem with the embedding definition and fixed it now. Thank you for your help.</p>