Sharing tensors across CPU and GPU?

rbelew: <p>TL;DR: I seem to be confused as to what’s required to make data and<br>
model agree about the (symbolic) tensor’s format in order to make use<br>
of GPU efficiencies?</p>
<p>I’m trying to set up a common environment across OSX using M2 Silicon<br>
hardware and an Linux using an i7 with an RTX 3050.</p>
<p>I’m using the<br>
<a href="https://keras.io/examples/nlp/text_classification_from_scratch/" rel="noopener nofollow ugc">Text classification from scratch</a><br>
example to drive my testing of what CPU vs GPU performance looks like.</p>
<p>I’m using Keras3, and specifying <code>os.environ["KERAS_BACKEND"] = "torch"</code>.  Package version specifics listed  below.</p>
<p>I’m using <code>DEVICE = torch.device("cpu")</code> or <code>DEVICE = torch.device("cuda")</code><br>
to switch between the alternatives on the Linux box, and<br>
<code>DEVICE = torch.device("cpu")</code> or <code>DEVICE = torch.device("mps")</code> on<br>
OSX.  So, I have four experiments: CPU vs GPU on OSX, and CPU vs GPU<br>
on linux.</p>
<p>The results table:</p>
<blockquote>
<pre><code>  Linux	CPU	    Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
        CUDA    Works!

  OSX	CPU	    Works!
        MPS	    Placeholder storage has not been allocated on MPS device!
</code></pre>
</blockquote>
<p>“Works!” means the <code>model.fit()</code> method runs as expected.  On both OSX and Linux,<br>
the issue arises in <code>torch.embedding()</code></p>
<ul>
<li>On Linux using CPU:</li>
</ul>
<blockquote>
<pre><code>  DEVICE=cpu type=&lt;class 'torch.device'&gt;
  ...
  Traceback (most recent call last):
  File ".../torch-kerasNLP.py", line 183, in &lt;module&gt;
  main()
  File ".../torch-kerasNLP.py", line 154, in main
  model.fit(train_ds, validation_data=val_ds, epochs=epochs)
  File ".../lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 123, in error_handler
  raise e.with_traceback(filtered_tb) from None
  File ".../lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  return forward_call(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  return forward_call(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/torch/nn/functional.py", line 2237, in embedding
  return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  RuntimeError: Exception encountered when calling Embedding.call().

  Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

  Arguments received by Embedding.call():
  • inputs=torch.Tensor(shape=torch.Size([32, 500]), dtype=int64)
</code></pre>
</blockquote>
<p>This is especially confusing, since I am specifying CPU as the    device, but nevertheless torch is considering a CUDA device?</p>
<ul>
<li>On OSX using MPS:</li>
</ul>
<blockquote>
<pre><code>  DEVICE=mps type=&lt;class 'torch.device'&gt;
  ...
  Traceback (most recent call last):
  File ".../torch-kerasNLP.py", line 178, in &lt;module&gt;
  main()
  File ".../torch-kerasNLP.py", line 149, in main
  model.fit(train_ds, validation_data=val_ds, epochs=epochs)
  File ".../lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 123, in error_handler
  raise e.with_traceback(filtered_tb) from None
  File ".../lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
  return forward_call(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
  return forward_call(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/torch/nn/functional.py", line 2233, in embedding
  return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  RuntimeError: Exception encountered when calling Embedding.call().

  Placeholder storage has not been allocated on MPS device!

  Arguments received by Embedding.call():
  • inputs=torch.Tensor(shape=torch.Size([32, 500]), dtype=int64)
</code></pre>
</blockquote>
<p>Are my issues with torch? with keras?  thanks for any suggestions, Rik</p>
<ul>
<li>Linux environment:</li>
</ul>
<blockquote>
<pre><code>  &gt;&gt;&gt; keras.__version__
  '3.0.5'
  &gt;&gt;&gt; tensorflow_text.__version__
  '2.15.0'
  &gt;&gt;&gt; keras_nlp.__version__
  '0.7.0'
  &gt;&gt;&gt; torch.__version__
  '2.0.1'
  &gt;&gt;&gt; torchtext.__version__
  '0.15.2'
  &gt;&gt;&gt; tensorflow.__version__
  '2.15.0'
</code></pre>
</blockquote>
<ul>
<li>OSX environment</li>
</ul>
<blockquote>
<pre><code>  &gt;&gt;&gt; keras.__version__
  '3.0.5'
  &gt;&gt;&gt; tensorflow_text.__version__
  '2.15.0'
  &gt;&gt;&gt; keras_nlp.__version__
  '0.7.0'
  &gt;&gt;&gt; torch.__version__
  '2.1.0.post100'
  &gt;&gt;&gt; torchtext.__version__
  '0.16.1'
  &gt;&gt;&gt; tensorflow.__version__
  '2.15.0'
</code></pre>
</blockquote>

ptrblck: <p>Based on the error message for Linux on CPU it seems the model was moved to the GPU while the inputs are still on the CPU. I don’t know if Keras or another part of your code will move things behind your back to the GPU, if it’s available, so you might need to double check this.</p>

eaah: <p>It would be useful for you to share your code atleast where you are loading your model.</p>
<p>If in your code you send the model to gpu at anytime you are expected to have all the inputs in gpu. If you do no send the model to gpu you should not see the gpu being loaded with the model. The memory in your gpu should not change when running on CPU.</p>
<p>You can run a test while supervising the gpu memory to identify when is the model being loaded to gpu and start debbugin from there.</p>
<p>I dont suggest running two experiments at the same time if you only have one GPU. (At least at the begining of the tests) I would use a singularity or docker container to run multiple experiments but only if I have multiple GPUs.</p>

rbelew: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/eaah">@eaah</a>, thanks for your help.</p>
<p>i knew i should have made the code accessible, but didn’t know where this forum likes paste examples (didn’t see this FAQ).  anyway, here it is:  <a href="https://github.com/rbelew/rikHak/blob/master/torch-kerasNLP-pub.py" class="inline-onebox" rel="noopener nofollow ugc">rikHak/torch-kerasNLP-pub.py at master · rbelew/rikHak · GitHub</a></p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>, re: Linux+CPU, how is it you know it is the MODEL that’s on the GPU?  i have an explicit <code>model.to(DEVICE)</code> before compiling it?</p>

eaah: <p>That is correct. The <code>model.to(DEVICE)</code> line sends the model to the GPU. Meaning the model weights and computations are now in the graphics card. To perform mathematical with your inputs they need to be done in the GPU as well.</p>
<p>When you run  <code>model.to(DEVICE)</code> you can see the memory in your GPU change as the model has now been loaded to the GPU.</p>
<p>In line 135 you have <code>model.fit(train_ds, validation_data=val_ds, epochs=epochs)</code> but the <code>train_ds</code> is still in the CPU.</p>
<p>To run everything in CPU remove the <code>model.to(DEVICE)</code>.<br>
To run in GPU move the dataset <code>train_ds</code> to the GPU. try <code>train_ds.to(DEVICE)</code> or <code>tran_ds.cuda()</code></p>

rbelew: <blockquote>
<p>The <code>model.to(DEVICE)</code> line sends the model to the GPU.</p>
</blockquote>
<p>despite the fact that on line#24 i have <code>DEVICE = torch.device("cpu")</code> ?!</p>

eaah: <p>Oh, apologies!! I can see you are choosing the device based on the host. It should be choosing the ‘cpu’ device, but i would find this line <code>mode.to(DEVICE)</code> redundant when operating in cpu. Have you tried removing that line? Just to test?</p>
<p>You should check which devices are available.</p>
<p>if <code>torch.cuda.is_available()</code> returns False you should be running on CPU.<br>
and <code>torch.cuda.device_count()</code>should be 0.</p>
<pre><code class="lang-auto">torch.cuda.current_device()
</code></pre>
<p>You can also check the <code>.device</code> attribute of all parameters and see where are they allocated.</p>
<p>I reviewed this post:</p>
<p><a href="https://medium.com/@btatmaja/trying-keras-core-with-pytorch-backend-4a643275911f" class="onebox" target="_blank" rel="noopener nofollow ugc">https://medium.com/@btatmaja/trying-keras-core-with-pytorch-backend-4a643275911f</a></p>
<p>here they don’t seem to be using model.to(DEVICE) with <code>keras_core</code></p>

rbelew: <p><a class="mention" href="/u/eaah">@eaah</a> Hah!  indeed, commenting out the <code>model.to(DEVICE)</code> line lets the CPU version proceed.  so that seems a bug (in keras?) to report.</p>
<p>now, down to just the OSX+MPS issue:).</p>
<p>thanks for your help.</p>

rbelew: <p>re: OSX+MPS , it seems clear the issue is that the IMBD tensors are on the CPU not the GPU/MPS.</p>
<p><a href="https://pytorch.org/docs/stable/notes/mps.html" rel="noopener nofollow ugc">MPS notes</a> show<br>
moving model to MPS, and raw tensors with <code>device=</code>.<br>
but the <code>keras.utils.text_dataset_from_directory()</code> utility doesn’t accept a <code>device=</code> argument, and the <code>.map(vectorize_text)</code> doesn’t either?</p>