QuestionAnsweringModel not working with GPU

username: kb2009
cooked: <p>I am using QuestionAnsweringModel from SimpleTransformers. When I run my code, and review my processes in Windows Task Manager, python is not using my GPU at all. I have included a code snippet to recreate the problem. Any help is highly appreciated.</p>
<p>import torch<br>
from simpletransformers.question_answering import QuestionAnsweringModel, QuestionAnsweringArgs</p>
<p>model_type=“bert”<br>
model_name= “bert-base-cased”<br>
model_args = QuestionAnsweringArgs()<br>
train_args = {<br>
“n_best_size”:1 ,<br>
‘overwrite_output_dir’: True,<br>
‘show_running_loss’:True,<br>
‘n_gpu’: 3</p>
<p>}<br>
model = QuestionAnsweringModel(model_type,model_name, args=train_args, use_cuda=True)</p>
<p>I have tried<br>
model.to(torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”))<br>
to set the device to cuda. But I am having no luck.<br>
I have also looked at a topic on the same issue here before. Recommendations were to update pytorch. But I have already done that as well.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>If <code>model.to()</code> was called with <code>cuda:0</code> as its argument, the GPU will be used since PyTorch won’t move the parameters back to the CPU behind your back. Use <code>nvidia-smi</code> to check the GPU usage or the “CUDA” tab in the task manager, as it doesn’t show the compute utilization by default.</p>

username: kb2009
cooked: <p>Thank you for the response! I have tried model.to(‘cuda:0’) before and I have checked nvidia-smi. One of my GPU gets used, but that has never run my python files. What I have not been able to check is CUDA under task manager. My task manager always has GPU usage under 1% and whatever it is using for is not python.  All of my python files run on CPU. My naive assumption is that my notebook recognizes that GPU is installed. But when it comes to running the model on one of the GPUs, it does not do that. model.to(“cuda:0”), and torch.cuda.set_device(“cuda”) does not work to make that happen.  torch.cuda.set_device(“cuda”)  doesn’t throw any error. But model.to(“cuda:0”) throws error that says “‘QuestionAnsweringModel’ object has no attribute ‘to’”. So I need to find a workaround for this.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf7c9783cb4766a3b71562d3d7a3682f872241b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf7c9783cb4766a3b71562d3d7a3682f872241b7" title="Screen Shot 2022-01-18 at 2.39.52 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf7c9783cb4766a3b71562d3d7a3682f872241b7_2_690x360.png" alt="Screen Shot 2022-01-18 at 2.39.52 PM" data-base62-sha1="rjY8hQ8GTeQzbNsF1JOAiyNtjBd" width="690" height="360" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf7c9783cb4766a3b71562d3d7a3682f872241b7_2_690x360.png, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf7c9783cb4766a3b71562d3d7a3682f872241b7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf7c9783cb4766a3b71562d3d7a3682f872241b7.png 2x" data-dominant-color="171717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-01-18 at 2.39.52 PM</span><span class="informations">920×481 152 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="kb2009" data-post="3" data-topic="141962">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/82dd89/48.png" class="avatar"> kb2009:</div>
<blockquote>
<p>But model.to(“cuda:0”) throws error that says “‘QuestionAnsweringModel’ object has no attribute ‘to’”. So I need to find a workaround for this.</p>
</blockquote>
</aside>
<p>I’m unsure how and if it was working before, but it seems you might be facing different issues and would have to debug what <code>QuestionAnsweringModel</code> is (it doesn’t seem to be an <code>nn.Module</code>) and how to push it to the device.<br>
In any case, once <code>to('cuda')</code> is called on the model and input, the GPU will be used and you would see it in <code>nvidia-smi</code> (unless an error is raised of course).<br>
A low GPU utilization could be caused by different bottlenecks in your code, e.g. the data loading, a lot of small kernel launches etc.</p>