'c10::Error' error with an unknown source

Sina_Moeini: <p>Hi I am getting the following error when</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f05eb34b617 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&amp;) + 0x64 (0x7f05eb30698d in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f05eb407128 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: &lt;unknown function&gt; + 0x16c99 (0x7f05eb3cfc99 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #4: &lt;unknown function&gt; + 0x19bad (0x7f05eb3d2bad in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #5: &lt;unknown function&gt; + 0x19fcd (0x7f05eb3d2fcd in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #6: &lt;unknown function&gt; + 0x510cf6 (0x7f05ad147cf6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #7: &lt;unknown function&gt; + 0x55ca7 (0x7f05eb330ca7 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x1e3 (0x7f05eb328cb3 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #9: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f05eb328e49 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #10: &lt;unknown function&gt; + 0x7c0dc8 (0x7f05ad3f7dc8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #11: &lt;unknown function&gt; + 0x15d4cd (0x55cf26ea64cd in /usr/bin/python)
frame #12: &lt;unknown function&gt; + 0x256210 (0x55cf26f9f210 in /usr/bin/python)
frame #13: _PyObject_GC_New + 0x266 (0x55cf26e993c6 in /usr/bin/python)
frame #14: _PyObject_GenericGetAttrWithDict + 0x1e8 (0x55cf26ed2258 in /usr/bin/python)
frame #15: PyObject_GetAttr + 0x4d (0x55cf26ed0a4d in /usr/bin/python)
frame #16: _PyEval_EvalFrameDefault + 0x52de (0x55cf26ec354e in /usr/bin/python)
frame #17: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #18: _PyEval_EvalFrameDefault + 0x6bf (0x55cf26ebe92f in /usr/bin/python)
frame #19: &lt;unknown function&gt; + 0x198791 (0x55cf26ee1791 in /usr/bin/python)
frame #20: _PyEval_EvalFrameDefault + 0x18d3 (0x55cf26ebfb43 in /usr/bin/python)
frame #21: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #22: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #23: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #24: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #25: &lt;unknown function&gt; + 0x198791 (0x55cf26ee1791 in /usr/bin/python)
frame #26: _PyEval_EvalFrameDefault + 0x6bf (0x55cf26ebe92f in /usr/bin/python)
frame #27: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #28: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #29: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #30: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #31: &lt;unknown function&gt; + 0x198791 (0x55cf26ee1791 in /usr/bin/python)
frame #32: _PyEval_EvalFrameDefault + 0x18d3 (0x55cf26ebfb43 in /usr/bin/python)
frame #33: &lt;unknown function&gt; + 0x198791 (0x55cf26ee1791 in /usr/bin/python)
frame #34: _PyEval_EvalFrameDefault + 0x6bf (0x55cf26ebe92f in /usr/bin/python)
frame #35: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #36: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #37: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #38: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #39: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #40: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #41: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #42: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #43: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #44: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #45: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #46: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #47: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #48: &lt;unknown function&gt; + 0x2ac834 (0x55cf26ff5834 in /usr/bin/python)
frame #49: _PyObject_MakeTpCall + 0x25b (0x55cf26eca75b in /usr/bin/python)
frame #50: _PyEval_EvalFrameDefault + 0x5c53 (0x55cf26ec3ec3 in /usr/bin/python)
frame #51: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #52: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #53: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #54: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
frame #55: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #56: _PyEval_EvalFrameDefault + 0x6bf (0x55cf26ebe92f in /usr/bin/python)
frame #57: _PyObject_FastCallDictTstate + 0xc4 (0x55cf26ec9934 in /usr/bin/python)
frame #58: _PyObject_Call_Prepend + 0x5c (0x55cf26ede8ec in /usr/bin/python)
frame #59: &lt;unknown function&gt; + 0x29cba4 (0x55cf26fe5ba4 in /usr/bin/python)
frame #60: _PyObject_MakeTpCall + 0x25b (0x55cf26eca75b in /usr/bin/python)
frame #61: _PyEval_EvalFrameDefault + 0x5f66 (0x55cf26ec41d6 in /usr/bin/python)
frame #62: _PyFunction_Vectorcall + 0x7c (0x55cf26ed466c in /usr/bin/python)
frame #63: _PyEval_EvalFrameDefault + 0x804 (0x55cf26ebea74 in /usr/bin/python)
</code></pre>
<p>It looks like the trace is getting truncated. Is there a way to increase the size of the trace so I can figure out where it is coming from?</p>
<p>Thanks</p>