Abort() has been called During DataLoader Iteration in Libtorch on Windows (MSVC + CUDA 11.8)

Saul_Leyva: <p>I am experiencing a crash during the iteration of a <code>DataLoader</code> when using Libtorch (C++ API for PyTorch) on Windows. The program fails consistently upon entering the loop for accessing the DataLoader content, regardless of dataset complexity, CUDA or CPU execution, or threading settings (num_workers). Here is a minimal reproducible example:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;

class DummyDataset : public torch::data::Dataset&lt;DummyDataset&gt; {
    private:
        size_t size_;
    public:
        explicit DummyDataset(size_t size) : size_(size) {}

        torch::data::Example&lt;&gt; get(size_t index) override {
            std::cout &lt;&lt; "Getting data for index: " &lt;&lt; index &lt;&lt; std::endl;
            torch::Tensor data = torch::ones({3, 224, 224}, torch::kFloat32); 
            torch::Tensor label = torch::tensor(static_cast&lt;int&gt;(index % 2), torch::kInt64); 
            return {data, label};
        }

        torch::optional&lt;size_t&gt; size() const override {
            return size_;
        }
};

int main() {

    const size_t dataset_size = 12; 
    const int batch_size = 4;
    const int num_workers = 2; 

    auto train_dataset = DummyDataset(dataset_size).map(torch::data::transforms::Stack&lt;&gt;());

    auto train_loader = torch::data::make_data_loader(
        std::move(train_dataset),
        torch::data::DataLoaderOptions().batch_size(batch_size).workers(num_workers)
    );

    std::cout &lt;&lt; "Iterating through DataLoader." &lt;&lt; std::endl;
   
    for (auto&amp; batch : *train_loader) {  // FAILS HERE!
        std::cout &lt;&lt; "Batch data size: " &lt;&lt; batch.data.sizes()
                    &lt;&lt; ", Batch label size: " &lt;&lt; batch.target.sizes() &lt;&lt; std::endl;
    }

    return 0;
}
</code></pre>
<p>The program crashes when entering the <code>for</code> loop of the <code>DataLoader</code>:</p>
<pre><code class="lang-auto">for (auto&amp; batch : *train_loader)
</code></pre>
<p>The expected output <code>Batch data size...</code> is never printed. Instead, a Windows error dialog appears stating:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/eac3861e1abbaf2ef43c6a9f8e35b41e9e862848.png" data-download-href="https://discuss.pytorch.org/uploads/default/eac3861e1abbaf2ef43c6a9f8e35b41e9e862848" title="Captura de pantalla 2024-11-15 220536"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/eac3861e1abbaf2ef43c6a9f8e35b41e9e862848.png" alt="Captura de pantalla 2024-11-15 220536" data-base62-sha1="xuODraWwYZKRK3q7zrNjrwVxnBK" width="301" height="170"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Captura de pantalla 2024-11-15 220536</span><span class="informations">603×341 41.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When debugging with gdb, I get the following backtrace:</p>
<pre><code class="lang-auto">Reading symbols from .\Debug\Planets_and_Moons.exe...
(No debugging symbols found in .\Debug\Planets_and_Moons.exe)
(gdb) run
Starting program: C:\Users\sauls\Desktop\a\build\Debug\Planets_and_Moons.exe
[New Thread 24504.0x5f08]
[New Thread 24504.0x38c4]
[New Thread 24504.0x5ca4]
[New Thread 24504.0x1fec]
Iterating through DataLoader.[New Thread 24504.0x4e00]

gdb: unknown target exception 0xe06d7363 at 0x7ff93afffa4c

Thread 1 received signal ?, Unknown signal.
0x00007ff93afffa4c in RaiseException () from C:\Windows\System32\KernelBase.dll
(gdb) bt
#0  0x00007ff93afffa4c in RaiseException () from C:\Windows\System32\KernelBase.dll
#1  0x00007ff91aa35267 in _CxxThrowException () from C:\Windows\SYSTEM32\vcruntime140.dll
#2  0x00007ff8e7eb6bfe in c10!?torchCheckFail@detail@c10@@YAXPEBD0I0@Z () from C:\Users\sauls\Desktop\a\build\Debug\c10.dll
#3  0x00007ff8e7e68d89 in c10!?torchInternalAssertFail@detail@c10@@YAXPEBD0I0UCompileTimeEmptyString@12@@Z ()
   from C:\Users\sauls\Desktop\a\build\Debug\c10.dll
#4  0x00007ff73b9cfdbf in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
#5  0x00007ff73b9d4f81 in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
#6  0x00007ff73b9d4e9d in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
#7  0x00007ff73b9d5e98 in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
#8  0x00007ff73b9d5f55 in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
#9  0x00007ff73b9c8c0e in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
#10 0x00007ff73b997705 in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
#11 0x00007ff73b9db969 in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
#12 0x00007ff73b9db80e in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
#13 0x00007ff73b9db6ce in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
--Type &lt;RET&gt; for more, q to quit, c to continue without paging--c
#14 0x00007ff73b9db9fe in Planets_and_Moons!??1Await@ivalue@c10@@UEAA@XZ ()
#15 0x00007ff93cf7259d in KERNEL32!BaseThreadInitThunk () from C:\Windows\System32\kernel32.dll
#16 0x00007ff93d7eaf38 in ntdll!RtlUserThreadStart () from C:\Windows\SYSTEM32\ntdll.dll
#17 0x0000000000000000 in ?? ()
</code></pre>
<p><strong>Environment:</strong></p>
<ul>
<li><strong>OS</strong>: Windows 11</li>
<li><strong>Compiler</strong>: MSVC 14.29 (Visual Studio 2019)</li>
<li><strong>Libtorch Version</strong>: 2.5.1+cu118 (CUDA) (also tested with 2.5.1+cpu, 2.4.1+cu118 and 2.1.2+cu118)</li>
<li><strong>CUDA Version</strong>: 11.8</li>
<li><strong>GPU</strong>: NVIDIA GeForce  RTX 3070 (laptop)</li>
</ul>
<p><strong>What I have tried:</strong></p>
<ol>
<li><strong>Disable Multithreading</strong>: Set num_workers = 0 to avoid threading issues (Issue persists).</li>
<li><strong>Change batch size</strong>: Set batch_size=1 (Issue persists).</li>
<li><strong>Simplified Dataset</strong>: Replaced dataset logic with dummy tensors (Issue persists).</li>
<li><strong>Force CPU Execution</strong>: Used CPU tensors only (no .to(torch::kCUDA)). (Issue persists).</li>
<li><strong>Checked .dll Placement</strong>: All necessary .dll files are in the same directory as the executable. (Issue persists).</li>
<li><strong>Reinstalled</strong> libtorch, cuda and visual studio tools (Issue persists).</li>
<li><strong>Access DataLoader Without Loop</strong> (Issue persists):</li>
</ol>
<pre><code class="lang-auto">auto iterator = data_loader-&gt;begin();
auto batch = *iterator; // FAILS HERE
</code></pre>
<ol start="8">
<li><strong>Small tests</strong> to see if <strong>cuda was available</strong> and/or if <strong>threading produced errors</strong>, but all worked fine.</li>
</ol>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;thread&gt;
#include &lt;torch/torch.h&gt;

void cuda_thread_test(int thread_id) {
    try {
        torch::Tensor tensor = torch::ones({100, 100}, torch::kCUDA);

        tensor = tensor * 2;

        std::cout &lt;&lt; "Thread " &lt;&lt; thread_id &lt;&lt; ": Tensor sum = " &lt;&lt; tensor.sum().item&lt;float&gt;() &lt;&lt; std::endl;
    } catch (const std::exception&amp; e) {
        std::cerr &lt;&lt; "Thread " &lt;&lt; thread_id &lt;&lt; ": Exception: " &lt;&lt; e.what() &lt;&lt; std::endl;
    }
}

int main() {
    if (!torch::cuda::is_available()) {
        std::cerr &lt;&lt; "CUDA is not available!" &lt;&lt; std::endl;
        return -1;
    }

    std::cout &lt;&lt; "CUDA is available!" &lt;&lt; std::endl;

    const int num_threads = 4;
    std::thread threads[num_threads];

    for (int i = 0; i &lt; num_threads; ++i) {
        threads[i] = std::thread(cuda_thread_test, i);
    }

    for (int i = 0; i &lt; num_threads; ++i) {
        threads[i].join();
    }

    std::cout &lt;&lt; "All threads completed successfully." &lt;&lt; std::endl;
    return 0;
}
</code></pre>
<h3><a name="p-459486-questions-1" class="anchor" href="#p-459486-questions-1"></a><strong>Questions</strong></h3>
<ol>
<li>What could cause a crash during the iteration of a <code>DataLoader</code>, considering the extensive testing and discarded possibilities (e.g., threading, dataset logic, CUDA availability)?</li>
<li>Are there additional debugging steps I can take to identify the root cause?</li>
<li>Is this a known issue with Libtorch on Windows?</li>
</ol>

Saul_Leyva: <p>I have found a solution to the problem I was experiencing with the <code>DataLoader</code> crash during iteration. In case anyone else encounters this issue in the future, here’s what worked for me.</p>
<p>Instead of building and running the application in Debug mode:</p>
<pre><code class="lang-auto">cmake --build .
./Debug/program.exe
</code></pre>
<p>You need to compile and run it in Release mode:</p>
<pre><code class="lang-auto">cmake --build . --config Release
./Release/program.exe
</code></pre>
<p>If anyone has an explanation of why this works, I would appreciate it.</p>