NumWorkers > 0 crashes

MartinPerry: <p>I am using libTorch with C++20. I create DataLoader like this:</p>
<pre><code class="lang-auto">template &lt;typename DatasetType&gt;
auto BuildDataLoader(const Settings&amp; sets, int&amp; bacthesCount)
{        
    auto ds = DefaultDataset();
    auto dsMapped = ds.map(torch::data::transforms::Stack&lt;typename DatasetType::ExampleType&gt;());
    
    auto datasetSize = dsMapped.size().value();
    bacthesCount = (datasetSize + sets.batchSize - 1) / sets.batchSize;

    auto loader = torch::data::make_data_loader(
        std::move(dsMapped),        
        torch::data::DataLoaderOptions()
            .batch_size(sets.batchSize)            
            .workers(sets.numWorkers)
            .enforce_ordering(false)            
    );    

    return loader;
}
</code></pre>
<p>where</p>
<pre><code class="lang-auto">DataLoaderData DefaultDataset::get(size_t index){    
    DataLoaderData ld;
    ld.input = at::ones({ 3, 64, 64 }, at::kFloat);
    ld.target = at::ones({ 1, 64, 64 }, at::kFloat);
    return ld;
}

torch::optional&lt;size_t&gt; DefaultDataset::size() const {
    return 30;
}
</code></pre>
<p>and</p>
<pre><code class="lang-auto">struct DataLoaderData
{    
    at::Tensor input;
    at::Tensor target;

    void setupDevice(const Settings&amp; sets)
    {
        input = input.to(sets.device, input.dtype(), sets.perf.useNonBlockingTransfers);
        target = target.to(sets.device, target.dtype(), sets.perf.useNonBlockingTransfers);
    }
};

//=================================================================================
/*
With this, we can use Dataset.map with
torch::data::transforms::Stack&lt;typename DatasetType::ExampleType&gt;()
which will stack batches together to one tensor (B, ...)
*/

namespace torch::data::transforms {
    template &lt;&gt;
    struct Stack&lt;DataLoaderData&gt; : public Collation&lt;DataLoaderData&gt; {
        DataLoaderData apply_batch(std::vector&lt;DataLoaderData&gt; ds) override {
           
            std::vector&lt;torch::Tensor&gt; inputs, targets;
            inputs.reserve(ds.size());
            targets.reserve(ds.size());            

            for (auto&amp; d : ds) 
            {
                inputs.push_back((d.input));
                targets.push_back((d.target));
            }

            DataLoaderData d;
            d.input = torch::stack(inputs);
            d.target = torch::stack(targets);
            
            return d;
        }
    };
}
</code></pre>
<p>However, when I run training with numWorkers &gt; 0, code randomly crashes during looping dataLoader.</p>
<pre><code class="lang-auto">for (auto&amp; batch : *dl) { .. some code.. }
</code></pre>
<p>Crash is from thread, stack trace:</p>
<pre><code class="lang-auto">LibTorchFramework.exe!free_dbg_nolock(void * const block=0x00000150060b4040, const int block_use=1) Line 996	C++
 	LibTorchFramework.exe!_free_dbg(void * block=0x00000150060b4040, int block_use=-1) Line 1030	C++
[External Code]	
LibTorchFramework.exe!torch::data::detail::Queue&lt;torch::data::DataLoaderBase&lt;torch::data::datasets::MapDataset&lt;DefaultDataset,torch::data::transforms::Stack&lt;DataLoaderData&gt;&gt;,DataLoaderData,std::vector&lt;unsigned __int64,std::allocator&lt;unsigned __int64&gt;&gt;&gt;::Job&gt;::pop(std::optional&lt;std::chrono::duration&lt;__int64,std::ratio&lt;1,1000&gt;&gt;&gt; timeout={...}) Line 57	C++
LibTorchFramework.exe!torch::data::detail::DataShuttle&lt;torch::data::DataLoaderBase&lt;torch::data::datasets::MapDataset&lt;DefaultDataset,torch::data::transforms::Stack&lt;DataLoaderData&gt;&gt;,DataLoaderData,std::vector&lt;unsigned __int64,std::allocator&lt;unsigned __int64&gt;&gt;&gt;::Job,torch::data::DataLoaderBase&lt;torch::data::datasets::MapDataset&lt;DefaultDataset,torch::data::transforms::Stack&lt;DataLoaderData&gt;&gt;,DataLoaderData,std::vector&lt;unsigned __int64,std::allocator&lt;unsigned __int64&gt;&gt;&gt;::Result&gt;::pop_job() Line 40	C++
LibTorchFramework.exe!torch::data::DataLoaderBase&lt;torch::data::datasets::MapDataset&lt;DefaultDataset,torch::data::transforms::Stack&lt;DataLoaderData&gt;&gt;,DataLoaderData,std::vector&lt;unsigned __int64,std::allocator&lt;unsigned __int64&gt;&gt;&gt;::worker_thread(torch::data::datasets::MapDataset&lt;DefaultDataset,torch::data::transforms::Stack&lt;DataLoaderData&gt;&gt; &amp; dataset={...}) Line 186	C++
LibTorchFramework.exe!torch::data::StatelessDataLoader&lt;torch::data::datasets::MapDataset&lt;DefaultDataset,torch::data::transforms::Stack&lt;DataLoaderData&gt;&gt;,torch::data::samplers::RandomSampler&gt;::{ctor}::__l6::&lt;lambda_1&gt;::operator()() Line 48	C++
[External Code]	
LibTorchFramework.exe!thread_start&lt;unsigned int (__cdecl*)(void *),1&gt;(void * const parameter=0x00000150060de940) Line 97	C++

</code></pre>
<p>I have tried to comment out most of the code, but I cannot pinpoint the problem <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=14" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> No matter what is inside dataLoader loop, it will crashes sooner or later with the same error.</p>

MartinPerry: <p>The problem was incorrect project configuration. One of the external libraries, not related to LibTorch, was build with <code>/MT</code> and  <code>/MTd</code> switch instead of <code>/MD</code> and <code>/MDd</code></p>

ptrblck: <p>Thanks for the update! How did you narrow down the issue (asking just out of curiosity to learn about your debug workflow)?</p>

MartinPerry: <p>First, I have removed all code and ended with a minimal working example - that was still crashing. After that, I removed all external libs and it started working. So I started to adding them back one by one, until it crashed again and I inspected the lib that caused it.</p>
<p>I have also tried to use AI, but it was suggesting incorrect solutions. The only usable was that it created the minimal working example from my code <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>