Access custom dataset inside the dataloader loop

lampadephoria: <p>Hello,<br>
I have created a custom dataset. Within get(), I recorded the sample index as well as the filename in std::vector.</p>
<pre><code class="lang-auto">torch::data::Example&lt;&gt; CustomDataset::get(size_t index) {
    torch::Tensor           tData;
    torch::Tensor           tLabel;
    std::string                tPath;
....
    //set tPath to an image filename. set tLabel as well.

    auto mat = cv::imread(tPath);
...
    tData = torch::from_blob(mat.data, {mat.rows, mat.cols, 3}, torch::kByte);
    tData = tData.to(torch::kFloat);
    tData = tData.permute({2, 0, 1}); // Channels x Height x Width

    path_.push_back(tPath);
    index_.push_back(index);
    return {tData.clone(), tLabel.clone()};
}
</code></pre>
<p>Since I can’t change get() signature, I rely on accessing CustomDataset in the data loading loop in order to access members of CustomDataset. In below example, I set the batch size to 10. During runtime, batch.data.size(0) is 10. However customDataset.path_.size() is 0.</p>
<pre><code class="lang-auto">    auto customDataset = CustomDataset();
    auto dataLoader = torch::data::make_data_loader&lt;torch::data::samplers::SequentialSampler&gt;(
            customDataset//std::move(customDataset)
                    .map(torch::data::transforms::Normalize&lt;&gt;({0.5, 0.5, 0.5},{0.5,0.5,0.5}))
                    .map(torch::data::transforms::Stack&lt;&gt;()),
            torch::data::DataLoaderOptions()
                    .batch_size(10)
                    .workers(actionParameters.nThreads)
                    .drop_last(true));

    for (auto&amp; batch : *dataLoader) {
        std::cout &lt;&lt; "score::runAction(): customDataset.path_.size() = " &lt;&lt; customDataset.path_.size() &lt;&lt; std::endl;
        std::cout &lt;&lt; "score::runAction(): batch.data.size(0) = " &lt;&lt; batch.data.size(0) &lt;&lt; std::endl;
...
        customDataset.path_.clear();
        customDataset.index_.clear();
    }
</code></pre>
<p>My impression is that dataloader created a copy of customDataset after make_data_loader(). The original customDataset object is no longer needed. Thus std::move() semantics is more efficient.</p>
<p>I saw this post in Python API discussing dataloader reset the state of dataset.</p><aside class="quote quote-modified" data-post="4" data-topic="27960">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-resets-dataset-state/27960/4">Dataloader resets dataset state</a> 
  </div>
  <blockquote>
    Yeah, I understand the issue and stumbled myself a few times over it. 
I think one possible approach would be to use shared memory in Python e.g. with <a href="https://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes" rel="noopener nofollow ugc">multiprocessing.Array</a>. 
You could initialize an array of your known size for the complete Dataset, fill it in the first iteration using all workers, and finally switch a flag indicating the cache/shared memory should be used in all following epochs. 
I’ve created a small dummy example showing this behavior. 
Currently the shared memory will be fil…
  </blockquote>
</aside>
<p>
This post has <a class="mention" href="/u/ptrblck">@ptrblck</a> suggested solution using <code>loader.dataset</code> to access the embedded dataset.</p>
<pre><code class="lang-auto">for epoch in range(2):
    for idx, data in enumerate(loader):
        print('Epoch {}, idx {}, data.shape {}'.format(epoch, idx, data.shape))
        
    if epoch==0:
        loader.dataset.set_use_cache(True)
</code></pre>
<p>I wonder what is the equivalent in the C++ world. How can I access the associated custom dataset after make_data_loader()?<br>
Thank you very much.</p>