Make_data_loader() calls dataset's destructor multiple times

username: lampadephoria
cooked: <p>Hello,<br>
I have a CustomDataset listed below with a destructor releases the memory allocated in the constructor.</p>
<pre><code class="lang-auto">class CustomDataset : public torch::data::Dataset&lt;CustomDataset&gt;{
public:
    CustomDataset() {
        testArray_ = new float[1024];
    };
    ~CustomDataset() override {
        if (testArray_) {
            delete testArray_;
            testArray_ = nullptr;
        }
    };

    torch::optional&lt;size_t&gt; size() const override;
    torch::data::Example&lt;&gt; get(size_t index) override;
private:
    float* testArray_;
};
</code></pre>
<p>When I register it to dataloader as below</p>
<pre><code class="lang-auto">    dataset = new CustomDataset();

    auto dataLoader = torch::data::make_data_loader&lt;torch::data::samplers::SequentialSampler&gt;(
            (*dataset).map(torch::data::transforms::Stack&lt;&gt;()),
            torch::data::DataLoaderOptions()
                    .batch_size(batchSize)
                    .workers(nWorkers)
                    .drop_last(true));//Shuffle by default
</code></pre>
<p>make_data_loader() appears to call ~CustomDataset() multiple times. CustomDataset() is only called once in the “new” call. Interestingly when ~CustomDataset() is called the second time, testArray_ value is exactly the same as in the first time. That triggers SIGABRT. Here is the call stack.</p>
<pre><code class="lang-auto">Thread 11 "cas" received signal SIGABRT, Aborted.
0x00007f01a3682387 in raise () from /lib64/libc.so.6
(gdb) where
#0  0x00007f01a3682387 in raise () from /lib64/libc.so.6
#1  0x00007f01a3683a78 in abort () from /lib64/libc.so.6
#2  0x00007f01a36c4ed7 in __libc_message () from /lib64/libc.so.6
#3  0x00007f01a36cd299 in _int_free () from /lib64/libc.so.6
#4  0x00007f0174d03b0f in CustomDataset::~CustomDataset (this=0x7f0177369130)
    at .../dataset.hpp:31
#5  0x00007f0174da4247 in torch::data::datasets::BatchDataset&lt;CustomDataset, std::vector&lt;torch::data::Example&lt;at::Tensor, at::Tensor&gt;, std::allocator&lt;torch::data::Example&lt;at::Tensor, at::Tensor&gt; &gt; &gt;, c10::ArrayRef&lt;unsigned long&gt; &gt;::map&lt;torch::data::transforms::Normalize&lt;at::Tensor&gt; &gt;(torch::data::transforms::Normalize&lt;at::Tensor&gt;) &amp; (
    this=0x7f01715d5b10, transform=...)
    at .../pytorch/torch/include/torch/csrc/api/include/torch/data/datasets/base.h:58
</code></pre>
<p>My questions are:</p>
<ol>
<li>Why is the destructor called (at least) twice and the constructor is called only once?</li>
<li>Why the values of a member variable are the same in both destructor calls?</li>
<li>Does this mean that CustomDataset class cannot implement destructor?</li>
</ol>
<p>Thank you very much.</p>

username: lampadephoria
cooked: <p>The first caller of the destructor is</p>
<pre><code class="lang-auto">(gdb) f 1
#1  0x00007f6bb426cddb in torch::data::datasets::map&lt;CustomDataset, torch::data::transforms::Normalize&lt;at::Tensor&gt; &gt; (dataset=..., transform=...)
    at .../pytorch/torch/include/torch/csrc/api/include/torch/data/datasets/map.h:112
112       return {std::move(dataset), std::move(transform)};
(gdb) list
107                   DatasetType::is_stateful,
108                   typename DatasetType::BatchType::value_type,
109                   typename DatasetType::BatchType&gt;::type,
110               typename TransformType::InputBatchType&gt;::value,
111           "BatchType type of dataset does not match input type of transform");
112       return {std::move(dataset), std::move(transform)};
113     }
114
115     } // namespace datasets
116     } // namespace data
</code></pre>
<p>The second caller of the destructor is</p>
<pre><code class="lang-auto">#1  0x00007f6bb4257247 in torch::data::datasets::BatchDataset&lt;CustomDataset, std::vector&lt;torch::data::Example&lt;at::Tensor, at::Tensor&gt;, std::allocator&lt;torch::data::Example&lt;at::Tensor, at::Tensor&gt; &gt; &gt;, c10::ArrayRef&lt;unsigned long&gt; &gt;::map&lt;torch::data::transforms::Normalize&lt;at::Tensor&gt; &gt;(torch::data::transforms::Normalize&lt;at::Tensor&gt;) &amp; (
    this=0x7f6bb15cc4c0, transform=...)
    at .../pytorch/torch/include/torch/csrc/api/include/torch/data/datasets/base.h:58
58          return datasets::map(static_cast&lt;Self&amp;&gt;(*this), std::move(transform));
(gdb) list
53        virtual optional&lt;size_t&gt; size() const = 0;
54
55        /// Creates a `MapDataset` that applies the given `transform` to this dataset.
56        template &lt;typename TransformType&gt;
57        MapDataset&lt;Self, TransformType&gt; map(TransformType transform) &amp; {
58          return datasets::map(static_cast&lt;Self&amp;&gt;(*this), std::move(transform));
59        }
60
61        /// Creates a `MapDataset` that applies the given `transform` to this dataset.
62        template &lt;typename TransformType&gt;
</code></pre>
<p>Any idea on my questions? Thank you for your suggestions.</p>

username: ptrblck
cooked: <p>CC <a class="mention" href="/u/yf225">@yf225</a> who might know, what’s going on.</p>