Custom DataLoader

avinash_m: <p>I saw the tutorial on custom dataloader. However, the class function has loading data functions too. I have tensors pair images, labels. How can I convert them into DataLoader format without using CustomDataset class??</p>

avinash_m: <p><a class="mention" href="/u/yf225">@yf225</a>. Please help me to resolve this issue.</p>

avinash_m: <p>I am getting this error:</p>
<p>error: invalid cast to abstract class type ‘CustomDataset’<br>
auto data = CustomDataset(train_tensor_indices, train_tensor_lab).map(torch::data::transforms::Stack&lt;&gt;());</p>
<p>note: because the following virtual functions are pure within ‘CustomDataset’:<br>
class CustomDataset : public torch::data::Dataset<br>
^<br>
In file included from /libtorch/include/torch/csrc/api/include/torch/data/datasets.h:3:0,<br>
from /libtorch/include/torch/csrc/api/include/torch/data.h:4,<br>
from /libtorch/include/torch/csrc/api/include/torch/all.h:4,<br>
from /libtorch/include/torch/csrc/api/include/torch/torch.h:3,</p>
<p>Custom Dataset class:<br>
<code><br>
class CustomDataset : public torch::data::Dataset {</code></p><code>
<pre><code>private:
    torch::Tensor states_, labels_;

public:
	CustomDataset(torch::Tensor emb_indices, torch::Tensor labels) {
		states_ = emb_indices;
     	labels = labels;
     };
     torch::data::Example&lt;&gt; get(size_t index) override {
     	torch::Tensor embedding_index = states_[index];
     	torch::Tensor Label = labels_[index];
     	return {embedding_index.clone(), Label.clone()};
     }; };
</code></pre>
<code>
auto data = CustomDataset(train_tensor_indices, train_tensor_lab).map(torch::data::transforms::Stack&lt;&gt;());</code></code>

glaringlee: <p><a class="mention" href="/u/avinash_m">@avinash_m</a><br>
I think you need to implement a size() as well. See here:<br>
‘<a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/data/datasets/base.h#L53" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/data/datasets/base.h#L53</a>’<br>
Dataset inherited from BatchDataset and it has a pure virtual function size() which is not overridden.</p>
<p>As the comment said, if you don’t need it, just return an empty c10::optional</p>

avinash_m: <p>Thank you for the suggestion. I will try and let you know.</p>

avinash_m: <p><a class="mention" href="/u/glaringlee">@glaringlee</a> Thanks for the suggestion. It worked. I need one more clarification.though not in scope to this question. The loss functions doesn’t support Long Tensors for backward pass?</p>

glaringlee: <p><a class="mention" href="/u/avinash_m">@avinash_m</a><br>
I need more details on ‘loss’ question.<br>
Do you have any log or error msg? I think we should support Long Tensors (the tensor’s datatype is Long correct? correct me if I am wrong)</p>

avinash_m: <p>Code:<br>
<code><br>
for(auto&amp; batch : trainloader) {<br>
auto inputs = at::_cast_Long(batch.data).to(device);<br>
auto targets  = at::_cast_Long(batch.target).to(device);</code></p><code>
<pre><code>            totalsize = totalsize + inputs.size(0);
            optimizer.zero_grad();
            auto output = Net.forward(inputs);
            auto loss = torch::nn::functional::cross_entropy(inputs, targets);
            loss.backward();
            optimizer.step();
</code></pre>
<p>Error Message:<br>
terminate called after throwing an instance of ‘c10::Error’<br>
what():  “log_softmax_lastdim_kernel_impl” not implemented for ‘Long’ (operator() at aten/src/ATen/native/cpu/SoftMaxKernel.cpp.AVX2.cpp:238)</p>
<p>frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;) + 0x6a (0x7f6e629dcaaa in /libtorch/lib/libc10.so)</p>
<p>frame <span class="hashtag">#1:</span>  + 0x154c8ff (0x7f6e54fe58ff in /libtorch/lib/libtorch_cpu.so)</p>
<p>frame <span class="hashtag">#2:</span> at::native::log_softmax_cpu(at::Tensor const&amp;, long, bool) + 0x54a (0x7f6e548ecc1a in /libtorch/lib/libtorch_cpu.so)</p>
<p>frame <span class="hashtag">#3:</span>  + 0x115ef07 (0x7f6e54bf7f07 in /libtorch/lib/libtorch_cpu.so)</p>
<p>frame <span class="hashtag">#4:</span>  + 0x11b16ab (0x7f6e54c4a6ab in /libtorch/lib/libtorch_cpu.so)</p>
<p>frame <span class="hashtag">#5:</span>  + 0x2d497a2 (0x7f6e567e27a2 in /libtorch/lib/libtorch_cpu.so)</p>
<p>frame <span class="hashtag">#6:</span>  + 0x11b16ab (0x7f6e54c4a6ab in /libtorch/lib/libtorch_cpu.so)</p>
<p>frame <span class="hashtag">#7:</span> at::native::log_softmax(at::Tensor const&amp;, long, c10::optional<a>c10::ScalarType</a>) + 0x160 (0x7f6e548ea0e0 in /libtorch/lib/libtorch_cpu.so)</p>
<p>frame <span class="hashtag">#8:</span>  + 0x1227333 (0x7f6e54cc0333 in /libtorch/lib/libtorch_cpu.so)</p>
<p>frame <span class="hashtag">#9:</span>  + 0x2ce3cf5 (0x7f6e5677ccf5 in /libtorch/lib/libtorch_cpu.so)</p>
<p>frame <span class="hashtag">#10:</span>  + 0x12987cc (0x7f6e54d317cc in /libtorch/lib/libtorch_cpu.so)</p></code>

avinash_m: <p>yes, data is long type.</p>

glaringlee: <p><a class="mention" href="/u/avinash_m">@avinash_m</a><br>
I see.  log_softmax_lastdim_kernel_impl is implemented for floating types. Specifically, it supports double, float and torch::Half only. You might need to use double instead of long.</p>

avinash_m: <p><a class="mention" href="/u/glaringlee">@glaringlee</a>. But Embedding layer only takes Long Tensors as input. I have to use Embedding layers. Is there any work around to avoid this issue? Should I raise an issue in the official github repo?</p>

glaringlee: <p><a class="mention" href="/u/avinash_m">@avinash_m</a><br>
Can you post more code?<br>
Based on the code piece you post here, I think you can cast your inputs and targets to double first. If you have other layers that need long, you can cast them to long later.</p>

avinash_m: <p>Code:<br>
<code><br>
struct RandCNN : nn::Module {<br>
RandCNN(int32_t &amp;vocabsize, int32_t &amp;embeddim, int32_t &amp;maxlen, int32_t &amp;numclasses)</code></p><code>
<pre><code>  : embed(nn::Embedding(nn::EmbeddingOptions(vocabsize, embeddim))),

    conv1(nn::Conv1d(nn::Conv1dOptions(embeddim, kNumfilters, kFilterSizes[0])
    	                 .stride(1)
    	                 .bias(false))),

    conv2(nn::Conv1d(nn::Conv1dOptions(embeddim, kNumfilters, kFilterSizes[1])
    	                 .stride(1)
    	                 .bias(false))),

    conv3(nn::Conv1d(nn::Conv1dOptions(embeddim, kNumfilters, kFilterSizes[2])
    	                 .stride(1)
    	                 .bias(false))),

    pool1(nn::MaxPool1d(nn::MaxPool1dOptions(maxlen - kFilterSizes[0] +1)
    	                    .stride(1))),

    pool2(nn::MaxPool1d(nn::MaxPool1dOptions(maxlen - kFilterSizes[1] +1)
    	                   .stride(1))),

    pool3(nn::MaxPool1d(nn::MaxPool1dOptions(maxlen - kFilterSizes[2] +1)
    	                    .stride(1))),

    fc(nn::Linear(nn::LinearOptions(kNumfilters * 3, numclasses))),

    drop(nn::Dropout(nn::DropoutOptions().p(kDropValue))) {
</code></pre>
<code>
   // register_module() is needed if we want to use the parameters() method later on
<pre><code>register_module("embed", embed);
register_module("conv1", conv1);
register_module("conv2", conv2);
register_module("conv3", conv3);
register_module("poo11", pool1);
register_module("pool2", pool2);
register_module("pool3", pool3);
register_module("fc", fc);
</code></pre>
<p>}</p>
<code>
 torch::Tensor forward(torch::Tensor x) {
<pre><code>x = embed(x);
x = at::transpose(x, 1, 2);
torch::Tensor out1, out2, out3, out;
out1 = torch::relu(conv1(x));
out2 = torch::relu(conv2(x));
out3 = torch::relu(conv3(x));
out1 = pool1(out1);
out2 = pool2(out2);
out3 = pool3(out3);
out = at::_cat({out1, out2, out3}, 1);
out = at::_unsafe_view(out, {at::size(out, 0), at::size(out, 1)});
out = drop(out);
out = fc(out);
//out = torch::log_softmax(out, /*dim=*/1);
</code></pre>
<p>return out;<br>
}</p></code></code></code>

avinash_m: <p>My dataformat will be a dataloader with Indices and Labels. The first layer of model Embedding converts them to embedding vectors. So, inputs will be Long Tensors.</p>

glaringlee: <p>I see.<br>
In your forward function, when calling the log_softmax,  you can do the conversion<br>
out = torch::log_softmax(out.to(torch::kDouble)).to(torch::kLong) if you need Long type as output</p>

avinash_m: <p>okay so i will use long type as input to embedding layer. then type cast to double after this so that it will be backward compatible? But the problem is back-propogation thorugh long tensors is giving error.</p>

glaringlee: <p><a class="mention" href="/u/avinash_m">@avinash_m</a><br>
A typo in my previous reply, it should be out = torch::log_softmax(out.to(torch::kDouble), 1).to(torch::kLong)</p>
<p>It should be backward compatible. Can I see your back-propogation error log?<br>
The previous error log you showed me is caused by calling log_softmax in forward.</p>

avinash_m: <aside class="quote no-group" data-username="avinash_m" data-post="8" data-topic="81874">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/85e7bf/48.png" class="avatar"> avinash_m:</div>
<blockquote>
<p><code>auto loss = torch::nn::functional::cross_entropy(inputs, targets);</code></p>
</blockquote>
</aside>
<p><a class="mention" href="/u/glaringlee">@glaringlee</a> I have identified the issue after careful debugging. In this line I should be passing outputs and targets to calculate loss however I was passing inputs. However, the error log is not useful to trace back the error. It should have reported shapes mismatch instead  was reporting about log_softmax.</p>
<p>Thank you for all the patience and time in helping me throughout.</p>

glaringlee: <p><a class="mention" href="/u/avinash_m">@avinash_m</a><br>
Great! I think the reason log_softmax complain is that your input is Long type <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .</p>

avinash_m: <p>I guess there should have been a better error log. Because clearly it is a shape mismatch.</p>