akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Bug or feature for longTensor?

username: Zhaoyi-Yan
cooked: <pre><code class="lang-python">import torch
a = (torch.rand(3,4)*10).long()
print(a)
print(a*0.9)
</code></pre>
<p>Output:</p>
<pre><code class="lang-bash">tensor([[8, 5, 3, 4],
        [1, 3, 6, 7],
        [8, 5, 8, 8]])
tensor([[0, 0, 0, 0],
        [0, 0, 0, 0],
        [0, 0, 0, 0]])
</code></pre>

username: Prerna_Dhareshwar
cooked: <p>Hi,</p>
<p>I don’t think thats a bug, it tries to retain the type of the tensor as long hence the rounding of 0.9 to 0.</p>
How to manage RAM capacity while loading DataLoader in deep learning?

username: mmdbrdrn
cooked: <p>Hello every one.</p>
<p>I am working on video processing and as data samples, i need short video clips (around 10 frame per each).<br>
I have a dataset, composed of consecutive images, and what i want to do is combining consecutive 10 frames to make a cuboid. Now the problem is: for big datasets or even for multiple normal datasets, when i create numpy arrays containing plenty of cuboids (10 * 227* 227 : 10 frames of size 227*227), the Computer RAM gets fully occupied and kernel dies!!</p>
<p>Maybe my algorithm has a bad mistake and i am unaware of a simple way, but i would be haapy if anyone give me useful hints to manage my RAM</p>
<p>Thank you very much</p>

username: ptrblck
cooked: <p>The storage of 10 frames of shape <code>227x227</code> shouldn’t cause any problems itself.<br>
Assuming that you are dealing with RGB images, you would only use approx. <code>10*227*227*3*4 / 1024**2 = 5.9MB</code>, if you store the images in FP32.</p>
<p>Do you run out of memory directly after loading the data or could some other code part (e.g. model forward/backward) cause this issue?</p>

username: mmdbrdrn
cooked: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a>  for your reply.<br>
First, there are plenty of these cuboids  ( 4 datasets such as UCSD, AVenue,…). Even considering the possible augmentations, it gets even higher.<br>
yes in fact i read, process the datasets first, to make cuboid and it makes me run out of memory even before  training! (whole 32GB RAM)<br>
some times i think about loading dataset while training ( not pre-loading the all dataset at once) but i think it makes the training so slow ( because GPU should wait for new data samples from CPU, if i am not wrong!).<br>
what is the standard approach for large datasets?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>You should not load whole datasets into your RAM if you do not have enough RAM to do that. You should use lazy loading provided in DataLoaders in PyTorch. As you mentioned, You have to prepare next data when your GPU is processing.</p>
<p>By the way, This is best and most efficient solution to task such this regarding huge datasets.</p>

username: mmdbrdrn
cooked: <p>Than you very much <a class="mention" href="/u/nikronic">@Nikronic</a>.<br>
thanks for your great information.</p>
Where is the best place to put nn.Sequential?

username: Yonghyun_Cho
cooked: <p>Which one is more PyTorch like (better) code for nn.Sequential?</p>
<pre><code class="lang-auto">   def __init__(self):
       self.layers = nn.Sequential (layer1, layer2, ...)

   def forward(self, x):
       self.layers(x)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">   def __init__(self):
       self.layer1, self.layer2, ...

   def forward(self, x):
       ss = nn.Sequential (layer1, layer2, ...)
</code></pre>
<p>Both would be okay?</p>

username: ajhanwar
cooked: <p>You wouldn’t put it in such a structure. You could just write it as <code>ss = nn.Sequential(layer1, activation, layer2, etc.)</code></p>
<p><code>nn.Sequential</code> is a sequential container that allows you to build a neural network by specifying the sequential building blocks of the model.</p>
Operation on a tensor without using loop

username: zahra
cooked: <p>Hi,</p>
<p>I have a tensor like this:</p>
<pre><code class="lang-auto">tensor([[[[ 1.,  2.,  3., 4., 5., 6., 7., 8.]]],
        [[[9., 10., 11., 12., 13., 14., 15., 16.]]]])
</code></pre>
<p>and I want to do a summation <strong>without any loop</strong> like this:</p>
<pre><code class="lang-auto">1+2 + 5+6
2+3 + 6+7
3+4 + 7+8
</code></pre>
<p>as the input tensor is in a batch mode, the operation should be done in batch mode.<br>
I think it is possible by changes the shape of tensor, but I do not know how to organize it.<br>
Do you have any idea about it?</p>
<p>Thanks<br>
Best Regards</p>

username: zahra
cooked: <p>I think I could solve it. For who has the same problem, here is the code:</p>
<pre><code class="lang-auto">a = torch.tensor([[[[ 1.,  2.,  3., 4., 5., 6., 7., 8.]]],
        [[[9., 10., 11., 12., 13., 14., 15., 16.]]]])
print(a)
b = a.view(2,1,2,4)
print(b)
c = b.unfold(2, 2, 2).unfold(3, 2, 2).contiguous().view(2, 2, 2, 2)
print(c)
d= c.sum(2, keepdim=True).sum(3, keepdim=True)
print(d)
</code></pre>
<p>Regards</p>
Selecting a square submatrix

username: calincru
cooked: <p>What is the most efficient way of selecting the square sub-matrix of a matrix <code>x</code> that is formed of the row and column intersections at some <code>indices</code>?  Is there a faster approach than <code>x[indices][:, indices]</code>?</p>
<p>I’m asking because I remember seeing this (seemingly unsolved) issue: <a href="https://github.com/pytorch/pytorch/issues/15245" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/15245</a>. But I’m not sure if it affects this case.</p>
Torch.nn.init.zeros_

username: ahhhhah
cooked: <p>I’m new to Pytorch, I feel puzzled when I run the code:<br>
<code>from torch.nn.init import zeros_</code></p>
<p>and use it to initialize some veriables:<br>
<code>zeros_(model.bias)</code></p>
<p>and I got an error:  <strong>ImportError: cannot import name ‘zeros_’</strong> <img src="https://discuss.pytorch.org/images/emoji/apple/tired_face.png?v=9" title=":tired_face:" class="emoji" alt=":tired_face:"></p>
<p>where torch.version is 0.4.0</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>I ran your code, and I did not get any error.<br>
If you want to init with constant values, here is a better approach:</p>
<pre><code class="lang-python">torch.nn.init.constant_(tensor, 0)
</code></pre>
<p>Bests<br>
Nik</p>

username: ahhhhah
cooked: <p>Hi,</p>
<p>Thanks for your reply!<br>
I will try to use the approach to init with constant values.</p>
<p>I solve this problem by updating the torch version.</p>
<p>when the torch version is 0.4.0</p>
<pre><code class="lang-auto">import torch
dir(torch.nn.init)
</code></pre>
<p>and can not find ‘zeros_’ in the result.  So I updata the torch version(0.4.1).</p>
<p>Thanks ! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

username: Nikronic
cooked: <p>You are welcome.</p>
<p>I think it is great idea to use latest version. There are a lot of optimization and development now. Furthermore, your code will be usable for different purposes.</p>
<p>Good luck</p>
How to calculate tensor mean?

username: wonchulSon
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/dafd335525fa14e05c65ba5dae664232957a9c10.png" alt="image" data-base62-sha1="vfgze1Po8fcRqFXarcCBoHD9w9q" width="591" height="173"></p>
<p>I want to calculate <code>mean</code> by <code>y-axis</code>, so the output should be 1x3.<br>
How should I do?</p>
<p>Thank you.</p>

username: LeviViana
cooked: <pre><code class="lang-python">import torch
x = torch.rand(3, 3)
print(x.mean(1))
</code></pre>

username: wonchulSon
cooked: <p><code>x.mean(0)</code> was what I wanted.<br>
Thank you so much^^!!</p>
List of high and low level IR

username: Milindn
cooked: <p>How do I get a list of all IR ops in glow?</p>

username: jfix
cooked: <p>You can look to a couple places in code depending on what exactly you’re looking for.</p>
<p>You can see all of Glow’s Nodes <a href="https://github.com/pytorch/glow/blob/master/tools/ClassGen/NodeGen.cpp#L37" rel="nofollow noopener">here</a>. But it generally makes more sense to see if a Node is supported on a specific backend, and with a specific precision.</p>
<p>So you can see whether a particular backend supports a Node with a specific precision by looking at <code>Backend::isOpSupported()</code>. For example, <a href="https://github.com/pytorch/glow/blob/2605951a3fabe63c23e3da159b8b71864a547458/lib/Backends/CPU/CPUBackend.cpp#L43" rel="nofollow noopener">here is that function for our CPU backend</a>.</p>

username: Milindn
cooked: <p>Thanks Jordan. Can you point me to any description (document or source code) for memory management pass within the compiler?</p>

username: jfix
cooked: <p>Sure! Just to be clear, we mainly do memory-related management/optimizations with our low-level IR, meaning only backends which use our low-level IR benefit from them. That includes our CPU, OpenCL, and Interpreter backends, but not Habana which skips low-level IR. Here are some links you might find useful:</p>
<ul>
<li><a href="https://github.com/pytorch/glow/blob/bdf0a045738fff1195055354123a8e29b3a9baac/lib/IR/ChildMemSizeBasedScheduler.cpp" rel="nofollow noopener">Our default low-level IR scheduler which tries to minimize memory usage</a></li>
<li>Our IR Optimizer does many memory related optimizations such as <a href="https://github.com/pytorch/glow/blob/bdf0a045738fff1195055354123a8e29b3a9baac/lib/Optimizer/IROptimizer/IROptimizer.cpp#L1174" rel="nofollow noopener">buffer sharing</a>, <a href="https://github.com/pytorch/glow/blob/bdf0a045738fff1195055354123a8e29b3a9baac/lib/Optimizer/IROptimizer/IROptimizer.cpp#L1331" rel="nofollow noopener">skipping unnecessary intermediate allocations</a>, <a href="https://github.com/pytorch/glow/blob/bdf0a045738fff1195055354123a8e29b3a9baac/lib/Optimizer/IROptimizer/IROptimizer.cpp#L256" rel="nofollow noopener">alloc activation sinking</a>, and many more found in that file, many (but not all) of which are listed/described <a href="https://github.com/pytorch/glow/blob/master/docs/Optimizations.md#set-of-supported-ir-optimizations" rel="nofollow noopener">here</a>
</li>
<li><a href="https://github.com/pytorch/glow/blob/master/docs/IR.md#low-level-ir" rel="nofollow noopener">And here’s a general description of our low-level IR</a></li>
</ul>
<p>Hope that helps!</p>
About Aten's dispatch mechanism

username: dongfangduoshou123
cooked: <p>The comment in AtenDispatch.h:<br>
[// This dispatch class serves as a replacement for our previous dispatch<br>
// mechanism, in which all functions were members of a Type class. A derived<br>
// class existed for each backend (and Variable), and the vtable was used to<br>
// dispatch to the correct implementation. This class is to be replaced by<br>
// the c10 dispatcher when it supports all argument and return types.<br>
// This implementation opts to store implementations in a table of void*.]</p>
<p>This means now there are 3 kinds of dispatch mechanisms for route the operators(LegacyTypeDispatch AtenDispatch and c10Dispatcher),  I see the c10Dispatcher is created many months ago, why along time has passed, this final dispatcher is still cannot used and not continuesly develope it?  why stop sometime to develop c10disparcher and switch todevelope a AtenDispatch, which is also a temp disign?</p>
Can' t import torch.optim.lr_scheduler

username: HYC
cooked: <p>When import torch.optim.lr_scheduler in PyCharm, it shows that AttributeError: module ‘torch.optim’ has no attribute ‘lr_scheduler’. But in the Pytorch’ s documents, there is torch.optim.lr_scheduler. So why torch.optim.lr_scheduler can’ t import?</p>

username: moskomule
cooked: <p>Hi, which version of PyTorch do you use? I think you see the doc for the master branch but use 0.12.</p>

username: HYC
cooked: <p>Thank you! You are right. If I want to use torch.optim.lr_scheduler, how to set up the corresponding version of PyTorch?</p>

username: moskomule
cooked: <p>Currently the latest version is 0.12 which you use. So if you like to use the latest PyTorch, I think <a href="https://github.com/pytorch/pytorch#from-source" rel="nofollow noopener">install from source</a> is the only way.</p>

username: HYC
cooked: <p>Okay, thank you very much!</p>

username: htd
cooked: <p>Check your local package, if necessary, add <a href="https://github.com/pytorch/pytorch/blob/d89d9d74bdd0b64c119980003aa29195a61f93a9/torch/optim/__init__.py#L18" rel="nofollow noopener">this line</a> to initialize lr_scheduler. I find my pip-package doesn’t have this line.</p>

username: Bailey
cooked: <p>i found my pip-package also doesn’t have this line. can i just add this line to my <em>init</em>.py ? how solve this problem?? thx</p>

username: Nikhil_Rai1
cooked: <p>I am using the the pytorch_version ‘0.1.12’ but getting the same error.<br>
One more thing is I am working in virtual environment.<br>
Is this is the problem with respect to virtual environment?<br>
Thank you in advance</p>

username: ptrblck
cooked: <p>You are using a very old PyTorch version. Have a look at <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">the website</a> for the install instructions for the latest version.</p>
Channels wise Linear Module

username: vHitsuji
cooked: <p>Hi everyone,<br>
in my case I have to apply many tensors (of same size) to many Linear Modules (respectively).<br>
These computations are independent and the order doesn’t matter.<br>
To make the Gpu the most efficiently, I wanted to apply these computations using the least number of calls to the Gpu.<br>
I decided to design a Channels wise Linear Module which is base on the Pytorch’s Linear Module :</p>
<pre><code class="lang-auto">class multiChannelsLinear(nn.Module):

    __constants__ = ['bias']

    def __init__(self, channels, in_features, out_features, bias=True):
        super(multiChannelsLinear, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.channels = channels
        self.weight = nn.Parameter(torch.Tensor(channels, out_features, in_features))
        if bias:
            self.bias = nn.Parameter(torch.Tensor(channels, out_features))
        else:
            self.register_parameter('bias', None)
        self.reset_parameters()

    def reset_parameters(self):
        nn.init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = nn.init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in)
            nn.init.uniform_(self.bias, -bound, bound)

    def forward(self, input):
        input = input.transpose(0, 2).transpose(0, 1)
        output = self.weight.matmul(input)
        output = output.transpose(0, 1).transpose(0, 2)
        if self.bias is not None:
            output += self.bias
        ret = output
        return ret

    def extra_repr(self):
        return 'channels={}, in_features={}, out_features={}, bias={}'.format(
            self.channels, self.in_features, self.out_features, self.bias is not None
        )

</code></pre>
<p>It seems that the code doesn’t produce the same thing than using many Linear Modules.<br>
I cannot find were I’m wrong,<br>
By the way, why this kind of module does not exist in Pytorch?<br>
I think it would be easier to reach better performance when we want to apply the same kind of transformation on different channels.</p>
Generate/manipulate computation graph

username: abhi-iyer
cooked: <p>Is there a way to generate and iterate successively through a computation graph in PyTorch, similar<br>
to how Tensorflow returns a Graph (or GraphDef) object?</p>
<p>I understand you can use torchviz/graphviz to visualize the computation graph, but is there a way to get the graph in the finest granularity possible, i.e. the individual mult/add/etc. ops?</p>
<p>thanks!</p>

username: abhi-iyer
cooked: <p>When I say generate a computational graph, I mean the following:</p>
<p>How do I express my net as a function <code>f(x)</code> such that <code>f</code> just consists of the most basic algebraic operations (add, subtraction, multiplication, division, etc.)?</p>
<p>Case A: Imagine my model just consists of linear layers and activation functions (say ReLU). I can break down my linear layers with the ops <em>mult</em> and <em>add</em> (because a linear layer is just A*x + b, where x = input), but how would I break down ReLU into algebraic ops?</p>
<p>Case B: Imagine my model is much more complicated, and now consists of Convs, MaxPools, and TanH’s (among other activation functions). How do I <em>break down</em> all of these into algebraic ops?</p>
<p>Given this, is there any way to trace the algebraic ops used in the net, starting from the input data to reach the output?</p>
<p>any help at all is appreciated. thanks!</p>
Is there any code torch.backends.cudnn.benchmark/torch.cuda.synchronize() similar in tensorflow?

username: jjmar98
cooked: <p>I am working on optimizing CUDA program’s performance. I used  <code>torch.backends.cudnn.benchmark</code> to optimize performance and  <code>torch.cuda.synchronize()</code>  to synchronize CUDA applications in pytorch. To do the same job in tensorflow I searched a lot time whether similar code is in tensorflow, however I could’nt find anything.</p>
<p>I wonder is there any equivalent code to  <code>torch.cuda.synchronize()</code>  and  <code>torch.backends.cudnn.benchmark</code>  exists in tensorflow.</p>

username: ptrblck
cooked: <p>I’m not sure, if you’ll get the best answer about Tensorflow in this discussion boad and I would recommend to use their discussion platforms (stack overflow and github issues).</p>
<p>That being said, may I ask, how you’re optimizing the performance using <code>torch.cuda.synchronize()</code> calls?</p>

username: jjmar98
cooked: <p>Thanks! Actually, I found that tf.sess.run is always synchronized. So tensorflow doesn’t need CUDA synchronization.<br>
I was running multiple processes using CUDA MPS, so it was essential to ensure every process is synchronized.<br>
Thank you for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Dropout weird behavior

username: vainaijr
cooked: <p>why does it drop sometimes more, sometimes less than 3 elements in a 2x5 matrix, when I have given probability 0.3<br>
y = torch.ones(2, 5)<br>
x = nn.Dropout(0.3)<br>
x(y)<br>
if got this output<br>
tensor([[1.4286, 1.4286, 1.4286, 0.0000, 0.0000],<br>
[1.4286, 1.4286, 1.4286, 1.4286, 1.4286]])</p>
<p>on later execution, this output<br>
tensor([[0.0000, 0.0000, 1.4286, 1.4286, 0.0000],<br>
[1.4286, 0.0000, 0.0000, 1.4286, 1.4286]])</p>

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<p>Each unit is dropped with a probability 0.3. In your case, you have 10 values.  You may expect exactly 3 nodes to be dropped every time. It does not need to be exactly 3. Sometimes, it might be 2. Sometimes it can be 4 or even 5. I can one intuitive example.</p>
<p>Suppose you have an unbiased coin. P(H)=P(T)=0.5 (Here H-Head, T-Tail). If you toss the coin 10 times, on average you get 5 Heads and 5 Tails. But it can be 4 Heads and 6 Tails. The probability enforces that the expected number of heads in 10 tosses is 5.</p>
<p>Thanks<br>
Regards<br>
Pranavan</p>

username: vainaijr
cooked: <p>so what do I need to do to drop exactly 3 of 10 elements, i.e. I want to drop exactly 3 elements every time?<br>
not more, not less.</p>

username: Pranavan_Theivendira
cooked: <p>In neural network, the randomness that we have is based on the probability. As far as I know, we do not have any modules which can enforce an exact behaviour. If you want the exact behaviour, you can select 3 indices at random and you can make it as zero. In that case, you have to write custom layers or modules in pytorch which extend the class <code>nn.Module</code>. Then you can plug it as a layer, in your code body.</p>
<p>Thanks</p>
nn.Upsample() don't support tuple scale_factor

username: 111116
cooked: <p>In the source code, there is float(scale_factor),but the document supports tuple.</p>

username: ptrblck
cooked: <p>The current master branch supports a tuple for the <code>scale_factor</code> (<a href="https://github.com/pytorch/pytorch/blob/83b0fbc38ddfabc8fe67353382abdfda88e65acb/torch/nn/modules/upsampling.py#L123" rel="nofollow noopener">line of code</a>).<br>
Which PyTorch version are you using?<br>
It looks like <a class="mention" href="/u/vishwakftw">@vishwakftw</a> added this functionality in <a href="https://github.com/pytorch/pytorch/pull/20581" rel="nofollow noopener">this PR</a>.</p>
Time increases per iteration of einsum

username: JohnM
cooked: <p>Hi,</p>
<p>I’m using torch.einsum() in a loop and noting some very strange behaviour, perhaps not related to einsum itself.</p>
<p>When I run it for 10, 100, and 1000 iterations, it takes ~0.003s, ~0.01s, and ~7.3s respectively:</p>
<blockquote>
<p>10 trials: 0.0027196407318115234 s<br>
100 trials: 0.010590791702270508 s<br>
1000 trials: 7.267224550247192 s</p>
</blockquote>
<p>It seems like the time per call increases drastically after about 250 iterations, see the following plot: <img src="https://discuss.pytorch.org/uploads/default/original/2X/e/eab734637b5d873d52a2fab39412aab368e2611e.png" alt="time" data-base62-sha1="xuof16nsHBIXHLHn5k3vkqznCs6" width="640" height="480"></p>
<p>To reproduce:</p>
<pre><code class="lang-auto">import time

import numpy as np
import torch

np.random.seed(123)
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

d = 4
p = 500
N = 8000

Z = torch.tensor(np.random.randn(N, p), device=device).float()
V = torch.tensor(np.random.randn(d, p, p), device=device).float()

for trials in [10, 100, 1000]:
	start = time.time()
	for i in range(trials):
		var = torch.einsum("dpq,np,nq-&gt;nd", V, Z, Z)
	print(f'{trials} trials: {time.time() - start} s')
</code></pre>
<p>Any ideas? I already tried using torch.no_grad() et.c. to no effect.</p>
<p>Thanks,<br>
John</p>

username: JoelSjogren
cooked: <p>Duplicate of <a href="https://github.com/pytorch/pytorch/issues/15793" class="inline-onebox" rel="noopener nofollow ugc">Sudden decrease in performance after N numbers of consecutive calls to forward (on GPU) · Issue #15793 · pytorch/pytorch · GitHub</a>.</p>
<blockquote>
<p>You are not measuring timing accurately. CUDA launches are asynchronous…</p>
</blockquote>
<blockquote>
<p>torch.cuda.synchronize()</p>
</blockquote>
Output varies when changing batch size (during test)

username: dirtbag
cooked: <p>I’m using a resnet18 network from torchvision.models.</p>
<p>During test time, I observe If I alter the batch size for the data loader, the accuracy of the model on test data changes. I can not comprehend why should this happen. Isn’t the case that the weights of the network are fixed during testing(I haven’t called any optimizer.step())? I also went through resnet’s architecture, there is no randomized output at any layer.</p>
<p>Any pointers where I could be wrong in my code or my information?</p>

username: dirtbag
cooked: <p>I think it’s because of the BatchNorm layer. The effect was more visible with smaller batch sizes. Accuracy dropped from 99.7% for a batch size of 25 to 14.85 % for a batch size of 1.</p>

username: ptrblck
cooked: <p>Have to set the model to evaluation mode with <code>model.eval()</code>? The batch size should not change the predictions!</p>

username: dirtbag
cooked: <p>Thanks! After <code>model.eval()</code> is called, the model starts using running mean and variance for normalization. I never went through this part of the documentation.</p>

username: luciolis
cooked: <p>Hey ! I got the same issue on my code. The results are very bad with a batch size of 1, which is not very practical for evaluation of a single image.</p>
<p>I put <code>model.eval()</code> as well as</p>
<pre><code class="lang-auto">    for child in model.children():
            if type(child) == nn.BatchNorm2d:
                child.track_running_stats = False
</code></pre>
<p>EDIT: does eval makes the use of the running stats but do not save the variance and mean ? Are the variance and mean supposed to scale with the batch size ?</p>
<p>Any ideas ? The network is a stock ResNet adapted for regression instead of classification</p>

username: Zhihan_Yang
cooked: <p>This one-liner basically saved my day. Thanks a lot!</p>
Keras like behaviour is needed

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<p>Implementing an autoregressor in Keras is something like the following code snippet.</p>
<pre><code class="lang-auto">def network_autoregressive(x):

    ''' Define the network that integrates information along the sequence '''

    x = keras.layers.GRU(units=256, return_sequences=False, name='ar_context')(x)


    return x
</code></pre>
<p>Here the input x is 32x4x128. And the output is 32x256. I am not able to produce a similar behaviour using pytorch GRU. Any help is much appreciated.</p>
<p>Thanks<br>
Regards<br>
Pranavan</p>

username: Pranavan_Theivendira
cooked: <p>Thanks for the people who read this topic. I found the solution. In keras, <code>return_sequences=False</code> actually returns the final output in the sequence dimension. Hence, it can be sliced from the GRU output of pytorch.</p>
Create dataset from a python list of path

username: Camille
cooked: <p>Good day to all.<br>
I’m currently building a NLP model for document classification.<br>
I have limited amount of computation power and storage, so I was wondering if it was possible to create a pytorch dataset from a python list of path, where every path leads to a different document.<br>
I apologise if my question is not clear enough, and am prepared to give any more information that could be helfpful.<br>
Thank you for your time.<br>
Camille</p>

username: Balamurali_M
cooked: <aside class="quote quote-modified" data-post="2" data-topic="48290">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/balamurali_m/48/12574_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-for-semantic-segmentation/48290/2">Dataloader for semantic segmentation</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I am assuming there is a folder which contains sub-folders image and mask. The filenames are same. This is a template you can modify it and use. All the best. 
import torch.utils.data as data

class DataLoaderSegmentation(data.Dataset):
    def __init__(self, folder_path):
        super(DataLoaderSegmentation, self).__init__()
        self.img_files = glob.glob(os.path.join(folder_path,'image','*.png')
        self.mask_files = []
        for img_path in img_files:
             self.mask_files.a…
  </blockquote>
</aside>

<p>Follow this, it is done for semantic segmentation. I think a similar thing can be worked out for your case.</p>
NLL loss with K dimensional input

username: Matej_Kompanek
cooked: <p>I am trying to rewrite following python sample to C++ (use of nll loss)</p>
<pre><code class="lang-auto">&gt;&gt;&gt; # 2D loss example (used, for example, with image inputs)
&gt;&gt;&gt; N, C = 5, 4
&gt;&gt;&gt; loss = nn.NLLLoss()
&gt;&gt;&gt; # input is of size N x C x height x width
&gt;&gt;&gt; data = torch.randn(N, 16, 10, 10)
&gt;&gt;&gt; conv = nn.Conv2d(16, C, (3, 3))
&gt;&gt;&gt; m = nn.LogSoftmax(dim=1)
&gt;&gt;&gt; # each element in target has to have 0 &lt;= value &lt; C
&gt;&gt;&gt; target = torch.empty(N, 8, 8, dtype=torch.long).random_(0, C)
&gt;&gt;&gt; output = loss(m(conv(data)), target)
&gt;&gt;&gt; output.backward()
</code></pre>
<p>I have the following code</p>
<pre><code class="lang-auto">long long N = 5;
long long C = 4;

auto data = torch::randn({ N, 16, 10, 10 });
auto conv = torch::nn::Conv2d(torch::nn::Conv2dOptions(16, C, 3));
auto target = torch::empty({ N, 8, 8 }, torch::kLong).random_(0, C);

data = conv-&gt;forward(data);
data = torch::log_softmax(data, 1);
		
auto output = torch::nll_loss(data, target);
</code></pre>
<p>It crashes with error: multi-target not supported at …\aten\src\THNN/generic/ClassNLLCriterion.c:20<br>
Any ideas ? Thanks</p>

username: ptrblck
cooked: <p>The code and shapes of your output and target are generally alright.<br>
However, since you are trying to calculate a loss using spatial inputs, you would have to use <code>torch::nll_loss2d</code> as the loss function.</p>

username: Matej_Kompanek
cooked: <p>Thanks for the reply!</p>
<p>However, I am actually working with 3D data (I don’t see a nll_loss3d function), so I will have to find a workaround.</p>

username: tom
cooked: <p>Hi Matej,</p>
<p><a href="https://github.com/pytorch/pytorch/blob/2938299de1773425c78009eb9041b06e80a302e9/torch/nn/functional.py#L1828-L1844" rel="nofollow noopener">check out the Python implementation of F.nll_loss</a> for inspiration.<br>
Of course, if you moved those bits (and even the dispatch to 2d potentially) into nll_loss on the C++ side in PyTorch itself, you could clean up your code. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Best regards</p>
<p>Thomas</p>
How to traverse all layers of the model

username: Cverlpeng
cooked: <p>hi,I want to init weights of all layers of the model,  How to traverse all layers of the model?<br>
if my  model  has nested nn.Sequential modules</p>
<pre><code class="lang-python">Sequential(
  (0): Bottleneck(
    (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace)
    (downsample): Sequential(
      (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (1): Bottleneck(
    (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace)
  )
</code></pre>
<p>thanks</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>First you need to define your weight initializer function regarding one layer.<br>
Here is an example:</p>
<pre><code class="lang-python">def init_weights(m):
    """
    Initialize weights of layers using Kaiming Normal (He et al.) as argument of "Apply" function of
    "nn.Module"

    :param m: Layer to initialize
    :return: None
    """

    if isinstance(m, nn.Conv2d) or isinstance(m, nn.ConvTranspose2d):
        torch.nn.init.kaiming_normal_(m.weight, mode='fan_out')
        nn.init.constant_(m.bias, 0)
    elif isinstance(m, nn.BatchNorm2d):
        nn.init.constant_(m.weight, 1)
        nn.init.constant_(m.bias, 0)
</code></pre>
<p>You just need to include different type of layers using <code>if/else</code> code.</p>
<p>Then after initializing your <code>model</code>, you call <code>.apply</code> and it will recursively initialize all of your model’s nested layers.</p>
<p>Here is example:</p>
<pre><code class="lang-python">
model = ModelNet()
model.apply(init_weights)

</code></pre>

username: Cverlpeng
cooked: <p>hi,<br>
Very detailed<br>
thank you very much</p>

username: Nikronic
cooked: <p>You’re welcome mate.</p>
<p>I just have noticed that using <code>.data</code> to initialize weights is not good practice while it works. It is better to remove <code>.data</code> from code above. I have edited the post.</p>
Precision missing while writing the GRU layer in c++

username: dinesh_mareedu
cooked: <p>Hi,</p>
<pre><code> I extracted the GRU weights from the trained model and followed the instructions in pytorch document.
</code></pre>
<p>But, while comparing the results of model’s GRU outputs and c++ GRU output precision is missing by 0.1(maximum of all results). Can you suggest me to overcome this.<br>
Advance thanks.</p>

username: yf225
cooked: <p>Could you post the full Python and C++ code for your experiment?</p>

username: yf225
cooked: <p>IIUC, is this resolved in <a href="https://discuss.pytorch.org/t/gru-calculations-for-gates-and-output-in-testing-mode/49606/19" class="inline-onebox">GRU calculations for gates and output in testing mode</a>?</p>

username: dinesh_mareedu
cooked: <p>Yes, it is resolved. I made an small error in matmuls.</p>
<p>Thanks,<br>
M. Dinesh</p>

username: dinesh_mareedu
cooked: <p>Can you give me suggestions on my current problem I am facing with bidirectional multilayered GRU?</p>
<aside class="quote quote-modified" data-post="1" data-topic="51568">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dinesh_mareedu/48/12060_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/bidirectional-gru-with-multilayers-what-are-the-inputs-to-the-2nd-layer/51568">Bidirectional GRU with multilayers (what are the inputs to the 2nd layer)</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" style="--category-badge-color: #9EB83B; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    [35610748-dd41a23a-06a5-11e8-803e-c577359c372b] 
 As per my understanding, GRU pytorch working as in the 2nd picture.  After each layer,  both forward and reverse directiona outputs are concatenated. Correct me if i was wrong. 

The initial hidden state of first layer initialised with zeroes. 
Is The last hidden state of 1st layer is the initial hidden state of the 2nd layer?. 
And also I want to know,  the second layer takes the concatenated 1st layer output as a input and operates on them, as …
  </blockquote>
</aside>
Remove last two layers (FC and average pooling) of ResNet152 and add Conv2d at the end

username: rashidch
cooked: <p>I want to remove last two layers of ResNet152 and add Conv2D at the end to handle variable size inputs. Can anyone help me?</p>
<p>Thanks!</p>

username: dingyongchao
cooked: <p>res = model.resnet18(pretrained=True)<br>
res = list(res.children())[:-2]<br>
_res = nn.Sequential(*res)</p>
<p>does it work?</p>

username: rashidch
cooked: <p>This works fine but how to add Conv2d at the end and then fine tune Conv2D only?</p>

username: dingyongchao
cooked: <p><code>_res .add_module("con-end", nn.conv2d(x, x, x))</code><br>
you can add the conv2d by above line<br>
And if you want to finetune this layer only, you should set the params on resNet into autogrand == False</p>

username: rashidch
cooked: <pre><code class="lang-auto">class ResNet152Bottom(nn.Module):
    def __init__(self, original_model):
        super(ResNet50Bottom, self).__init__()
        self.features = nn.Sequential(*list(original_model.children())[:-2])
        self.conv = nn.Conv2d(2048, 10, kernel_size=(7, 7), stride=(1, 1), bias=False)
        
    def forward(self, x):
        x = self.features(x)
        x = self.conv(x)
        return x

res50_model = torchvision.models.resnet152(pretrained=True)
for param in res50_model.parameters():
    param.requires_grad = False
res50_conv2 = ResNet50Bottom(res50_model)
inputs, classes = next(iter(dataloaders['train']))
outputs = res50_conv2(inputs)
outputs.data.shape  
out = torch.squeeze(outputs, dim=2)
out2 = torch.squeeze(out, dim=2)
out2.data.shape
</code></pre>
<p>This code works fine. It removes last two layers of ResNet152 and add conv2d layer at bottom. But I want to train the conv2d layer. When I tried to train conv2d, I got the following error, <strong>RuntimeError: 1only batches of spatial targets supported (non-empty 3D tensors) but got targets of size: : [16]</strong></p>
<p>Can you tell me why I got this error? I tried to search on internet but could not find any appropriate answer.</p>
How to extract torch.nn.conv1d complete python source code

username: A.V_Yugesh
cooked: <p>where do i find torch.nn.conv1d source code as shown below?</p>
<pre><code class="lang-auto">fun= torch.nn.Conv1d(in_channels,out_channels,kernel_size,1,groups=in_channels,bias=False,padding=2,dilation=2)
</code></pre>

username: Balamurali_M
cooked: <p>In the pytorch documentation, each class will have a source in the right side. Just click that and read about it</p>
How can I train student model with teacher model's output?

username: wonchulSon
cooked: <p>I finished trained teacher model and I can save and reload teacher model’s weight, feature map etc.</p>
<p>So I want to do is making similar student’s first layer output feature map with teacher’s.<br>
Teacher have 5x5 filter, student have 3x3 filter.</p>
<p>But I don’t know how to coding the loss or feature map comparing part.<br>
Thank you.</p>

username: Balamurali_M
cooked: <p>You are taking about gan ?</p>
Is it safe to have device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") at the top of every of my files?

username: pinocchio
cooked: <p>Is it safe to have:</p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
</code></pre>
<p>there is not some unexpected error that can happen right?</p>
<p>If I import stuff those files will use the right <code>device</code> cuz they are all ran on the same machine, right?</p>
<p>(cuz I don’t really like using globals so I was trying to avoid that…but I can’t figure out what the standard in pytorch is or what is safe or distinguish between them…)</p>

username: Balamurali_M
cooked: <p>If cuda is available this code will automatically run in cuda. If you want to have options you can manually mention using some flags.</p>
How does torchtext treat unseen token?

username: Jeffrey
cooked: <p>Say I am using pretrained GloVe model:</p>
<p><code>TEXT.build_vocab(train, vectors=GloVe(name='6B', dim=100))</code></p>
<p>If there is a token in the test dataset that is not in the train dataset, but the token is actually in GloVe(name=‘6B’, dim=100). Torchtext will return 0 as index, instead of the actual index in GloVe(name=‘6B’, dim=100).</p>
<p>Is this a bug or do I miss anything?</p>
Bidirectional GRU with multilayers (what are the inputs to the 2nd layer)

username: dinesh_mareedu
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8517075e59678ac614056ab82f89b3afdcca685c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8517075e59678ac614056ab82f89b3afdcca685c" title="35610748-dd41a23a-06a5-11e8-803e-c577359c372b"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8517075e59678ac614056ab82f89b3afdcca685c_2_690x452.jpeg" alt="35610748-dd41a23a-06a5-11e8-803e-c577359c372b" data-base62-sha1="iZmQeBS0V91qlGKNm7qXnqwBx00" width="690" height="452" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8517075e59678ac614056ab82f89b3afdcca685c_2_690x452.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/8/8517075e59678ac614056ab82f89b3afdcca685c.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/8/8517075e59678ac614056ab82f89b3afdcca685c.jpeg 2x" data-dominant-color="E0E6EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">35610748-dd41a23a-06a5-11e8-803e-c577359c372b</span><span class="informations">1030×676 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code> As per my understanding, GRU pytorch working as in the 2nd picture.  After each layer,  both forward and reverse directiona outputs are concatenated. Correct me if i was wrong. 
</code></pre>
<p>The initial hidden state of first layer initialised with zeroes.<br>
Is The last hidden state of 1st layer is the initial hidden state of the 2nd layer?.<br>
And also I want to know,  the second layer takes the concatenated 1st layer output as a input and operates on them, as it is operated in 1st layer? Or any operations different.</p>
<p>Thank you,<br>
Dinesh</p>
How do i get the ultimate output tensor in a semantic segmention task

username: Mellow
cooked: <p>I’m practicing a semantic segmentation task.<br>
As there are two class for each pixel, the output of the model is of shape(1, 2, 512, 512). See there are two channels, but i need to convert it into shape(1, 1, 512, 512), to match the shape of my label. Wondering what method should i take to complete this transformation?</p>

username: MariosOreo
cooked: <p>Hello,</p>
<p>I think shape of output of the network is depending on which loss function employed.<br>
In binary classification, you can use both <code>CELoss</code> and <code>BCELoss</code>.<br>
For <code>CELoss</code>, the shape of the output should be <code>[batch_size, nb_classes, H, W]</code> then produce a probability map by <code>softmax</code>.<br>
For <code>BCELoss</code>, the shape should be <code>[batch_size, H, W]</code> which is the same to label, and it should incorperate with <code>sigmoid</code>.</p>
<p>Note: <code>CELoss</code> has contain <code>logsoftmax</code>, so you could only pass the model output and label.<br>
<code>BCELossWithLogits</code> = <code>BCELoss</code>+<code>sigmoid</code><br>
<a href="https://pytorch.org/docs/master/nn.html#torch.nn.CrossEntropyLoss" rel="nofollow noopener">for more details</a></p>

username: XiaoAHeng
cooked: <p>How to get  the ultimate output tensor ,i.e. (1, 2, 512, 512) to transform  into the ground truth shape (1, 1, 512, 512)? I know how to use CELoss and <code>BCELoss</code></p>

username: XiaoAHeng
cooked: <p>Have you solved the problem?</p>

username: Nikronic
cooked: <p>Hi,<br>
<a class="mention" href="/u/mellow">@Mellow</a></p>
<p>This line worked for me.</p>
<pre><code class="lang-python">
_, pred = torch.max(scores, dim=1)
</code></pre>
<p><a href="https://pytorch.org/docs/stable/torch.html#torch.max" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.max</a></p>
<p>Where <code>scores</code> is your ultimate tensor containing probs.</p>
<p>For instance, <code>scores</code> tensor has size of <code>[10, 150, 256, 256]</code> which means I have 150 classes to segment and using above code give me <code>[10, 256, 256]</code> tensor. Then you can <code>.unsqueeze(1)</code> to get your desired dimension.</p>
<p>PS: <code>torch.max</code> return a tuple which the second value is the target tensor.</p>

username: XiaoAHeng
cooked: <p>thank you very much!</p>
CPU/GPU agnostic. Why using .to() instead of the simple CUDA_VISIBLE_DEVICES?

username: philipperemy
cooked: <p>Hey guys,</p>
<p>I’m pretty new at Pytorch. I’m coming from Tensorflow.</p>
<p>I want my code to be device agnostic.</p>
<p>I am following this tutorial: <a href="https://pytorch.org/docs/stable/notes/cuda.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/cuda.html</a></p>
<p>Basically, using <code>.to(device)</code> everywhere.</p>
<p>However, I realized that I can just:</p>
<ul>
<li>Run on CPU with <code>export CUDA_VISIBLE_DEVICES=;</code>
</li>
<li>Run on GPU with <code>export CUDA_VISIBLE_DEVICES=0;</code>
</li>
</ul>
<p>Exactly like in Tensorflow where the code for CPU/GPU is -roughly- the same.</p>
<p>Why do people bother with <code>.to(device)</code> or <code>.type()</code>?</p>
<p>I may understand why if we deal with multiple GPUs and want to have a finer-grained management of the GPUs.</p>
<p>But in the simple case, doesn’t the code get bloated and less readable?</p>
<p>Philippe</p>

username: ptrblck
cooked: <p>If you don’t use <code>to()</code> to push your data and models to a GPU, all tensors will be created on the CPU by default.<br>
Setting <code>CUDA_VISIBLE_DEVICES</code> won’t change this behavior, so you would still need to somehow declare which tensor should be pushed to the GPU and which should stay on the CPU.</p>
<p>You could try to set the default tensor type to a CUDA one, if a GPU was found, but that feels like an unnecessary hack, since now all calculations will be performed on the GPU by default.<br>
Asynchronous CUDA calls with some CPU computation will hardly be possible.</p>
<p>Could you explain your idea a bit more, as I might misunderstand it?</p>

username: philipperemy
cooked: <p>My bad you are right! I thought for a couple of seconds that we could do the same as in tensorflow.</p>
<p>Basically, writing code without mentioning anything about device/cuda and still being able to run it on either a GPU or CPU by just setting <code>CUDA_VISIBLE_DEVICES</code>.</p>
<p>Thanks for your help!</p>
Load_state_dict with strict=False returns IncompatibleKeys tuple

username: vip
cooked: <p>The current load_state_dict API with strict=False returns an IncompatibleKeys named tuple which is misleading in the case of successful loading of weights. I don’t think IncompatibleKeys is the correct name for a return object for loading the state dict. I was thrown off for a while trying to figure out why I was repeatedly getting the same IncompatibleKeys error with strict=False before realizing that was actually the return object itself.</p>

username: ptrblck
cooked: <p>This issue was tracked <a href="https://github.com/pytorch/pytorch/issues/20128" rel="nofollow noopener">here</a> and should be fixed in the current master or nightly build by now.<br>
Could you check it again with one of the current versions?</p>
DCGAN beginner curiosities: HWC size? Batch size?

username: peteaj
cooked: <p>Hi, I am new to PyTorch and working with GANs. I looked around online already but I had a couple of basic beginner curiosities if anyone could help.</p>
<p>Not sure I understand three dimensional jpegs. HWC size? Is that similar to jpeg + frame size? I noticed a transformer resizes training images. Does the transformer crop images that aren’t square in a certain way?</p>
<p>Also any advice on batch size minimum to train? I created my own dataset but it’s generating static for fake images. I don’t think I have enough input. Is there a good amount of images to think about in terms of minimum training data? And anywhere I could learn more about GAN inputs?</p>
<p>Any ideas are welcomed!<br>
Thanks</p>
Train CNN-RNN network for multi label video classification with sliding window technique

username: IliasPap
cooked: <p>I’m implementing a model in which a CNN model is used to extract feature sequences from videos , and RNN is used to analyze the generated feature sequences, and output a multi label classification result. Frame-wise annotation of actions is not given but only the temporal order  so the model is being trained with nn.CTCLoss.<br>
I want to use sliding window tecnique to generate more video segments (overlapping windows)  to capture dependencies over different timesteps.<br>
Should i use x.unfold( ) in the input video or  in the generated features after the cnn and before the rnn input ?<br>
does it have any effect  in the backward operation ?</p>
Keras Data Generator to Pytorch Data Generator

username: Pranavan_Theivendira
cooked: <p>Hi ,</p>
<p>How can I change a Keras Data Generator to a pytorch data Generator? Just curious whether there are any shortcuts or not. I added one example below.</p>
<p>In Keras case, we usually use Keras generators to fit the model as follows.</p>
<pre><code class="lang-auto">model.fit_generator(generator=train_data, steps_per_epoch=len(train_data), validation_data=validation_data, validation_steps=len(validation_data), epochs=epochs,verbose=1,callbacks=callbacks)
</code></pre>
<p>Thanks</p>