akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
KFOLD Cross Validation Provides extremely unexpected Results?

alitahir: <p>I’m using transfer learning with k-fold cross validation. k=9 and k folds are being generated using sklearn. The problem i’m facing is that after evaluation of first fold performance decreases drastically. In first fold i get  54% validation accuracy and after that even the training accuracy doesn’t exceed 4% for all the remaining folds.</p>
<p>First guess was that it is probably because of data difference in all folds. So i have run the model multiple times but this behavior still exists. It doesn’t seem to be because of data as new run generates shuffled new data for all folds. I’m completely out of ideas why its happening. Any help is appreciated. Thank You.</p>
<blockquote>
<p>Blockquote<br>
def loadModel():<br>
model = sphere20a(classnum=1000).to(device)<br>
model_state = model.state_dict()<br>
for name, param in model.named_parameters():<br>
if (name[:4] == ‘conv’ or name[:4] == ‘relu’) and name[4] is not ‘4’:<br>
param.requires_grad = False</p>
</blockquote>
<pre><code>pretrained_state = torch.load("/kaggle/input/kinmodel/sphere20a_20171020.pth")
pretrained_state = {k:v for k,v in pretrained_state.items() if k in model_state and v.size() == model_state[k].size()}

model_state.update(pretrained_state)
model.load_state_dict(model_state)
return model
</code></pre>
<p>training loop.</p>
<p>for train_index, val_index in kf.split(data):</p>
<pre><code>n_epochs=10
learn_rate = 0.01
counter = counter + 1
model = loadModel() #load new model everytime and discard old one
</code></pre>
<p>I am deleting the old model after every fold completes. using del model</p>

alitahir: <p>If somebody is up for the challenge of finding the problem i can share the complete notebook.</p>

ptrblck: <p>Could you post the complete code, so that I could have a look?<br>
I’m always up for a challenge! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

alitahir: <p>Sorry for replying late. I got caught up in some stuff. I have made the notebook public. Here is the link: <a href="https://www.kaggle.com/ali14tahir/sphereface-model" rel="nofollow noopener">https://www.kaggle.com/ali14tahir/sphereface-model</a></p>
<p>I can also share it with you kaggle if you like. I haven’t changed anything since then as I don’t understand what’s going on anymore</p>
Silent error Object detection Finetuning Tutorial on CPU

msoulio: <p>Hello, I am following the object recognition tutorial from pytorch<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html </a>. I use Windows 10 with CPU only and I had to make a couple of tweaks to the auxiliary scripts (utils, engine) to avoid calls to cuda.<br>
However, the following line is giving silent error <strong>when run for the 2nd time in a loop</strong>, so I guess it is a memory problem. Both in Jupyter and in the terminal, no error shown, just restarting kernel or going to next command.</p>
<blockquote>
<p>coco_dt = loadRes(self.coco_gt, results) if results else COCO()</p>
</blockquote>

ptrblck: <p>Could you check your system RAM and swap?<br>
Could your machine be running out of memory and thus killing the process?</p>
Resume plotting scalars to tensorboard when resume training in pytorch

guko: <p>i am training my model using pytorch , and i already added saving and resume training functionalities, but the problem when i want to override the step value on tensorboard , the graph missed up ( connecting last step to the current step )</p>
<p>ex: i am logging scalars to tensorboard every 10 iterations, and saving models every 100 iteration where iteration=step</p>
<p>in image below , i trained model for 150 iteration (so i have 15 step point in tensorboard graph), and then resume training from 100 iteration model and start logging in step 100 , i was expected to override all step points after step 110 but it seems to connect the last step point (150) to step 110, how i can override all steps beyond step 100, and fix the graph ???</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e26af736305fbb69b36f20f3411af1320423f7e8.png" alt="Untitled" data-base62-sha1="wiZ52bwPvzjg63hNbszbudGj3Pq" width="364" height="317"></p>
<p>logging to tensorboard code:</p>
<pre><code class="lang-auto">if (OLD_ITERATIONS + i)%10 is 0 and i is not 0:
            wirter.add_scalar(tag='Accuracy/Train', scalar_value=accs/10, global_step=OLD_ITERATIONS+i)
            wirter.add_scalar(tag='Accuracy/Valid', scalar_value=val_accs/10, global_step=OLD_ITERATIONS+i)
            wirter.add_scalar(tag='Loss/Train', scalar_value=losses/10, global_step=OLD_ITERATIONS+i)
            wirter.add_scalar(tag='Loss/Valid', scalar_value=val_losses/10, global_step=OLD_ITERATIONS+i)

            accs = 0
            losses = 0
            val_accs = 0
            val_losses = 0
</code></pre>
Average different layers in pytorch

Vishu_Gupta: <p>How do I perform an average of 2 or more layers i.e. <code>z = keras.layers.average([x, y, w])</code> as done in Keras. Should I first add them via <code>torch.add</code> and then divide by the number of layers or is there any direct way to perform it?</p>

ptrblck: <p>If you want to calculate the average of the activations of different layers, you could simply add them and divide by the number of activations (3 in your case).<br>
I’m not sure, what “average of layers” means, but if you want to average the parameters of different layers, you could use <a href="https://discuss.pytorch.org/t/weights-are-not-converging/84251/4">this approach</a>.</p>

Vishu_Gupta: <p>You mean to say that to implement above mentioned the keras average layer in pytorch, it would be <code>z = (x+y+w)/3</code> ?</p>

ptrblck: <p>If the Keras layer calculates the mean of the activations, then yes, your approach should work.</p>
(Resolved)Libtorch1.5_cu92 win7: torch.cat RuntimeError: error in LoadLibraryA

zhijian_li: <p>Win7 64bit<br>
CUDA9.2<br>
Libtorch1.5+cu92</p>
<p>when I use “module.forward(inputs).toTensor()”, crashed.<br>
I print out the exception, something like it below:<br>
"<br>
input = torch.cat([argument_2, x1], 1)<br>
~~~~~~~~~ &lt;— HERE<br>
return (_17).forward(input, )<br>
…<br>
…<br>
…<br>
RuntimeError: error in LoadLibraryA<br>
"</p>
<p>But I can run cpu traced model well.</p>

ptrblck: <p>This error should be fixed in PyTorch <code>1.5.0</code>, if I’m not mistaken.<br>
If you are using an older version, you could update or use <a href="https://discuss.pytorch.org/t/torch-cat-runtimeerror-error-in-loadlibrarya/71188/9">this workaround</a>.</p>

peterjc123: <p>Someone else found out that it could also be caused by the antivirus. See <a href="https://blog.csdn.net/weixin_38314865/article/details/103840899" rel="nofollow noopener">https://blog.csdn.net/weixin_38314865/article/details/103840899</a>.</p>

zhijian_li: <p>I double checked that, I use <strong>libtorch1.5</strong>, and I use <strong>pytorch 1.5.0</strong> to trace and saved my model.<br>
Today I downloaded the latest nightly build <strong>libtorch</strong> version, has the same issue.</p>

zhijian_li: <p>I tried it, not work for me. <strong>Pytorch</strong> is OK, no issue, <strong>Libtorch</strong> has this situation.</p>

peterjc123: <p>For libtorch, just copy all the DLLs to the location of your executable.</p>

zhijian_li: <p>I will try it. thanks</p>

zhijian_li: <p>Fixed, thank you very much.</p>
Image transforms

MonIA: <p>Is there a need to resize images using (transforms.RandomResizedCrop(224)) in validation and testing datasets when using transfer learning with VGG16? or only using normalization and nothing else?</p>

Nikronic: <p>Hi,</p>
<p>If your input images are bigger than <code>[224, 224]</code> just normalization would be enough, otherwise you need to resize all image to the aforementioned size.</p>
<p>Here is the official source code of evaluation step:</p>
<pre><code class="lang-auto">normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225])

transforms.Compose([
            transforms.Scale(256),  # optional
            transforms.CenterCrop(224),  # optional
            transforms.ToTensor(),
            normalize,
        ])

</code></pre>
<p><a href="https://github.com/pytorch/examples/blob/42e5b996718797e45c46a25c55b031e6768f8440/imagenet/main.py#L89-L111" rel="nofollow noopener">Source code</a> of training using ImageNet dataset</p>
<p>Bests</p>

MonIA: <p>Many thanks!<br>
Could you tell me how can I check the images sizes in a certain dataset?</p>

Nikronic: <p>If I understand correctly, you can find info about image sizes in the webpage of each dataset. If you are using your own custom dataset, you can check it in <code>__getitem__</code> method by using <code>CustomDataset</code>. Here is post for such a situation: <a href="https://discuss.pytorch.org/t/custom-dataset-with-some-preprocessing/30385/2" class="inline-onebox">Custom Dataset with some preprocessing</a></p>
<p>Also, <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">dataloading tutorial</a> may help.</p>
<p>But a simple solution would be adding <code>transforms.Resize(224)</code> in the previous post right before <code>ToTensor()</code>.</p>

MonIA: <p>Thank you, that’s what I’m looking for <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
How to install PyTorch Nighly with requirements.txt?

WaterKnight: <p>I am trying to install PyTorch Nightly in my docker container. For doing it I am pip installing from requirements.txt.</p>
<p>The requirements.txt file looks as follows:</p>
<pre><code class="lang-auto"># ML Framework
numpy
--find-links https://download.pytorch.org/whl/nightly/cu102/torch_nightly.html
torch
torchvision

# Web Framework
Flask

</code></pre>
<p>However, It looks like 1.5 pytorch is being installing</p>
<p>However, torch and torchvision is not getting installed!</p>
Mapping a list of input to a module

Alex_Finkelshtein: <p>Hi everyone,<br>
I was reading around and couldn’t figure it out.<br>
Is  there is a way to Map multiple inputs in <strong>parallel</strong>, to a module that expects a single input?</p>
<p>Specifically with the following scenario in mind:</p>
<p><strong>EXAMPLE:</strong></p>
<p>def MyCustomModule(Module):<br>
def <strong>init</strong>():<br>
pass<br>
def forward(self, single_x)<br>
some computation<br>
return single_output</p>
<p>def OtherLayer(Module):<br>
def <strong>init</strong>():<br>
pass<br>
def forward(self, x)<br>
some computation<br>
return multiple_x</p>
<p>MAIN:<br>
define network:<br>
x = OtherLayer(x)<br>
output = map(MyCustomModule, x)</p>
<p>Is it possible to define such a thing where the feeding of the custom module will happen in parallel?<br>
Or alternatively with a torch custom function?</p>
<p>Thanks!</p>
Train loss decreased but validation loss doesn't change

Array: <p>I use pre-trained ResNet to extract 1000 dimensional features for each image, then put these images into my self-built net to do classification tasks and use triplet loss function.</p>
<p>There is a part of my code:</p>
<pre><code class="lang-auto">class Network(torch.nn.Module):
    def __init__(self,n_feature = 1000, n_hidden_1 = 200, n_hidden_2 = 100, n_hidden_3 = 50, n_hidden_4 = 20,n_output = 3):
        super(Network, self).__init__()
        self.net = torch.nn.Sequential(    
                        torch.nn.Linear(n_feature, n_hidden_1),
                        torch.nn.BatchNorm1d(n_hidden_1),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden_1, n_hidden_2),
                        torch.nn.BatchNorm1d(n_hidden_2),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden_2, n_hidden_3),
                        torch.nn.BatchNorm1d(n_hidden_3),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden_3, n_hidden_4),
                        torch.nn.BatchNorm1d(n_hidden_4),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden_4, n_output),
                        torch.nn.Sigmoid()
                                )
    def forward(self, x):
        x = self.net(x)
        return x
</code></pre>
<p>and there are training and validation process:</p>
<pre><code class="lang-auto">    for epoch in tqdm(range(n_epoch)):
        model.train()
        for step, (batch_anchor, batch_positive, batch_negative )in enumerate(train_loader):

            anchor_out = model(batch_anchor)
            positive_out = model(batch_positive)
            negative_out = model(batch_negative)

            loss = loss_func(anchor_out, positive_out, negative_out)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        model.eval()
        for step, (batch_anchor_val, batch_positive_val, batch_negative_val) in enumerate(val_loader):

            anchor_out_val = model(batch_anchor_val)
            positive_out_val = model(batch_positive_val)
            negative_out_val = model(batch_negative_val)

            loss_val = loss_func(anchor_out_val, positive_out_val, negative_out_val)

</code></pre>
<p>where I define loss function and optimiser like following:</p>
<pre><code class="lang-auto">optimizer = optim.Adam(model.parameters(), lr=0.002)
loss_func = torch.nn.TripletMarginLoss(p=2, margin=1)
</code></pre>
<p>and some results:</p>
<pre><code class="lang-auto">Epoch: 1/50 - Loss: 0.8764 - Val_loss: 0.9920
Epoch: 2/50 - Loss: 0.7035 - Val_loss: 0.9897
Epoch: 3/50 - Loss: 0.6313 - Val_loss: 0.9972
Epoch: 4/50 - Loss: 0.5958 - Val_loss: 0.9980
Epoch: 5/50 - Loss: 0.5724 - Val_loss: 0.9930
Epoch: 6/50 - Loss: 0.5541 - Val_loss: 1.0123
...
</code></pre>
<p>While training, train loss always decrease, but val loss doesn’t change, I don’t know why, maybe someone know what are potential reasons? I have read some blogs, and check my data set, I am sure my data set are correctly split.</p>
Argmax from multiple tensors without replacement

Tin: <p>Current problem: I have a batch of sequences of equal length, and, for each item in the sequence, a probability distribution over all items.<br>
I.e. I have a tensor of shape <code>(batch_size, seq_length, num_items)</code>.</p>
<p>For each item in the sequence, I want to compute the argmax using these distributions.<br>
This is trivial with no further restrictions: <code>torch.argmax</code> supports batched computation and a dim argument.<br>
I would, however, like to sample sequences with no duplicates.<br>
A simple heuristic would be: at item j, set the probability of items i &lt; j = 0.<br>
Implementing it using for loops:</p>
<pre><code class="lang-auto">batch_samples = []
for i in range(batch_size):
  # seq_dists dimension (seq_length, num_items)
  seq_dists = my_batch[i, :, :]
  seq = []
  for j in range(seq_length):
    seq_dist = seq_dists[j]
    # set prob. of previous samples to 0
    seq_dist = seq_dist.scatter_(dim=-1, index=torch.LongTensor(seq), src=torch.tensor(0))
    item = torch.argmax(seq_dist)
    seq.append(item)

  batch_samples.append(torch.tensor(seq))

batch_samples = torch.stack(batch_samples)
</code></pre>
<p>However, as expected, this process is incredibly slow.<br>
Is there anything I can do achieve the same result in a more efficient way?</p>

Tin: <p>Bump.<br>
Any ideas on how this can be solved?</p>

Tin: <p>Bump… no way to mitigate this problem?</p>
How to deploy quantized model to C++ frontend?

seeker: <p>Is there any tutorial or instructions about how to deploy our quantized model to C++ frontend? I can’t find any on the Internet.</p>

jerryzh168: <p>maybe <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_export.html</a>? there is nothing specific to quantized models, it should work the same way as other models</p>
Need an efficient way to store data

Muhammad_Ahmed: <p>I’m currently trying to run knn on lfw dataset.<br>
I have trained dataset and got the vectors from forward pass.<br>
The problem is, I can’t save all the vectors as I get memory out of range.<br>
I get a 128*1 vector after forward pass, and i need to save around 9000.<br>
How should i save the vectors for knn.<br>
I don’t think I can’t store it in form of trainloader<br>
Please help<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Which pytorch version do I need?

S_Ellis: <p>I am trying to run the code for Fader Networks, available <a href="https://github.com/facebookresearch/FaderNetworks" rel="nofollow noopener">here</a>. But whichever version of pytorch I use I get attribute errors.</p>
<p>When I try version 0.3.0 I get this attribute error:</p>
<p><code>AttributeError: Can't get attribute '_rebuild_tensor_v2' on &lt;module 'torch._utils'&gt;</code></p>
<p>When I try 0.4 or higher I get a different attribute error:</p>
<p><code>AttributeError: 'BatchNorm2d' object has no attribute 'track_running_stats'</code></p>
<p>My question is which version of pytorch had both torch._utils._rebuild_rensor_v2 and BatchNorm2d.track_running_stats.</p>
<p>PS I have left an issue on the project’s github repo, but repo seems inactive so I don’t expect a response there.</p>

harsha_g: <aside class="quote no-group quote-modified" data-username="S_Ellis" data-post="1" data-topic="84920">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e274bd/48.png" class="avatar"> S_Ellis:</div>
<blockquote>
<p>AttributeError: ‘BatchNorm2d’ object has no attribute ‘track_running_stats’</p>
</blockquote>
</aside>
<p>How about taking the middle road? Did you try 0.3.1? Got it from <a href="https://discuss.pytorch.org/t/batchnorm2d-object-has-no-attribute-track-running-stats/17525">here</a>.</p>

harsha_g: <p>You may also try writing to the authors whose e-mail addresses are in the article (link in github).</p>

S_Ellis: <p>Yeah, I did also try 0.3.1, with no luck. I could try the email from the paper, but I’m doubtful that I’ll get a response regarding a 2/3 yo paper so I thought I’d ask the community first.</p>

S_Ellis: <p>UPDATE: While I’ve not exactly found the solution, I think I have found the cause of the issue. I was in effect trying to load pre-trained models and apply them. These models must’ve been trained with an older version of pytorch, where BatchNorm2d did not have (and hence did not save) the ‘track_running_stats’ attribute.</p>
<p>The confusing thing is that using an older version of pytorch failed due to a reliance on the ‘_rebuild_tensor_v2’ attribute of torch._utils, which seemingly references a pytorch version later than that used to train the example models.</p>
<p>My only explanation is that the models available were saved with an older version of pytorch, development was then made with a newer version of pytorch, and backwards compatibility with available pre-trained models was not checked.</p>
<p>The fix for me is hopefully just to re-train all models from scratch using a new version of pytorch, so that my own saved models will have BatchNorm2d objects with the track_runnning_stats attributes (c.f. <a href="https://github.com/xingyizhou/pytorch-pose-hg-3d/issues/36" rel="nofollow noopener">https://github.com/xingyizhou/pytorch-pose-hg-3d/issues/36</a>).</p>
<p>Thanks!</p>
Obtaining abnormal changes in loss and accuracy

Droid786: <p>I am training a multi-class classification problem.Instead of declining at a steady rate, y loss and accuracy metrics are moving in zig-zag rate. I am not able to figure out the reason behind this behaviour. Can anyone please help me out?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/47252703dfeb2062b0a581df5572071657aa82c5.png" data-download-href="https://discuss.pytorch.org/uploads/default/47252703dfeb2062b0a581df5572071657aa82c5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47252703dfeb2062b0a581df5572071657aa82c5_2_345x244.png" alt="image" data-base62-sha1="a9nuWntHtp4uolHdBfoj8Hz2tFz" width="345" height="244" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47252703dfeb2062b0a581df5572071657aa82c5_2_345x244.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47252703dfeb2062b0a581df5572071657aa82c5_2_517x366.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47252703dfeb2062b0a581df5572071657aa82c5_2_690x488.png 2x" data-dominant-color="FBFCFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">850×602 26.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If I keep on running my model for longer epochs, then the same zigzag behaviour is being obtained.</p>

Chirath_Pansilu: <p>Try lowering your learning rate .<br>
It will be helpful if you can upload some code snippets</p>

Droid786: <p>I did , from 0.1 to 0.0001.<br>
My code is here - <a href="https://gist.github.com/droid786/4069fe9cb16e909eb09ed087b427fa26" rel="nofollow noopener">https://gist.github.com/droid786/4069fe9cb16e909eb09ed087b427fa26</a></p>
Question about the need for distributed autograd context

rahul003: <p>Hi,</p>
<p>I am trying to understand distributed autograd so I can use it correctly for my use case. Specifically, I’m confused about</p>
<p>“”"<br>
Multiple nodes running distributed backward passes might accumulate gradients on the same tensor and as a result the <code>.grad</code> field of the tensor would have gradients from a variety of distributed backward passes before we have the opportunity to run the optimizer. This is similar to calling <a href="https://pytorch.org/docs/master/autograd.html#torch.autograd.backward" rel="nofollow noopener"> <code>torch.autograd.backward()</code> </a> multiple times locally. In order to provide a way of separating out the gradients for each backward pass, the gradients are accumulated in the <a href="https://pytorch.org/docs/master/rpc.html#torch.distributed.autograd.context" rel="nofollow noopener"> <code>torch.distributed.autograd.context</code> </a> for each backward pass.<br>
“”"</p>
<p>Why would multiple backward passes be executing backward pass at the same time? Are they for different batches? All the examples seem to show optimizer update at the end of processing each batch.</p>

mrshenli: <p>Hey <a class="mention" href="/u/rahul003">@rahul003</a></p>
<p>This is for the use case where you have multiple independent trainers accessing the same parameter server (PS), where each training runs its own forward-backward-optimize in every iteration. With this setup, the parameters on the PS needs to remember gradients for different trainers, as they would run optimizer step independently. Hence, we cannot directly accumulate grads into <code>param.grad</code> field.</p>
<p>This is one example: <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html</a></p>
<p>The idea is partially borrowed from Hogwild training: <a href="https://people.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf" rel="nofollow noopener">https://people.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf</a></p>
<p>BTW, could you please add a “distributed-rpc” tag for RPC-related questions so that the team can get back to you promptly, thanks!</p>
Quantized Mobilenet model pipeline

juanmanpr: <p>Hello,</p>
<p>I am interested in learning the proper quantization + mobile deployment with Pytorch.<br>
In the Android tutorial there is a quantized mobilenet trained on Imagenet. I was wondering if there is a tutorial on how to obtain such model. Ideally, I would prefer to try a post-training quantization tutorial, if any.</p>
<p>Thanks!</p>
How to know the filter size and stride in AdaptiveAvgPool2d

JIALI_MA: <p>I know that AdaptiveAvgPool2d is able to give the outputs of any defined size, but I don’t know what is the size of filter and stride it indeed use during calculation.<br>
For example: if the input size is (14, 14), and the output size is (3, 3). What is the corresponding filter parameter? Suppose padding is 0, it seems there are multiple solutions.</p>
<ul>
<li>filter 10x10 with stride 2x2;</li>
<li>filter 8x8 with stride 3x3;</li>
<li>filter 6x6 with stride 4x4;</li>
</ul>
<p>Which one is correct? What is the calculation principle?</p>

ptrblck: <p>Based on <a href="https://github.com/pytorch/pytorch/blob/877c96cddfebee00385307f9e1b1f3b4ec72bfdc/aten/src/ATen/native/AdaptiveAveragePooling.cpp#L12-L72">this implementation</a>, the input of <code>14x14</code> and output of <code>3x3</code> would create the first start and end index as:</p>
<pre><code class="lang-python"># First iteration
istartH = floor((0*14)/3) = 0
iendH = ceil((1*14)/3) = 5

# Second iteration
istartH = floor((1*14)/3) = 4
iendH = ceil((2*14)/3) = 10
</code></pre>
<p>So it seems that the window size is not constant (if I don’t have a mistake in the calculations).</p>

JIALI_MA: <p>Thanks very much for your reply, that really surprises me!</p>
Backprop >20x slower than forward pass for U-Net

Sorench: <p>Hi,<br>
I have been reading through the posts that mention slow back prop and my undertanding is that backprop should take about 2x the forward pass. Mine takes 20x.<br>
I am running a 3D U-Net on a (1,1,32,512,512) test image volume and I am consistently gettig these numbers (Titan RTX and pytroch 1.5.0 from pip):<br>
Transferring images: 0.01<br>
Forward: 0.00<br>
Loss Calc: 0.32<br>
Back prop: 7.59<br>
Step: 0.00</p>
<p>Would anyone be able help point me in the right direction? I assume it is simply slower than expected?<br>
Minimal example below.<br>
Thanks!<br>
Soren</p>
<p>device = torch.device(‘cuda:0’ if torch.cuda.is_available() else ‘cpu’)<br>
net = UNet3D(n_channels=1, n_classes=1)<br>
net.to(device=device)</p>
<p>test_size=(1,1,32,512,512)<br>
dumin=torch.from_numpy( np.zeros(test_size,np.float32) )<br>
dumGT=torch.from_numpy( np.zeros(test_size,np.float32) )<br>
net.cuda()<br>
summary(net, input_size=dumin.shape[1:])</p>
<p>criterion = torch.nn.BCELoss()</p>
<p>optimizer = optim.RMSprop(net.parameters(), lr=0.1, weight_decay=1e-8)</p>
<p>losses=[]<br>
for epoch in range(5):<br>
net.train()<br>
print(epoch)</p>
<pre><code>for indx,item in enumerate(range(100)):


    #to GPU
    torch.cuda.synchronize()
    c = time.perf_counter()
    true_masks = dumin.to(device=device)  # item["seg"].to(device=device)
    imgs = dumin.to(device=device)   # item["data"].to(device=device)
    torch.cuda.synchronize()
    print(f"Transferring images: {time.perf_counter()-c:2.2f}")

    #Cost

    c = time.perf_counter()
    masks_pred = net(imgs)
    print(f"Forward: {time.perf_counter() - c:2.2f}")
    c = time.perf_counter()
    beloss = criterion(masks_pred, true_masks)
    torch.cuda.synchronize()
    print(f"Loss Calc: {time.perf_counter() - c:2.2f}")


    #backprop
    c = time.perf_counter()
    optimizer.zero_grad()
    beloss.backward()
    torch.cuda.synchronize()
    print(f"Back prop: {time.perf_counter() - c:2.2f}")

    #update
    torch.cuda.synchronize()
    c = time.perf_counter()
    optimizer.step()
    torch.cuda.synchronize()
    print(f"Step: {time.perf_counter() - c:2.2f}")</code></pre>

ptrblck: <p>Assuming you are using cudnn, you could add <code>torch.backends.cudnn.benchmark = True</code> at the beginning of your script to use the cudnn heuristics to pick the fastest algorithms for your workload.<br>
Note that the first iteration for each new input shape will be slow, as cudnn is benchmarking the kernels, so you should profile the model after a few warmup iterations.</p>

Sorench: <p>Fantastic - thank you so much!  Increased it close to 10x which brings it close to where I expected it to be.<br>
Thanks again!<br>
Soren</p>
Why the values changed when converting a Variable to numpy?

DumnBird: <p>I am trying to convert a parameter of resnet34 to numpy , but I found that  the values would change after converting, as shown in the figure.<br>
Why would this happen? What can I do to get  precise values in numpy format?</p>
<p>( I am trying to get parameter in torch pretrained models and put them in tensorflow 1.x model , since I can’t find a pretrained resnet34 models in tensorflow1 afteing searching for several days. I am afraid this changed of values would affect the accuracy of the model.  )</p>
<p>(I hate tensorflow…)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9cf9bbc1b619edfe3a9922046cc3eb0738393f3.png" alt="微信截图_20200610132341" data-base62-sha1="zDVGIlHf6RY1b6FEGSbGSGwTXqP" width="413" height="190"></p>

ptrblck: <p>This is most likely a printing issue and you could use <code>torch.set_printoptions(precision=15)</code> to increase the decimals.</p>
Backpropagation gets slower in mixprecision

dujiangsu: <p>Hi,</p>
<p>After converting my program into mixed-precision using amp, the forward time gets shorter while the backward time gets longer when I record running time by “import time”.</p>
<p>Then I use “torch.profiler”  to record the running time.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4eee5c5737235a1add66e5c006949d76b789071.png" data-download-href="https://discuss.pytorch.org/uploads/default/a4eee5c5737235a1add66e5c006949d76b789071" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4eee5c5737235a1add66e5c006949d76b789071_2_690x462.png" alt="image" data-base62-sha1="nx48Ja8sxbAiQdaq78KkNHpGdgZ" width="690" height="462" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4eee5c5737235a1add66e5c006949d76b789071_2_690x462.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4eee5c5737235a1add66e5c006949d76b789071_2_1035x693.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4eee5c5737235a1add66e5c006949d76b789071_2_1380x924.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1704×1143 56 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, it seems that the result of torch.profiler meets my expectation: cuda.time becomes around 1/4 of the fp32.</p>
<p>Here my question is: what is the cuda.time means and why the backpropagation time becomes longer in mix precision?</p>
<p>mixed-precision:<br>
import time :<br>
Entire Epoch: [1]	Train: [0]	SUM: 1828.379	DT: 13.270	FW: 71.987	BK: 1711.248	CM: 31.873	<br>
torch profiler:<br>
Self CPU time total: 56.883s<br>
CUDA time total: 893.112s</p>
<p>FP32:<br>
import time :<br>
Entire Epoch: [3]	Train: [0]	SUM: 1584.260	DT: 15.714	FW: 368.550	BK: 1119.231	CM: 80.766<br>
torch profiler:<br>
Self CPU time total: 105.049s<br>
CUDA time total: 3021.355s</p>
<p>Detailed Logs of my program are  listed below:<br>
<a href="http://49.234.107.127:81/index.php/s/qa3Yjo8WJwNZjCS" rel="noopener nofollow ugc">http://49.234.107.127:81/index.php/s/qa3Yjo8WJwNZjCS</a> (mixed precision)<br>
<a href="http://49.234.107.127:81/index.php/s/y8SpyfiM3d5SZp7" rel="noopener nofollow ugc">http://49.234.107.127:81/index.php/s/y8SpyfiM3d5SZp7</a></p>
<p>My model uses conv3d and I run my code on Tesla V100.</p>
<p>Many thanks.</p>

ptrblck: <p>Could you post the results of your manual profiling here, please?</p>
What is the difference between pytorch0.4.1 and pytorch0.4.1.post2?

Lucksong: <p>Just as the topic says.</p>

ptrblck: <p>The <code>post2</code> version has most likely minor fixes. Since these versions were released 2 years ago, I would recommend to not use or look into them. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Clarification on loading weights after early stopping

John_J_Watson: <p>I have written a class/method for early stopping, but I wanted some clarification on how to load weights to after early stopping. So, we have:</p>
<pre><code class="lang-auto">model = Net()
optim=Adam(model.params()) # rough syntax
lrs= &lt;some_lrs&gt;

for epoch in epochs:
    # do something and check for validation loss, assuming this has decreased, do:
    torch.save(model.state_dict(), "es_ckpt.pt")
   # check for early_stopping base don iterations of no_change, assuming no_change, load best weights for model:
   stop()

# load best model like so:
model.load_state_dict(torch.load('es_ckpt.pt'))
</code></pre>
<p>is this it? Do I need to create a fresh instance of the model or do I simply load the weights to the existing instance like above?</p>
<p>I also presume I do not need optimiser or lrs paramters in this es_ckpt.pt? since I have them already.</p>
<p>I just wnat to make sure there are no gotchas here, since it sounds too simple <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Cuda usage high but GPU usage low

Lea_Cly: <p>cuda usage is 95% but GPU usage always 0~1%<br>
I used profile to check my net, seems most time it works on GPU<br>
Self CPU time total: 4.838ms<br>
CUDA time total: 102.627ms</p>
<p>I tried to increase batchsize but nothing happened</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1d35e1d17ea2f6884fd73e40f1117c029c42e3d.png" data-download-href="https://discuss.pytorch.org/uploads/default/d1d35e1d17ea2f6884fd73e40f1117c029c42e3d" title="QQ图片20200610153046"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1d35e1d17ea2f6884fd73e40f1117c029c42e3d_2_690x435.png" alt="QQ图片20200610153046" data-base62-sha1="tWcDX18XyuHU9kYuWp6oPPP1B8N" width="690" height="435" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1d35e1d17ea2f6884fd73e40f1117c029c42e3d_2_690x435.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1d35e1d17ea2f6884fd73e40f1117c029c42e3d_2_1035x652.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1d35e1d17ea2f6884fd73e40f1117c029c42e3d_2_1380x870.png 2x" data-dominant-color="F6F7F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ图片20200610153046</span><span class="informations">1836×1160 124 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Lea_Cly: <p>I checked nvidia-smi，GPU-Util is 80%， windows monitor is not accurate</p>
What is the most standard way to put data in batch?

tcsn_wty: <p>I’m doing NLP projects, mostly using RNN, LSTM and BERT. I’ve never systematically learned PyTorch, and have seen many ways of putting data into torch tensors before passing to neural network. However, it seems that different ways sometimes can also influence the training process. I would like to know if anyone happen to know a most standard way to do so.</p>
<p>Say I’m doing text classification. Here’s the two ways I’ve tried that both worked for some projects.</p>
<p>First way:</p>
<pre><code class="lang-auto">from sklearn.model_selection import train_test_split

# Use 90% for training and 10% for validation.
# x is my input (numpy.ndarray), y is my label (numpy.ndarray).
x_train, x_test, y_train, y_test = train_test_split(x, y, random_state=42, test_size=0.9)

# Convert all inputs and labels into torch tensors, the required datatype for our model.
x_train = torch.tensor(x_train)
x_test = torch.tensor(x_test)
y_train = torch.tensor(y_train)
y_test = torch.tensor(y_test)

from torch.utils.data import DataLoader, TensorDataset

batch_size = 32

# Create the DataLoader for our training set.
train_data = TensorDataset(train_inputs, train_labels)
train_dataloader = DataLoader(train_data, batch_size=batch_size)

# Create the DataLoader for our validation set.
validation_data = TensorDataset(validation_inputs, validation_labels)
validation_dataloader = DataLoader(validation_data, batch_size=batch_size)
</code></pre>
<p>Then during training we’ll do</p>
<pre><code class="lang-auto">for step, batch in enumerate(train_dataloader):
        inputs = batch[0].cuda()
        labels = batch[1].cuda()
</code></pre>
<p>The other way I’ve used is:</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader, TensorDataset

# create tensor dataset
train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))
test_data = TensorDataset(torch.from_numpy(test_x), torch.from_numpy(test_y))

batch_size = 32

# shuffle data
train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size, drop_last=True)
test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size, drop_last=True)

for inputs, labels in train_loader:
</code></pre>
<p>I wonder if there is an document claiming a standard way to do such thing in PyTorch. More specifically, is there a standard way to make my input (say, list of embedded sentences) into batches and also acceptable by PyTorch neural network?</p>

ptrblck: <p>I don’t know exactly what difference you would like to highlight.<br>
In the first approach <code>train_inputs</code> and <code>train_labels</code> seem to be undefined (as well as <code>validation_x</code>), so I assume you would like to use <code>x_train</code> etc.?<br>
Also, the <code>DataLoader</code> loop is different, since you are unpacking the values in the first approach inside the loop (but that doesn’t matter and is just different coding style).</p>
<p>Are these the differences or what would you like to discuss?</p>
Training Error in Inception-V3

Mudasser_Afzal: <p>Hi ,<br>
I have got another error saying<br>
<code>AttributeError: 'InceptionOutputs' object has no attribute 'size'</code><br>
In incpetion_v3 pretrained model. I am using focal loss which works well on other models. focal loss code is here</p>
<pre><code class="lang-auto">def focal_loss(targets,logits,eps,l):
    ce_loss = torch.nn.functional.binary_cross_entropy_with_logits(logits, targets, reduction= 'none')
    pt = torch.exp(-ce_loss)
    loss = (eps * (1-pt)**l * ce_loss).mean()
    return loss
</code></pre>
<p>Please need help. It throws error on ce_loss statement</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/error-in-training-inception-v3/23933/10">here</a> with potential answer.</p>
Extract several intermediate layers of pytorch-transformer models to form a new model

willanxywc: <p>I am using the huggingface pytorch-transformers for knowledge distilation,<br>
How can I, for example, extract 8 layers from the 12 BertLayers of the bert-base-uncased to form a new model? I want to use the embedding and pooler layer of orginal model, but use only a portion of the encoder layers.</p>

ptrblck: <p>Depending on the original model architecture, you should be able to access the internal modules directly e.g. via:</p>
<pre><code class="lang-python">emb = model.encoder.embedding
fc = model.fc
</code></pre>
<p>These attribute names are just randomly picked by me and you would have to look into the model architecture to get the corresponding names.</p>
<p>Once you have all modules, you could pass these to your custom model and just reuse them in the <code>forward</code> method. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Adding and Concatenating layers

Vishu_Gupta: <p>Is<code> z = torch.add(x, y)</code> and <code>z = torch.cat(x, y)</code> in pytorch same as<br>
<code>z = keras.layers.add([x, y])</code> and <code>z = keras.layers.concatenate([x, y])</code> in keras?</p>

Nikronic: <p>Hi,</p>
<p><code>torch.add(x, y)</code> is equivalent to <code>z = x + y</code>. Similar to keras but only accepts 2 tensors.</p>
<p><code>torch.cat((x, y), dim)</code> (note that you need one more pair of parentheses like brackets in keras) will concatenate in given dimension, same as keras.</p>
<p>Bests</p>

Vishu_Gupta: <p>You said that <code>torch.add(x, y)</code> can add only 2 tensors. Why are we not able to person the operation on 3 or more tensors at once? Is it because the value may be affected or is it just how this thing is defined?</p>

Nikronic: <p>Actually, I do not understand why keras has implemented addition as an independent layer. In PyTorch, you can add any number of tensors by simply using <code>+</code> sign between them. <code>w = x + y + z</code> will do the same as <code>w = torch.add(x, torch.add(y, z))</code>.</p>
FileNotFoundError

Maccido: <p>Hello,</p>
<p>I am trying to reproduce the code implemented on</p>
<p><a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a></p>
<p>but i kept seeing this error FileNotFoundError.</p>
<p>The issue i believe is at the data loading stage. The original code loaded the data on dataroot = “data/celeba”. I am trying to do the same.</p>
<p>I created a folder celebrities_images inside which i saved the folder containing the training images named celeba and the python file is also saved in celebrities_images folder. I loading the data i used the command: dataroot =  “/Celebrities_Image/celeba” but i kept getting FileNotFoundError: [Errno 2] No such file or directory: ‘/Celebrities_Image/celeba’</p>
<p>PS: I am working on Linux and Celebrities_Image folder in home directory</p>
<p>Regards,<br>
Mohammad</p>

simaiden: <p>Use <code>dataroot = “./Celebrities_Image/celeba”</code> or <code>dataroot = “Celebrities_Image/celeba”</code></p>

shahadatku: <p><em>I am using the following code, but it showing the FileNotFoundError. I have tried several times, but could not run the code correctly. Can anyone please fix the error?</em></p>
<p>data_dir = ‘Cat_Dog_data/train’</p>
<p>transform = transforms.Compose([transforms.Resize((224,224)),<br>
transforms.CenterCrop(224),<br>
transforms.ToTensor()])<br>
dataset = datasets.ImageFolder(data_dir, transform=transform)<br>
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7eb35a0ebd67d9bc6ad72dbca54839d278153238.png" data-download-href="https://discuss.pytorch.org/uploads/default/7eb35a0ebd67d9bc6ad72dbca54839d278153238" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7eb35a0ebd67d9bc6ad72dbca54839d278153238.png" alt="Capture" data-base62-sha1="i4Qq5cIsfEnMWNvYchQ8IQYEDLW" width="690" height="362" data-dominant-color="F4F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">998×524 37.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>Cat_Dog_data/train</code> cannot be found on your system so you would have to verify that you are indeed in the right working directory and that the specified folder path really exists.</p>
How to interpret class weights when using CrossEntropyLoss

alx: <p>Hi all, my objective is to have the model prioritize class 1 and 2.<br>
I’m currently doing the following:</p>
<pre><code class="lang-auto">weights = torch.tensor([5., 10., 10., 5., 5.])
class_weights = torch.FloatTensor(weights).cuda()
criterion_weighted = nn.CrossEntropyLoss(weight=class_weights)
</code></pre>
<p>I’d like to know by how much is the model currently prioritizing those classes. Are the class weights relative to the sum of them? (5/35), (10/35)?</p>

ptrblck: <p>The loss for the current sample will be multiplied with the corresponding class weight and the final loss will be normalized by the sum of all used weightes (if <code>reduction='mean'</code> is used).<br>
The <a href="https://pytorch.org/docs/stable/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss">docs</a> give the formula, while <a href="https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/10">this post</a> gives you a manual example.</p>
<p>So the “prioritization” is relative to the used samples, not absolute.</p>
Failed to load model trained by DDP for inference

Wilson_Ho: <p>I have trained my model with 4 GPU by DDP (torch.nn.parallel.DistributedDataParallel)</p>
<p>But I wanna load the model for inference on machine with only single GPU , the codes are listed as below:</p>
<pre><code class="lang-auto">m = Model()
m.load_state_dict(torch.load('model.pth'))
</code></pre>
<p>it raises errors listed as below</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for ObjectDetector:
	Missing key(s) in state_dict: "backbone.block1.0.block.0.weight", "backbone.block1.0.block.0.bias", "backbone.block1.0.block.1.weight", "backbone.block1.0.block.1.bias", "backbone.block1.0.block.1.running_mean", "backbone.block1.0.block.1.running_var", "backbone.block1.1.block.0.weight", "backbone.block1.1.block.0.bias", "backbone.block1.1.block.1.weight", "backbone.block1.1.block.1.bias", "backbone.block1.1.block.1.running_mean", "backbone.block1.1.block.1.running_var", "backbone.block1.2.conv_block1.block.0.weight", "backbone.block1.2.conv_block1.block.0.bias", "backbone.block1.2.conv_block1.block.1.weight", "backbone.block1.2.conv_block1.block.1.bias", "backbone.block1.2.conv_block1.block.1.running_mean", "backbone.block1.2.conv_block1.block.1.running_var", "backbone.block1.2.conv_block2.block.0.weight", "backbone.block1.2.conv_block2.block.0.bias", "backbone.block1.2.conv_block2.block.1.weight", "backbone.block1.2.conv_block2.block.1.bias", "backbone.block1.2.conv_block2.block.1.running_mean", "backbone.block1.2.conv_block2.block.1.running_var", "backbone.block2.0.block.0.weight", "backbone.block2.0.block.0.bias", "backbone.block2.0.block.1.weight", "backbone.block2.0.block.1.bias", "backbone.block2.0.block.1.running_mean", "backbone.block2.0.block.1.running_var", "backbone.block2.1.conv_block1.block.0.weight", "backbone.block2.1.conv_block1.block.0.bias", "backbone.block2.1.conv_block1.block.1.weight", 

.......
"backbone.block5.1.conv_block1.block.1.running_var", 
</code></pre>
<p>I have tried</p>
<pre><code class="lang-auto">m.load_state_dict(torch.load('model.pth'), strict=False)
</code></pre>
<p>But the inference results are very strange not as expected</p>
<p>How should I fix it? thanks</p>

Wilson_Ho: <p>I have resolved the issues. It’s related to  <a href="https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686">1686</a></p>
<pre><code class="lang-auto">state_dict = torch.load(weight_path)

from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
    name = k[7:] # remove 'module.' of DataParallel/DistributedDataParallel
    new_state_dict[name] = v

m.load_state_dict(new_state_dict)
</code></pre>
Dataloader help

jamesmdai: <p>I currently have a dataset that is split into training, validation, and test sets. The dataset is consistent of  frames from various videos.</p>
<p>I’m running into issues about how to load this data in. The dataset is currently saved as video_1_0000, video_1_0001, …, video_2_0000, video_2_0001,…, etc. And each video may have different numbers of frames.</p>
<p>How do I go about loading this data, especially since the frames are all in the same directory? Do I need to do some preprocessing with the directory management or is there some library I can take advantage of?</p>
<p>Thanks!</p>

ptrblck: <p>How would you like to load these frames?<br>
Should each frame be randomly picked to create a batch or would you like to implement a sequential logic?<br>
Also, how would the targets be defined?</p>