akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Do pytorch have somewhere leaderboard/benchmark for testing minor modifications?

username: hadaev8
cooked: <p>Would be nice to have premade env, data, and model so I can just test if my modification makes sense.<br>
Of course, kaggle comes to mind, but I want to test GAN model, besides the Kaggle competition usually does not have enough data and in my experience, the influence of random seed is great (and ofc things like fine-tuning to test data, insane ensembling, etc).</p>
Multiple digit recognition

username: Kakar_Nyori
cooked: <p>Hi,</p>
<p>I have completed 60 MINUTE BLITZ with PyTorch and have learned how to train a classifier to distinguish between 10 different objects. Also, I have completed tutorial on recognising digits with PyTorch. Since then I have  been fascinated by PyTorch and machine learning in general!</p>
<p>I would like to continue my journey to learn more on deep learning, and would like to train a classifier to  recognise number/s (multiple digits) in an image? And later may be recognise whole text alphabets and digits.</p>
<p>I am sorry if this is a very noob question, but your help would really help me. How should I go about doing it? Any tutorials or suggestion are appreciated!</p>
<p>Edit:<br>
I know how to recognise a single digit in an image. For example this image can be recognised as <strong>4</strong>.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6f45a35c6b4d70b17bcd046730485199da41e3e.jpeg" alt="4" data-base62-sha1="q6uqaTdl6HKqtA2FQx81CWSbGMK" width="55" height="55"></p>
<p>But how can this single image with multiple digits be recognised this as <strong>504</strong>:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3cb9ef6106e990b9a82fcca8f61873b059a0d86f.jpeg" alt="504" data-base62-sha1="8Fd0LeaMO3MbYYIHIG6c6Ly1kL5" width="150" height="55"></p>
<p>Thank you!</p>

username: humble_D
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://www.kaggle.com/static/images/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://www.kaggle.com/c/digit-recognizer/notebooks?sortBy=relevance&amp;group=everyone&amp;search=pytorch&amp;page=1&amp;pageSize=20&amp;competitionId=3004" target="_blank" rel="nofollow noopener">kaggle.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://storage.googleapis.com/kaggle-competitions/kaggle/3004/logos/thumb76_76.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://www.kaggle.com/c/digit-recognizer/notebooks?sortBy=relevance&amp;group=everyone&amp;search=pytorch&amp;page=1&amp;pageSize=20&amp;competitionId=3004" target="_blank" rel="nofollow noopener">Digit Recognizer</a></h3>

<p>Learn computer vision fundamentals with the famous MNIST data</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Try this… Go through the notebooks that people have shared.</p>
Why same code behaves differently?

username: Dzhange
cooked: <p>Hi<br>
My model used to perform well with batch size 2, but since someday the loss looks like this with batch size 2(<strong>The code and data are the same!</strong>).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee0591b02b4189e6c89ba8b7a8ec8fa187d6f218.png" alt="image" data-base62-sha1="xXDzjUDyVHbbdsqnPYxf9WkT9Vu" width="568" height="463"></p>
<p>However, if I change the batch size into 16, the loss curve looks well, but the loss drops much slower than previous training.</p>
<p>The model I used is <a href="https://github.com/drsrinathsridhar/xnocs/blob/master/models/SegNet.py" class="inline-onebox" rel="noopener nofollow ugc">xnocs/models/SegNet.py at master · drsrinathsridhar/xnocs · GitHub</a><br>
And the loss function was  <a href="https://github.com/drsrinathsridhar/tk3dv/blob/2028b5ea77aca3410d9bf794daf14566ac6bf589/tk3dv/ptTools/loaders/GenericImageDataset.py#L78" class="inline-onebox" rel="noopener nofollow ugc">tk3dv/tk3dv/ptTools/loaders/GenericImageDataset.py at 2028b5ea77aca3410d9bf794daf14566ac6bf589 · drsrinathsridhar/tk3dv · GitHub</a></p>
<p>In another scenario, where I used the model for a harder object, the result with batch 2 will always converge to an all-white image(used to perform well), and when changing batch size into 16 the result became right but with high loss.</p>
<p>I’ve been struggled for days to figure out the reason but failed. At first, I thought this is caused by environmental problems but this bug exists in EVERY env. Can anyone give any suggestions?</p>
<p>Lots of thanks!</p>

username: Shisho_Sama
cooked: <p>Its very possible that you made a mistake back then. It happens. you think you set it to sth like 2, but infact it might have really been 16, you trained it with 16 and then sometime later, changed it back to 2 (or maybe changed sth else, used a higher dropout ratio, weight decay, increased or decreased a featuremap, etc) and forgot about it. in order to be sure you indeed ran on batch of 2, look at your logs if you have one.<br>
Another scenario is, the random nature of the seed you use. try using the same seed for your operations and see how that goes.</p>

username: Dzhange
cooked: <p>Thanks for the reply!<br>
I’m sure that I’ve been always using batch 2 and the hyperparameters are also fixed.<br>
Also the random seed is specified to 0 each time</p>

username: Dzhange
cooked: <p>On my initial post I said the code won’t work in any environment, but I just noticed that <strong>the results of this model vary from different environments</strong>. On one AWS server(with DeepLearning AMI, version 20, ubuntu 16.04), the model did give non-all-white output with batch size of 2. But the same setting just won’t work on another machine(All the difference I can tell is the Nvidia driver, one is 410.79 and another is 410.129).</p>
<p>Could this be a bug of pytorch?</p>
Should weighted CrossEntropyLoss and WeightedRandomSampler be used at the same time?

username: ProGamerGov
cooked: <p>Is it a good idea to use both weighted CrossEntropyLoss and WeightedRandomSampler at the same time? Or should I only be using one at a time for imbalanced datasets? Should I use weighted CrossEntropyLoss for the training data and WeightedRandomSampler for the validation data?</p>
<pre><code class="lang-auto">t_sampler = torch.utils.data.sampler.WeightedRandomSampler(train_weights, len(t_data))
criterion = nn.CrossEntropyLoss(weight=train_weights)
</code></pre>
<p>The weights are created with:</p>
<pre><code class="lang-auto">from collections import Counter
def count_classes(dataset):
    class_counts = dict(Counter(sample_tup[1] for sample_tup in dataset))
    class_counts = dict(sorted(class_counts.items()))
    return class_counts

train_class_counts = count_classes(t_data)
train_weights = [1 / train_class_counts[class_id] for class_id in range(num_classes)]
train_weights = torch.FloatTensor(train_weights)
</code></pre>
Running multiple basic matrix operations (add/sum etc) on GPU asynchronously?

username: BSODjunkie
cooked: <p>TL;DR: I want to subtract a a list of vectors from a Matrix asynchronously.</p>
<p>What I have is a matrix of size N x 512. I have another matrix of size M x 512 where M &lt; N always.</p>
<p>What I want is to subtract all the vectors from the matrix element wise asynchronously (I want to save M results).</p>
<p>When M = 1 the case is simple and here is an example:</p>
<pre><code class="lang-auto">a = torch.from_numpy(np.arange(50).reshape(5, 10))
b = torch.from_numpy(np.arange(10).reshape(1, 10))
a.sub(b)
</code></pre>
<p>which gives the correct output;</p>
<pre><code class="lang-auto">tensor([[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0],
        [10, 10, 10, 10, 10, 10, 10, 10, 10, 10],
        [20, 20, 20, 20, 20, 20, 20, 20, 20, 20],
        [30, 30, 30, 30, 30, 30, 30, 30, 30, 30],
        [40, 40, 40, 40, 40, 40, 40, 40, 40, 40]])
</code></pre>
<p>Is there any way for me to then do this for M &gt; 1 and save the results in a non-sequential manner?</p>
Workaround for index_add / masked_scatter

username: IgnacioPickering
cooked: <p>I’m currently trying to export a model to ONNX and run it using an external library that parses ONNX. Unfortunately my model uses <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.index_add_" rel="nofollow noopener">index_add</a> and <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.masked_scatter_" rel="nofollow noopener">masked_scatter</a> and the external library can’t parse the ONNX operations these two are transformed into (I believe index_add doesn’t even have an ONNX equivalent for opset 11 which is what I’m using). Is there any way I can rewrite these operations into “simpler” ONNX compatible operations, without a major dip in performance (i. e. without having to write explicit for loops in python)?</p>
Torchvision.io.read_video vs extracting images prior to creating dataset

username: braindotai
cooked: <p>I’d like to know which one is faster… torchvision.io.read_video vs extracting the images of the videos and then reading them.</p>
<p>For both the output video shape should be (channels, frames, height, width)</p>

username: JuanFMontesinos
cooked: <p>torchvision is quite fast.<br>
Extracting images (on-the-fly) requires hardisk IO thus i doubt it were faster.<br>
If you canextract images before hand that would be faster for sure as video is compressed and requires time to be decoded.</p>
Pytorch way to write following nested loop

username: ad6398
cooked: <p>I want to change value of a tensor matrix (mat1) as per the value of tensors in pairs and mat2. I am doing it by running a nested loop as below. Is there any way to write this without using a loop like using gather or any other faster inbuilt commands?<br>
size of mat1 = (bs, seq_len)<br>
size of pairs = (bs, x, 2)<br>
size of mat2= (bs,x, y)</p>
<pre><code class="lang-auto">for i in range(mat1.size(0)):
        for j in range(pairs[i].size(0)):
            s,e= pairs[i][j][0], pairs[i][j][1]
            mat1[i][s:e+1] = mat2[i][j][0:e-s+1]
</code></pre>

username: ad6398
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, any help would be appreciated.</p>
I can't import OneCycleLR from torch.optim.lr_scheduler

username: mercer_Alex
cooked: <p>pytorch vision: 1.5.1<br>
Only OneCycleLR can’t be import and others are normal<br>
Am I writing it wrong ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90c9929b8242e7b07494de15070730dc0f407cf1.png" data-download-href="https://discuss.pytorch.org/uploads/default/90c9929b8242e7b07494de15070730dc0f407cf1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90c9929b8242e7b07494de15070730dc0f407cf1.png" alt="image" data-base62-sha1="kEQCDHtyuSJ49O8lJ1v8ECc4Fqx" width="690" height="84" data-dominant-color="3A3C3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1001×123 6.98 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Are you getting an error when you execute the <code>import</code> statement or is this just a warning from your IDE?<br>
<code>OneCycleLR</code> was added ~11 months ago, so it should be in <code>1.5.1</code>.</p>

username: mercer_Alex
cooked: <p>I tryed to run the code, it worked!</p>
Torch_geometric node2vec get vector

username: Steven_Duan
cooked: <p>For example if I have a graph of 3 nodes A, B, and C. What I want to do is to feed this graph into a node2vec model and learn an embedding of each node. Here is the pipe line</p>
<ol>
<li>Construct a graph using nextworkx</li>
<li>Convert it to a pytorch graph data using from_networkx</li>
<li>Construct a node2vec model and train it.<br>
After training I should get a 3XN matrix where N is the embedding dimension I desired. My question is, how do I know which of the 3 embedding is the embedding for node A</li>
</ol>
How to know if model is half or full precision?

username: bendangnuksung
cooked: <p>Is there a way to find if a model is a half-precision or full precision model?</p>

username: ptrblck
cooked: <p>By default PyTorch will initialize all tensors and parameters with “single precision”, i.e. <code>float32</code>.<br>
If you are not using the mixed precision training utilities or are calling <code>.half()</code>, <code>to(torch.float16)</code>etc. in your code, the model should stay in FP32.</p>
<p>To check it, you could iterate all parameters (and buffers) and print their <code>dtype</code>:</p>
<pre><code class="lang-python">for param in model.parameters():
    print(param.dtype)
</code></pre>
Why does GoogLeNet additionally process input via _transform_input?

username: algo
cooked: <p>Was GoogLeNet trained the same way as other torchvision models?</p>
<p>If so why is there additional preprocessing step via _transform_input function?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/308d239038d9c398c1290abca8eb7198ddf4db6c.png" alt="image" data-base62-sha1="6Vvj3lnP4RzRT1N97AquLtIuKEk" width="560" height="174"></p>
<p>It’s called in forward() before the actual work in _forward() function is executed.</p>
RuntimeError: cublas runtime error : library not initialized at /opt/conda/conda-bld/pytorch_1565272279342/work/aten/src/THC/THCGeneral.cpp:216

username: zongneng
cooked: <p>netD = Discriminator()</p>
<p>netD.zero_grad()</p>
<h1>train with real</h1>
<p>D_real = netD(real_data_v)</p>
<p>Traceback (most recent call last):<br>
File “/tmp/pycharm_project_314/wgan-gp/gan_toy.py”, line 274, in <br>
D_real = netD(real_data_v)<br>
File “/public/home/yanjianhua/anaconda3/envs/Pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/tmp/pycharm_project_314/wgan-gp/gan_toy.py”, line 82, in forward<br>
output = self.main(inputs)<br>
File “/public/home/yanjianhua/anaconda3/envs/Pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/public/home/yanjianhua/anaconda3/envs/Pytorch/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 92, in forward<br>
input = module(input)<br>
File “/public/home/yanjianhua/anaconda3/envs/Pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/public/home/yanjianhua/anaconda3/envs/Pytorch/lib/python3.6/site-packages/torch/nn/modules/linear.py”, line 87, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/public/home/yanjianhua/anaconda3/envs/Pytorch/lib/python3.6/site-packages/torch/nn/functional.py”, line 1369, in linear<br>
ret = torch.addmm(bias, input, weight.t())<br>
RuntimeError: cublas runtime error : library not initialized at /opt/conda/conda-bld/pytorch_1565272279342/work/aten/src/THC/THCGeneral.cpp:216</p>

username: ptrblck
cooked: <p>Could you check, if you might be running out of memory?<br>
Sometimes these library issues are raised even though you are in fact running out of memory.<br>
If that’s the case, I would recommend to reduce the batch size.</p>
Is there a way to globally disable autograd?

username: zStupan
cooked: <p>This might be a stupid question but here it goes…In my Module, I would like to calculate and update the gradients myself. Is there a way I can disable autograd globally rather than setting <code>requires_grad = False</code>  for every tensor?</p>

username: ebarsoum
cooked: <p>You can use:</p>
<pre><code class="lang-auto">with torch.no_grad():
    your code here.
</code></pre>

username: ptrblck
cooked: <p>Additionally to <a class="mention" href="/u/ebarsoum">@ebarsoum</a>’s solution, you could also use <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_grad_enabled"><code>torch.autograd.set_grad_enabled(mode)</code></a> to globally disable the gradient calculation.</p>
[fairseq] Possible bug: Mutlingual training with max_tokens blows up in utils.resolve_max_positions

username: Enamex
cooked: <p>See <code>fairseq/utils.py</code>:<code>367</code>.</p>
<pre><code class="lang-auto"># fairseq/utils.py
def resolve_max_positions(*args):
    """Resolve max position constraints from multiple sources."""

    [...]

    max_positions = None
    for arg in args:
        if max_positions is None:
            max_positions = arg
        elif arg is not None:
            max_positions, arg = _match_types(max_positions, arg)
            if isinstance(arg, float) or isinstance(arg, int):
                max_positions = min(max_positions, arg)
            elif isinstance(arg, dict):
                max_positions = map_value_update(max_positions, arg)
            else:
                max_positions = tuple(map(nullsafe_min, zip(max_positions, arg)))

    return max_positions
</code></pre>
<p>And consider where the multilingual translation task uses the following for max positions:</p>
<pre><code class="lang-auto"># fairseq/tasks/multilingual_translation.py
# class MultilingualTranslationTask(FairseqTask):
[...]
    def max_positions(self):
        """Return the max sentence length allowed by the task."""
        if len(self.datasets.values()) == 0:
            return {'%s-%s' % (self.args.source_lang, self.args.target_lang):
                    (self.args.max_source_positions, self.args.max_target_positions)}
        return OrderedDict([
            (key, (self.args.max_source_positions, self.args.max_target_positions))
            for split in self.datasets.keys()
            for key in self.datasets[split].datasets.keys()
        ])
</code></pre>
<p>Which returns a dict like <code>{lang_pairs...: (max1, max2)}</code>.</p>
<p>Then the resolve function tries to match types between all the passed max_positions values. Which can be a number, a tuple, or a dict. Once it sees a dict, <code>max_positions</code> becomes a dict and any later values will error as there’s no meaningful upgrade from dict to dict.</p>
Official PyTorch Resnet50 in Keras

username: syomantak
cooked: <p>There is a package in Keras which I need to use so I need to shift to Keras half-way through a project.<br>
Right now, I load the trained model like this</p>
<pre><code class="lang-auto">import torchvision.models as models

model = models.__dict__[arch](num_classes=365)
model_file = 'resnet50_places365.pth.tar'
checkpoint = torch.load(model_file, map_location=lambda storage, loc: storage)
state_dict = {str.replace(k,'module.',''): v for k,v in checkpoint['state_dict'].items()}
model.load_state_dict(state_dict)
model.eval()
</code></pre>
<p>For full code, refer <a href="https://github.com/CSAILVision/places365/blob/master/run_placesCNN_basic.py" rel="nofollow noopener">here</a>.<br>
I now need the same model with the same weight in keras. I have no idea how to convert the model! Any help would be great!</p>

username: ptrblck
cooked: <p>I would recommend to post this question on StackOverflow, as you should find more Keras users there. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: syomantak
cooked: <p>I ended up implementing the package in PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>
Explain algorithm, code for training time

username: shangguan91
cooked: <p>it is super wierd that start= time.time()</p>
<p>time.time() minus time.time() =0?</p>
<p>but the result do not equal 0 why?</p>
<pre><code class="lang-auto">import time

start = time.time()

model.train() # Training statement

print("Total time: ", time.time() - start, "seconds")
</code></pre>
<p>This code is wrong,</p>

username: alpha_omega_alpha
cooked: <p>Hi <a class="mention" href="/u/shangguan91">@shangguan91</a>, the code is correct.<br>
Calling <code>model.train()</code> doesn’t train your model, it just sets your model in training mode, so virtually no cost in time.<br>
For more information, please refer to <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module.train" rel="nofollow noopener">here</a>.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="shangguan91" data-post="1" data-topic="88446">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a3d4f5/48.png" class="avatar"> shangguan91:</div>
<blockquote>
<p>but the result do not equal 0 why?</p>
</blockquote>
</aside>
<p><code>time.time()</code> will return your current system time in each call. Since the <code>model.train()</code> operation will take a bit of time to execute, your total time will not be 0.</p>

username: alpha_omega_alpha
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I also tried timing <code>model.train()</code> using <code>time.time()</code> which returned 0, too. So, my initial thought was that as <code>model</code> was already in training mode by default, so basically running <code>model.train()</code> would not incur cost in time, hence 0.<br>
Would be interested to know why it’s returning 0, though.</p>

username: ptrblck
cooked: <p>Even if the operation won’t change any internal flags, still some Python code would be executed, so I don’t know why it shows a zero. Maybe the timer has not enough resolution to properly time the fast execution. On my laptop, I’ll get sometimes a result of <code>0.0005006</code> and other times <code>0.0</code>.<br>
Running multiple iterations and calculating the average time solves this issue usually.</p>
Transfer learning, extra input channels

username: Giannhs_Vlasso
cooked: <p>tldr:<br>
I want to transfer trained weights from a conv layer like:<br>
self.conv_init = nn.Conv1d(num_in_channels, num_hidden, 1)</p>
<p>to a new one with more input channels (initializing the rest with 0’s or according to some initialization scheme) - and everything else remaining same, like eg. :<br>
self.conv_init = nn.Conv1d(num_in_channels+5, num_hidden, 1).</p>
<p>As far as i know load_state_dict will either ignore that layer or throw an error for the size mismatch. Is there a built in way to do the above? …Or another way?</p>
<hr>
<p>story for context:<br>
I’m working on building a net for time series data. The dataset contains sensor (let’s call them group a) readings since early 00’s. At some point after '17 extra sensors (‘group b’) where added.<br>
I’d like to train a model on the 1st original group (to take advantage of the 2 decades of data) , and then use that as a ‘starting point’ to train another including group b.<br>
The input is an 1d multichannel tensor, that’s a fixed size sliding window on the data where every channel is a sensor’s readings.<br>
The 1st conv layer (shown/mentioned above) has kernel size 1, and is used to create the hidden channels that the rest of the layers will use, that will stay the same between the two models.</p>

username: ptrblck
cooked: <p>You could load the <code>state_dict</code> before any manipulation of the first conv layer to make sure all other layers are properly restored.<br>
Once this is done, you could replace the first conv layer with a new one and reassign the parameters manually. This code snippet give you an example:</p>
<pre><code class="lang-python">ref = nn.Conv2d(3, 64, 3, 1, 1)
conv = nn.Conv2d(5, 64, 3, 1, 1)

with torch.no_grad():
    conv.weight[:, :3] = ref.weight
    conv.weight[:, 3:] = torch.zeros_like(conv.weight[:, 3:])
    conv.bias = ref.bias
    
x = torch.randn(2, 3, 24, 24)
out1 = ref(x)
out2 = conv(torch.cat((x, torch.randn(2, 2, 24, 24)), dim=1))

print((out1 == out2).all())
&gt; tensor(True)
</code></pre>
Check type of Modules in nn.ModuleList

username: Kale-ab_Tessera
cooked: <p>I have a ModuleList, with Linear and Batchnorm layers. I wanted to call named <code>named_parameters()</code> and only get the grads of the linear layers as follows:</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
            if(isinstance(param, nn.Linear)):            
                  grads[name] = param.grad
</code></pre>
<p>This doesn’t work since <code>type(param)</code> always returns <code>torch.nn.parameter.Parameter</code>. Also <code>if "weight" in name:</code> doesn’t work since batchnorm layers also have “weight” in their name.</p>
<p>Are there any workarounds to this? I also looked at giving custom names to modules in the ModuleList, but I couldn’t find how.</p>

username: chetan_patil
cooked: <p>Hi, ModuleList does not come with any learnable parameters. Items inside the list are just like name of layers.<br>
You can replace <code>nn.ModuleList()</code> with <code>nn.Sequential()</code> instead.</p>

username: Kale-ab_Tessera
cooked: <p>I have a dynamic number of hidden layers and so I don’t think nn.Sequential will work.</p>

username: ptrblck
cooked: <p>You could iterate <code>model.named_modules()</code> and check the returned <code>module</code> to be an <code>nn.Linear</code> instance:</p>
<pre><code class="lang-python">for name, module in model.named_modules():
    if isinstance(module, nn.Linear):
        print(name, module.weight.grad)
</code></pre>
Result reproducibility in PyTorch

username: Megh_Bhalerao
cooked: <p>Hi everyone,<br>
I have a question with regards to reproducibility of results in PyTorch. In my main file (<code>trainer.py</code> file), I am doing the following:</p>
<pre><code class="lang-auto">torch.cuda.manual_seed(seed_val)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
</code></pre>
<p>I am doing this ensure that my results match atleast upto some precision across different runs.</p>
<p>So my question is, do I have to write these 3 lines in <strong>all my files</strong> which import pytorch, for example my file where I have defined the neural network model, or maybe file where I have defined my loss function, the data loader files and so on,  or is it enough to include these 3 lines <strong>ONLY</strong> in my <code>trainer.py</code>, where I am instantiating (and initializing), the loss function, the NN model, and other stuff?</p>
<p>Please help me out.</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>It should be sufficient to set the seed after the first import of PyTorch.<br>
As long as the control flow stays the same (i.e. the calls to the pseudo-random number generator are the same), you should get the same outputs (besides of course the limitations mentioned in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a>).</p>
Getting different results running the code in Colab and JupyterNotebook

username: samin_hamidi
cooked: <p>I am training and testing an autoencoder. I run the exact same code in Jupyter notebooks and also in Googlecolab. I have these settings:</p>
<blockquote>
<p>np.random.seed(0)<br>
random.seed(0)<br>
torch.manual_seed(0)<br>
torch.cuda.manual_seed(0)<br>
torch.cuda.manual_seed_all(0)<br>
torch.backends.cudnn.enabled = False<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = True</p>
</blockquote>
<p>and</p>
<blockquote>
<p>train_loader = torch.utils.data.DataLoader(<br>
dataset=train_set,<br>
batch_size=batch_size,<br>
shuffle=False,<br>
num_workers=2,<br>
worker_init_fn=np.random.seed(0))</p>
</blockquote>
<blockquote>
<p>test_loader = torch.utils.data.DataLoader(<br>
dataset=test_set,<br>
batch_size=batch_size,<br>
shuffle=False,<br>
num_workers=2,<br>
worker_init_fn=np.random.seed(0))</p>
</blockquote>
<p>Also, I am using GPU on both environments.</p>
<p>The performance in each environment is deterministic but they don’t produce the exact same accuracy: Colab=0.7068, JupyterNotebook=0.6689<br>
I wonder why they behave like this? shouldn’t they produce the exact same values?</p>
<p>I appreciate any help and guidance.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="samin_hamidi" data-post="1" data-topic="88655">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a698b9/48.png" class="avatar"> samin_hamidi:</div>
<blockquote>
<p>I wonder why they behave like this? shouldn’t they produce the exact same values?</p>
</blockquote>
</aside>
<p>Not necessarily, if different hardware and potentially software versions (CUDA, PyTorch) are used.<br>
You could run some tests and see, if you would be getting the same random numbers on both platforms in the first place.</p>
Leaky ReLU only gives negative values or values close to 0

username: zimmer550
cooked: <p>I have normalized my data using zero mean and variance of one. I want to do a sanity check to see if I can overfit a small amount of data. I created one fully connected layer with input size of 1024 and output size of 9. I use <code>torch.nn.init.kaiming_uniform_</code><br>
to initialize the weights and I also use batch norm before calling the Leaky ReLU activation function for this layer.</p>
<p>I have been training for almost 50 epochs with batch size of 32 and total batches are just 100. However, my network’s output is either negative or close to zero. I am trying to regress to label values that lie between 0 and 1. Before I tried using ReLU (with same weight initialization as I mentioned now) but the output, in that case, was almost always entirely zero. What could I do here to steer my output towards more positive values?</p>

username: ptrblck
cooked: <p>Try to remove the last activation function and use the output of the last linear layer as the prediction (if that’s not already the case).<br>
If that doesn’t work, try to play around with more hyperparamters, such as the learning rate etc.</p>
Create VGG19_bn architecture class with Forward function

username: rashidbaloch
cooked: <p>Hi Experts,<br>
I need help in creating a custom model architecture just like VGG19_bn. Instead of using VGG19(pretrained=True) I want to create Identical VGG architecture with Class and Forward functions etc so that I can get 4096 dimensional feature vector so that these can output_feature=20 based on my custom data for Image classification.<br>
I am trying to do image classification based on custom dataset from Scratch without using transfer learning(pretrained networks). I just want to use the architecture of VGG19_bn that’s it.<br>
Thank you</p>
<p>Example:<br>
class VGG19(nn.Module):</p>
<pre><code>def __init__(self,num_classes=20):

    super(VGG19,self).__init__()
    # All layers of VGG19_bn along with its Relu Max Pooling bias weights etc everything.
self.fc = nn.Linear(in_features=512*7*7,out_features=num_classes)
</code></pre>
<p>def forward(self, input):<br>
output = <a href="http://self.net" rel="nofollow noopener">self.net</a>(input)<br>
output = output.view(-1,128<em>7</em>7)<br>
output = self.fc(output)<br>
return output</p>

username: ptrblck
cooked: <p>You could copy the <code>VGG</code> implementation from <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py"><code>torchvision.models.vgg</code></a> and change the layers as you want, which would be the simplest approach for your custom architecture.</p>