akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Changing batch size

slavavs: <p>Why, when changing batch size, do the weights of neurons change? If I test new data with a batch size equal to the size with which I trained NN, then the results are good. If you change the batch size, the results are bad.</p>
<pre><code class="lang-auto">class MyResNet(ResNet):
    def __init__(self):
        super(MyResNet, self).__init__(BasicBlock, [2, 2, 2, 2], num_classes=3)
        self.conv1 = torch.nn.Conv2d(1, 64, 
            kernel_size=(7, 7), 
            stride=(2, 2), 
            padding=(3, 3), bias=False)
...
model.load_state_dict(torch.load('save.pth'))
criterion = nn.CrossEntropyLoss(reduction='sum')
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
...
outputs = model(x)
loss1 = criterion(outputs, y)
optimizer.zero_grad()
loss1.backward()
optimizer.step()
</code></pre>

ptrblck: <aside class="quote no-group" data-username="slavavs" data-post="1" data-topic="56700">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/slavavs/48/15429_2.png" class="avatar"> slavavs:</div>
<blockquote>
<p>Why, when changing batch size, do the weights of neurons change?</p>
</blockquote>
</aside>
<p>The weights do not change based on the batch size or the forward pass alone.<br>
Could you explain this issue a bit more?</p>
<aside class="quote no-group" data-username="slavavs" data-post="1" data-topic="56700">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/slavavs/48/15429_2.png" class="avatar"> slavavs:</div>
<blockquote>
<p>If I test new data with a batch size equal to the size with which I trained NN, then the results are good. If you change the batch size, the results are bad.</p>
</blockquote>
</aside>
<p>Make sure to call <code>model.eval()</code> before evaluating your model, as otherwise e.g. the running estimates of batchnorm layers will be updated, which depends on the used batch size.</p>

slavavs: <p>I don’t know how, but my can teaches samples in batch.<br>
I did not shuffle.<br>
After training, if I change the batch size, then the network can no longer correctly predict.<br>
If I call model.eval () the result is very bad</p>

ptrblck: <p>This might be due to skewed running estimates in your batch norm layer.<br>
Try to use a higher batch size during training or adapt the <code>momentum</code>.</p>
Does there exist embedding sparse

songziqin: <p>python have sparse embedding api(let sparse=True) I want to know how to finish the same task in C++.<br>
thanks.</p>

Koro: <p>Hi, have you tried to set the internal weight Tensor to sparse directly?<br>
If I do something like this here for example:</p>
<pre><code class="lang-auto">auto embedd = torch::nn::Embedding(3, 2);
embedd-&gt;weight = embedd-&gt;weight.to_sparse();
</code></pre>
<p>And then print embedd-&gt;weight before and after, it shows the following:<br>
BEFORE:</p>
<pre><code class="lang-auto">0.1794  1.1953
1.6296  0.2926
3.0924 -1.9228
[ Variable[CPUFloatType]{3,2} ]
</code></pre>
<p>AFTER:</p>
<pre><code class="lang-auto">[ Variable[SparseCPUFloatType]{}
indices:
 0  0  1  1  2  2
 0  1  0  1  0  1
[ Variable[CPULongType]{2,6} ]
values:
 0.1794
 1.1953
 1.6296
 0.2926
 3.0924
-1.9228
[ Variable[CPUFloatType]{6} ]
size:
[3, 2]
]
</code></pre>
<p>with kind regards, Florian Korotschenko</p>
CNN weights initialization

Mohammed_Awney: <p>How I could initialize fixed weights for cnn each time when I start training?</p>

albanD: <p>Hi,</p>
<p>You want every initialization of your network to be the same when you start training?<br>
You can set random seeds for that.<br>
Be careful though that many things are non-deterministic and so won’t behave exactly the same between runs. You can read more about this <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">here</a>.</p>

Mohammed_Awney: <p>I want to compare performance between 3 CNNs but I want to fix the weights to ensure that the changes in performance doesn’t comes from weights random initialization of each</p>

albanD: <p>Hi,<br>
The thing is that the random value of the weights might favor one architecture/training procedure over the others. So fixing one random initialization is not really a fair comparison.<br>
To prevent weight initialization from interfering, you would need to run each model to a few random seeds and compare average and variance of their performances.</p>
Make changes to resnet18

slavavs: <p>I want to use resnet18. According to the documentation, she receives an image 224x224x3 in size at the entrance. My semple has a size of 224x3x3. I want to add a linear (3x224) layer to the resnet18 input to get semple (224x224). Help me change def forward.</p>
<pre><code class="lang-auto">class MyResNet(ResNet):
    def __init__(self):
        super(MyResNet, self).__init__(BasicBlock, [2, 2, 2, 2], num_classes=3)
        self.myfc1 = nn.Linear(3, 224)
        nn.init.xavier_uniform_(self.myfc1.weight)
        self.conv1 = torch.nn.Conv2d(1, 64, 
            kernel_size=(7, 7), 
            stride=(2, 2), 
            padding=(3, 3), bias=False)
</code></pre>

slavavs: <p>Interestingly, if I upload to a resnet sample of size 224x3, there is no error. Why is that? The resnet network itself converts my semples</p>
How can i change the preset model to accept custom shape?

Steve_Hu: <p>Recently, i want to use the torchvision.models.video.r2plus1d_18 , but i found the needed input shape is<br>
(3 x T x H x W), but my input is   (2 x T x H x W), so there is a solution to modify the preset model to accept this kind of shape? i am confused now…thx!!!</p>

spanev: <p>Hi <a class="mention" href="/u/steve_hu">@Steve_Hu</a>,</p>
<p>You can first try to zero pad the input:</p>
<pre><code class="lang-auto"># Given input = (B, 2, T, H, W)
input = torch.cat((input, torch.zeros(B, 1, T, H, W)), axis=1)
</code></pre>
<p>You can also modify the first layer and fine tune it with your 2 channels input. To modify it:</p>
<pre><code class="lang-auto">net = torchvision.models.video.r2plus1d_18()
r2.stem[0] = Conv3d(2, 45, kernel_size=(1, 7, 7), stride=(1, 2, 2), padding=(0, 3, 3), bias=False)
</code></pre>

Steve_Hu: <p>thx very very much:heart:! i will try the second way:grin:</p>
Conda not installing or updating to latest pytorch version

Kong: <p>I ran this command <code>conda install pytorch torchvision cudatoolkit=10.0 -c pytorch</code> but it was <code>0.3.0.post4</code> that got installed. Then I ran <code>conda update pytorch torchvision</code> but got the following message below. How do I update to the latest version of PyTorch? I have a GTX 1080 and the Cuda driver version is 10.1.</p>
<blockquote>
<p>Collecting package metadata (current_repodata.json): done<br>
Solving environment: done</p>
<p>==&gt; WARNING: A newer version of conda exists. &lt;==<br>
current version: 4.7.10<br>
latest version: 4.7.12</p>
<p>Please update conda by running</p>
<pre><code>$ conda update -n base -c defaults conda
</code></pre>
<h2>Package Plan</h2>
<p>environment location: /home/kong/anaconda3/envs/ffp</p>
<p>added / updated specs:<br>
- pytorch<br>
- torchvision</p>
<p>The following packages will be downloaded:</p>
<pre><code>package                    |            build
---------------------------|-----------------
libgcc-7.2.0               |       h69d50b8_2         269 KB
pytorch-0.2.0              |py35h01b9ba7_4cu75       311.5 MB  soumith
torchvision-0.1.9          |   py35h72e4c6f_1          87 KB  soumith
------------------------------------------------------------
                                       Total:       311.8 MB
</code></pre>
<p>The following NEW packages will be INSTALLED:</p>
<p>libgcc             pkgs/main/linux-64::libgcc-7.2.0-h69d50b8_2</p>
<p>The following packages will be SUPERSEDED by a higher-priority channel:</p>
<p>pytorch            pytorch::pytorch-0.3.0-py35_cuda8.0.6~ → soumith::pytorch-0.2.0-py35h01b9ba7_4cu75<br>
torchvision        pytorch::torchvision-0.2.0-py35heaa39~ → soumith::torchvision-0.1.9-py35h72e4c6f_1</p>
</blockquote>
<p>Running <code>nvcc -V</code> gives me</p>
<blockquote>
<p>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2019 NVIDIA Corporation<br>
Built on Sun_Jul_28_19:07:16_PDT_2019<br>
Cuda compilation tools, release 10.1, V10.1.243</p>
</blockquote>

ptrblck: <p>Could you update <code>conda</code> as given by the command in your output and try to install PyTorch again?</p>
Train loss fluctuate regularly and doesn't descend

ljx: <p>Hi, I am new in Pytorch and using DL in image segmentation.  However, the train_loss does not descend apparently and always fluctuate regularly and the test dice coefficient does not ascend apparently.<br>
I wonder the problem is on the programming itself or the network structure or the data set or the hyper parameters. Could anyone give some advise and experiences ?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0c9e3ecd9376eccdae12dc1371471bca90adbd68.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c9e3ecd9376eccdae12dc1371471bca90adbd68" title="Screenshot%20from%202019-05-26%2017-13-41"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0c9e3ecd9376eccdae12dc1371471bca90adbd68_2_325x500.png" alt="Screenshot%20from%202019-05-26%2017-13-41" data-base62-sha1="1NCLEQ9hPS8lfI7b5clJHR5tmFq" width="325" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0c9e3ecd9376eccdae12dc1371471bca90adbd68_2_325x500.png, https://discuss.pytorch.org/uploads/default/original/2X/0/0c9e3ecd9376eccdae12dc1371471bca90adbd68.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0c9e3ecd9376eccdae12dc1371471bca90adbd68.png 2x" data-dominant-color="F6F7F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-05-26%2017-13-41</span><span class="informations">351×539 24.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

SoucheChapich: <p>Have you solved your problem ?</p>
Help find a bug in the code

slavavs: <p>IndexError: list index out of range<br>
I can’t understand why this error is being called.</p>
<pre><code class="lang-auto">import numpy as np
import random

wn1 = 2
epochs = 10
st_s1 = np.random.rand(10,9)
st_t1 = np.random.rand(10,1)
len_st = len(st_s1)
b = len_st-wn1

for epoch in range(epochs):
    shuffle_list = list(range(1, b))
    random.shuffle(shuffle_list)
    for wn_start in range(b):
        wn_all2 = []
        los_l = []
        shuffle_temp = shuffle_list[wn_start]
        los_l.append(st_t1[shuffle_temp+wn1])
        wn_all2.append(st_s1[shuffle_temp:shuffle_temp+wn1,3:6])
</code></pre>

spanev: <p>Hi <a class="mention" href="/u/slavavs">@slavavs</a>,</p>
<p>Here you declare a list of length 7:</p>
<pre><code class="lang-auto">    shuffle_list = list(range(1, b))
</code></pre>
<p>So in the inner loop you should iterate of <code>range(b-1)</code> and not <code>range(b)</code>.</p>
Torch.onnx.export can not disable logging

jinfagang: <pre><code class="lang-auto">torch.onnx.export(model,
                      x,
                      model_path,
                      verbose=False,
                      opset_version=10,
                      export_params=True)
                    #   verbose=False,
                    #   opset_version=10,
                    #   strip_doc_string=True,
                    #   do_constant_folding=True)
    print('onnx model exported.')
</code></pre>
<p>I export maskrcnn-benmark maskrcnn model to onnx. but it keeps give me huge logging message.</p>
<p>Even with verbose=False.</p>
<p>How to disable it???</p>
<p>Anyone nknows?</p>

jinfagang: <p>Hope some body could help me. One bitcoin as bonners,</p>

jinfagang: <p>Close it , I know why.</p>
[solved] RNN - access hidden states of every time step

moonlightlane: <p>I’m training a vanilla RNN on MNIST (sequential, len=784, one scalar value at each time step), and I would like to visualize the hidden states at every time step, not just the final time step. How can I access the hidden states prior to the one at the final time step? It seems that <code>torch.nn.RNN</code> api only allows me to access the hidden state at the final time step.</p>
<p>I can probably accomplish this by using <code>torch.nn.RNNCell</code> instead of <code>torch.nn.RNN</code>: looping through the input sequence manually and saving all the hidden states; however, I’m a bit concerned about the performance drop if I implement an RNN using <code>RNNCell</code> instead of <code>RNN</code>, both in terms of speed (see <a href="https://discuss.pytorch.org/t/rnncell-vs-rnn/3881">here</a>) and classification accuracy (see <a href="https://discuss.pytorch.org/t/very-different-performance-between-lstm-and-lstmcell-i/14025">here</a>).</p>
<p>Any suggestions/advice would be greatly appreciated!</p>

John_Smith: <p>When you run a rnn net over a sequence of data, the output has two tensors: (output, h_n). While h_n is the hidden state from the last timestamp, the tensor output actually has the hidden states for every timestamps in the sequence.</p>
<pre><code class="lang-auto">rnn_net = nn.RNN(4, 3)
X = torch.rand((6,5,4))
out, hidden = rnn_net(X)
out.shape
hidden.shape
</code></pre>
<p>You will see the shape of out is torch.Size([6, 5, 3]) which represents (seq_len, batch, hidden_size)</p>

moonlightlane: <p>Thanks for pointing that out! Looks like I have been very sloppy when reading documentation…</p>

Namita_Kalra: <p>But output contains hidden states from only the last LSTM layer. What to do in case we want hidden states from all the layers of each time step?</p>
<p>Looping makes it a lot slower!</p>
LSTM Produces Random Predictions

skiddles: <p>I have trained an LSTM in PyTorch on financial data where a series of 14 values predicts the 15th. I split the data into Train, Test, and Validation sets.  I trained the model until the loss stabilized.  Everything looked good to me when using the model to predict on the Validation data.</p>
<p>When I was writing up my research to explain to my manager, I happened to get different predicted values each time I ran the model (prediction only) on the same input values. This is not what I expected so I read some literature but was not able to explain my results.  Intuitively, my results indicate there is some random variable, node, gate, that is influences the prediction, but I cannot figure out where this is or if/how this can be configured.</p>
<p>Here is my model definition:</p>
<pre><code>class TimeSeriesNNModel(nn.Module):
    def __init__(self):
        super(TimeSeriesNNModel, self).__init__()
        self.lstm1 = nn.LSTM(input_size=14, hidden_size=50, num_layers=1)
        self.lstm2 = nn.LSTM(input_size=50, hidden_size=25, num_layers=1)
        self.linear = nn.Linear(in_features=25, out_features=1)

        self.h_t1 = None
        self.c_t1 = None
        self.h_t2 = None
        self.c_t2 = None

    def initialize_model(self):
        self.h_t1 = torch.rand(1, 1, 50, dtype=torch.double)
        self.c_t1 = torch.rand(1, 1, 50, dtype=torch.double)
        self.h_t2 = torch.rand(1, 1, 25, dtype=torch.double)
        self.c_t2 = torch.rand(1, 1, 25, dtype=torch.double)

    def forward(self, input_data, future=0):
        outputs = []
        self.initialize_model()

        output = None
        for i, input_t in enumerate(input_data.chunk(input_data.size(1), dim=1)):
            self.h_t1, self.c_t1 = self.lstm1(input_t, (self.h_t1, self.c_t1))
            self.h_t2, self.c_t2 = self.lstm2(self.h_t1, (self.h_t2, self.c_t2))
            output = self.linear(self.h_t2)
            outputs += [output]

        outputs = torch.stack(outputs, 1).squeeze(2)
        return outputs
</code></pre>
<p>If anyone can point out what is wrong with my model or my understanding, I’d be really grateful.</p>

Yibin_Sheng: <p>RNN do give the same result once its parameters is given.<br>
I ran into the same problem yesterday.</p>
<p>The seemingly random behavior may be caused by:</p>
<ol>
<li>
<p>dropout: When training, dropout layer randomly choose some units to block, which may cause random behavior. see <a href="https://pytorch.org/docs/stable/nn.html#dropout-layers" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#dropout-layers</a> for more information. In the predicting phase, you should set model.training to false by model.eval() to distribute your weight over all layer units.</p>
</li>
<li>
<p>random initialization: You should not only reconstruct your network, but also load your parameters.(My case)</p>
</li>
</ol>

Abhilash_Srivastava: <p>This might be useful: <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/randomness.html</a><br>
It talks about reproducibility.</p>