akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Linspace auto detect end

username: onesmart
cooked: <p>Is there an existing pytorch function to generate <strong>n</strong> equally spaced items starting from <strong>x</strong> with interval of <strong>k</strong> without having to specify the end?</p>
<p>Eg.<br>
n = 5, x = 1, k = 0.1 -&gt; [1.0, 1.1, 1.2, 1.3, 1.4]<br>
n = 5, x = 1, k = 2 -&gt; [1, 3, 5, 7, 9]</p>
<p>So the end is determined by the interval (<strong>k</strong>) and number of items (<strong>n</strong>)</p>

username: alexgo
cooked: <p>Is it not good enough to calculate the end from n, x and k and using the available function?</p>
<p><code>end = k*(n-1) + x</code></p>

username: onesmart
cooked: <p>That’s okay. I only wanted to avoid extra calculations as much as possible, no matter how trivial it seems to calculate. But I guess for now, I cannot avoid it. Thanks</p>
Scalar matrix multiplication for a tensor and an array of scalars

username: NadiaMe
cooked: <p>If I have a 3d tensor of size<br>
<code>T1 --&gt; torch.Size([196, 14, 14])</code><br>
and a 1d tensor of size<br>
<code>T2 --&gt; torch.Size([196])</code></p>
<p>How do I multiply matrices of size<code> [14, 14]</code> in <code>T1</code> by the corresponding scalar in <code>T2</code>?</p>
<p>e.g if <code>T2[0] == 3,</code> then the first multiplication would be<br>
<code>T1[0] * 3</code> , etc.</p>
<p>So the output should be with the original shape of <code>T1</code>.</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>You just need to change the view of second tensor to match shapes in corresponding dimension, then PyTorch’s tensor <em>broadcasting</em> will take care of that. e.g.</p>
<pre><code class="lang-auto">t1 = torch.arange(0, 196*14*14).view((196, 14, 14))  # shape (196, 14, 14)
t2 = torch.arange(196).view(-1, 1, 1)  # shape (196, 1, 1)
result = t1*t2
</code></pre>
<p>Bests</p>
2d output of torch.max instead of 1d array

username: NadiaMe
cooked: <p>I have the following tensor <code>T=[100, 10, 10]</code><br>
I want to find max for each of the <code>100</code> matrices and get output <code>[100]</code>.</p>
<p>When I run<br>
<code>out = torch.max(T, dim=0)[0]</code></p>
<p>I get 2d output of size <code>[10, 10]</code>.<br>
What does it mean and why I dont get 1d output of size 100 ?</p>
<p>Thanks!</p>

username: KFrank
cooked: <p>Hi Nadia!</p>
<aside class="quote no-group" data-username="NadiaMe" data-post="1" data-topic="100163" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nadiame/48/8864_2.png" class="avatar"> NadiaMe:</div>
<blockquote>
<p>I have the following tensor <code>T=[100, 10, 10]</code><br>
I want to find max for each of the <code>100</code> matrices and get output <code>[100]</code>.</p>
</blockquote>
</aside>
<p><code>T.max (dim = -1)[0].max (dim = -1)[0]</code> should work, or you can<br>
use the equivalent <code>torch.max()</code> version, if you prefer.  (This could also<br>
be written <code>T.max (dim = 2)[0].max (dim = 1)[0]</code>.)</p>
<p>(If you want to be courageous, you could try the unstable, nightly build,<br>
e.g., version “1.8.0.dev20201021,” and run <code>T.amax (dim = (1, 2))</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>When I run<br>
<code>out = torch.max(T, dim=0)[0]</code></p>
<p>I get 2d output of size <code>[10, 10]</code>.<br>
What does it mean and why I dont get 1d output of size 100 ?</p>
</blockquote>
</aside>
<p><code>dim = 0</code> doesn’t mean “keep dim 0.”  It means take the <code>max()</code> over<br>
dim 0.  So it builds tensor of shape <code>[10, 10]</code>, each element of which<br>
is the <code>max()</code> of 100 values, namely the <code>max()</code> of the corresponding<br>
elements of the 100 shape-<code>[10, 10]</code> slices of your original tensor.</p>
<p>Best.</p>
<p>K. Frank</p>
Pruning torch.nn.modules.activation.multihead

username: rodasoares
cooked: <p>Hello,</p>
<p>I am trying to prune torch.nn.modules.activation.multihead, however the “in_proj_weight” and “q_proj_weight”, “k_proj_weight” and “v_proj_weight” matrices are not of the type Module but are of the type Parameter.</p>
<p>As such they don’t appear when using:<br>
modules = model.named_modules()</p>
<p>And as far as I know, one can only prune modules.</p>
<p>Do I have to create a new Multi-Head implementation where I use a Key, Query and Value that are modules?<br>
Or is there a workaround?</p>
<p>Thanks,<br>
Rodrigo</p>

username: rodasoares
cooked: <p>I can use ‘in_proj_weight’ etc, instead of ‘weight’.</p>
Dqn.update() fails: Found dtype Double but expected Float

username: notreadyyet
cooked: <p>Similar error has been posted in several other topics, however I don’t see how recommendations from other articles apply to my case. In short, I was following <a href="https://github.com/iffiX/machin/blob/master/examples/tutorials/your_first_program/main.py" rel="noopener nofollow ugc">this example</a> and tried to use my own environment class. I’d appreciate if someone would point me to a mistake I am making. Please find source t002.py, requirements.txt and a csv file with data on this GitHub <a href="https://github.com/notreadyyet/pytorch_dqn" rel="noopener nofollow ugc">project</a>. Thank you for your time.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You might want to double check your code to make sure that all the Tensors and models are of the same dtype. In this case it looks like you expect float everywhere.<br>
You should be especially careful if you create Tensors from numpy arrays as they are in double precision by default and you will have to explicitly change them to float.</p>

username: notreadyyet
cooked: <p>For anyone interested, explicit casting of reward, returned by environment’s step function, to float32, allowed dqn.update function to succeed.</p>
Tensor values fading off through CNN Encoder

username: dllacer
cooked: <p>Hi everyone,</p>
<p>I’m working on an Autoencoder. My problem is that the output of the Decoder is always flat (all pixels are the same value) and does not reconstruct the image well.</p>
<p>I’ve explored the output of each layer and I’ve noticed that the tensor values become zero as the image goes through the Encoder layers.</p>
<p>Do somebody have any idea of why is this happening?</p>
<p>Thanks!</p>
How to modify loss value after forward pass (but before backprop.)?

username: Anshul_Thakur
cooked: <p>Is it possible to alter the magnitude of loss after forward pass? I tried using loss.data to assign a new value. But the gradients are still being computed on the old value.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The gradients do not depend on the actual value of the loss in most cases. So that’s why you don’t see any change when you change it.<br>
Note that I would advise again using <code>.data</code> as it will most likely lead to wrong gradients being computed!</p>
How to create a tensor of masks where only one entry is 1

username: NadiaMe
cooked: <p>Say we have a tensor T of size <code>[s, s]</code>.<br>
How do I create a mask tensor of size <code>[s*s, s, s]</code> where for each tensor only 1 entry is equal to 1.<br>
E.g for <code>s = 3</code> mask tensor would look like</p>
<pre><code class="lang-auto">[
[[1, 0, 0], [0,0,0], [0, 0, 0]],
[[0, 1, 0], [0,0,0], [0, 0, 0]],
[[0, 0, 1], [0,0,0], [0, 0, 0]],
...
[[0, 0, 0], [0,0,0], [0, 0, 1]]
]
</code></pre>
<p>Thanks!</p>

username: pchandrasekaran
cooked: <pre><code class="lang-auto">mask_setup = torch.ones(s, s) #Shape -&gt; [s, s]

mask = torch.diag(mask_setup) #Shape -&gt; [s*s, s*s]

mask = mask.reshape(s*s, s, s) #Shape -&gt; [s*s, s, s]
</code></pre>
<p>This should give you what you need. There may be a simpler way to get it done though.</p>

username: NadiaMe
cooked: <aside class="quote no-group" data-username="pchandrasekaran" data-post="2" data-topic="100152">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pchandrasekaran/48/29605_2.png" class="avatar"> pchandrasekaran:</div>
<blockquote>
<p><code>mask = mask.reshape(s*s, s, s)</code></p>
</blockquote>
</aside>
<p>So in my case s = 14</p>
<pre><code class="lang-auto">mask = mask.reshape(s * s, s, s)
RuntimeError: shape '[196, 14, 14]' is invalid for input of size 14
</code></pre>
<pre><code class="lang-auto">
mask = torch.diag(mask_setup)
</code></pre>
<p>– &gt; torch.Size([14])</p>
<p>so it does not work, but maybe you can explain me your idea?</p>

username: pchandrasekaran
cooked: <p>I apologize. The first line is wrong.</p>
<pre><code class="lang-auto">mask_setup = torch.ones(s*s,) #Shape -&gt; [s*s,]

mask = torch.diag(mask_setup) #Shape -&gt; [s*s, s*s]

mask = mask.reshape(s*s, s, s) #Shape -&gt; [s*s, s, s]
</code></pre>
<p>The idea is to create a diagonal matrix of size (s * s, s * s) and then reshape it (s*s, s, s).</p>
Need help with GPU memory issues

username: PedsB
cooked: <p>I have a loss function that requires multiple internal passes:</p>
<pre><code class="lang-auto">def my_loss_func(logits, sigma, labels, num_passes):
    total_loss = 0
    img_batch_size = logits.shape[0]
    logits_shape = list(logits.shape)
    vol_std = np.zeros((img_batch_size, num_passes))
    for fpass in range(num_passes):
        noise_array = torch.normal(mean=0.0, std=1.0, size=logits_shape, device=torch.device('cuda:0'))
        stochastic_output = logits + sigma * noise_array
        del noise_array
        temp_vol = torch.softmax(stochastic_output, dim=1)
        temp_vol = temp_vol[:, 0, ...]
        vol_std[:, fpass] = temp_vol.view(4, -1).sum(1).detach().cpu().numpy()
        del temp_vol
        exponent_B = torch.log(torch.sum(torch.exp(stochastic_output), dim=-1, keepdim=True))
        inner_logits = exponent_B - stochastic_output
        soft_inner_logits = labels * inner_logits
        total_loss += torch.exp(soft_inner_logits)
        del exponent_B, inner_logits, soft_inner_logits
    mean_loss = total_loss / num_passes
    actual_loss = torch.mean(torch.log(mean_loss))
    batch_std = np.std(vol_std, axis=1)
    return actual_loss, batch_std
</code></pre>
<p>Both logits and sigma are networks outputs and therefore have associated gradients. I run into memory issues when num_passes exceeds 50, are there any other ways in which I could fully optimise memory allocation to allow for a greater number of passes? I’m not at all concerned with readability/ ugly solutions, anything will do.</p>

username: YASJAY
cooked: <p><a class="mention" href="/u/pedsb">@PedsB</a>, I am not exactly sure if this is the most efficient way to do the multiple internal passes but firstly have you checked for other running processes that are taking up the GPU memory?</p>
<p>Maybe, kill 'em all: <code>kill -9 $(nvidia-smi | sed -n 's/|\s*[0-9]*\s*\([0-9]*\)\s*.*/\1/p' | sort | uniq | sed '/^$/d')</code></p>
<p>Secondly, the problem could be at this line of code: <code>vol_std = np.zeros((img_batch_size, num_passes))</code></p>
<p>What’s your batch_size?</p>
<p>Maybe, use a smaller batch size.</p>
<p>Thanks</p>

username: PedsB
cooked: <p>Hi <a class="mention" href="/u/yasjay">@YASJAY</a>, I should’ve clarified:  I’m running this job on a GPU cluster so I’m confident memory is completely free prior to job submission. My batch size cannot be changed at all, but it is fairly small, four.</p>
<p>My issue is that I quickly run out of memory, likely because of the total += torch.exp(soft_inner_logits) step; I believe that keeps piling on computation graphs on top of each other, using up a lot of memory. I was more looking for ways to re-write what I’ve done in a more memory-efficient way: E.g.: backpropagating in the loop to free up intermediary memory (Which unfortunately I can’t do in this case since the loss cannot be calculated until I’ve gone through N passes).</p>
Create masking tensor using lower and upper boundary tensors

username: billiout
cooked: <p>I want to create a masking tensor of shape [N, K] that contains ones in between different ranges of K for each N and zeros outside. To create this, I have two tensors with shape [N, 1] that contain the lower and upper index for each example that I should give “one”. In other words:</p>
<p>Assuming N=3 and K=5, as well as the lower limit tensor is [[2], [0], [4]] and the upper limit tensor is [[3], [2], [4]] then, the masking tensor should look like this:</p>
<p>[<br>
[0, 0, 1, 1, 0],<br>
[1, 1, 1, 0, 0],<br>
[0, 0, 0, 0, 1]<br>
]</p>
<p>Is there a way of creating this masking tensor with the provided Pytorch functions?</p>

username: phan_phan
cooked: <p>Hi,<br>
You can use  <code>torch.ge</code> (greater or equal) and  <code>torch.le</code> (lower or equal) with :</p>
<pre><code class="lang-auto">lower = torch.LongTensor([2, 0, 4]).unsqueeze(dim=-1)
upper = torch.LongTensor([3, 2, 4]).unsqueeze(dim=-1)

idx = torch.arange(5)
mask = idx.ge(lower) &amp; idx.le(upper)
</code></pre>
<p>It will give you a tensor with <code>True</code> and <code>False</code><br>
Use <code>mask.long()</code> if you need ones and zeros</p>
RGB to Greyscale mean and standard deviation

username: snowe
cooked: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I am working on a project where I want to compare the performance of CNNs on RGB images and the converted greyscale images. In PyTorch there is a handy transform <code>torchvision.transforms.Grayscale(num_output_channels=1)</code> to convert an RGB image into its greyscale version.<br>
In the source code I found that they use the luma transform to do so: L = R * 299/1000 + G * 587/1000 + B * 114/1000. Essentially, this converts an RGB image into the Y plane of a YCbCr image (if I am not mistaken).<br>
My question: do the mean and the standard deviation of a dataset of converted greyscale images change compared to the same dataset with RGB images? The mean should be the same right, since grey just means that R=G=B. What about the standard deviation?</p>
<p>Any help is very much appreciated!</p>
<p>All the best<br>
snowe</p>

username: RaLo4
cooked: <p>You might want to check out <a href="https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560/6">this</a> or <a href="https://discuss.pytorch.org/t/computing-the-mean-and-std-of-dataset/34949/2">this </a> post.</p>

username: snowe
cooked: <p>Thank you <a class="mention" href="/u/ralo4">@RaLo4</a>! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I just tried it out and the mean and the std for both RGB and greyscale datasets seem to be the same with a difference of maybe 0.01 - 0.02.</p>
<p>I am just curious… from a theoretical perspective, does that make sense?</p>
<p>All the best<br>
snowe</p>
Trying to load model and switch to eval

username: rgodbey
cooked: <p>In my script the model successfully trains and I save it at the end. When I try to load and evaluate it this is what happens:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File ".\GRU.py", line 139, in &lt;module&gt;
    gru_outputs, targets, gru_sMAPE = evaluate(gmodel, X_test, Y_test, label_scalars)
  File ".\GRU.py", line 115, in evaluate
    model.eval()
AttributeError: 'NoneType' object has no attribute 'eval'
</code></pre>
<p>This is the last portion of the code:</p>
<pre><code class="lang-auto">def evaluate(model, X_test, Y_test, label_scalars):
    model.eval()
    outputs = []
    targets = []
    start_time = time.clock()
    for i in X_test.keys():
        inp = torch.from_numpy(np.array(X_test[i]))
        labs = torch.from_numpy(np.array(Y_test[i]))
        h = model.init_hidden(inp.shape[0])
        out, h = model(inp.to(device).float(), h)
        outputs.append(label_scalars[i].inverse_transform(out.cpu().detach().numpy()).reshape(-1))
        targets.append(label_scalars[i].inverse_transform(labs.numpy()).reshape(-1))
    print("Evaluation Time: {}".format(str(time.clock()-start_time)))
    sMAPE = 0
    for i in range(len(outputs)):
        sMAPE += np.mean(abs(outputs[i]-targets[i])/(targets[i]+outputs[i])/2)/len(outputs)
    print("sMAPE: {}%".format(sMAPE*100))
    return outputs, targets, sMAPE

lr = 0.002

#Gru_model = train(train_loader, lr)

gmodel = torch.load('./grunet.pkl')

gru_outputs, targets, gru_sMAPE = evaluate(gmodel, X_test, Y_test, label_scalars)
</code></pre>

username: ptrblck
cooked: <p>What does <code>print(gmodel)</code> return after the <code>torch.load</code> call?<br>
Based on the error it seems it’s <code>None</code>, which indicates the loading fails somehow.</p>
Why does Faster R-CNN ResNet-50 FPN have 91 classes insteand of 80?

username: bwang1991
cooked: <p>As far as I know, COCO only provides annotations for 80 classes. My question is how was Faster R-CNN ResNet-50 FPN form torchvision trained with 91 classes?</p>

username: RaLo4
cooked: <p>I am guessing this is due to the 91 stuff categories the COCO dataset has.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b06af17a2d264754a5568597185941d6db8b5ba.png" alt="Screenshot from 2020-10-21 11-26-21" data-base62-sha1="8qaurE0rcoGM06CaiE13ohoRlOi" width="267" height="378"><br>
<em>Screenshot from the COCO website</em></p>
<p>Not 100% sure tho…</p>
How to break a single large input among different GPUs for using DDP?

username: vr25
cooked: <p>Hi,</p>
<p>I am using a text dataset which has some really long documents and so fitting one such input on a single machine runs into OOM error.</p>
<p>Any suggestions on how to break the input across multiple GPUs and also efficiently do the backward pass?</p>
<p>Thanks!</p>

username: a_d
cooked: <p>I suppose the model and the input should be on the same device. If you have multiple GPUs, one thing you could do is use the <code>mode parallel </code> in pytorch and distribute your model layers across the multiple GPUs and keep only a few or even one layer of the model on the GPU which has your input.<br>
Might not be what you wanted but may help. The <code>mode parallel</code> tutorial is here -<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" target="_blank" rel="noopener nofollow ugc">Single-Machine Model Parallel Best Practices — PyTorch Tutorials 1.6.0...</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: vr25
cooked: <p>Thank you for your reply. I tried that. I am using multiple BERT so breaking it down across several GPUs is leading to more issues.</p>

username: Abhilash_Srivastava
cooked: <p>Try using <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="noopener nofollow ugc">Data Parallelism</a>. It’s easy to use if you have access to multiple GPUs. Make sure that the batches are still small enough to fit on a single machine.</p>
Why does PyTorch over fit after the first experiment

username: Meriem
cooked: <p>I can’t help but notice that PyTorch models are most likely to over fit after the first experiment. For example, when using k-fold cross validation, the first fold learns and tests normally (loss decreases slowly until it gets saturated) but the second, third, … folds already start with a low loss value, get to the best performance in the very first epochs, and then the model starts to over fit. Is there any explanation for this behaviour?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I guess you might not be resetting the experiment setup properly, i.e. the <code>model</code>, <code>optimizer</code>, <code>lr_scheduler</code> or any other object which “learns” during the first fold.</p>
When using multi-GPU training, torch.nn.DataParallel stuck in the model input part

username: WangZL
cooked: <p>During model training, due to GPU memory overflow, only pytorch’s nn.DataParallel was used for setting, and a simple program was written to test. The result is feasible, but it is stuck in the model input part during real model training,that is “self.model(inputs)”.No error message, just stuck. Can you help me ? Thanks. The code is as follows:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/195cd7a3130c028e848cd39f341afdc6e4161a7c.png" data-download-href="https://discuss.pytorch.org/uploads/default/195cd7a3130c028e848cd39f341afdc6e4161a7c" title="{OUH7G@%6J7UB`GP_9)AA"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/195cd7a3130c028e848cd39f341afdc6e4161a7c.png" alt="{OUH7G@%6J7UB`GP_9)AA" data-base62-sha1="3CmQJtzgXm1F32EaayWl7mzGrfe" width="690" height="286" data-dominant-color="F6F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">{OUH7G@%6J7UB`GP_9)AA</span><span class="informations">1281×532 32.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Which PyTorch version are you using?<br>
If it’s an older one, could you update to the latest stable version (<code>1.5</code>)?</p>
<p>Also, is your model working on a single GPU?</p>

username: mangoxb
cooked: <p>I also have same problem.<br>
After reading <a class="mention" href="/u/ptrblck">@ptrblck</a> 's reply, I updated pytorch as version 1.6.<br>
But still it is not fixed.</p>

username: mangoxb
cooked: <p>I found that it was caused by deadlock of another part.<br>
Now I fixed my problem.<br>
Maybe you need to check other part too.</p>
Image Gradients

username: Hmrishav_Bandyopadhy
cooked: <p>Hi,<br>
I am working on finding gradients of an image using the sobel operator and convolutional filters. Right now, my approach is</p>
<pre><code class="lang-auto">kernel_x = [[1., 0., -1.], [2., 0., -2.], [1., 0., -1.]]
kernel_x = torch.FloatTensor(kernel_x)
kernel_y = [[1., 2., 1.] ,[0. ,0., 0.], [-1., -2., -1.]]
kernel_y = torch.FloatTensor(kernel_y)
kernel_x = kernel_x.unsqueeze(0).unsqueeze(0)
kernel_y = kernel_y.unsqueeze(0).unsqueeze(0)
self.weight_x = nn.Parameter(data=kernel_x, requires_grad=False)
self.weight_y = nn.Parameter(data=kernel_y, requires_grad=False)
grad_x = F.conv2d(x, self.weight_x)
grad_y = F.conv2d(x, self.weight_y)
</code></pre>
<p>The main reason i am confused is that the weights are transposed during convolutions–and i want to calculate the x and y gradients specifically. Is this approach right?–like would grad_x calculate the x-directional gradients and grad_y the y-directional?</p>
<p>Input Format : [1,1,64,64]</p>
How to make nGram in pytorch

username: Bahaa_Kattan
cooked: <p>I want to train dnn using bigram but I failed to create a way that can run using torch text or data loader.</p>
<p>Is there any way to make bigram that can be engaged with torch text and its classes like tabular data etc…</p>

username: Bahaa_Kattan
cooked: <p>I reatched to this result</p>
<pre><code class="lang-auto"> ["[('email', 'yeah'), ('yeah', 'fast'), ('fast', 'extrem'), ('extrem', 'high'), ('high', 'damag'), ('damag', 'want'), ('want', 'fast'), ('fast', 'im'), ('im', 'gonna'), ('gonna', 'getyou')]"
  1]]
</code></pre>
<p>but I don’t know how to train a pytorch network on this data, like how to numericalize it without accepting  <code>[]</code> and <code>()</code> as string  .<br>
any idea that can fix it ?</p>

username: Abhilash_Srivastava
cooked: <p>Did you try using?<br>
<code>from nltk import bigrams</code></p>

username: Bahaa_Kattan
cooked: <p>yes I tried, but it will be hard to work with torchtext because it will not know how to pad sequance   or build vocabs etc…</p>
<p>is there any other way ?</p>
<p>if you know a method that work I will be so pleasure to here.</p>
Memory management for variables stored during forward pass

username: Sushma_Rao
cooked: <p>Is there a difference in the way memory is allocated/freed using <code>ctx.save_for_backward(x)</code> vs <code>ctx.x = x</code></p>

username: YASJAY
cooked: <p>Hi <a class="mention" href="/u/sushma_rao">@Sushma_Rao</a>, <a href="https://discuss.pytorch.org/t/when-should-you-save-for-backward-vs-storing-in-ctx/6522/8">this</a> thread covers your question.</p>
<p>It specifies ctx.x = x could lead to memory leak, and GPU memory exhausion.</p>
Saving warmed up torchscript models directly

username: Prashant
cooked: <p>Heya I’ve been doing some googling and it looks like the first execution of a torchscript model includes a heavy warmup period. In my case, its almost 5 minutes.</p>
<p>This is pretty painful and I’m wondering if I can directly save the optimized model in the first place so I don’t have to go through this warmup period each time.</p>

username: eellison
cooked: <p>Hi, could you try running it on master or the nightly build? For the 1.7 release compilation times were improved.</p>

username: Michael_Suo
cooked: <p>Sorry to hear that. Unfortunately, we can’t save the warmed-up model today due to a number of optimizations relying on in-memory data structures that can’t be serialized. We have some longer-term infrastructural work planned to improve compile times but it hasn’t landed yet.</p>
<p>Two things that would help us improve the situation:</p>
<ol>
<li>We recently made a number of improvements to compilation and optimization time, can you try the nightly or the 1.7 RC and see if you get improvements (try running the model a few times).</li>
<li>If possible, can you provide the <code>.pt</code> file for the model that is taking a really long time to warm up, as well as some example inputs? It will help us understand what sort of models are causing long compilation times.</li>
</ol>

username: Prashant
cooked: <p>Thanks to both you and eellison for the replies.</p>
<p>Using 1.7 seems to have done the trick! Compile times went from 200s -&gt; 2s. When do you think it will make it to stable?</p>

username: eellison
cooked: <p>The 1.7 release is coming soon,  I can’t give you an exact date. Stay tuned…</p>
Types of ResNet

username: Kaneda
cooked: <p>I got a quick question about ResNet, i don’t really get the idea behind how to define that we want 20 stacked, or 32 stacked layers for example</p>
<pre><code class="lang-auto">def resnet20():
    return ResNet(BasicBlock, [3, 3, 3])


def resnet32():
    return ResNet(BasicBlock, [5, 5, 5])
</code></pre>
<p>why [3,3,3] for resnet20 and [5,5,5] for resnet32, exactly what those numbers do?</p>
<p><em>code <a href="https://github.com/akamaster/pytorch_resnet_cifar10/blob/master/resnet.py" rel="noopener nofollow ugc">https://github.com/akamaster/pytorch_resnet_cifar10/blob/master/resnet.py</a></em></p>

username: YASJAY
cooked: <p>Hi <a class="mention" href="/u/kaneda">@Kaneda</a>, from the code it looks like the [3,3,3] or the [5,5,5] is an argument “num_blocks” to the function _make_layer which is being used to decide strides for different layers.</p>
<blockquote>
<p>def _make_layer(self, block, planes, num_blocks, stride):<br>
strides = [stride] + [1]*(num_blocks-1)<br>
layers = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for stride in strides:<br>
layers.append(block(self.in_planes, planes, stride))<br>
self.in_planes = planes * block.expansion</p>
<pre><code>    return nn.Sequential(*layers)
</code></pre>
</blockquote>
PyTorch quantization resnet50 model

username: samhithaaaa
cooked: <p>Hi all,<br>
I am trying the resnet50 model quantization with PyTorch and I tried these 3 lines of code :<br>
the import, model=qn.resnet50(pretrain=true), and model.state_dict()), and why the coefficients being shown are all float values if this is the quarantined version of the model?<br>
Noticed this while trying to figure out how to save/load the coefficients for a quantized model, and is there anything special you need to do convert model coefficients between float and int8.</p>
<p>Please let me know. Appreciate any help/suggestions , Thanks!</p>

username: samhithaaaa
cooked: <p>Can anyone help me with this please?</p>

username: Vasiliy_Kuznetsov
cooked: <p>check out <a href="https://pytorch.org/docs/stable/quantization.html#quantized-torch-tensor-operations" rel="nofollow noopener">https://pytorch.org/docs/stable/quantization.html#quantized-torch-tensor-operations</a>, in partucular the “int_repr” function.  By default, if you print out a quantized tensor you will see the dequantized values that the tensor represents.  To see the raw int8 values, you can use <code>x.int_repr()</code>.</p>

username: samhithaaaa
cooked: <p>Thank you so much for your response,I will check it.Thanks again.</p>

username: samhithaaaa
cooked: <p>HI,I tried this but it gives me this error:<br>
RuntimeError: Could not run ‘aten::int_repr’ with arguments from the ‘CPUTensorId’ backend. ‘aten::int_repr’ is only available for these backends: [QuantizedCPUTensorId, VariableTensorId].</p>
2D to 3D Network

username: banikr
cooked: <p>Hello,<br>
I have used a <strong>2D GAN</strong> code to synthesize CT images from MR.<br>
Since both MR CT comes in 3D, I had to slice the images into 2D images and feed into the network.<br>
Now I want to try the same synthesis project in 3D. I want to use the same GAN algorithm.<br>
How do I shift the <strong>same project from 2D to 3D</strong>?<br>
What should I be concerned about? How loss, optimization, forward, backward change if data dimension increases?</p>

username: CS.Enthu
cooked: <p>Hello,<br>
Can you let me know how did you extracted the slices from the 3d image.I am working on same thing.<br>
Or can you share any reference .<br>
Currently I have my images as 3D , and I am stuck at trying to slice them</p>

username: banikr
cooked: <p>I just extracted them axially. Suppose if 3D image dimension is 172x220x156. I created 156 2D images with 156x220 dimension and saved them in .tif format.<br>
It should be pretty straightforward and can be done using Matlab or PIL python library.<br>
Let me know how it goes for you.</p>

username: CS.Enthu
cooked: <p>Thank you, I was able to do that for 2D network. Have you also tried to create 3D patches.Since I want to create 3D patches for my CT and PET images</p>

username: CS.Enthu
cooked: <p>Hello Banik,</p>
<p>Were you successful in converting your 2D project to 3D? As I am also trying to implement the same. But stuck with some errors.</p>

username: banikr
cooked: <p>Hi Enthu,<br>
I had converted the 2D synthesis into 3D. If I remember correctly I was having issues in reconstruction from patches.</p>
How to know if your evaluation metrics for your deep learning model is 'good' to proceed for a flask application?

username: Aniruddh
cooked: <p>Hi there! I am currently working on a skin cancer classification problem. The data consists of 9 target values and the model is trained with densenet for 20 epochs. The model file has been saved and the evaluation metrics are observed. But I want to have confidence on what I am seeing and proceed in creating a flask application and go straight to testing. The highest training accuracy is 97% and validation accuracy is 87%. Does that mean my model is evaluating well? This is the value counts of each label after sampling weights.</p>
<pre><code class="lang-auto">AK          8503
VASC        8496
unknown     8493
nevus       8450
melanoma    8410
DF          8400
SCC         8245
BCC         8208
BKL         7443
</code></pre>
<p>This is the best record of my model file.</p>
<pre><code class="lang-auto">best record: [epoch 11], [val loss 0.40036], [val acc 0.87697]
</code></pre>
<p>Here is the confusion matrix, the classification report</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3becc0b663f419404716e57389879a358278d4a5.png" alt="image" data-base62-sha1="8y7puRxuHSLtySOLohzjUbw7ABT" width="349" height="294"></p>
<pre><code class="lang-auto">              precision    recall  f1-score   support

          AK       0.66      0.66      0.66       204
         BCC       0.79      0.91      0.85       700
         BKL       0.67      0.69      0.68       657
          DF       0.66      0.75      0.70        56
       nevus       0.68      0.73      0.70       126
        VASC       0.90      0.86      0.88        63
    melanoma       0.62      0.69      0.65      1067
         SCC       0.92      0.85      0.88      4497
     unknown       0.95      0.96      0.95      5425

    accuracy                           0.87     12795
   macro avg       0.76      0.79      0.77     12795
weighted avg       0.88      0.87      0.87     12795
</code></pre>
<p>Notice in the confusion matrix, the SCC label is incorrectly predicted as melanoma 291 times! This made me take a step back and question my model again as it was a little freaky.<br>
I would really appreciate anyone who can give me a general observation on the model. This would help me make any changes for better results! Thanks a lot for the help.</p>
KL divergence between Uniform and MultivariateNormal

username: Adam_Conkey
cooked: <p>There is a KL divergence function registered for <code>KL(Uniform || Normal)</code>, but not for <code>KL(Uniform || MultivariateNormal)</code>. Is there any fundamental reason for this, or it just hasn’t been implemented yet?</p>
<p>Both <code>KL(Normal || Uniform)</code> and <code>KL(MultivariateNormal || Uniform)</code> are infinite since Uniform will have bounded support and you’ll divide by zero anywhere outside that region. I’m just trying to understand if there’s anything that should prevent the <code>KL(Uniform || MultivariateNormal)</code> case. I haven’t tried working through the math yet, I also haven’t found any good references on these derivations. Any pointers are appreciated.</p>
The fastest way to save multiple images from torch tensor

username: hanbit
cooked: <p>Hi, I’m trying to save multiple images (number of batch_size) from tensors.</p>
<p>Let’s say, we want to add an adversarial noise on each image.</p>
<p>The most efficient way I can think of is that</p>
<ol>
<li>load images of batch size</li>
<li>calculate adversarial noise and add them --&gt; which makes Tensor([B, C, W, H])</li>
<li>using for loop to save each image from the tensor.</li>
</ol>
<p>Is there a faster or more ‘data-parallel’ way to save images?</p>

username: JuanFMontesinos
cooked: <p>I recommend you to use imageio which is a very fast and nice library together with multiprocessing.</p>

username: albertotono
cooked: <aside class="quote no-group" data-username="hanbit" data-post="1" data-topic="37829">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/858c86/48.png" class="avatar"> hanbit:</div>
<blockquote>
<p>Is there a faster or more ‘data-parallel’ way to save i</p>
</blockquote>
</aside>
<p>I am using matplot pyplot, I am not sure what I am doing wrong. it always print the first image. Any help?</p>
<pre><code class="lang-auto">for t in range(0,batch_size):
    a = image[t].cpu().numpy()
    plt.savefig("./test.png")
    plt.imshow(a)
</code></pre>
Modify nn::AnyModule embedded in nn::Sequential

username: abhinavkulkarni
cooked: <p>Hi,</p>
<p>I have a custom module (let’s call it <code>MyModule</code>) that has conditional logic in its <code>forward</code> method. This custom module is part of an <code>nn::Sequential</code>.</p>
<p>I want to be able to do something along the following lines:</p>
<pre><code class="lang-cpp">
auto flag = true;
auto module = MyModule(flag);
auto sequential = nn::Sequential(
    module
);

auto x = torch::rand({T, inChannels});
auto y = sequential-&gt;forward(x);


sequential-&gt;modules()[0]-&gt;as&lt;MyModule&gt;-&gt;flag = false;

x = torch::rand({T, inChannels});
y = sequential-&gt;forward(x);


</code></pre>
<p>How do I go about this? The line</p>
<pre><code class="lang-cpp">sequential-&gt;modules()[0]-&gt;as&lt;MyModule&gt;-&gt;flag = false;
</code></pre>
<p>doesn’t really set the <code>flag</code> (I think it only returns a temporary rvalue and changes it’s flag, but not of the underlying module).</p>
<p>Thanks.</p>

username: abhinavkulkarni
cooked: <p>Thanks, I am able to modify the underlying <code>MyModule</code> and my comment about <code>rvalue</code> is incorrect. I’m closing this issue.</p>
Conditional Image segmentation (organ segmentation with body part regressor)

username: stadtgiraffe
cooked: <p>Hello there,<br>
I am a graduate student working on an conditional multi-organ segmentation model!</p>
<p>I have finished writing my code to train and validate segmentation architectures (like U-Net, PIPO-Net, DFCN). Currently I am trying to improve my multi-segmentation net with a Body Part Regressor (BPR - much like this one <a href="https://arxiv.org/pdf/1707.03891.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1707.03891.pdf</a>) which maps a CT image-slice to its corresponding location in the body, i.e. outputs a score which represents the height in the body where the input slice is positioned.<br>
So my BPR can tell me what body part/area my volume (or sub-volume) shows. This means I could use partial evidence for my segmentation net, much like telling the net: “don´t look for liver or kidney while segmentating, since the BPR says the volume is only in the upper lung area”. But I do not want to set those partial labels manually.</p>
<p>Likewise, I could maybe combine the BPR and the segmentation net while training, making this partial evidence a learnable parameter.</p>
<p>Does somebody have an idea on how to combine the segmentation masks and the BPR scores or knows how to make a single architecture of those two?</p>
<p>Thanks for the ideas!</p>