akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Do anyone knows where is this model?

username: blackbirdbarber
cooked: <p>            </p><aside class="onebox reddit">
              <header class="source">
                <img src="https://www.redditstatic.com/desktop2x/img/favicon/android-icon-192x192.png" class="site-icon" width="16" height="16">
                <a href="https://www.reddit.com/r/programming/comments/gnkiej/openai_model_generates_python_code/" target="_blank" rel="nofollow noopener">reddit</a>
              </header>
              <article class="onebox-body">
                <h3><a href="https://www.reddit.com/r/programming/comments/gnkiej/openai_model_generates_python_code/" target="_blank" rel="nofollow noopener">r/programming - OpenAI Model Generates Python Code</a></h3>
                <div class="aspect-image-full-size">
                  <a href="https://www.reddit.com/r/programming/comments/gnkiej/openai_model_generates_python_code/" target="_blank" rel="nofollow noopener">
                    <img src="https://external-preview.redd.it/d5KMSebeOcjHn9oVf1bRkMVwKv_ZsbSJK5MGn1Tk1qg.jpg?auto=webp&amp;s=ac22926a57055bcedb9b52b23826a31482d05624" class="scale-image" width="520" height="390">
                    <span class="instagram-video-icon"></span>
                  </a>
                </div>
                <div class="description"><p>339 votes and 78 comments so far on Reddit</p></div>
              </article>
            </aside>

<p>The video  explains OpenAi generates Python code. So far I haven’t fond this model. Did anyone found it?</p>

username: blackbirdbarber
cooked: <p>I haven’t found this model still, since OpenAI is closed still, but i found this transcoder stuff:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="15" height="15">
      <a href="https://github.com/facebookresearch/TransCoder" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/16943930?s=400&amp;v=4" class="thumbnail onebox-avatar" width="59" height="59">

<h3><a href="https://github.com/facebookresearch/TransCoder" target="_blank" rel="nofollow noopener">facebookresearch/TransCoder</a></h3>

<p>Public release of the TransCoder research project https://arxiv.org/pdf/2006.03511.pdf - facebookresearch/TransCoder</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Bravo for FB.</p>
Custom Loss Function to maximise the difference

username: saumya0303
cooked: <p>Hi team,<br>
I want to define a loss function as to maximise a-b where b’s value is fixed in the starting of the function while a’s value keep changing at each iteration. Please help me to define a correct loss function so as to maximise a-b.Currently I have defined L as a loss function.Is this correct to represent the above condition.<br>
<strong>L = max(0, a-b)</strong><br>
where a = torch.tensor(3.04)<br>
and b =torch.tensor(7.698) at first iteration.<br>
How to represent this L ?Thanks in advance. Please help me to resolve this issue.</p>
Preprocessing used for ImageNet-pretrained models

username: ks7g2h3
cooked: <p>What transforms (random crops, flips, etc.) were applied to the training data for the standard imagenet-pretrained models (vgg-16, alexnet, etc.) available through the model zoo?</p>

username: RylanSchaeffer
cooked: <p>Did you ever find the answer to this?</p>

username: Kale-ab_Tessera
cooked: <p>I believe this:</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(
        datasets.ImageFolder(traindir, transforms.Compose([
            transforms.RandomSizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            normalize,
        ])),
        batch_size=args.batch_size, shuffle=True,
        num_workers=args.workers, pin_memory=True)

    val_loader = torch.utils.data.DataLoader(
        datasets.ImageFolder(valdir, transforms.Compose([
            transforms.Scale(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            normalize,
        ])),
        batch_size=args.batch_size, shuffle=False,
        num_workers=args.workers, pin_memory=True)
</code></pre>
<p>From <a href="https://github.com/pytorch/examples/blob/42e5b996718797e45c46a25c55b031e6768f8440/imagenet/main.py#L95" rel="nofollow noopener">https://github.com/pytorch/examples/blob/42e5b996718797e45c46a25c55b031e6768f8440/imagenet/main.py#L95</a></p>
Visualize the distribution of Activations

username: Kale-ab_Tessera
cooked: <p>I want to get the distribution of the activations in a MLP that uses ReLU. Any convenient ways to do this?</p>
<p>It can be the activations per layer, or activations of the network as a whole. I know these activations fire per input, so I am not really sure how to approach this.</p>

username: Diego
cooked: <p>Take a look at <a href="https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html#forward-and-backward-function-hooks" rel="nofollow noopener">pytorch hooks</a> I think it is what you are looking for.</p>
FBGEMM & Python Performance

username: sbsky
cooked: <p>So I’m in the process of deploying a torch.jit traced model into an x86 environment. I’d ideally like to do this in Python, but I see the tutorials are in C++. Is it possible to unlock the full performance of the FBGEMM backend from Python, or should I really be using C++?</p>
<p>A 20-30% performance hit is acceptable, but if C++ will be multiple times faster I’ll go with that.</p>
How to load a parameter file saved as zip

username: alan_ayu
cooked: <p>Hello,</p>
<p>I have two version of pytorch. The old one installed on my PC computer, and the new one on Linux server. I didn’t realize the change on the function torch.save, so the parameters trained on server were saved as zip files. When I downloaded them and tried to evaluate on my PC, some running error occurred. Is the torch.save and net.load_state_dict in two versions incompatible? What I should do to use my training results on server?</p>
<p>Thank you !</p>

username: RaLo4
cooked: <p>Hi,<br>
Had the same issue. I don’t know what the correct fix would be, but as a temporary fix you could load the saved .zip files on your server and then save them again with <code>torch.save(obj, f, _use_new_zipfile_serialization=False)</code>.<br>
This will give you the old format.<br>
You can read more about this in the notes <a href="https://pytorch.org/docs/stable/generated/torch.save.html#torch.save" rel="nofollow noopener">here</a></p>

username: alan_ayu
cooked: <p>Thank you very much for the solution <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
Thoughts on discrete valued data for synthetic data hashing project

username: godot
cooked: <p>Hi!</p>
<p>I have been studying various architectures for synthetic data generation specific to Locality Sensitive Hashing (LSH). Basically I want to train a GAN or AAE with LSH hashes and their corresponding LSH label, to see if it would be feasible to use PyTorch for LSH function approximation. The first problem seems to be the discrete vs. continuous nature of the LSH sequences with corresponding label. I’ve looked at SeqGAN as a potential for this, with the idea that once trained the Generator network could then be sampled from according to label.</p>
<p>Is using a GAN specifically purposed for discrete data a requirement for this? I am assuming I need a the least to use a GAN which uses the concept of labels with additional style vectors that are a part of the Generator / Discriminator training.</p>
<p>Any help would be greatly appreciated!</p>
What is "std.data.new(n,z_dim)" and how to write it in C++

username: cj0741
cooked: <p>Hi,<br>
I am trying write the following python code in C++.</p>
<p>“eps = torch.autograd.Variable(std.data.new(n,z_dim).normal_())”</p>
<p>I got stuck on the “std.data.new(n,z_dim)”, I didn’t find it in torch API and python, anyone have idea on it, and how write it in C++?<br>
Many thanks~</p>
Undefined symbol google::base::CheckOpMessageBuilder::NewString when compiling with torch extensions

username: be7f984b2f2e66ba7969
cooked: <p>Hi,</p>
<p>I integrate glog into my python project which using torch extensions.</p>
<p>but when importing the module it always say that it can’t find glog symbol (_ZN6google4base21CheckOpMessageBuilder9NewStringEv)</p>
<p>when using c++filt, the real name is google::base::CheckOpMessageBuilder::NewString()</p>
<pre><code class="lang-auto">import torch
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension

setup(
    name='charmander',
    ext_modules=[
        CUDAExtension('charmander', [
            'src/charmander.cpp',
            'src/cuda_image.cu'
            ],
            libraries=['glog']
        ),
    ],
    cmdclass={
        'build_ext': BuildExtension
    })
</code></pre>
<p>the interesting thing is that when using prebuilt pytorch, the problem is gone. but when using pytorch installed from pip, we have this problem.</p>
<p>any idea?</p>
How to Install NumPy with Python Distribution

username: Aarti_Yadav
cooked: <p>Hello All, Can anyone tell me how to install NumPy matrix with Python distribution? I have already download anaconda python distribution but still, I don’t know whats the next step is? Can anyone suggest me step to step process?</p>

username: mathematics
cooked: <p>I think this question is not relevant here.<br>
But Anyway, you can use Anaconda to download packages all required for data science at once.<br>
<a href="https://www.anaconda.com/products/individual" rel="nofollow noopener">https://www.anaconda.com/products/individual</a></p>
<p>and download and install</p>
<p>You can see lot of tutorials and videos regarding installing Anaconda.</p>

username: Aarti_Yadav
cooked: <p>I have to install anaconda but I am stuck in some places but my mostly requires was resolved with the help of this <a href="https://hackr.io/blog/numpy-matrix-multiplication" rel="nofollow noopener">post</a> and thanks for your reference which is useful to me.</p>
Run two models in parallel on same GPU

username: pb10
cooked: <p>Hi,<br>
I have two neural networks. I wish to run them in parallel on the same gpu using same data. How should I go about it?</p>
<p>model1 =  Net1().cuda()<br>
model2 = Net2().cuda()</p>
<p>out1 = model1(input)<br>
out2 = model2(input)</p>
<p>How can I get out1 and out2 in parallel? Will running them in parallel be faster than the current sequential operations?</p>

username: Kushaj
cooked: <p>I think this is answered <a href="https://discuss.pytorch.org/t/inference-of-two-models-isnt-parallelized-on-same-gpu/81908">here</a>.</p>

username: pb10
cooked: <p>Thank you so much!!!</p>
RuntimeError: expected scalar type Half but found Float

username: pldnxj
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn
print(torch.__version__)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

class Net(nn.Module):
  def __init__(self,):
    super(Net, self).__init__()
    self.rnn = nn.LSTMCell(10,20)

  @torch.cuda.amp.autocast()
  def forward(self, input):
    hx, cx = self.rnn(input)
    return hx

model = Net().to(device)
input = torch.randn(3, 10).to(device)
scaler = torch.cuda.amp.GradScaler()
with torch.cuda.amp.autocast():
  output = model(input)
print(output.size())
</code></pre>
<p>error</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
print(torch.__version__)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

class Net(nn.Module):
  def __init__(self,):
    super(Net, self).__init__()
    self.rnn = nn.Linear(10,20)

  @torch.cuda.amp.autocast()
  def forward(self, input):
    hx = self.rnn(input)
    return hx

model = Net().to(device)
input = torch.randn(3, 10).to(device)
# scaler = torch.cuda.amp.GradScaler()
with torch.cuda.amp.autocast():
  output = model(input)
print(output.size())
</code></pre>
<p>How to fix the first piece of code</p>

username: pldnxj
cooked: <p>Is there a boss to solve this problem?</p>

username: ptrblck
cooked: <p>CC <a class="mention" href="/u/mcarilli">@mcarilli</a> to have a look at this, as it’s reproducible in <code>1.7.0.dev20200728</code>.</p>

username: pldnxj
cooked: <p>Thanks, I will send the message</p>

username: mcarilli
cooked: <p>It appears LSTMCell and GRUCell have <a href="https://github.com/pytorch/pytorch/blob/db52cd7322ccf3f2121f1a6e592ddb436968aa50/tools/autograd/derivatives.yaml#L1756-L1761" rel="nofollow noopener">dedicated autograd operations</a> that type-check inputs.  I should add those to the FP16 cast list.</p>
<p>Filed issue <a href="https://github.com/pytorch/pytorch/issues/42605" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/42605</a>.  Should be a straightforward fix, thanks for reporting this.  I thought the cells were implemented in terms of autograd-exposed primitive ops (eg. gemms) that autocast already covers.</p>

username: pldnxj
cooked: <p>Thank you for your answer</p>
Questions about Gradient checkpointing

username: Martin_cb
cooked: <p>Hey I have 2 questions with regards to using gradient checkpointing</p>
<p><strong>1. Do Dropout and Batchnorm layers work now with Checkpointing?</strong></p>
<ul>
<li>in this tutorial, it is mentioned that these layers didn’t work with chekpointing:<br>
<a href="https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb" rel="nofollow noopener">https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb</a>
</li>
</ul>
<p>Is this still the case? Some of the pytorch documentation suggests that this may have changed for at least for dropout? Can someone provide a definitive answer?</p>
<p><strong>2. What is the preferred way to use data parallel with Checkpoint_Sequential?</strong><br>
Can someone give me a template for both DataParallel and DistributedDataParallel, that actually works with multiple GPUs?</p>
Converting a list of tensors to a tensor and set grad_fn

username: samin_hamidi
cooked: <p>I have a list of tensors in this form:</p>
<blockquote>
<p>[tensor(-2.9222, grad_fn=), tensor(-2.8192, grad_fn=SqueezeBackward1&gt;), tensor(-3.1894, grad_fn=), tensor(-2.9048, grad_fn=)]</p>
</blockquote>
<p>and what I need is this list to be in the form of a tensor like this:</p>
<blockquote>
<p>tensor([-0.5575, -0.9004, -0.8491,  …, -0.7345, -0.6729, -0.7553],grad_fn=SqueezeBackward1&gt;)</p>
</blockquote>
<p>Is it possible to do such a conversion and keep the grad_fn set as it is per tensor in the original image?</p>

username: Kushaj
cooked: <p>In the second case there is only one tensor. <code>grad_fn</code> is stored as an attribute of a tensor. So you cannot store the grad_fn of multiple tensors in a single tensor (you can use a proxy to sum all the tensors, but your case that is not valid).</p>
Reaching `transformer` attribute while model is wrapped in DataParallel

username: Ernst_Stavro_Blofeld
cooked: <p>Hi folks.</p>
<p>I successfully managed to use Huggingface transformers with Pytorch using <strong>a single GPU</strong>.</p>
<p>Now, I’m trying to use multiple gpus with <code>DataParallel</code>. While wrapped in <code>DataParallel</code>, my model begins as follows:</p>
<pre><code class="lang-auto">DataParallel(
  (module): DataParallel(
    (module): CustomTransformerModel(
      (transformer): RobertaForSequenceClassification(
        (roberta): RobertaModel(
          (embeddings): RobertaEmbeddings(
            (word_embeddings): Embedding(50265, 768, padding_idx=1)
            (position_embeddings): Embedding(514, 768, padding_idx=1)
            (token_type_embeddings): Embedding(1, 768)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
[...]
etc..
</code></pre>
<p>Whareas while NOT wrapped in DataParallel, the model look like this:</p>
<pre><code class="lang-auto">CustomTransformerModel(
  (transformer): RobertaForSequenceClassification(
    (roberta): RobertaModel(
      (embeddings): RobertaEmbeddings(
        (word_embeddings): Embedding(50265, 768, padding_idx=1)
        (position_embeddings): Embedding(514, 768, padding_idx=1)
        (token_type_embeddings): Embedding(1, 768)
        (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
[...]
etc
</code></pre>
<p>As I try to reach the <code>transformer</code> as it is wrapped in <code>DataParallel</code>, I incur in the (predictable) error: <code>AttributeError: 'DataParallel' object has no attribute 'transformer'</code>.</p>
<p>I ran a quick search and found this thread: <a href="https://discuss.pytorch.org/t/how-to-reach-model-attributes-wrapped-by-nn-dataparallel/1373" class="inline-onebox">How to reach model attributes wrapped by nn.DataParallel?</a></p>
<p>but still, <code>model.module.transformer</code> outputs <code>AttributeError: 'DataParallel' object has no attribute 'transformer'</code>.</p>
<p>The thread is old though (2017).</p>
<p>How can I reach <code>transformer</code> attribute with Pytorch 1.3.1? Thanks!!</p>

username: cgnarendiran
cooked: <p>Looks like you have applied the Dataparallel class twice to the model</p>
Calculate accuracy in Binary classification

username: netaglazer
cooked: <p>Hi<br>
I have a NN binary classifier, and the last layer is sigmoid, I use BCEloss<br>
this is my accuracy calculation:</p>
<pre><code class="lang-auto">def get_evaluation(y_true, y_prob, list_metrics, epoch):
    # accuracy = accuracy_score(y_true, y_prob)
    y_prob = np.array(y_prob)
    y_prob = np.where(y_prob &lt;= 0.5, 0, y_prob)
    y_prob = np.where(y_prob &gt; 0.5, 1, y_prob)

    accuracy = metrics.accuracy_score(y_true, y_prob)
    return accuracy
</code></pre>
<p>is this the correct way to calculate accuracy?</p>
<p>tnx</p>

username: mariosasko
cooked: <p>It seems good to me.</p>
<p>You can use conditional indexing to make it even shorther.</p>
<pre><code class="lang-auto">def get_accuracy(y_true, y_prob):
    accuracy = metrics.accuracy_score(y_true, y_prob &gt; 0.5)
    return accuracy
</code></pre>
<p>If you want to work with Pytorch tensors, the same functionality can be achieved with the following code:</p>
<pre><code class="lang-auto">def get_accuracy(y_true, y_prob):
    assert y_true.ndim == 1 and y_true.size() == y_prob.size()
    y_prob = y_prob &gt; 0.5
    return (y_true == y_prob).sum().item() / y_true.size(0)
</code></pre>
Is torchvision.transforms incompatible with tensors?

username: mgpoirot
cooked: <p>I’ve never used the built-in transforms system before, but it seemed really useful, so I wanted to give it a try.</p>
<p>My dataloader grabs a 4D batch (batch, channels, x, y) np.ndarray from an HDF5 file, I believe this is a very common setup for many 2D networks.</p>
<p>However, the transforms system can only process 3D (x,y,channel) shaped PIL-images?<br>
So to use this system I’d have to build a for loop in which np.ndarray is converted to PIL image, transpose, transform and convert back to np?</p>
<p>This sounds so ridiculous I can hardly believe it. I would have had expected that this torch module would at least accept torch tensors.</p>

username: ptrblck
cooked: <p>Some transformations are working on tensors directly (<code>Normalization</code>), while most work on <code>PIL.Image</code>s, so you are correct in your assumption.<br>
Alternatively, you could use e.g. opencv to directly transform your numpy arrays.</p>

username: mgpoirot
cooked: <p>Thanks for your response, there really is a plethora of alternatives in reprocessing modules (opencv, skimage.transform, scipy.ndimage, imutils, even numpy), so I’ll go for one of those.</p>

username: theRealSuperMario
cooked: <p>Interestingly, the RandomCrop Transform is supposed to work on PIL images and Tensors according to <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomCrop" rel="nofollow noopener">this</a>.</p>
<p>But it is only working in PIL images, at least in my torch==1.6.0 and torchvision==0.7.0 environment.</p>
<p>Same for <code>transforms.functional.crop</code></p>
<p>Update:</p>
<p>Apparently, the feature got introduces just a couple of weeks ago (<a href="https://github.com/pytorch/vision/pull/2342" rel="nofollow noopener">https://github.com/pytorch/vision/pull/2342</a>)<br>
and is not yet on anaconda</p>

username: ptrblck
cooked: <p>This functionality is available in the nightly binaries, but it seems it wasn’t picked for the <code>0.7</code> release.<br>
I’ve created a <a href="https://discuss.pytorch.org/t/documentation-inconsistent-for-torch-vision-transform-functional-crop/91469/2">similar post</a>, where I’ve tagged Francisco to take a look, if this was missed. Otherwise the docs should be updated.</p>
Torch tensor reshape by removing a channel

username: Palash_Shah
cooked: <p>so i’m training a 3D Unet where my input size is (1, 1, 64, 64, 64) and my output is (1, 2, 64, 64, 64). How do I get rid of one of the channels for the output so that i can compare the two?</p>
<p>I want my tensor to go from (1, 2, 64, 64, 64) to (1, 1, 64, 64, 64) by removing one of the channels.</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>I think it is not good idea to just <em>get rid of</em> a channel. If your model generates outputs of size <code>[1, 2, ...]</code> then simply you can change last layer to generate 1 channel outputs. Otherwise, you have to find a function that maps 2 channel to 1 channel, otherwise you might lose a lot of information.<br>
Doing it on UNet model is really easy.</p>
<p>Bests</p>

username: Palash_Shah
cooked: <p>so i tried this by implementing my own function and it doesn’t seem to work:</p>
<pre><code class="lang-auto">def combine_channel(output):

  first, second = output[0][0], output[0][1]

  first = torch.reshape(first, shape=(1, 262144))
  second = torch.reshape(second, shape=(1, 262144))
  arr = []

  for fir, sec in zip(first[0], second[0]):
    if(fir &gt; sec):
      val = 0
      arr.append(val)
    else:
      val = 1
      arr.append(val)

  final = torch.reshape(torch.from_numpy(np.asarray(arr)), shape=(1, 262144))
  image = torch.reshape(final, shape=(1, 1, 64, 64, 64)).type(torch.float)
  return image
</code></pre>
<p>this just compares the two channels (since they’re probabilities of class 1 or 2) and then creates a new tensor with the larger one.</p>

username: Palash_Shah
cooked: <p>made a new issue for the new error i’m getting <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Nikronic
cooked: <aside class="quote no-group" data-username="Palash_Shah" data-post="3" data-topic="91651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/palash_shah/48/27323_2.png" class="avatar"> Palash_Shah:</div>
<blockquote>
<p>(since they’re probabilities of class 1 or 2) and then creates a new tensor with the larger one.</p>
</blockquote>
</aside>
<p>You can do this by <code>torch.max</code></p>
<pre><code class="lang-auto">output = torch.randn(1, 2, 64, 64, 64)
output,_ = torch.max(output, 1, keepdim=True)
print(output.shape) # [1, 1, 64, 64, 64]
</code></pre>
Four p2.xlarge vs two p3.2xlarge

username: qap
cooked: <p>Each EC2 <em>p3.2xlarge</em> instance has 8 CPU and 1 GPU.  I’m allowed a maximum of 16 CPUs on AWS at any given time, so I’ve been doing distributed training over 2 GPUs.  However, I’ve just noticed that each <em>p2.xlarge</em> instance has 4 CPU and 1 GPU.  This means that I could have 4 of these, which means training over 4 GPUs.  Would this make training faster?</p>
<p>What factors should be taken into consideration?  4 instances do take more time to set up than 2.  Cost is not an issue.  I’m doing mixed precision distributed training with apex.</p>
<p>Thanks</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="qap" data-post="1" data-topic="91743">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qap/48/27108_2.png" class="avatar"> qap:</div>
<blockquote>
<p>This means that I could have 4 of these, which means training over 4 GPUs. Would this make training faster?</p>
</blockquote>
</aside>
<p>This is possible. Hope Figure 9 in this paper can offer some insight: <a href="https://arxiv.org/pdf/2006.15704.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2006.15704.pdf</a></p>
<blockquote>
<p>What factors should be taken into consideration?</p>
</blockquote>
<p>If the GPUs are the same, the network bandwidth is one of the dominating factor of training speed.</p>
<blockquote>
<p>4 instances do take more time to set up than 2.</p>
</blockquote>
<p>This is one-time setup overhead, instead of per-iteration overhead, right? If so, it should be fine.</p>
Random integers

username: stefanonardo
cooked: <p>Is there anything similar to NumPy <code>random.randint</code> to generate random integers between a certain range?</p>

username: adrianjav
cooked: <p><a href="http://pytorch.org/docs/0.3.1/torch.html?highlight=torch%20random#torch.rand" rel="nofollow noopener"><code>torch.rand</code></a> outputs a tensor fill out with random numbers within <code>[0,1)</code>. You can use that and convert it to the range <code>[l,r)</code> using a formula like <code>l + torch.rand() * (r - l)</code> and then converting them to integers as usual.</p>

username: ptrblck
cooked: <p>You could also directly set the range using:</p>
<pre><code>torch.LongTensor(10).random_(0, 10)
</code></pre>
<p>where numbers define the lower and upper bound, respectively.</p>

username: Adam_Dziedzic
cooked: <p>PyTorch also supports randint: <a href="https://pytorch.org/docs/stable/torch.html#torch.randint" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.randint</a></p>
<pre><code class="lang-auto">torch.randint(3, 5, (3,))
tensor([ 4.,  3.,  4.])
</code></pre>

username: Brando_Miranda
cooked: <p>this should work:</p>
<pre><code class="lang-auto">import torch

batch_size = 16
y = torch.randint(low=0,high=Dout,size=(batch_size,))
</code></pre>
<p>docs: <a href="https://pytorch.org/docs/stable/generated/torch.randint.html#torch.randint" rel="nofollow noopener">https://pytorch.org/docs/stable/generated/torch.randint.html#torch.randint</a></p>
How to average loss whilst retaining grad_fn information?

username: tdfalc
cooked: <p>I am trying to average the output of the nn.MSELoss() over the following batch before firing batch_loss.backward().</p>
<pre><code class="lang-auto">[tensor(0.0031, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;),
 tensor(0.0010, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;),
 tensor(0.0015, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;),
 tensor(0.0027, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;),
 tensor(0.0042, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;),
 tensor(0.0045, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;),
 tensor(0.0044, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;),
 tensor(0.0005, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;),
 tensor(0.0034, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;),
 tensor(0.0023, device='cuda:0', grad_fn=&lt;MseLossBackward&gt;)]
</code></pre>
<p>How can I do this whilst retaining the grad_fn information? As if I use torch.mean(torch.tensor()), I lose this information.</p>
<p>Thanks!</p>

username: Kushaj
cooked: <p><code>sum(output).backward()</code>. It will work if output is a list.</p>
Input to cuda not register

username: yosyos
cooked: <pre><code>    image = (np.random.random((416, 416, 3)) * 250).astype('uint8')
    im = Image.fromarray(image)
    x = TF.to_tensor(im)
    x.cuda()
    x = x.unsqueeze_(0)
    x = Variable(x)

    # Get detections
    with torch.no_grad():
        detections = model(input_imgs)
        detections = non_max_suppression(detections, opt.conf_thres, opt.nms_thres)
</code></pre>
<p>got error:<br>
Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same</p>
<p>so I check and got:<br>
x.is_cuda = False</p>

username: ptrblck
cooked: <p><code>x.cuda()</code> is not an inplace method, so you would have to reassign the return value:</p>
<pre><code class="lang-python">x = x.cuda()
</code></pre>

username: yosyos
cooked: <p>Thank you<br>
(The post must be 20 char long?l    <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_raised_eyebrow.png?v=9" title=":face_with_raised_eyebrow:" class="emoji" alt=":face_with_raised_eyebrow:">)</p>
CUDA out of memory. Tried to allocate 98.00 MiB (GPU 0; 15.90 GiB total capacity; 14.80 GiB already allocated; 47.88 MiB free; 15.16 GiB reserved in total by PyTorch)

username: Umair_Javaid
cooked: <pre><code class="lang-auto">def sub_forward(self, x)
      x = self.conv1(x)
      x = self.bn1(x)
      x = self.relu(x)
      x = self.maxpool(x)
      print("ok")
      x = self.layer1(x)
      x = self.layer2(x)
      x = self.layer3(x)
      x = self.layer4(x)
      print("ok1")
      return x

    def forward(self, x, labels=None, return_cam=False):
        batch_size = x.shape[0]
        x1 = self.sub_forward(x)
        x1 = self.conv6(x1)
        x1 = self.relu(x1)
        x1 = self.conv7(x1)
        x1 = self.relu(x1)
        x2 = self.sub_forward(x)
        x2 = self.conv8(x2)
        x2 = self.relu(x2)
        x2 = self.conv9(x2)
        x2 = self.relu(x2)
        x = x1 + x2
        if return_cam:
          normalized_feature_map = normalize_tensor(x.detach().clone())
          cams = normalized_feature_map[range(batch_size), labels]
          return cams

        x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        print("ok2")
        return {'logits': x}
</code></pre>
<p>I am training a resnet model. I can not change the architecture of the model, nor can I reduce the batch size. Is there anything else I can do to avoid this error?</p>
<p><strong>Output:</strong></p>
<pre><code class="lang-auto">ok
ok1
ok
ok1
ok2
ok
</code></pre>
<p>It can be seen that it completes the first forward pass but fails in second forward pass</p>
user_suspended: yes

username: Umair_Javaid
cooked: <p><a class="mention" href="/u/alband">@albanD</a> any suggestions?</p>
user_suspended: yes

username: albanD
cooked: <p>Hi,</p>
<p>Not much, that’s why I did not answer anything <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
You can check this related post for potential solutions: <a href="https://discuss.pytorch.org/t/how-can-you-train-your-model-on-large-batches-when-your-gpus-can-only-hold-couple-of-batches/80696" class="inline-onebox">How can you train your model on large batches when your GPUs can only hold couple of batches</a></p>

username: Umair_Javaid
cooked: <p>I am working on 13G ram google colab gpu. Would moving to a larger ram gpu solve this problem?</p>
user_suspended: yes

username: albanD
cooked: <p>Yes having more memory will remove this issue. How much more you need though, I don’t know <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
EDIT: As mentioned below, yes, more GPU memory will help, not more ram. I misread the message above.</p>

username: Oscar_Rangel
cooked: <p><a class="mention" href="/u/alband">@albanD</a> it is not RAM memory you are needing is GPU memory… try smaller batch sizes , 32, 16, 8, 4 ,2 1…</p>
<p>But with smaller batch size you really not going to get good generalization…</p>
1d csv data or image 2d data for DCGAN

username: kkedev01
cooked: <p>Hi everyone,<br>
i hope that everyone is fine.<br>
i am wondering if you can help me, i have (1600,21,1) signal processing csv data that i made myself(by experimentation ), i am trying hard to train and predict with DCGAN this data but i think that there are lot of thing are missing.<br>
so the question is, do you think that it is better to convert this data into 2d spectogram image and train with it my DCGAN in order to get better performance?</p>
<p>thank you for your answer</p>
<p>kwhiw</p>
Improving NMT model outputs

username: Aiman_Mutasem-bellh
cooked: <p>Dear <span class="mention">@All</span></p>
<p>I have an MTN project based on TRANSFORMER everything is working fine, for the final translation output the model translates number to zeros <code>0000</code>, and non-English names to <code>&lt;UNK&gt;</code>.</p>
<p>I need to solve these issues to increase my blue score. any suggestions?</p>

username: vdw
cooked: <p>Rare words or out-of-vocabulary words are a fundamental challenge for NMT. You still find very recent academic papers addressing this.</p>
<p>For example, for a very simple NMT task, I used an off-the-shelf NER system to replace, say, person names. So 2 sentences “I met Alice” and “I met Bob” would be converted to "I met "; same for the target sentences. After the translation, I would simple replace  with the actual name. Replacing numbers with  would also be very easy with a RegEx. It worked fine enough for my use case, but it’s probably too naive for the general case.</p>

username: Aiman_Mutasem-bellh
cooked: <p>Thank you Sir <a class="mention" href="/u/vdw">@vdw</a> , That what I’m looking for.</p>
nn.ReLU inpalce operation error

username: Leslie_Wang
cooked: <p>I want to pass some params to nn.Relu without setting the inplace=True. But still got error <code>RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.</code><br>
here is my code, just want to figure out if pytorch will set inplace=True in train mode automatically?</p>
<pre><code class="lang-auto">self.p6_w1 = nn.Parameter(torch.ones(2, dtype=torch.float32), requires_grad=True)
        self.p6_w1_relu = nn.ReLU()
</code></pre>
<pre><code class="lang-auto">  p6_w1 = self.p6_w1_relu(self.p6_w1)
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Leslie_Wang" data-post="1" data-topic="91597">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leslie_wang/48/27460_2.png" class="avatar"> Leslie_Wang:</div>
<blockquote>
<p>just want to figure out if pytorch will set inplace=True in train mode automatically?</p>
</blockquote>
</aside>
<p>No, the <code>inplace</code> argument won’t be changed and will keep the value you’ve set during the instantiation.<br>
If you are not using inplace ReLUs, some other operation might manipulate the data inplace, such as assignments using indexing etc.</p>
Import adapted Resnet model

username: jacobo
cooked: <p>Hi everyone, I would like to use a pre-trained Resnet model, but with some modifications. I would like to change the input from 3 channels (RGB) to gray scale (1 input). I would also like to change the output from 1000 classes to 10. Furthermore, it would be interesting to implement a deconvnet to see the feature maps at the different filters.</p>
<p>I do all this changes in the original Resnet code, and save it as resnet_adapted in the same folder. How do I import this model pre-trained? Could I freeze certain layers, just like in normal transfer learning?</p>
<p>I have been advised to use:</p>
<pre><code class="lang-auto">import sys
sys.path.append(r'C:\Users\Jacobo\anaconda3\envs\deep learning 3\Lib\site-packages\torchvision\models')
import resnet_adapted
</code></pre>
<p>but I think I cannot obtain a pre-trained model from that, nor freeze specific layers.</p>
<p>Thanks in advance for the help!</p>

username: ptrblck
cooked: <p>Depending how much has changed in your custom Resnet implementation, you could either try to directly load the pretrained resnet <code>state_dict</code> with the <code>strict=False</code> option:</p>
<pre><code class="lang-python">resnet_adapted = MyResnet()
pretrained = torchvision.models.resnet50(pretrained=True)
resnet_adapted.load_state_dict(pretrained.state_dict(), strict=False)
</code></pre>
<p>or load specific layers manually by iterating the keys in the pretrained <code>state_dict</code>.</p>
<p>Why do you think you cannot freeze specific layers in your model? Are you seeing any errors?</p>
Data augmentation with several efects

username: tenet2001tenet
cooked: <p>Hi, i have a dataset of X different classes. My main goal is to augmentate my data with:</p>
<ul>
<li>gaussian blur</li>
<li>rotation</li>
<li>contrast</li>
<li>gamma+random noise<br>
Generating 20 images per image, i.e, 5 times this effects for each image.</li>
</ul>
<p>I tried this:</p>
<pre><code class="lang-auto">def gaussian_blur(img):

    image = np.array(img)

    image_blur = cv2.GaussianBlur(image,(65,65),10)

    new_image = image_blur

    im = Image.fromarray(new_image) 

    return im

data_transforms = {

    'train': transforms.Compose([

        transforms.RandomRotation([-8,+8]),

        transforms.Lambda(gaussian_blur),

        transforms.ColorJitter(brightness=0, contrast=0.4, saturation=0, hue=0),

        transforms.Compose([transforms.Lambda(lambda x : x + torch.randn_like(x))]),

        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

    ]),

    'val': transforms.Compose([

        transforms.RandomRotation([-8,+8]),

        transforms.Lambda(gaussian_blur),

        transforms.ColorJitter(brightness=0, contrast=0.4, saturation=0, hue=0),

        transforms.Compose([transforms.Lambda(lambda x : x + torch.randn_like(x))]),

        transforms.ToTensor(),

        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

    ]),

}
</code></pre>
<p>But i have errors, also because i dont know how to use it.<br>
Any suggestion?</p>

username: ptrblck
cooked: <p>What kind of errors are you seeing?<br>
Could you post them with the complete stack trace, please?</p>
Acquire cuDNN debug trace for PyTorch

username: dan_soutar
cooked: <p>Not sure how best to categorise this topic, although nothing stands out as an obvious topic for this questiont. Feel free to clarify.</p>
<p>I am currently trying to run a simple pooling layer through PyTorch 1.6.0 as follows:</p>
<pre><code class="lang-auto"># run-torch-pooling.py
import torch
import numpy as np

# Get input tensor 
t = torch.from_numpy(np.array([i for i in range(224*224*3)], dtype=np.float32).reshape((1, 3, 224, 224)))

pool = torch.nn.MaxPool2d(3, stride=2)

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

if torch.cuda.is_available():
  t = t.to(device)

with torch.cuda.device(0):
  output = pool(t)

# Do stuff with output
</code></pre>
<p>Simple program, and this works fine. My issue is that if I were to run the following command (assuming an appropriate environment in Ubuntu:</p>
<pre><code class="lang-auto">$ (pyenv) CUDNN_LOGINFO_DBG=1 CUDNN_LOGDEST_DBG=stdout python run-torch-pooling.py
</code></pre>
<p>Nothing happens regarding the expected cuDNN trace. Am I right in thinking that this is because cuDNN is opened in a separate process and so I can’t introspect the raw cuDNN dump? For what it’s worth, the analogous program for Tensorflow works as expected and dumps the cuDNN kernel calls.</p>
<p>I can verify that my GPU is being used, as I can observe memory allocations being done in <code>nvidia-smi</code>.</p>
<p>If anyone could give advice or answers to how I can activate this kind of trace that would be greatly appreciated.</p>

username: ptrblck
cooked: <p>I haven’t looked into the source code, but maybe this particular workload is not using cudnn at all.<br>
Have you checked the kernel calls with <code>nvprof</code> of <code>nsys</code> to make sure cudnn is used?</p>