akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
nn.GRU last output and h_n (state) doesn't match

dovah: <p>Hi,<br>
I’m having troubles to understand the relationship between the output of <a href="https://pytorch.org/docs/stable/nn.html#gru" rel="noopener nofollow ugc">nn.GRU</a> and h_n (last state).</p>
<p>Per documentation:</p>
<blockquote>
<p>output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features h_t from the last layer of the GRU, for each t.</p>
</blockquote>
<blockquote>
<p>h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len</p>
</blockquote>
<p>The documentation additionally specifies how to separate the various dimensions properly, so I followed that hint and made this script:</p>
<pre><code class="lang-auto">with torch.no_grad():
  rnn = nn.GRU(16, 16, num_layers=2, bidirectional=True)
  input = torch.randn(5, 10, 16)
  memory, state = rnn(input)

  memory_view =\
    memory.view(
      5,
      10,
      2,
      16)

  state_view =\
    state.view(
      2,
      2,
      10,
      16)

  # True
  print(torch.allclose(memory_view[-1, 0, 0, :], state_view[-1, 0, 0, :]))
  # False
  print(torch.allclose(memory_view[-1, 0, 1, :], state_view[-1, 1, 0, :]))
</code></pre>
<p>How come the backward part of the last <code>memory</code> vector doesn’t match the backward part of <code>state</code>?<br>
It’s not clear to me where I’m mistaking and I hope you’ll kindly help me out this confusion.<br>
Thanks</p>
How can I constraint a layer input to [0,1]?

calliope38: <p>I have a nonlinear activation function that induces nonlinearity in the domain range of [0,1], so Id like to have the input be in that range. the input to this nonlinear layer is from a nn.Linear layer. batchnorm1d doesnt seem to do the job</p>

JuanFMontesinos: <p>Use sigmoid, clamping or rescaling. Depending on the problem you are facing.</p>
Can I share nn.Embeddings between processes on the same GPU?

jts: <p>I’m working on a project that uses multiple instances of the same model on one GPU (using torch.multiprocessing).  They are RNNs that process blocks of text and they do not interact with each other.</p>
<p>I’m using nn.Embedding and loading up the full glove vector file (about 2 million words, vector size 300) in each model.  This takes up a lot of GPU memory.  I’m trying to see if I can share the embedding memory between the processes on the same GPU.  I understand I could limit the number of words but that is not the goal of this exercise.</p>
<p>Looking in tensor.py I see the share_memory_() function and it appears that the nn.Embedding is made up of tensors, but is there a way to share the entire nn.Embedding instance?  Simply sharing the glove vector tensors won’t help because the nn.Embedding appears to reserve its own space.</p>
<p>My other option would be to not use the nn.Embedding and just convert the word indices to vectors on the CPU before being passed into the RNN but this would be slower.</p>
<p>One curious thing I see in share_memory_() in storage.py is this:</p>
<blockquote>
<pre><code>    if self.is_cuda:
        pass  # CUDA doesn't use POSIX shared memory
</code></pre>
</blockquote>
<p>In tensor.py this is the share_memory_() function:</p>
<blockquote>
<pre><code>    self.storage().share_memory_()
    return self
</code></pre>
</blockquote>
<p>To me this looks like calling share_memory_() on the GPU does nothing.  What am I missing?</p>
Using libtorch to implement DQN

stephenxudong: <p>I need to implement DQN algorithm using libtorch. But it seems that libtorch didn’t provide<br>
the method of copying the parameters from the training network to the target network, which is necessary for DQN.</p>
<p>The corresponding Python code is like this:</p>
<pre><code class="lang-auto"> def update_target_q_network(self):
        self.target_model.load_state_dict(self.model.state_dict())
</code></pre>
<p>Are there any practical solutions to this question?</p>

EmmiOcean: <p>try this:</p>
<pre><code class="lang-auto">for (size_t i = 0; i &lt; target_model.parameters().size(); i++)
{
        target_model.parameters()[i].copy_(model-&gt;parameters()[i]);
}
</code></pre>
Pytorch equivalant of tensorflow GradientTape

chaslie: <p>Hi,</p>
<p>I was wondering what the equivlent in pytorch of the following tensor flow is:</p>
<pre><code class="lang-auto">def compute_apply_gradients(model, x, optimizer):  
     with tf.GradientTape() as tape:    
          loss = compute_loss(model, x)  
     gradients = tape.gradient(loss, model.trainable_variables)  
     optimizer.apply_gradients(zip(gradients, model.trainable_variables))
</code></pre>
<p>many thanks,</p>
<p>chaslie</p>

albanD: <p>Hi,</p>
<p>You can look at the tutorials on the website. But that would be something like:</p>
<pre><code class="lang-auto">loss = compute_loss(model, x)
optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>

chaslie: <p>hi albanD,</p>
<p>thanks for the reply. This is what i had assumed but i wasn’t sure if there were any switches that i had missed.</p>
<p>Hope you manage to avoid Covid-19.</p>
<p>chaslie</p>
Import pytorch fails on my windows 10 PC

joepareti54: <p>I have installed pytorch using (miniconda3) the following command:</p>
<p><em>conda install pytorch torchvision cpuonly -c pytorch</em></p>
<p>In the jupyter notebook, the ‘import torch’ fails:</p>
<p><em><strong>ModuleNotFoundError: No module named 'torch._C’</strong></em></p>
<p>Further (using cygdrive), I see the following:</p>
<p>$ find . -name ‘python.exe’ -print<br>
./Users/joepareti/AppData/Local/amlworkbench/Python/pkgs/python-3.5.6-he025d50_0/python.exe<br>
./Users/joepareti/AppData/Local/amlworkbench/Python/python.exe<br>
./Users/joepareti/AppData/Local/Microsoft/WindowsApps/Microsoft.DesktopAppInstaller_8wekyb3d8bbwe/python.exe<br>
./Users/joepareti/AppData/Local/Microsoft/WindowsApps/python.exe<br>
./Users/joepareti/Miniconda3/envs/myenv/python.exe<br>
./Users/joepareti/Miniconda3/envs/myenv1/python.exe<br>
./Users/joepareti/Miniconda3/pkgs/python-3.7.1-h8c8aaf0_6/python.exe<br>
./Users/joepareti/Miniconda3/python.exe</p>

peterjc123: <p>Please do a cleanup of the python environments and try again.</p>
Convergence with custom layers

calliope38: <p>im trying to implement a three layer network with a custom layer (which is essentially a curve fitted function using scipy). my network converges when i use a standard torch class but when i replace it with my custom layer, the network doesnt converge. my function is mathematically the same, as in im replacing a sigmoid with a fitted sigmoid. what could be going wrong?</p>

charan_Vjy: <p>Can you display both the pytorch code and the modified version?</p>
Deep Q always giving same output no matter the input

jandfranc: <p>I am attempting to replicate (using PyTorch) another person’s Deep Q Network (which used TensorFlow) to learn the OpenAI gym environment ‘BreakoutDeterministic-v4’. The one I am attempting to replicate is here: <a href="https://github.com/fg91/Deep-Q-Learning/blob/master/DQN.ipynb" rel="nofollow noopener">https://github.com/fg91/Deep-Q-Learning/blob/master/DQN.ipynb</a></p>
<p>When looking at the actions chosen for the batch size, the action chosen is always the same, no matter how long the network has trained. This action sometimes changes, but it is always the same for every state in the batch, despite these states being randomly selected.</p>
<p>All parameters in the network are directly taken from the network I am trying to replicate, and the input image is cropped in the exact same manner also. The network I am replicating does work as I have tested it.</p>
<p>Below is the code used for the DQN.</p>
<pre><code class="lang-auto">class DQN(nn.Module):
def __init__(self, input_shape, n_actions):
    super(DQN, self).__init__()
    self.conv = nn.Sequential(
        nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),
        nn.ReLU(),
        nn.Conv2d(32, 64, kernel_size=4, stride=2),
        nn.ReLU(),
        nn.Conv2d(64, 64, kernel_size=3, stride=1),
        nn.ReLU()
    )

    self.val_conv = nn.Sequential(
        nn.Conv2d(64, 512, kernel_size=7, stride=1),
        nn.ReLU()
    )

    self.adv_conv = nn.Sequential(
        nn.Conv2d(64, 512, kernel_size=7, stride=1),
        nn.ReLU()
    )

    conv_out_size = self._get_conv_out(input_shape)
    self.value_stream = nn.Sequential(
        nn.Linear(conv_out_size, 1)
    )

    self.advantage_stream = nn.Sequential(
        nn.Linear(conv_out_size, n_actions)
    )

def _get_conv_out(self, shape):
    o = self.conv(torch.zeros(1, *shape))
    o = self.val_conv(torch.zeros(*tuple(o.size())))
    return int(np.prod(o.size()))

def forward(self, state):
    print(state.cpu()[1][1])
    features = self.conv(state)
    feat_val = self.val_conv(features)
    feat_adv = self.adv_conv(features)
    feat_val = feat_val.view(feat_val.size(0), -1)
    feat_adv = feat_adv.view(feat_adv.size(0), -1)
    values = self.value_stream(feat_val)
    advantages = self.advantage_stream(feat_adv)
    qvals = values + (advantages - advantages.mean())

    return qvals
</code></pre>
<p>I am unsure if this is helpful but the Optimizer I am using is Adam, learning rate of 0.00001, and the epsilon decreases at the exact same rate as the one I am trying to replicate.</p>

jandfranc: <p>Also, I just checked and found that the loss of the network is very low, even from the beginning.</p>
Compute weight gradients for first forward pass and input gradients for the rest

atouchet: <p>Hi,</p>
<p>I’m trying to implement a more efficient iterative method similar to adversarial attacks.</p>
<p>For each model I want to compute standard loss for weight updates, but I want to add a penalized loss before performing <code>optimizer.step()</code>. Computing the penalized loss requires iteratively computing the input gradients, but I don’t need to store the weight gradients for each step in this iterative phase.</p>
<p>Is it possible to perform <code>loss.backward()</code> after the first forward pass, and freeze the weight gradients so that computing gradients with respect to inputs doesn’t modify them (i.e. ch.autograd.grad(model(x), [x]) doesn’t change <code>model.arbitrary_layer.weight.data</code>?).</p>
<p>At the end of the iterative phase a loss function will be called again however this time I want the weight gradients to be accumulated with the previous loss.</p>
<pre><code class="lang-auto">logits, otheroutput = model(x, return_otheroutput=True)
loss = loss_fn(logits, x_label)
loss.backwards()

z = x.clone().detach().requires_grad_(True)
for i in range(iterations):

    z = z.clone().detach().requires_grad_(True)

    logits = model(z, return_otheroutput=False)

    loss = loss_fn(logits, other_label)

    grad = torch.autograd.grad(loss, [z])

    with torch.no_grad():
        z = compute_new_x(z, grad)

logits = model(z, return_otheroutput=False)
loss = loss_fn(z, other_label)
loss = -2*loss
loss.backwards()
optimizer.step()
</code></pre>
<p>The end loss function should be <code>&lt;standard_loss&gt; - 2&lt;loss of adversarial image&gt;</code>. In this case I say its adversarial image but not really. It’s quite similar though. Also the <code>-2</code> is a standard in for the usual <code>lambda</code> scalar that is attached to some penalized loss functions</p>

albanD: <p>Hi,</p>
<p>You can use <code>autograd.grad</code> to get the gradient wrt a specific set of Tensors without touching any <code>.grad</code> field.<br>
But you seem to be doing that properly in your sample.</p>
<p>Few things:</p>
<ul>
<li>you want <code>.backward()</code> (without the s)</li>
<li>The way you compute your loss does not match the formula you gave</li>
<li>This code should do what you want.</li>
</ul>
Non thread-safe dataset with DataLoader

npielawski: <p>I have a really big dataset containing images of cells in a really weird format. I am using the library bio-formats (which works in Java) to read the images, but is not thread-safe. They are advising to create a new image reader per instance.</p>
<p>For now, if I don’t do anything specific, when num_worker &gt; 0, nothing happens (it seems to be blocked in an infinite loop).</p>
<p>I have torch 1.4.0, and I tried torch.multiprocessing.set_start_method(“forkserver”/“spawn”) but to no avail, it gives the following error:</p>
<pre><code class="lang-bash">... in _javabridge.JB_Object.__reduce_cython__()

TypeError: no default __reduce__ due to non-trivial __cinit__
</code></pre>
<p>Is there any way of having separate instances in the dataloader?</p>

albanD: <p>Hi,</p>
<p>Note that when you use <code>num_worker &gt; 0</code>, we use multiprocessing and not multithreading to load the different objects on the Dataset.<br>
So you want to make sure that your java loader works fine in the child process .</p>
Moving tensor to cuda | with and without Apex

shrutee: <p>putting tensor to cuda, with 1 gpu only</p>
<pre><code class="lang-auto">import torch
torch.randn(1, 1, 32000).to(device='cuda:0')
</code></pre>
<p>In google colab, with</p>
<p><strong>Apex</strong></p>
<pre><code class="lang-auto">!git clone https://github.com/NVIDIA/apex
!pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./apex
</code></pre>
<p><strong>CUDA Version 10.0.130</strong><br>
<strong>torch 1.4.0+cu100</strong><br>
<code>!pip install torch==1.4.0+cu100 torchvision==0.5.0+cu100 -f https://download.pytorch.org/whl/torch_stable.html</code><br>
<strong>CUDNN 7.6.5</strong></p>
<p>I am getting following error<br>
<code>RuntimeError: CUDA error: an illegal memory access was encountered</code></p>
<p>However, the code runs fine with</p>
<pre><code class="lang-auto">CUDA Version 10.0.130
torch 1.4.0
CUDNN 7.6.5
</code></pre>

ptrblck: <p>I cannot reproduce this issue in Colab using this code snippet:</p>
<pre><code class="lang-python">!pip install torch==1.4.0+cu100 torchvision==0.5.0+cu100 -f https://download.pytorch.org/whl/torch_stable.html
!git clone https://github.com/NVIDIA/apex
!pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./apex
import torch
torch.randn(1, 1, 32000).to(device='cuda:0')
</code></pre>
<p>However, if you are fine with updating to the latest nightly binary, you could use the <a href="https://pytorch.org/docs/master/amp.html#module-torch.cuda.amp">core amp implementation</a>, so that you wouldn’t need to install <code>apex</code> to use mixed-precision training.</p>

shrutee: <p>I installed nightly</p>
<p><code>!pip3 install torch_nightly -f ttps://download.pytorch.org/whl/nightly/cu100/torch_nightly.html --user</code></p>
<p>however, <code>import torch.cuda.amp</code> throws error<br>
<code>AttributeError: module 'torch.cuda' has no attribute 'amp'</code></p>
<p>how should I import this?</p>
DistributedDataParallel and SubsetRandomSampler

tchainzzz: <p>I am currently using SubsetRandomSampler to enforce a train-val split on my custom dataset, which works well on my current single-GPU configuration. However, in anticipation of moving to training on multiple nodes and GPUs, I wanted to see if it’s possible to “wrap” the splits created by SubsetRandomSampler somehow such that within my train split, I can replicate the functionality of DistributedSampler.</p>
<p>If not – what alternatives do I have for creating a train-val split?  Must I create separate Dataset objects for the train and the val set?</p>

mrshenli: <p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> for dataloader question. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Sum over indices with value (1)01

moj: <p>Hi,</p>
<p>I have a tensor with a shape (2, 2, 3) like:<br>
a= tensor ( [ [ [2, 0, 2],<br>
[1, 0, 0] ],<br>
[ [1, 0, 1],<br>
[0, 1, 0] ] ] )</p>
<p>I want to find the indices of the values (1), then make 2 to the power of those indices and finally add the results for the last dimension, so the final result should be like:</p>
<p>tensor ( [ [ [ ],<br>
[2^0]],<br>
[ [2^0+2^2],<br>
[2^1] ] ] )</p>
<p>My actual tensor is much bigger than this example, so I don’t want to use “for” loop, and I have to use broadcasting…</p>
<p>I was thinking of something like torch.pow(2,(a == 1).nonzero()).sum(), but it doesn’t work. I have to find a way to apply  (a== 1).nonzero() only for the last dimension, any suggestion? Thanks.</p>
Conda channel "pytorch" super slow

noname1: <p>conda channel “pytorch” so slow that even unusable.  100 megabit connection - 100 kilobytes/s download. What’s the problem it can be? I’m doing docker build, I’ve tried also with “–network host”.</p>

ptrblck: <p>I tried to download the latest conda binary and got a reasonable speed.<br>
Do you see this issue for the first time, i.e. was the download faster before?<br>
Could you check you general network speed?</p>

noname1: <p>Most of the times problem with a pytorch package, I’m experiencing slow speed for a few days. 100 megabit connection and nvme disk, no significant other IO operations. I’ll report details about versions later.</p>
Model Performance Dependent on PyTorch Weights Initialisation

CCL: <p>I have a trained CNN with 70% accuracy, I apply transfer learning, add Global Average Pooling to it, and finetune this new model. However, the train &amp; validation accuracy differ wildly depending on PyTorch’s random weight initialisation values (I’ve seen from 10% to 50% so far), even across different learning rates (<code>1e-01</code> to <code>1e-06</code>). The train &amp; valid loss are simply stuck at around their initial value, even after 400 epochs of backpropagation.</p>
<p>I have made sure to set the model’s parameters with <code>requires_grad() = True</code>. What could be a possible reason for this bizarre behaviour? Thank you!</p>

ptrblck: <p>I wouldn’t call it a bizarre behavior and would expect that the loss might get stuck using a wrong initialization method.<br>
In my opinion, sane weight init functions were a key to be able to train deep neural networks at all.</p>

CCL: <p>I’m using the default weights initialisation from PyTorch, which worked for the CNN (but not after I added GAP). Would it be a good idea to experiment with different weights initialisation methods? If so, which do you recommend? Thanks!</p>
Training models together? How?

alx: <p>I was browsing on Kaggle when I found the following:</p>
<blockquote>
<p><strong>Models and Input sizes</strong></p>
<p>My final submission was a simple average of the following eight models. Inceptions and ResNets usually blend well. If I could have two more weeks I would definitely add some EfficientNets.</p>
<pre><code class="lang-auto">2 x inception_resnet_v2, input size 512
2 x inception_v4, input size 512
2 x seresnext50, input size 512
2 x seresnext101, input size 384
</code></pre>
</blockquote>
<p>Its the first time I hear about training models together, what does it mean? does pytorch support that? any exemples?</p>
<p>How does one average 8 models !?</p>

ptrblck: <p>Usually the model are trained independently, and their penultimate activation is used in a model ensemble to create the final classifier.<br>
<a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">Here</a> is a small exampe of a model ensemble.<br>
Based on the description, some blending was used, so maybe the class logits/probabilities were reduced instead of using a final classifier.</p>
Accuracy for Training and Validation dont update or change

JakeAndFinn: <p>I am tring to fine turn a Resnet 152 model on the CIFAR100 dataset (from torchvision) and trying to track the accuracy of the training and validation but it always stays at 0.1 which is not correct as when I test the model it gets an average of 54% on the 100 images.<br>
I thought I was tracking it correctly but now I am not too sure.</p>
<p>Here is my training code</p>
<pre><code class="lang-auto">valid_loss_min = np.Inf
for epoch in range(10):
    train_loss = 0.0
    valid_loss = 0.0
    
    train_total = 0
    val_total = 0
    
    train_correct = 0
    val_correct = 0
    
    # Training
    model.train()
    for idx, (images, labels) in enumerate(train_loader):
        if use_gpu:
            images, labels = images.cuda(), labels.cuda()
        optimizer.zero_grad() # Reset gradient 
        
        output = model(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()
        
        # Calculate loss and accuracy
        train_loss += loss.item()*images.size(0)
        
        scores, predictions = torch.max(output.data, 1)
        train_total += labels.size(0)
        train_correct += int(sum(predictions == labels))
        acc = round((train_correct / train_total) / 100, 2)
    
    #Validate Model
    model.eval()
    for idx, (images, labels) in enumerate(val_loader):
        if use_gpu:
            images, labels = images.cuda(), labels.cuda()
        output = model(images)
        loss = criterion(output, labels)
        valid_loss += loss.item()*images.size(0)
        scores, predictions = torch.max(output.data, 1)
        val_total += labels.size(0)
        val_correct += int(sum(predictions == labels))
        val_acc = round((val_correct / val_total) / 100, 2)
    
    # Now display the models accuracy and loss
    train_loss = train_loss/len(train_loader.sampler)
    valid_loss = valid_loss/len(val_loader.sampler)
    print("Epoch:{} TL:{} TA:{}  VL:{}  VA:{}".format(epoch, 
                                                      train_loss,
                                                      acc,
                                                      valid_loss,
                                                      val_acc))
    
    # Save the best possible model
    if valid_loss &lt; valid_loss_min:
        print("Saving Model")
        torch.save(model.state_dict(), 'CLIRF100_Weights.pt')
        valid_loss_min = valid_loss
        
</code></pre>
<p>What am I doing wrong here?<br>
Thank you for any help with this.</p>

yash1994: <p>Hi,</p>
<p>Is there any reason why you are dividing <code>train_correct / train_total</code> by 100 in your accuracy calculation if want your accuracy in percentage then you should multiply your <code>train_correct / train_total</code> by 100. Otherwise, accuracy is as simple as <code>train_correct / train_total</code>.</p>
Branching (across channels) the output of a network into 2 losses and combining to a single loss

Deepak: <p><strong>Context:</strong><br>
The output of my network has 6 channels. In my custom loss function, I break the 6 channels into two groups of 3 channels each. Process the first group using l1_loss (got a scalar output), the second group using bce_loss (got a scalar output), &amp; add the two scalars obtained to get the final loss. Then I use <code>loss.backward()</code>.</p>
<p><strong>Question:</strong><br>
Would autograd handle the gradient flow automatically in above case? If yes, how is autograd doing this?</p>
<p><strong>Code:</strong></p>
<pre><code class="lang-auto">class MyLoss(torch.nn.Module):
    
    def __init__(self):
        super(MyLoss, self).__init__()
		.
		.
	def forward(self, device, output_batch, target_batch):
		
		output_batch_part_1 = output_batch[:,0:3,:,:] # [batch, channel, W, H]
		output_batch_part_2 = output_batch[:,3:6,:,:]
		
		target_batch_part_1 = target_batch[:,0:3,:,:]
		target_batch_part_2 = target_batch[:,3:6,:,:]
		
		loss_term_1 = self.l1_loss_calculation(output_batch_part_1, target_batch_part_1)
		
		#... some processing on output_batch_part_2 ...
		.
		.
		loss_term_2 = self.bce_loss_calulation(output_batch_part_2, target_batch_part_2)
		
		#... some normalizations on loss_term_2...
		.
		.
		final_loss = loss_term_1 + loss_term_2
		
		return final_loss
</code></pre>

ptrblck: <p>Yes, Autograd tracks the indexing operation as well as additions, so your model should get valid gradients as shown in this example:</p>
<pre><code class="lang-python">x = torch.randn(1, 6, 4, 4, requires_grad=True)
x0 = x[:, :3]
x1 = x[:, 3:]

x0.mean().backward()
print(x.grad)
x1.mean().backward()
print(x.grad)
</code></pre>
Debugging training from the remote server

lycaenidae: <p>Hi,</p>
<p>It may seem dummy for some of the developers, but this is the first time that I am dealing with around 50gb training data. Until now, I was only implementing homeworks and can store the data in my local.</p>
<p>Now, I want to debug the baseline code, visualize the data and training&amp; validation curves by using tensorboard (if you are using a different one, I am open to suggestions), but the code is running on the remote server on the GPU. The dataset is available on the remote server. Since I am using mac there is no chance to run in my local the code also store the dataset.</p>
<p>I am wondering, how do you handle this kind of situations in bigger projects in terms of the scale?</p>
<ul>
<li>How to visualize data</li>
<li>How implementation works (Are you implementing in your local but what about testing?)</li>
</ul>
<p>I am looking for general advice. Could you please help me?</p>
<p>Thanks a lot</p>

ptrblck: <p>If you are working on a remote server (with docker), you could open a specific port (e.g. 6006) and use it to for the tensorboard visualization.</p>
<p>The workflow might differ, but you could of course directly work on the server, with a code editor in the terminal or you could use a remote session with e.g. VSCode and develop of your Laptop.<br>
I’m not familiar with Mac, but Atom should also have this remote feature.</p>
RuntimeError: CUDA error: out of memoryy

Mughees: <p>Hey.</p>
<p>I started working on a new workplace virtual system and did all the necessary installations. But currently, I am unable to create a simple cuda variable. I don’t know what requirement is I am lacking.</p>
<p>I am using Windows Server 2016 and just installed latest official pytorch</p>
<p>Attached is the screenshot and any guidance is highly appreciated.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b866f9d0218d9fe6c65815f1cf877efc7e948eba.png" data-download-href="https://discuss.pytorch.org/uploads/default/b866f9d0218d9fe6c65815f1cf877efc7e948eba" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b866f9d0218d9fe6c65815f1cf877efc7e948eba.png" alt="image" data-base62-sha1="qjitIilJJaLxrTB542CZ1ST2bKa" width="690" height="206" data-dominant-color="090909"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">849×254 9.14 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17c586e56cd637a81fc579933999ef92890de576.png" data-download-href="https://discuss.pytorch.org/uploads/default/17c586e56cd637a81fc579933999ef92890de576" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17c586e56cd637a81fc579933999ef92890de576.png" alt="image" data-base62-sha1="3oibf2Ut09fDMjYTvwN0Wn8SkMm" width="690" height="319" data-dominant-color="070707"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">879×407 11.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
thanks alot.</p>

ptrblck: <p>Could you check the memory usage on the server via <code>nvidia-smi</code> and make sure you have enough device memory left?<br>
If you are using multiple GPUs, you can specify a specific via<br>
<code>x = torch.randn(1, device='cuda:id')</code>, where you would have to replace <code>id</code> with the device ID.</p>
Combining multiple loss functions

mohamed_nabil: <p>I’m making a simple autoencoder on Mnist Digits<br>
So the problem i’m facing is that i’m defining 3 different losses<br>
and when i combine them the loss doesn’t decrease while using only one seems to work fine</p>
<pre><code class="lang-auto">epochs = 3
model_2.train()
for e in range(epochs):
    loss_r = [0,0,0]
    r = 0
    for train , test in zip(train_loader_2,train_loader) :
        images = train[0].view(-1,14*14)
        labels_1 = train[1]
        labels_2 = test[0].view(-1,784)
        optimizer.zero_grad()
        x , x_en , x_dec = model_2(images)
        loss_1 = criterion_1(x,labels_2)
        loss_2 = criterion_2(x_en,labels_2)
        loss_3 = criterion_3(x_dec,labels_1)
        
        torch.autograd.backward([loss_1,loss_2,loss_3])
        
        optimizer.step()
        loss_r[0] += loss_1.item()
        loss_r[1] += loss_2.item()
        loss_r[2] += loss_3.item()
        if r%10 == 9 :
            print("epoch = {} batch = {} final_loss={} aux_loss = {} classification_loss={}"
                  .format(e+1,r+1,loss_r[0]/10,loss_r[1]/10,loss_r[2]/10))
            loss_r = [0,0,0]
        
        r+=1
</code></pre>
<p>what seems to be the problem here</p>

Nikronic: <p>Hi,</p>
<p>I think you need to specifically define the combination.</p>
<aside class="quote no-group" data-username="mohamed_nabil" data-post="1" data-topic="63420">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohamed_nabil/48/12207_2.png" class="avatar"> mohamed_nabil:</div>
<blockquote>
<p>torch.autograd.backward([loss_1,loss_2,loss_3])</p>
</blockquote>
</aside>
<p>For instance,</p>
<pre><code class="lang-auto">loss = loss_1 + loss_2 + loss_3
loss.backward()
</code></pre>
<p>I have never seen the approach you haved used to combine multiple losses so maybe the function you calling is not doing what you expected.</p>
<p>Bests,<br>
Nik</p>

ptrblck: <p>Both approaches should result in accumulated gradients.<br>
Since the gradients are accumulated, you might need to lower the learning rate or scale the losses.</p>

mohamed_nabil: <p>Thanks for your help<br>
can i use the same loss fuction or shoud i define different one for every loss</p>

ptrblck: <p>You can use the same, if you are fine with its setup (e.g. <code>reduction</code>, <code>weight</code>, if passed, etc.).</p>
How pytorch consumes GPU

DoubtWang: <p>How pytorch’s GPU-consuming mechanism works ? When loading a small model (64M storage) during testing, it takes up a lot of GPU (870M GPU).<br>
The architecture of the model is as follows:</p>
<pre><code class="lang-auto">rep_linear.weight    torch.Size([768, 21128])
rep_linear.bias    torch.Size([768])
LayerNorm.weight    torch.Size([768])
LayerNorm.bias    torch.Size([768])
classifier.weight    torch.Size([119, 768])
classifier.bias    torch.Size([119])
</code></pre>
<p>Therefore, I am very confused why does it take up so many GPU resources (nearly ten times)?</p>
<p>Also, I put an embeddings (61M storage) on the GPU and the incredible thing happened as well.</p>
<pre><code class="lang-auto">Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
import torch
CUDA_VISIBLE_DEVICES="0"
torch.nn.Embedding(20000, 768).cuda().requires_grad_(False)
Embedding(20000, 768)
20000*768*4 / 1000 /1000 = 61MB
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/a/daa0ef23a2fbf14232da869b6afa4c17001de662.png" alt="image" data-base62-sha1="vc4T4QqtpiEDIXW5GZzg3TX6qLU" width="550" height="85"></p>
<p>Why does this happen?<br>
I hope you will give your opinions. Thanks.</p>
<p>In addition, for the same matrix, I found that different pytorch versions allocate different GPU sizes.</p>

DoubtWang: <p>However, I find that a small tensor will also take up a lot of gpu space.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b022fc1071face41c51addf8d5c44a4d1dcf3efd.png" alt="image" data-base62-sha1="p8aZjLPgCkg2F2oN5lNJTVFoJ9H" width="560" height="114"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0d2c4e16225ce0b7ba8c1fd5757554cd7a4c8c2.png" alt="image" data-base62-sha1="tNkSPYROLOIezioRzcDmI0Dtc4i" width="645" height="57"></p>
<p>SO, cuDNN library occupies around 750Mb memory. <a href="https://forums.developer.nvidia.com/t/cuda-memory-release/76778/6" rel="noopener nofollow ugc">detail</a></p>
Why need on-hot vector as input for seq2seq model

sgaur: <p>I am just wondering that why we need one-hot vector as an input for seq2seq models.<br>
Why can’t we use basic numeric tokens as an input for encorder or decorder.</p>
<p>Example:</p>
<p>Encorder_input = [3,4,5,1,2,0,0,0] —&gt; Why can’t we use this as input</p>
<p>I see lots of tutorials but all of them input should be one-hot encoded vectors??</p>
<p>Thanks in advance</p>

vdw: <p>The <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">PyTorch Seq2Seq tutorial</a> uses token indices as inputs. That should be the default method for anything using <code>nn.Embedding</code>.</p>
ValueError: Classification metrics can't handle a mix of binary and continuous-multioutput targets

Sarah_W: <p>Hi!</p>
<p>I have some troubles to get sklearn’s cross_val_predict run for my ResNet18 (used for image classification). The scoring function is ‘accuracy’ and I get the error:<br>
ValueError: Classification metrics can’t handle a mix of binary and continuous-multioutput targets.</p>
<p>My net returns the probabilities for each image to belong to one of my ten classes as float - I assume that the scoring is not working as the probabilites are floats… I also tried to use .round() but than I get the error: ValueError: Classification metrics can’t handle a mix of binary and multilabel-indicator targets, as often more than one class is set to one.</p>
<p>Can you help me to solve this problem? Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tom: <p>“Accuracy” sounds like you would be expected to feed in the class number (the one with maximal probability) rather than a vector of probabilities.</p>
<p>Best regards</p>
<p>Thomas</p>

Sarah_W: <p>I solved the problem by writing my own accuracy function which can handle continous-mulitoutput targets. It determines the maximum of the multioutput first.</p>

vidivinc: <p>Could you please explain me how you did that?<br>
I think that I have the same problem but I am not able to fix it.</p>
<p>Really Thanks</p>
Forcing output of torch.nn.linear layer between 0 to 1

spaul13: <p>How to force output of torch.nn.linear() layer between 0 to 1 without any normalization (i.e. sigmoid)</p>
<p>I have 3 output nodes of the linear layer it can have negative value or values are quite different but if I apply any normalizing function specifically sigmoid it forces the output value for all three nodes to be between 0.30 - 0.35 (what i observed)</p>
<p>any suggestions will be highly helpful</p>

JuanFMontesinos: <p>If you network maps those values to 0.30-0.35 it means it doesn’t work. The network should slowly converge to binary values if it is designed to do so.<br>
Be aware it reaches the optimal value.<br>
You can just clamp th values so that values between 0-1 remains the same after the transform. You can rescale the whole interval into 0-1.</p>
How to attach a cost to a graph

George3d6: <p>I have a process that looks something like this:</p>
<pre><code class="lang-auto">module_one = #.... some NN
module_two = #.... some NN

X2 = module_one(X1)
Y = module_two(X2)
</code></pre>
<p>I want to manually get the cost for X2 as resulting from the backwards pass of model one and pass it as the cost to module_two</p>
<p>So something like</p>
<pre><code class="lang-auto">X2.required_grad = True
Y = module_two(X2)
cost = X2.grad.cpu().detach()

module_one.attach_cost(cost)
cost.bakcwards()
module_one_optim.step()
</code></pre>
<p>I am unsure how to accomplish the <code>attach_cost</code> step.</p>
<p>I know that for this simple example I could just combine the forward pass for both modules into one… in practice, due to the structure of my code this is rather hard, so I’d prefer to just manually attach the cost and do the backprop based on that.</p>
<p>Is this possible?</p>

albanD: <p>HI,</p>
<p>Could you define what you mean by “cost” here? Are they gradients?</p>
<p>If you want to backprop each module independently, you can do:</p>
<pre><code class="lang-python">X2_out = module_one(X1)

X2_in = X2_out.detach().requires_grad_()
Y = module_two(X2_in)
loss = crit(Y, target)


grad_X2 = autograd.grad(loss, X2_in) # backprop module 2
grad_X1 = autograd.grad(X2_out, X1, grad_X2) # backprop module 1

</code></pre>

George3d6: <p>Yes, I meant gradients: <code>cost = X2.grad.cpu().detach()</code></p>
<p>I hope my understanding here is not fundamentally broken and those are the same thing … ?<br>
Is there a difference between taking the the <code>.grad</code> from the inputs of the first module (if I set those to require gradient) and doing it via <code>autograd.grad</code>?</p>
<p>But anyway, I’ll try doing the backprop on the first module via <code>autograd.grad</code> and passing the <code>.grad</code> from the inputs of the second module (after backprop has been done on that the usual way).</p>

albanD: <blockquote>
<p>Is there a difference between taking the the <code>.grad</code> from the inputs of the first module (if I set those to require gradient) and doing it via <code>autograd.grad</code> ?</p>
</blockquote>
<p>The value will be the same but it is trickier to do with .backward() because you need to make sure to reset the value properly and saving in .grad a Tensor that requires gradient is not recommended.</p>

George3d6: <p>Alright, well I guess both of those are analogous for my purposes.<br>
But… are the original inputs required to do <code>autograd.grad</code> ? Is it not possible to just call is in a similar way to <code>backward</code> where you just pass the loss and not the inputs that lead to it ?</p>

albanD: <p>The whole point of autograd.grad is that you specify what you want gradient for. If you want all, use .backward(). Otherwise how do you know which tensors the gradients returned by autograd.grad correspond to?</p>
When using DataParallel, the other half of the batch disappeared

xinsss: <p>I have two GPUs and I am trying to use <code>torch.nn.DataParallel</code> to wrap my model. So my model could run on two GPUs. I successfully did this many times before. However, something strange happened.</p>
<p>When the model is in training, I tried to print the input tensor inside the model’s forward function. There is only half batch size data there. For example, the batch_size =16, the model is only using 8 in training.</p>
<p>When the model is in eval mode, I also tried to print the input tensor inside the model’s forward function. <strong>Sometimes</strong> it will print twice (each with batch of 8) as expected.</p>
<p>Any ideas?</p>
<p>Here is my code</p>
<pre><code class="lang-auto">...
device = torch.device("cuda:0")
model.to(device)
model = torch.nn.DataParallel(model, device_ids=[0, 1])

# Epochs
for _ range(int(num_train_epochs)):
    for step, batch in enumerate(epoch_iterator):
        model.train()
        loss = model(**inputs) # I compute the loss inside forward for balancing GPU memory.
....   
</code></pre>

jmandivarapu1: <aside class="quote no-group" data-username="xinsss" data-post="1" data-topic="71685">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/82dd89/48.png" class="avatar"> xinsss:</div>
<blockquote>
<p>For example, the batch_size =16, the model is only using 8 in training.</p>
<p>When the model is in eval mode, I also tried to print the input tensor inside the model’s forward function. <strong>Sometimes</strong> it will print twice (each with batch of 8) as expected.</p>
<p>Any ideas?</p>
</blockquote>
</aside>
<p>I think may be because of the nn.DataParallel it got split into two GPUS 8 in each</p>

xinsss: <aside class="quote no-group" data-username="jmandivarapu1" data-post="2" data-topic="71685">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jmandivarapu1/48/3544_2.png" class="avatar"> jmandivarapu1:</div>
<blockquote>
<p>thi</p>
</blockquote>
</aside>
<p>That is what I wanted it to do. If it did split the batch to two GPUs (each has batch of 8), then the print function in the model’s forward function will be called twice to print batch of 8 each time.</p>

Balamurali_M: <p>Did you identify the problem ? I am facing similar issues.</p>

xinsss: <p>Yes, I was able to solve the issue. I found my input tensors have different size. So the DataParallel couldn’t properly split the input tensors into GPUs. So maybe check if your inputs size is consistent?</p>
How to output the loss gradient backpropagation path through a PyTorch computational graph

spaul13: <p>I have implemented a new loss function in PyTorch.</p>
<pre><code class="lang-auto">#model_1 needs to be trained
outputs = model_1(input)
loss = myloss(outputs,labels)

#output is how much to resize an image
#label give the image file index
#Below I am explaining myloss() function
org_file_name = pic_ + str(labels[0]) + ".png"
new_image = compress(org_image,outputs)
accuracy_loss = run_pretrained_yolov3(org_image, new_image)

#next two lines are modifying the same DAG
prev_loss = torch.mean((outputs-labels)**2)
new_loss = (accuracy_loss/prev_loss.item())*prev_loss
new_loss.backward()
</code></pre>
<p>Can anyone plz help me suggesting  <strong>how can I know regarding how the loss gradient backpropagation through the computational graph?</strong></p>
<p>[i.e., Actually, inside the myloss() function, I used some other pre-trained model applied in testing mode to get the difference or final loss value.] Now I want to know whether my new_loss.grad backpropagated through model1 or first through yolov3 then through model1? pretrained yolov3 is used on testing mode only.</p>
<p>I have tried tensorboard, it’s not providing me that option. Any suggestions will be highly helpful.</p>

albanD: <p>Hi,</p>
<p>The back-propagation will just happen in the reverse order of your forward function.<br>
So it will go through these models in reverse order that you call them in the forward.</p>

spaul13: <p>as the yolov3 is being used in inference mode with torch.no_grad() so that means no computational graph will be created for that. How it will compute the gradient for yolov3<br>
as Loss = f(x) - f(k(g(x))) where f = yolov3 model, g = classification model I want to train, k= compression, x = original image and k(g(x)) = new compressed image<br>
now loss gradient with respect to one parameter,<br>
dLoss/dW = df/dk * dk/dg *dg/dW</p>
<p>how df/dk and dk/dg will be computed or represented in computational graph as those are not traditional differentiable operators.</p>

albanD: <p>Well if you run f in no_grad mode, then your loss won’t require gradients and you won’t be able to run backprop.<br>
If you want to learn g properly, you will need to run the k and f applied after it in a differentiable manner to be able to get gradient.</p>
[ImageFolder] How to load image data with flexible label definition

rightly0716: <p>I am following the ImageFolder+DataLoader tutorial to load data and assign labels. The code is:</p>
<pre><code class="lang-auto">normalize = T.Normalize(mean=[0.4, 0.4, 0.4], std=[0.2, 0.2, 0.2])
transform  = T.Compose([
         T.RandomResizedCrop(224),
         T.RandomHorizontalFlip(),
         T.ToTensor(),
         normalize,
])
dataset = ImageFolder('data/dogcat/', transform=transform)
dataloader = DataLoader(dataset, batch_size=3, sampler=sampler, num_workers=0, drop_last=False)
</code></pre>
<p>It works perfect when there are two folders under the targeted dir like this:</p>
<pre><code class="lang-auto">data/dogcat/
|-- cat
|   |-- cat.12484.jpg
|   |-- cat.12485.jpg
|   |-- cat.12486.jpg
|   `-- cat.12487.jpg
`-- dog
    |-- dog.12496.jpg
    |-- dog.12497.jpg
    |-- dog.12498.jpg
    `-- dog.12499.jpg
</code></pre>
<p>Now, I collect more dog/cat images and put them in this way:</p>
<pre><code class="lang-auto">data/dogcat/
|-- cat
|   |-- cat.12484.jpg
|   |-- cat.12485.jpg
|   |-- cat.12486.jpg
|   `-- cat.12487.jpg
`-- dog
    |-- dog.12496.jpg
    |-- dog.12497.jpg
    |-- dog.12498.jpg
    `-- dog.12499.jpg
|-- newcat
|   |-- newcat.12484.jpg
|   |-- newcat.12485.jpg
`-- newdog
    |-- newdog.12496.jpg
    |-- newdog.12497.jpg
</code></pre>
<p>May I ask:</p>
<ol>
<li>Whether there is a quick way I can easily switch between different loading strategies, such as loading (cat vs “dog+newdog” to model), or loading (“cat+newcat” vs “dog+newdog” to model)</li>
<li>I think the newdog and newcat have higher quality but fewer numbers. Is there a way I can assign higher weights to them when trying different loading ideas as 1 shows? The difficult I am facing is, such overweighting is not on the entire dog or cat but only on the new subclass. I find it hard to implement using sampler in DataLoader as: <a href="https://discuss.pytorch.org/t/balanced-sampling-between-classes-with-torchvision-dataloader/2703/2" class="inline-onebox">Balanced Sampling between classes with torchvision DataLoader</a>
</li>
</ol>
<p>Thanks so much!</p>
Pytorch distributed with gloo backend send float32 or float64

Liron_Mor_Yosef: <p>Hello,<br>
I am implementing MPI and compare to pytorch distributed,<br>
it’s seems that there is mismatch between my implemetation and the one of pytorch with gloo.</p>
<p>does pytorch distributed send float32 or float64 tensors?</p>

mrshenli: <aside class="quote no-group" data-username="Liron_Mor_Yosef" data-post="1" data-topic="74769">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/liron_mor_yosef/48/12953_2.png" class="avatar"> Liron_Mor_Yosef:</div>
<blockquote>
<p>does pytorch distributed send float32 or float64 tensors?</p>
</blockquote>
</aside>
<p>It depends on what the scalar type of the tensor you passed to the communication API. Check out <a href="https://github.com/pytorch/pytorch/blob/a8ca340ad6faa8ebb51fc364b891985843b5fe14/torch/lib/c10d/ProcessGroupGloo.cpp#L36-L61" rel="noopener nofollow ugc">this</a>.</p>