akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Different behaviors: nn.Module vs ScriptModule

Chenchao_Zhao: <p>I noticed some annoying differences between nn.Module and ScriptModule</p>
<p><code>torch.__version__ = 1.4.0</code></p>
<p>Example:</p>
<pre><code class="lang-auto">class DataClass(nn.Module):
    my_data: List[int]
    
    def __init__(self):
        super(DataClass, self).__init__()
        self.my_data = []
    
    def forward(self, new: int):
        self.my_data.append(new)
        return self.my_data
</code></pre>
<p>nn.Module:</p>
<pre><code class="lang-auto">dat = DataClass()
dat(0)
dat.my_data.append(1)
print(dat.my_data)
&gt;&gt;&gt;  [0, 1]
</code></pre>
<p>ScriptModule</p>
<pre><code class="lang-auto">dat = DataClass()
script_dat = torch.jit.script(dat)
script_dat(0)
script_dat.my_data.append(1)
print(script_dat.my_data)

&gt;&gt;&gt;  [0]
</code></pre>
How do I run the inference in model I have written in pytorch on glow?

tazz: <p>Hi,</p>
<p>I exported a mobilenetv2 model form pytorch that I have trained to do binary classification in the onnx format. I understand the the image classifier example is used to load the model and the image and test it.<br>
But are there any other steps or sample code you have with better documentation that will help me understand to utilize the model better?</p>
<p>I understand you have the model-compiler tool to compile the model to an executable. Also is there a way to deploy this model on GPU? What steps do I need to follow for that?</p>

jfix: <p>Hi, are you interested in ahead-of-time compilation (called creating a “bundle” in Glow terminology), or in JIT compilation? You can use <code>image-classifier</code> for both purposes, or <code>model-compiler</code> for ahead of time.</p>
<p>There are some examples for <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md" rel="nofollow noopener">AOT here</a> and <a href="https://github.com/pytorch/glow/blob/master/docs/Testing.md" rel="nofollow noopener">JIT here</a>.</p>
<p>For GPUs, the best we have right now is our OpenCL backend. Otherwise there’s not much open source GPU support currently, as we have focused Glow more on supporting specialized accelerator HW for inference such as Intel’s NNPI, Habana’s Goya, etc.</p>

tazz: <p>Hi <a class="mention" href="/u/jfix">@jfix</a> ,</p>
<p>I was able to execute the image classifier tool. Its seems that there seems to be a lot more information for AOT.  Although a basic command line example with the parameters that the image-classifier to might make the process easier. Thanks!</p>

jfix: <p>I’d suggest taking a look at the <a href="https://github.com/pytorch/glow/tree/master/examples/bundles/resnet50" rel="nofollow noopener">Resnet50 bundle example</a> – it simply has a <code>main.cpp</code> and <code>CMake</code> that drives all of the commands you need to create an AOT bundle and run it.</p>
Do we have a set_default_dtype in libtorch

7starsea: <p>In python, we have torch.set_default_dtype, but I did not find a similar method in libtorch, do we have one?</p>
<p>Thanks.</p>

dfalbel: <p>It’s defined here: <a href="https://github.com/pytorch/pytorch/blob/master/c10/core/DefaultDtype.h#L10" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/c10/core/DefaultDtype.h#L10</a><br>
And can be accessed trough <code>torch::set_default_dtype</code>.</p>
Kinetics400 Dataset

ilkarman: <p>The Kinetics400 VisionDataset seems very specific. For example, returning audio and requiring step_between_clips (basically forcing a sliding-window).<br>
Quoting</p>
<blockquote>
<p>To give an example, for 2 videos with 10 and 15 frames respectively, if <code>frames_per_clip=5</code><br>
and <code>step_between_clips=5</code>, the dataset size will be (2 + 3) = 5, where the first two<br>
elements will come from video 1, and the next three elements from video 2.<br>
Note that we drop clips which do not have exactly <code>frames_per_clip</code> elements, so not all<br>
frames in a video might be present.</p>
</blockquote>
<p>The use case of taking two videos with 10, 15 frames respectively and returning (assuming clip-length of 5) a random temporal sample of frames [2,3,4,5,6] from video 1 and [11,12,13,14,15] from video 2 seems much more common (at least for i3d, r2+1d, etc models).</p>
<p>Are there any examples of converting the kinetics data-loader to work in these cases? To randomly sample one clip per video not do a sliding window?</p>
<p>I have found <a href="https://github.com/antoine77340/MIL-NCE_HowTo100M/blob/9ba876bd67160e24a5ce379a07d18a8036be0d36/video_loader.py" rel="noopener nofollow ugc">this example</a> using ffmpeg-python would be great to use torchvision:</p>
Batch Permutation op problem when converting caffe2 to onnx

ash95: <p>As the title suggests, I have a caffe2 model (model_init.pb and model.pb) and wanted to convert it into onnx via the following code snippet.</p>
<pre><code class="lang-auto">import onnx
import caffe2.python.onnx.frontend
from caffe2.proto import caffe2_pb2
from PIL import Image
import numpy as np

# We need to provide type and shape of the model inputs, 
# see above Note section for explanation
data_type = onnx.TensorProto.FLOAT

IMAGE_LOCATION = "/home/an1/caffe2_model/input.jpg"
INIT_NET = "/home/an1/caffe2_model/model_init.pb"
PREDICT_NET = "/home/an1/caffe2_model/model.pb"

# Read single image
img = Image.open(IMAGE_LOCATION)
img = np.array(img)

# Convert HWC -&gt; CHW
img = img.swapaxes(1, 2).swapaxes(0, 1)

# Convert CHW -&gt; NCHW
img = np.array([img])

# Im info N x 3 tensor of (height, width, scale)
im_info = np.reshape(np.array([img.shape[2], img.shape[3], 1.0]), (1,-1))

value_info = {
    'data': (data_type, img.shape),
    'im_info': (data_type, (1,3))
}

predict_net = caffe2_pb2.NetDef()
with open(PREDICT_NET, 'rb') as f:
    predict_net.ParseFromString(f.read())

init_net = caffe2_pb2.NetDef()
with open(INIT_NET, 'rb') as f:
    init_net.ParseFromString(f.read())

onnx_model = caffe2.python.onnx.frontend.caffe2_net_to_onnx_model(
    predict_net,
    init_net,
    value_info,
)

onnx.checker.check_model(onnx_model)
</code></pre>
<p>However, I’m encountering an error with the Batch Permutation op:<br>
<code>RuntimeError: [enforce fail at batch_permutation_op.cc:64] X.dim32(0) &gt; 0. 0 vs 0</code></p>
<p>Does anyone have an idea what could be causing this?</p>

9thDimension: <p>I’m trying to do the same thing, and getting the same error. Did you ever fix this?</p>
<p>EDIT: The solution is to use a larger image input size, I suspect because otherwise the successive pooling operations will reduce it to have a negative size or something. Also it must be a multiple of 32.</p>

GJoshi27: <p>Did anyone find the solution for this? I am still facing it.</p>
Error with Dataloader with multiple classes

eranb: <p>Hi everyone,<br>
My data is composed of  3 classes:<br>
‘x_ray’: [NumpyToTorch(image) for image in x_ray],<br>
‘bone1’: [NumpyToTorch(image) for image in bone1],<br>
‘bone2’: [NumpyToTorch(image) for image in bone2]<br>
I create a data loader and want to loop on this data for my network.<br>
dataloader = DataLoader(data, batch_size=batch_size,<br>
shuffle=False)</p>
<p>I get the error:<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
KeyError: 0</p>
<p>Someone can explain to me how to work with a data loader with multiple classes.</p>
<p>Best,<br>
Eran</p>

ptrblck: <p>Could you try to index the <code>Dataset</code> directly with index0 and see, if it would raise an error?</p>
<pre><code class="lang-python">out = data[0]
</code></pre>
<p>should probably yield a better error message.</p>

eranb: <p>Yes, thank you.<br>
I figure it out, the error was related to the dataloader.</p>
MLPerf ResNet-50 pre-trained model

gilsho: <p>Hi there,</p>
<p>I wonder, have someone migrated MLPerf ResNet-50 checkpoint to PyTorch?<br>
It achieves top-1 accuracy of 76.456%, and it will help me a lot having that checkpoint as a baseline.</p>
<p><a href="https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection" rel="nofollow noopener">https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection</a></p>
<p>Thanks!</p>

ptrblck: <p>I don’t know, how these mlperf models are stored or used, but NVIDIA provides Resnet50v1.5 with a similar or higher Top1 accuracy <a href="https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5#training-accuracy-nvidia-dgx-1-8x-v100-16g">here</a>.</p>

gilsho: <p>Thanks! I will definitely check it out.</p>
<p>While we’re at it, does PyTorch has something for MobileNet-V1?<br>
Again, MLPerf offers one with top-1 accuracy of 71.676%, but I can’t find any pre-trained model for PyTorch with this accuracy. Any suggestions?</p>
<p>And thanks again!</p>

gilsho: <p>If someone will come across this post, I managed to migrate the weights from MLPerf to PyTorch models. They can be found here: <a href="https://github.com/gilshm/mlperf-pytorch" rel="nofollow noopener">https://github.com/gilshm/mlperf-pytorch</a></p>
No gradient for making the max value 1 and others 0

cqhoneybear: <p>Hi,</p>
<p>I would like to keep the maximum value to be 1 and others to be 0 in a tensor. But the backward function does not produce gradient for the tensor.</p>
<p>I attached the example. Thanks!</p>
<p>import torch</p>
<p>x = torch.Tensor([[0, 1, 2, 3]]).requires_grad_()</p>
<p>b = (x==x.max(dim=1,keepdim=True)[0]).type(torch.FloatTensor).requires_grad_()</p>
<p>out = b.sum()</p>
<p>out.backward()</p>
<p>x.grad <span class="hashtag">#why</span> is there no grad for x?</p>
<p>b.grad</p>

albanD: <p>Hi,</p>
<p>The problem is that you have a non-floating point Tensor that you set by hand to <code>requires_grad_()</code> again.<br>
You will need to only use differentiable ops here.</p>
<p>To keep the values between 0/1 I would recommend the softmax function that will make sure you get non-zero gradients everywhere (but you won’t be able to reach 0 or 1).<br>
Or you can use clamp. But then the gradients for the values &lt;0 and &gt;1 will be 0.</p>

cqhoneybear: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="86864">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>clamp</p>
</blockquote>
</aside>
<p>Thanks <a class="mention" href="/u/alband">@albanD</a>.  Does it mean “b = (x==x.max(dim=1,keepdim=True)[0]).type(torch.FloatTensor)” is actually not differentiable and so that  x.grad is none?</p>
<p>And is there a way to keep the maximum value to be 1 and the others to be 0 in a tensor but the tensor still could have gradient? I found a similar question here <a href="https://discuss.pytorch.org/t/set-max-value-to-1-others-to-0/44350/2" class="inline-onebox">Set Max value to 1, others to 0 - #2 by KFrank</a> but it seems one hot is also not differentiable.</p>

albanD: <blockquote>
<p>“b = (x==x.max(dim=1,keepdim=True)[0]).type(torch.FloatTensor)” is actually not differentiable and so that x.grad is none?</p>
</blockquote>
<p>If returns a Tensor that does not require gradients even though the input does. This means it is not differentiable yes.</p>
<blockquote>
<p>And is there a way to keep the maximum value to be 1 and the others to be 0 in a tensor but the tensor still could have gradient?</p>
</blockquote>
<p>If you think about gradient as “how much the output change if I change this input a little”, then you can see that if the output of your function is “the maximum value to be 1 and the others to be 0”, then all the gradients will be 0. Because when you change any input a little (if the maximum one does not change), then all the outputs will remain the same.</p>

cqhoneybear: <p>Very clear explanation, thanks <a class="mention" href="/u/alband">@albanD</a> !</p>
Backpropagation not propagating through all computations

nierth: <p>Hi,</p>
<p>I am having the weird problem that backpropagation does not update all parameters as expected. The important code snippet is shown below. All actor/critic/env nets are simple MLPs. I checked the weights by outputting the sum of certain layers and the <code>critic_1 </code> / <code>critic_2 </code> networks get properly updated. The  weights of the  <code>env </code> net however stay the same all the time. Am I doing something completely wrong? I always thought that setting the <code>requires_grad</code> flag to true is sufficient to enable backpropagation.</p>
<p>Any help is highly appreciated</p>
<p>P.S. I also don’t understand why I have to run the <code>state = state.to(device)</code> line. Right before returning the state in the <code>env.step(last_action)</code> method the state is still on the GPU, but after returning it it is on the CPU</p>
<pre><code class="lang-auto"># Sample replay buffer, all elements are detached by using the sample function
last_state, last_action, next_state, reward, done = replay_buffer.sample(self.batch_size)

# enable gradient computation
last_state.requires_grad_(True)
last_action.requires_grad_(True)

# run
input_seed = env.get_input_seed().repeat(len(last_state)).unsqueeze(1)
env.set_state(last_state)
env.set_input_seed(input_seed)
state, _, _ = env.step(last_action)
state = state.to(device) # why have to move to device again?

action = self.actor(state)

with torch.no_grad():
   # Select action according to policy
   next_action = (self.actor_target(next_state))
   reward = reward

   # Compute the target Q value
   target_Q1 = self.critic_target_1(next_state, next_action)
   target_Q2 = self.critic_target_2(next_state, next_action)
   target_Q = torch.min(target_Q1, target_Q2)
   target_Q = reward + (1-done) * self.gamma * target_Q

# Get current Q estimates
current_Q1 = self.critic_1(state, action)
current_Q2 = self.critic_2(state, action)

# Compute critic loss
critic_loss = F.mse_loss(current_Q1, target_Q) + F.mse_loss(current_Q2, target_Q)

# Optimize the critic and environment
self.critic_optimizer.zero_grad()
critic_loss.backward()
self.critic_optimizer.step()
</code></pre>

rajcscw: <p>There are couple of problems.</p>
<p>First, I do not see that your env model is attached to any kind of optimizer. Questions that arise to me are: where is the forward pass of the model?  Are you sure the loss you apply at the end is connected to the computation graph of the env model in some way?</p>
<p>Second,<br>
You have the context <code>with torch.no_grad()</code> which would disable gradient computation. When you are in training mode, it should not be used at all.</p>
<p>Hope this helps!</p>

nierth: <p>thanks, forgot to attach the parameters to the optimizer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to gather a list of strings from different rank machines in DDP mode with NCCL backend?

sherylwang: <p>Hi,<br>
These days I’ve accelerated the training of models with DistributedDataParallel. NCCL is used as the backend of torch.distributed. Currently, I try to do validation with a list of strings stored in the memory. However, with the multi-process mechanism, it’s hard to share the list across different ranks than in DP mode. Is there any good way to solve the problem?</p>

mrshenli: <p>There was an PR to provide such a feature for general Python objects, but not landed yet. You can copy that code for now.</p>
<aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/pull/28811/files" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Pull Request">
    <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/pull/28811" target="_blank" rel="nofollow noopener">[distributed] implement all_gather for arbitrary python objects</a>
    </h4>

    <div class="branches">
      <code>pytorch:gh/rohan-varma/25/base</code> ← <code>pytorch:gh/rohan-varma/25/head</code>
    </div>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-10-28" data-time="21:40:01" data-timezone="UTC">09:40PM - 28 Oct 19 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/rohan-varma" target="_blank" rel="nofollow noopener">
          <img alt="rohan-varma" src="https://avatars2.githubusercontent.com/u/8039770?v=4" class="onebox-avatar-inline" width="20" height="20">
          rohan-varma
        </a>
      </div>

      <div class="lines" title="9 commits changed 2 files with 93 additions and 0 deletions">
        <a href="https://github.com/pytorch/pytorch/pull/28811/files" target="_blank" rel="nofollow noopener">
          <span class="added">+93</span>
          <span class="removed">-0</span>
        </a>
      </div>
    </div>

  </div>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


sherylwang: <p>Thanks for your reply. This design is interesting. <a class="mention" href="/u/mrshenli">@mrshenli</a></p>
Load own model on android app

GreenSun: <p>Hi,<br>
I want create my Android app by using my convolutional model written in Pytorch 1.4.0.<br>
I was following the guidelines written in <a href="https://pytorch.org/mobile/android/" rel="nofollow noopener">https://pytorch.org/mobile/android/</a>, in detail I was following example code indicating here: <a href="https://github.com/pytorch/android-demo-app/tree/master/HelloWorldApp" rel="nofollow noopener">https://github.com/pytorch/android-demo-app/tree/master/HelloWorldApp</a>.</p>
<p>I have problem when I import my model. Error is:</p>
<p>Process: org.pytorch.helloworld, PID: 15383<br>
java.lang.RuntimeException: Unable to start activity ComponentInfo{org.pytorch.MainActivity}: com.facebook.jni.CppException: [enforce fail at inline_container.cc:143] . PytorchStreamReader failed reading zip archive: failed finding central directory<br>
(no backtrace available)</p>
<p>on Android studio (versione 4.0). I read that this can be due to erroneous procedure to save model_traced.pt.</p>
<p>I saved Mymodel.pt  as below:</p>
<p>model.load_state_dict(torch.load(Mymodel.pt)<br>
model.eval()<br>
example_inputs=torch.rand(1,1,66,100)<br>
Mymodel_traced=torch.jit.trace(model,example_inputs=example_inputs)<br>
Mymodel_traced.save(“model_traced.pt”)</p>
<p>In my gradle I have:</p>
<p>dependencies {<br>
implementation ‘androidx.appcompat:appcompat:1.1.0’<br>
implementation ‘org.pytorch:pytorch_android:1.4.0’<br>
implementation ‘org.pytorch:pytorch_android_torchvision:1.4.0’<br>
}</p>
<p>How can I solve? Thank you.</p>
Distance between two sets of neural net weights

belhal: <p>Hi all,</p>
<p>Is there a quick way to access (then plot) the l2 norm of the distance between the initial set of weights w_0 and a set of weights at iteration t, w_t ?<br>
I’d like to access this quantity to then plot it in 2D (along a random dimension).</p>
<p>I believe <code>net.parameters()</code> where net is defined as below</p>
<pre><code class="lang-auto">class MnistNetSmall(nn.Module):
    def __init__(self):
        super(MnistNetSmall, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.fc1 = nn.Linear(20 * 12 * 12, 10)
        #self.fc1 = nn.Linear(28 * 28, 10)


    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 20 * 12 * 12)
        #x = x.view(-1, 28 * 28)
        x = self.fc1(x)
        return F.log_softmax(x, dim=1)

net = MnistNetSmall()

</code></pre>
<p>is what I need. Though this quantity is a generator object.</p>
<p>Any ideas how to collect <code>net.parameters()</code> at each iteration for plotting purposes?</p>
<p>Thanks all</p>

Chirath_Pansilu: <p>You Can use this approach:</p>
<p>If you want conv1 layers weights:</p>
<pre><code class="lang-auto">conv_weights = net.conv1.weight
</code></pre>
<p>or fc1 layers weights:</p>
<pre><code class="lang-auto">fc1_weights = net.fc1.weight
</code></pre>
<p>as these layers are defined in<code> __init__</code> you can access them by name and get weights</p>
<p>Hope this helps.</p>

belhal: <p>Indeed that was the syntax.<br>
Though, when I try to store the weights of say my first fully connected layer as in</p>
<pre><code class="lang-auto">all_weights = []

for epoch in range(start_epoch, start_epoch+epochs):
    train_loss = 0
    correct = 0
    total = 0
    ip_loss = []; 
    grad_loss = []
    for batch_idx, (inputs, targets) in enumerate(trainloader):
        inputs, targets = inputs.to(device), targets.to(device)
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        _, diag_args = optimizer.step()
        ip_loss.append(diag_args['ip_loss'])
        grad_loss.append(diag_args['grad_loss'])
        train_loss += loss.item()
        _, predicted = outputs.max(1)
        total += targets.size(0)
        correct += predicted.eq(targets).sum().item()
        logger_diag.append([lr, momentum,newstat, train_loss, 100.*correct/total, np.sum(ip_loss)])

        #Append weights of fc1 through the iterations
        all_weights.append(net.fc1.weight.data.detach().numpy())
</code></pre>
<p>Those weights appear to be constant through the iterations/epochs.<br>
The weights should be updated at each <code>optimizer.step()</code><br>
Any idea why?</p>

Chirath_Pansilu: <p>Are you sure the model is training ? Does losses go down as training ?</p>

belhal: <p>Yes training is happening.<br>
Copying the array of weights and appending made it work.<br>
Don’t know why but I guess copy() is always a good friend.</p>
<pre><code class="lang-auto">a  = net.fc1.weight.data.detach().numpy().copy()
all_weights.append(a)
</code></pre>
Modify DataLoader to make request to CPU

Tenese: <p>Hello, I’m new to PyTorch. I want to modify the Data Loader to send a request to CPU to do data augmentation on a batch then send back to the GPU for training. However, I do not know how to approach this issue. Is there any tutorial on this?</p>
<p>Any help is appreciated. Thank you.</p>

Nikronic: <p>Hi,</p>
<p>Dataloaders by default run all operations in CPU unless you explicitly do otherwise. For instance, you can combine your augmentations using <code>torchvision.transforms</code> then all generated images have been produced in CPU.</p>

Tenese: <p>Hello, will the augmentation in CPU run in parallel with the training process in GPU?</p>
<p>Thanks for the reply!</p>

Nikronic: <p>Yes exactly, that is why in <code>Dataset</code> definition, there is a function called <code>__getitem__</code> which only returns a single item of dataset after augmentation. Then dataloaders take advantage of this and can run this method in parallel. And after this, you convert tensors in batch, obtained from dataloader to tensors in GPU so this pipeline will be held.</p>

Tenese: <p>Thanks for the explanation.</p>
CycleGAN Error during training - Trying to backward through the graph a second time

chaslie: <p>Hi,</p>
<p>I hope you are all well…</p>
<p>I am trying to write a CycleGAN,</p>
<p>in the first epoch i want to use stable data, after this I would like to swich to the outputs of the 2 generators.</p>
<p>When i do this i get the following error:</p>
<pre><code class="lang-auto">
**RuntimeError**: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>For the First Epoch<br>
Generator A</p>
<blockquote>
<pre><code>        Gen_model_A.zero_grad()
        loss_A,out_A = loss_fn_I(Gen_model_A, real_data_A,lab_A)
        temp=loss_A.detach().cpu().numpy()
        if np.isnan(temp)==True:
            exit()
        #    print(out.element_size()*out.nelement())
        D_G_z2A = out_A.mean().item()
        loss_A.backward(retain_graph=True)
        optimizer_gen_A.step()
</code></pre>
</blockquote>
<p>Generator B:</p>
<blockquote>
<pre><code>        Gen_model_B.zero_grad()
        loss_B,out_B = loss_fn(Gen_model_B, real_data_B,lab_B)
        temp=loss_B.detach().cpu().numpy()
        if np.isnan(temp)==True:
            exit()
        #    print(out.element_size()*out.nelement())
        D_G_z2B = out_B.mean().item()
        loss_B.backward(retain_graph=True)
        optimizer_gen_B.step()
</code></pre>
</blockquote>
<p>Then from epoch 2 onwards:<br>
Generator A:</p>
<blockquote>
<pre><code>        Gen_model_A.zero_grad()
        loss_A,out_A = loss_fn(Gen_model_A, out_B,lab_A)
        temp=loss_A.detach().cpu().numpy()
        if np.isnan(temp)==True:
            exit()
        #    print(out.element_size()*out.nelement())
        D_G_z2A = out_A.mean().item()
        loss_A.backward()
        optimizer_disc_A.step()
</code></pre>
</blockquote>
<p>Generator B:</p>
<blockquote>
<pre><code>        Gen_model_B.zero_grad()
        loss_B,out_B = loss_fn_I(Gen_model_B, out_A,lab_B)
        temp=loss_B.detach().cpu().numpy()
        if np.isnan(temp)==True:
            exit()
        #    print(out.element_size()*out.nelement())
        D_G_z2B = out_B.mean().item()
        loss_B.backward()      &lt;-----   This is where i get the error
        optimizer_disc_B.step()
</code></pre>
</blockquote>
<p>I think it maybe that i am not passing the graph between eoch 1 &amp; 2 and i was wondering if the problem would be solved using <code>hidden.detach_()</code>, but i cannot find anything in the pytorch documentation, can anyone help please?</p>
<p>Many thanks,</p>
<p>Chaslie</p>
Cant import torch from pycharm

SilverHorse: <p>hello every one, im new for useing pytorch.<br>
i was plaing with pytorch on jupyter notebook. and it’s seems i have it all.</p>
<ol>
<li>torch.cuda.is_available() = True</li>
<li>torch.<strong>version</strong> = 1.5</li>
<li>torch.version.cuda = 10.2</li>
<li>and i have an nvidia cpu</li>
</ol>
<p>so i should have all the libraries i need on this computer.<br>
so i wanted to move to pycharm to code there. but I cant import torch on pycharm.<br>
anyone know why?</p>

SilverHorse: <p>it’s seem i need to change the python interpreter to where i instale pytorch.<br>
how i can see the path im useing and how do i fing the conda path?</p>

Rahul_pillai: <div class="youtube-onebox lazy-video-container" data-video-id="56bPIGf4us0" data-video-title="Pycharm Tutorial #1 - Setup &amp; Basics" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=56bPIGf4us0" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/56bPIGf4us0/maxresdefault.jpg" title="Pycharm Tutorial #1 - Setup &amp; Basics" width="690" height="388">
  </a>
</div>

<p>I recommend you to go through this.</p>
<p>There’s also a inbuilt IDE called spyder which comes with the conda package.<br>
Personally i like spyder.<br>
if you want to access spyder go to the anaconda prompt and type ‘spyder’.</p>

Rahul_pillai: <p>If you doing ML or data science spyder is the better option, you can use R, you can have figures within window and you have easier data viewers. Pycharm is more for programming if you are into something other than data science go for pycharm.<br>
<a href="https://www.quora.com/What-are-the-pros-and-cons-of-Spyder-versus-PyCharm-as-Python-IDEs#:~:text=They%20are%20different%20because%20they,with%20ssh%2C%20with%20shell%20console" rel="nofollow noopener">https://www.quora.com/What-are-the-pros-and-cons-of-Spyder-versus-PyCharm-as-Python-IDEs#:~:text=They%20are%20different%20because%20they,with%20ssh%2C%20with%20shell%20console</a>.</p>
Matrix Multiply along one dimension, broadcasting batches

Jonas_F: <p>Hi,</p>
<p>I am currently trying to do a matrix multiplication of two matrices <em>A</em> of size <em>b x l x k</em> and <em>B</em> of size <em>l x k x p</em>, such that I get a matrix <em>C</em> of size <em>b x l x p</em> with <em>C[i,j,:] = A[i,j,:] dot B[j,:,:], i=1,…,b</em> and <em>j=1,…,l.</em><br>
Essentially what I need is a mixture of broadcasted matmul and bmm, that is, I want a matrix multiplication of <em>A</em> and <em>B</em> for each slice <em>j=1,…,l</em> and broadcasted to the batches <em>b</em>.</p>
<p>Thanks in advance!</p>
Why use a different tensor for computing gradients of another tensor

paul_shuvo: <p>I’m very new to pytorch and this might come as a stupid question; while going through the <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="noopener nofollow ugc">autograd</a> tutorial I came to this section where it’s written-</p>
<blockquote>
<p>Now in this case  <code>y</code>  is no longer a scalar.  <code>torch.autograd</code>  could not compute the full Jacobian directly, but if we just want the vector-Jacobian product, simply pass the vector to  <code>backward</code>  as an argument:</p>
</blockquote>
<pre><code class="lang-auto">v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(v)

print(x.grad)
</code></pre>
<p>Now, I don’t understand why are we using a different tensor <code>v</code> in order to compute gradients for <code>x</code>.</p>

ezyang: <p>The short answer is, you can’t compute a gradient unless it is with respect to a scalar quantity, because the gradient says how much the parameter would change if you vary the output: it makes sense that you can bump a scalar up and down, but if you have a matrix, how would you vary that? The tensor <code>v</code> specifies exactly in what sense you are “varying” the matrix, so that you can still compute a gradient. We don’t ever compute Jacobians, because in DL they are not usually needed.</p>
Making a custom activation function more memory-efficient

Andreas_Bloch: <p>I have the following function, which quantizes an input according to a monotonic step function to the values: [-0.1, -0.05, 0.0, 0.05, 0.1]. In the backward step I just pretend that the forward function was the identity (thus giving back the same gradient).</p>
<pre><code class="lang-auto">class Quantize(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input):
        input[0.075 &lt; input] = 0.1
        input[(0.025 &lt; input) &amp; (input &lt;= 0.075)] = 0.05
        input[(-0.025 &lt; input) &amp; (input &lt;= 0.025)] = 0.0
        input[(-0.075 &lt; input) &amp; (input &lt;= -0.025)] = -0.05
        input[input &lt;= -0.075] = -0.1
        return input

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output
</code></pre>
<p>Unfortunately, i get an out-of-memory error on the GPU when using this function. Before resorting to reducing the batch size I’d like to see if approaches exist to reduce the memory footprint of this function.</p>
<ul>
<li>Is there even a built-in function that does the above quantization?</li>
<li>Are there some built-in functions that could be used to achieve the functionality of above?</li>
<li>Is there a more memory-efficient approach to achieve the assignment of above?</li>
<li>Is there some way to define a function that maps over each entry of a tensor? (E.g., like PyTorch’s .exp, .softmax or similar)</li>
<li>Or is there some completely different API to write a custom “Activation”-Function instead of a Function?</li>
</ul>
<p>Looping over the input is not an option - that takes way too long. The best would be if I could parallelize the operation like a ‘map’.</p>
Should i take the mean of loss before doing backward operation for multiple tasks?

ayadav01: <p>I have a CNN network that returns three different model outputs, so before doing doing the backward operation of the network, should i be taking the mean of the losses or just sum them up and do the backward pass. Something like this:</p>
<pre><code class="lang-auto">criterion1 = nn.CrossEntropyLoss(weights_label_0)
criterion2 = nn.CrossEntropyLoss(weights_label_1)
criterion3 = nn.CrossEntropyLoss(weights_label_2)

loss_1 = criterion1(output[0], label_0)
loss_2 = criterion2(output[1], label_1)
loss_3 = criterion3(output[2], label_2)

loss = loss_1+loss_2+loss_3
loss.backward()

# or 

loss = (loss_1+loss_2+loss_3)/3
loss.backward()
</code></pre>
<p>which of the two would be correct ? Also, i have a slight confusion regarding calculating the weights for the labels. should the weights be calculated per batch or per dataset ?</p>

Scott_Hoang: <p>I would opted for the mean version.</p>

Scott_Hoang: <p>also as for the your second question, the weights should be determine per dataset.</p>

tstandley: <p>If you are always training with these 3 tasks, it doesn’t matter what you do. It’s equivalent to training with a learning rate that is 3x bigger/smaller.</p>
<p>If you plan on comparing to single task learning or switching up your tasks, you should use the mean version, which should keep the gradients more similar across conditions.</p>
Ǹorm of a tensor

diravan: <p>Hi all,</p>
<p>I have a <code>64x10x3x32x32</code> tensor <code>g</code> where the first coordinate is the batch_size. For every <code>10x3x32x32</code> subtensor I would like to compute the 2-norm and then sum them up. One way do it is like this</p>
<pre><code class="lang-auto">    sum_2_normes = torch.zeros(1)
    ......
    for i in range(batch_size):
        sum_2_normes += g[i].norm(p=2)
</code></pre>
<p>I was wondering if there is a way to do it in one line and/or more efficiently. I tried <code>torch.norm(g, p=2, dim=0)</code> but, as expected according to the documentation I get a <code>10x3x32x32</code> tensor, instead of what I need which is a <code>64x1</code> tensor</p>

ptrblck: <p>You could flatten the vector and calculate the norm on its view.</p>
<pre><code class="lang-auto">a = torch.randn(64, 10, 3, 32, 32)
a = a.view(64, -1)
b = torch.norm(a, p=2, dim=1)
torch.sum(b)
</code></pre>

diravan: <p>Thanks, that works. Now I realized that I was thinking about <code>dim</code> as an numpy <code>axes</code>, which is wrong.</p>

ptrblck: <p>Well, you can think of <code>dim</code> as a numpy <code>axis</code>. Could you elaborate what you expected?<br>
Numpy will behave in the same way for this problem:</p>
<pre><code>x = np.random.randn(64, 10, 3, 32, 32)
y = np.linalg.norm(x, ord=2, axis=1)
print(y.shape)</code></pre>

dragen: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I feel confused about matrix norm and vector norm. I kknow vector norm and matrix norm has different formulation. So how to tell <code>torch.norm()</code> to use which kind of nomr?</p>

ptrblck: <p>Have a look at the <a href="https://pytorch.org/docs/stable/torch.html#torch.norm" rel="nofollow noopener">torch.norm docs</a> to see all possible flags for <code>p</code> and which norm will be calculated for matrices and vectors.</p>

pinocchio: <p>why is there a accept on the word norm here? U know it makes it harder to search right?</p>

optyang: <p>Hi, is the solution you proposed more effective than the “for loop” solution, due to, e.g., parallel computation (in theory computing the L2 norm for one batch is independent from another batch)?</p>
<p>Thanks a lot!</p>

ptrblck: <p>Python for loops might be often slower than a single call into a specific function, as it could use vectorized code under the hood.<br>
You could profile both approaches for your current workloads and chose the faster one.</p>
<p>For the norm calculation I would assume that avoiding the loop will yield a better performance.</p>
Compute blocks' std and backpropagation

ptnewbie: <p>I want to compute a block-based std on the image, but I don’t know which way is efficient and whether the gradient of std can be backpropagated.</p>
<p>For example, I have images: 10x1x20x20 (NCHW), I want to compute the std within 3x3 window on the images and compute each std in this 3x3 matrix and do backpropagation. Can anyone help me? Thanks!</p>

ptrblck: <p>You could use <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.unfold"><code>F.unfold</code></a> to create these <code>3x3</code> windows from the image and compute the <code>std</code> for each of these windows.<br>
You should be able to backpropagate through <code>torch.std</code>.</p>

ptnewbie: <p>Thank you. I tried to use unfold, and then computed torch.std on each patch, but got nan…</p>

ptrblck: <p>Could you check the input values of this particular patch, which is creating the NaN output?<br>
<code>torch.std</code> should return a zero, if the input is constant.</p>
How to make Detectron2 demo work on combination of vgg, ImageNet, and COCO/COCO Stuff class names?

Mona_Jalal: <p>Currently, this demo only works for 1600 VGG class names. How can I make it work for union of VGG, COCO, COCO stuff, and ImageNet class names?</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/airsplay/py-bottom-up-attention/blob/master/demo/demo_vg_detection.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/airsplay/py-bottom-up-attention/blob/master/demo/demo_vg_detection.ipynb" target="_blank" rel="nofollow noopener">airsplay/py-bottom-up-attention/blob/master/demo/demo_vg_detection.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You may need to restart your runtime prior to this, to let your installation take effect\n",
    "# Some basic setup\n",
    "# Setup detectron2 logger\n",
    "import detectron2\n",
    "from detectron2.utils.logger import setup_logger\n",
    "setup_logger()\n",
    "import os\n",
    "\n",
    "# import some common libraries\n",
    "import numpy as np\n",
    "import cv2\n",
    "import torch\n",
</code></pre>

  This file has been truncated. <a href="https://github.com/airsplay/py-bottom-up-attention/blob/master/demo/demo_vg_detection.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Additionally, if say I have 700 other classes that are not in the union of these 4, what would be some straighforward ways to go after it and be able to use in this demo?</p>

ptrblck: <p>My best guess is that you would have to add the class names to the <a href="https://github.com/airsplay/py-bottom-up-attention/blob/834fa8b8123657fe6fa6b27c069015b824e07646/detectron2/data/catalog.py#L168"><code>MetadataCatalog</code></a>.</p>
Backward error and Runtime Error

MaggieX: <p>Any suggestions on how to debug the following? Thank you!!</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 90, in <br>
g_loss.backward()<br>
File “/Users/zxiao/opt/anaconda3/lib/python3.7/site-packages/torch/tensor.py”, line 198, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/Users/zxiao/opt/anaconda3/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 100, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 1024, 1, 1]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce this issue, please?</p>
How to implement code in pytorchpoint.net

shangguan91: <p>how do i fill the blank of dataset. Should i download?  or throgh the data-loader</p>
<p><a href="https://github.com/fxia22/pointnet.pytorch" rel="nofollow noopener">https://github.com/fxia22/pointnet.pytorch</a></p>
<pre><code class="lang-auto">python train_classification.py --dataset &lt;dataset path&gt; --nepoch=&lt;number epochs&gt; --dataset_type &lt;modelnet40 | shapenet&gt;
</code></pre>
<p>How do I operate?</p>

harsha_g: <p>The github repo provides a shell script to <strong>download</strong> data.</p>

shangguan91: <pre><code class="lang-auto">https://github.com/fxia22/pointnet.pytorch/blob/master/scripts/download.sh
</code></pre>
<p>Thank you, then what should do to implement the classfication?<br>
dataset=xxx<br>
path=xxx<br>
!python train_train_classification.py ?</p>

shangguan91: <aside class="quote no-group" data-username="shangguan91" data-post="1" data-topic="86672">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a3d4f5/48.png" class="avatar"> shangguan91:</div>
<blockquote>
<p><code>&lt;dataset path&gt; </code></p>
</blockquote>
</aside>
<p>what does dataset  mean? what should i substitue, suppose i want to Run modelnet10 with 1000 points?</p>

harsha_g: <p>The link to the <a href="https://shapenet.cs.stanford.edu/ericyi/shapenetcore_partanno_segmentation_benchmark_v0.zip" rel="nofollow noopener">datasource</a> is provided in the download.sh file. Download the zip file to some location (let’s call it <code>/whatever/data/path/is</code>) on your machine and unzip it. Now, from the command line go to <code>utils</code> folder in this project and run</p>
<p><code>python train_classification.py --dataset /whatever/data/path/is/shapenetcore_partanno_segmentation_benchmark_v0 --nepoch=1000 --dataset_type modelnet40</code></p>
<p>I can tell you that as soon as you run the above command you will run into issues with some file missing. You may have to labor through this code manually to get this running. Be patient and try to read the errors carefully to understand how you can address them one at a time. Good luck.</p>
reuse dataloader

rabbitwayne: <p>My training code is like the following:</p>
<p>for epoch in range(n_epoch):<br>
for i, batch in enumerate(dataset):<br>
train()</p>
<p>dataset is created using Dataloader and I have only one data file. I saw that Dataloader loads everything in this file into memory and then extracts batches. I want to train this data file for multiple epochs. But I noticed that every time a new epoch begins, the data file will be loaded from disk again and loading this file takes a lot of time. How to load this file only once for the first epoch and reuse the dataset without fetching from disk repeatedly for the following epochs? Thanks a lot!</p>

ptrblck: <p>I assume you are loading the data in the <code>Dataset.__init__</code> method?<br>
If that’s the case, you could preload the data before creating the <code>Dataset</code> and pass it to its <code>__init__</code> method:</p>
<pre><code class="lang-python">data = torch.load(...)
dataset = MyDataset(data)
loader = DataLoader(dataset)
</code></pre>
<p>This should avoid reloading the data. However, the <code>Dataset.__init__</code> method would still be called in each epoch, if I’m not mistaken, so you should make sure that no heavy loading is there.</p>

rabbitwayne: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for your reply! The training uses the same data file and trains it for many epochs. How can I avoid reloading the file on every new epoch? How to reuse the data loaded in the previous epoch? Thanks!</p>

ptrblck: <p>Is my suggestion not working for you?<br>
If not, what would be the reason?</p>
How to resample a new batch data with Dataset and Dataloader?

dingfeng_shi: <p>I have an image dataset and I try to extract a large number of patches from every image(the number of patches is different from different images, I hope all the patches can be used). I want a new batch, not about images but patches. I try to get patches before constructing a Dataset class, but it will take a very long time. Can I have an efficient way(such as multiprocess) to get a new batch of data while training?<br>
Any help would be greatly appreciated.</p>

ptrblck: <p>Is the number of patches and their location for each image fixed?<br>
If so, you could create these patches only once and could reuse them afterwards.</p>
<p>If that’s not possible, it could be a bit tricky to use multiple processes to create the patches, as you would have to provide the image index as well as the patch “index” (location etc.) to the <code>Dataset.__getitem__</code> method e.g. by using a custom sampler.</p>

dingfeng_shi: <p>Thank you for your advice. The number of the patches is fixed, but it’s too large, so if a create these before train, I have to wait a very long time. But I think the second idea can be useful in my code, thank you!</p>
Got stuck at model.to(cuda) , CUDA8.0

Kirito0816: <p>I tried to train a neural network using Pytorch. However, the code was stuck at  model.to(cuda). I am using CUDA 8.0 and try pytorch 1.0.0 and 0.4.1 with same problem. I checked the driver and it is good. And I tried the model.to(torch.device(cuda)). Could u pls help with me? Thank you very much.</p>

Kirito0816: <p>conda list | grep pytorch<br>
cuda80 1.0 h205658b_0 pytorch<br>
pytorch 1.0.0 py3.7_cuda8.0.61_cudnn7.1.2_1 [cuda80] pytorch</p>

Kirito0816: <aside class="quote no-group" data-username="Kirito0816" data-post="2" data-topic="86810">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirito0816/48/26001_2.png" class="avatar"> Kirito0816:</div>
<blockquote>
<p>onda list | grep pytorch<br>
cuda80 1.0 h205658b_0 pytorch<br>
pytorch 1.0.0 py3.7_cuda8.0.61_cudnn7.1.2_1 [cuda80]</p>
</blockquote>
</aside>
<p>nvidia-smi<br>
Thu Jun 25 10:02:47 2020<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 375.26                 Driver Version: 375.26                    |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|===============================+======================+======================|<br>
|   0  GeForce GTX TIT…  Off  | 0000:83:00.0     Off |                  N/A |<br>
| 27%   66C    P0    93W / 250W |   1602MiB / 12206MiB |    100%      Default |<br>
±------------------------------±---------------------±---------------------+<br>
|   1  GeForce GTX TIT…  Off  | 0000:84:00.0     Off |                  N/A |<br>
| 22%   37C    P8    17W / 250W |   1343MiB / 12206MiB |      0%      Default |<br>
±------------------------------±---------------------±---------------------+<br>
|   2  GeForce GTX TIT…  Off  | 0000:87:00.0     Off |                  N/A |<br>
| 38%   80C    P2   173W / 250W |   9726MiB / 12206MiB |     79%      Default |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                       GPU Memory |<br>
|  GPU       PID  Type  Process name                               Usage      |<br>
|=============================================================================|<br>
|    2     22922    C   python3                                       9724MiB |<br>
±----------------------------------------------------------------------------+</p>

Rahul_pillai: <p>Try updating CUDA to 9.2 or 10 with the latest stable pytorch</p>