akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Expanding multidimensional tensor by non-singleton dimension

martinr: <p>I am trying to expand a [200, 176, 2] binary mask to select from [200, 176, 14] tensor, so that first 7 elements from the tensor’s 3rd dimension (size 14) would be selected by mask[:, :, 0] and last 7 elements by mask[:, :, 1]. E.g. if my mask at third dimension is [0,1] then a selection is made as if it was [0,0,0,0,0,0,0,1,1,1,1,1,1,1]. I managed to solve it by this piece of lengthy code, but I imagine there must be a shorter and more straightforward way (and also without using Numpy as I intend to process this on GPU).</p>
<p>Goal in short: use [200, 176, 2] binary mask <em><strong>b</strong></em> to select from [200, 176, 14] tensor <em><strong>a</strong></em></p>
<p>My current code (works as expected, but very lengthy):</p>
<pre><code class="lang-auto"># tensor to select from
a = torch.rand([200,176,14])

# mask
b = torch.zeros([200,176,2], dtype=torch.uint8)

# split mask by the last dimension
mask_parts = torch.split(b, 1, dim=2)

# first part, size torch.Size([200, 176, 1])
mask1 = mask_parts[0]
# expand to torch.Size([200, 176, 7])
mask1 = mask1.expand(-1,-1,7)

# second part, identical processing to the first
mask2 = mask_parts[1]
mask2 = mask2.expand(-1,-1,7)

# join masks, get [200, 176, 14]
mask = torch.cat((mask1, mask2), 2)

# now the goal - use the mask to select elements from a
result = a[mask]
</code></pre>
<p>Is there a better way to achieve the same result or is it OK?</p>

InnovArul: <p>To expand [200, 176, 2] mask to size [200, 176, 14], you can do the following:</p>
<pre><code class="lang-auto">new_mask = b.unsqueeze(-1).repeat(1, 1, 1, 7).view(200, 176, -1)
print( torch.all((new_mask.float() - mask.float()) == 0)) # 1
</code></pre>

martinr: <p>Thank you! Expanding into 4th dimension and back was something I couldn’t think of myself!</p>
AttributeError: module 'torch' has no attribute 'device'

Fatia: <p>%load_ext autoreload<br>
%autoreload 2</p>
<p>import os, json, argparse, torch, sys<br>
import numpy as np<br>
from tqdm import tqdm<br>
from glob import glob<br>
import torch.nn.functional as F<br>
from multiprocessing import Pool, cpu_count<br>
sys.path.append("…/")</p>
<p>from utils import image<br>
from utils.MobileNetV2_pretrained_imagenet import MobileNetV2<br>
from utils.data import NumpyImageLoader<br>
from utils.metrics import BinaryClassificationMetrics</p>
<h1>Print parameters</h1>
<p>params = {}<br>
params[“channel”] = “YCbCr”<br>
params[“threshold”] = 0.80<br>
params[“test_subset”] = 5</p>
<p>params[“patch_test_au_dir”] = “…/backup/MBN2-pre-YCbCr/test/au”<br>
params[“patch_test_tp_dir”] = “…/backup/MBN2-pre-YCbCr/test/tp”</p>
<p>params[“training_log_dir”] = “…/backup/MBN2-pre-YCbCr/checkpoints/”<br>
MODEL_FILE = os.path.join(params[“training_log_dir”], “model.ckpt”)<br>
DEVICE = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)</p>
<p>params[“au_subsets_file”] = “…/dataset/au_subsets.json”<br>
params[“tp_subsets_file”] = “…/dataset/tp_subsets.json”</p>
<p>params[“casia2_au”] = “/media/antiaegis/storing/datasets/CASIA2/Au”<br>
params[“casia2_tp”] = “/media/antiaegis/storing/datasets/CASIA2/Tp”</p>

ptrblck: <p>Could you check your PyTorch version (<code>print(torch.__version__)</code>)?<br>
I think <code>torch.device</code> was introduced in <code>0.4</code>.<br>
You can find the install instructions for the current releast (<code>1.1</code>) <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">here</a>.</p>

Fatia: <p>PyTorch version is 0.3.1, i think that’s the problem, thanks</p>
How to Eliminate Warnings While Creating VGG_11 Trace File

sercan: <p>I try to create trace file of a model without any warning. I use the model VGG_11 at this <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py" rel="noopener nofollow ugc">link</a>. I trained the model with my custom dataset then saved the model as a .pth file. Then, I loaded this .pth file and created trace file as a .pt file. Then, it gave those warnings:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/45bf0125016bcf078c1c4cd621ae7b71204acac5.png" data-download-href="https://discuss.pytorch.org/uploads/default/45bf0125016bcf078c1c4cd621ae7b71204acac5" title="VGG_Trace_Warnings"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/45bf0125016bcf078c1c4cd621ae7b71204acac5_2_690x116.png" alt="VGG_Trace_Warnings" data-base62-sha1="9X0azZhTWZtqvUOArbZ1ocb0srr" width="690" height="116" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/45bf0125016bcf078c1c4cd621ae7b71204acac5_2_690x116.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/45bf0125016bcf078c1c4cd621ae7b71204acac5_2_1035x174.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/45bf0125016bcf078c1c4cd621ae7b71204acac5_2_1380x232.png 2x" data-dominant-color="323030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">VGG_Trace_Warnings</span><span class="informations">1853×313 29.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Could you offer a solution about how to eliminate those warnings?</p>

Michael_Suo: <p>The warning is telling you that there are non-deterministic nodes in your model, specifically dropouts. If you’re tracing a model that you already trained, you may have forgotten to set <code>eval()</code> on the module.</p>
How to get the output from an intermediate layer of a custom made model in Pytorch?

Anirudh257: <p>I am a beginner to Pytorch and I have trained a custom built model. It looks like this:</p>
<pre><code class="lang-auto">stackedIndRNN_encoder(
  (DIs): ModuleList(
    (0): Linear(in_features=150, out_features=512, bias=True)
    (1): Linear(in_features=512, out_features=512, bias=True)
    (2): Linear(in_features=512, out_features=512, bias=True)
    (3): Linear(in_features=512, out_features=512, bias=True)
    (4): Linear(in_features=512, out_features=512, bias=True)
    (5): Linear(in_features=512, out_features=512, bias=True)
  )
  (BNs): ModuleList(
    (0): Batch_norm_step(
      (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): Batch_norm_step(
      (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (2): Batch_norm_step(
      (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (3): Batch_norm_step(
      (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (4): Batch_norm_step(
      (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (5): Batch_norm_step(
      (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (RNNs): ModuleList(
    (0): IndRNN_onlyrecurrent()
    (1): IndRNN_onlyrecurrent()
    (2): IndRNN_onlyrecurrent()
    (3): IndRNN_onlyrecurrent()
    (4): IndRNN_onlyrecurrent()
    (5): IndRNN_onlyrecurrent()
  )
  (lastfc): Linear(in_features=512, out_features=60, bias=True)
)

</code></pre>
<p>I want to access the output from the last layer, RNNs that is a ** 512-dimensional vector** that is inputted into the final fully connected layer, <strong>lastfc</strong>. I have tried forward hook in the following way:</p>
<pre><code class="lang-auto">RNNs_output = None

def RNN_hook(module, input_, output):
	global RNNs_output
	RNNs_output = output

model.lastfc.register_forward_hook(RNN_hook)	

model(skeletons)
print(RNNs_output)

</code></pre>
<p>but I get None in RNNS_output. What am I doing wrong?</p>

ptrblck: <p>After registering a forward hook, you would need to perform a forward pass, so that the input and output will be filled with tensors.<br>
Also, if <code>lastfc</code> is your last layer, you’ll also get its output as the return value from your model’s forward:</p>
<pre><code class="lang-python">output = model(data)
</code></pre>

Anirudh257: <p>Thanks for your reply, <a class="mention" href="/u/ptrblck">@ptrblck</a>. Let me try and get back to you.</p>
Unexpected behaviour in parameter initialization

jmaronas: <p>Hi.</p>
<p>I have been surprised by the behavior of PyTorch. What I have done is initialize the bias of the output layer of a neural network using some data statistics. So considering, for instance:</p>
<pre><code class="lang-auto">mean=data.mean(0) #numpy array
std = data.std(0) #numpy array

#inside the model there is a sequential for the output layer. The model receives mean and std as arguments
f=#here is the sequential
f[0].bias=nn.Parameter(torch.from_numpy(numpy.log(std**2)))#initialize
</code></pre>
<p>I was surprised because if I put this as above, instead of std.copy(), the underlying std tensor changes when the bias changes in the optimizer updates. I thought that, in these cases, a new instance of the tensor is created (and same for numpy). As far as I know:</p>
<pre><code class="lang-auto">a=numpy.random.randn(1000,1)
b=torch.from_numpy(a)
a[0]=10# this also changes b

c=b**2 # a new instance in memory is created 
</code></pre>
<p>Any suggestions? Thanks.</p>

MariosOreo: <p>Hi,</p>
<p>Did you get any proper explaination on it?</p>

InnovArul: <p>Numpy arrays and pytorch tensors interoperate by sharing memory.</p>
<aside class="quote" data-post="1" data-topic="7395">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/7bcc69/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-is-the-overhead-of-transforming-numpy-to-torch-and-vice-versa/7395">What is the overhead of transforming numpy to torch and vice versa?</a> 
  </div>
  <blockquote>
    What is the overhead of transforming numpy to torch and vice versa?
  </blockquote>
</aside>


jmaronas: <p>yes. But any operation with reassignment should allocate new memory. If you do this on a numpy tensor:</p>
<pre><code class="lang-auto">a=numpy.random.randn(100,1)
b=a**2
</code></pre>
<p>b and a use different memory.</p>
Confused about BCEWithLogitsLoss use (as discriminator loss)

trougnouf: <p>I don’t understand what is the proper use of BCEWithLogitsLoss for binary classification. I used it as the loss function of a discriminator which outputs around 0 or 1. I removed the Sigmoid activation function.</p>
<p>Doing a simple experiment where I set lf = nn.BCEWithLogitsLoss(), I get<br>
lf(res=0, target=0   ) = 0.6931<br>
lf(res=0, target=0.5) = 0.6931<br>
lf(res=0, target=1)    = 0.6931<br>
lf(res=0.5, target=0) = 0.9741<br>
lf(res=0.5, target=0.5) = 0.7241<br>
lf(res=0.5, target=1)    = 0.4741<br>
lf(res=1, target=0)       = 1.3133<br>
lf(res=1, target=0.5)    = 0.8133<br>
lf(res=1, target=1)       = 0.3133</p>
<p>Clearly I am using the loss function wrong, because the loss is a constant 0.6931 whenever the model returns 0, and so it returns around 0 all the time.</p>
<p>What is a correct use case? Or relevant documentation? Or is that loss function not relevant here?</p>

ptrblck: <p><code>nn.BCEWithLogitsLoss</code> expects logits as the model output, not probabilities.<br>
Internally, this loss function will be used (taken from the docs and just removed the weight):</p>
<blockquote>
<p>-[y_n​⋅logσ(x_n​)+(1−y_n​)⋅log(1−σ(x_n​))]</p>
</blockquote>
<p>If you pass <code>x_n</code> as <code>0</code>, we can simplify the formula to:</p>
<blockquote>
<p>-[y_n​⋅log(0.5)+(1−y_n​)⋅log(0.5)]</p>
</blockquote>
<p>using the fact that <code>sigmoid(0.) = 0.5</code>. Further we’ll get:</p>
<blockquote>
<p>-log(0.5) * [y_n + 1 - y_n] = -log(0.5) = 0.6931</p>
</blockquote>
<p>If you would like to pass the input as probabilities, use <code>nn.BCELoss</code> instead.</p>

trougnouf: <p>I think I understand, thank you! The given target is still a probability then and the model’s output as well as the loss’ result are unbounded.</p>
Data reading framework for PyTorch (Hive, MySQL, S3 etc.)

pritamdamania87: <p>At Facebook we are building a data reading framework for PyTorch which can efficiently read from data stores like Hive, MySQL, our internal blob store and any other tabular data sources. The framework allows for specifying complex input pipelines to read from different sources. For example if you have a table which stores handles for images, you can write SQL like code to read from the table, apply filters to select certain handles and then retrieve those handles from another data source with a few lines of code.</p>
<p>In addition to this, the framework supports running user-defined transforms which can either be pure python (ex: torchvision.transforms) or torchscript code. This framework can also be used with the <a href="https://pytorch.org/docs/stable/distributed.html" rel="nofollow noopener">torch.distributed</a> package to distribute the data across multiple nodes for training. The input pipeline that the user specifies can be defined once, serialized as a plan and run on multiple remote machines if required.</p>
<p>The framework builds upon the OSS dataloader and dataset framework. In particular it uses <a href="https://github.com/pytorch/pytorch/pull/19228" rel="nofollow noopener">IterableDataset</a> to provide a stream based interface for data retrieved from input pipelines.</p>
<p>Sample code to illustrate what reading and pre-processing images would look like:</p>
<pre><code class="lang-auto"># Hive table has columns handle and partition_no. The partition column 
# for the table is partition_no
df = data.data_warehouse("mynamespace", "mytable")

# Filter to partition the data across multiple workers.
partition_filter = "hash(partition_no) % {0} = {1}".format(worker_info.num_workers, worker_id)
df = df.filter(partition_filter)

# Fetch the handle from a blobstore
df = df.map(["fetch_handle(handle) as img"])

# Rebatch the data
df = df.rebatch(batch_size=16)

# transform_image is a user supplied function to run image transforms.
ds = MyDataset(df=df, transforms=transform_image)
dl = torch.utils.data.DataLoader(ds)

for batch in dl:
    pass
</code></pre>
<p>We are evaluating whether it makes sense to open source this framework. For OSS users, this framework might be useful for training jobs which store large amount of data in Hive or S3 (images). Although, we would love to hear from the community whether this would be useful and also some use cases that might benefit from a framework like this.</p>
How to solve ModuleNotFoundError: No module named 'tensorboardX'?

Niki: <p>For this line (in Python 2.7.10):</p>
<pre><code class="lang-auto">from tensorboardX import SummaryWriter
</code></pre>
<p>I am getting this error:</p>
<pre><code class="lang-auto">ModuleNotFoundError: No module named 'tensorboardX'
</code></pre>
<p>I installed:<br>
<code>$ pip install tensorboardX</code></p>
<p>and the result for <code>pip list|grep tensor</code>:</p>
<pre><code class="lang-auto">tensorboard          1.13.1         
tensorboardX         1.7            
tensorflow           1.13.1         
tensorflow-estimator 1.13.0
</code></pre>
<p>Any ideas how can I solve this error?</p>

bigmb: <p>Are you using Pycharm ?<br>
If so, try restarting it or directly run the program and see if it works.</p>
<p>I had this issue , but got fixed after restarting pycharm.</p>

alex.veuthey: <p>If you are using the latest PyTorch version (1.1.0), <a href="https://pytorch.org/docs/stable/tensorboard.html" rel="nofollow noopener">tensorboard support is now integrated</a>, using the same API as <code>tensorboardX</code>. You can call it with <code>from torch.utils.tensorboard import SummaryWriter</code>, so that you don’t need an external library.</p>
<p>If indeed you run in PyCharm and haven’t restarted yet, try <a class="mention" href="/u/bigmb">@bigmb</a>’s solution.</p>

bigmb: <p><a class="mention" href="/u/alex.veuthey">@alex.veuthey</a> have you tried visualizing Unet models in pytorch(1.1.0) ? Because for me it works for simple configuration but if it has skip connection then it shows me error, but the same model works in tensorboardX.</p>

alex.veuthey: <p>That’s good to know! I haven’t tried yet. The link says “<strong>It also currently does not support all model types for <code>add_graph</code> , which we are actively working on.</strong>” so I guess that explains it…</p>

Niki: <p><a class="mention" href="/u/bigmb">@bigmb</a> <a class="mention" href="/u/alex.veuthey">@alex.veuthey</a> thank you for your reply.</p>
<p><a class="mention" href="/u/bigmb">@bigmb</a> I don’t use Pycharm. I used Python and also Jupyter notebook.<br>
<a class="mention" href="/u/alex.veuthey">@alex.veuthey</a> I upgraded torch version to 1.1.0, and tried <code>from torch.utils.tensorboard import SummaryWriter</code> but it gave the same error.</p>

bigmb: <p><a class="mention" href="/u/niki">@Niki</a> : just make sure that if you using anaconda , then you are installing it in conda and not in python. Also, check if the python you are using is linked to the site-packages.</p>

Niki: <p>Ok. Thank you for your help <a class="mention" href="/u/bigmb">@bigmb</a>.</p>
Converting a Pytorch model to Torch Script program via Tracing Using Pytorch 1.0: Not working

safi: <p>I am trying the new <a href="https://pytorch.org/tutorials/advanced/cpp_export.html#loading-a-pytorch-model-in-c" rel="nofollow noopener">tutorial</a> from Pytorch (dev version) which shows how to load a Pytorch model in C++ without any Python dependencies. I am using the tracing method:</p>
<pre><code class="lang-auto">import torch
import torchvision

# An instance of your model.
model = A UNET MODEL FROM FASTAI which has hooks as required by UNET

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)
</code></pre>
<p>However, I got a value error:</p>
<p><code>ValueError: Modules that have hooks assigned can't be compiled</code></p>
<pre><code class="lang-auto">   1118         if orig._backward_hooks or orig._forward_hooks or orig._forward_pre_hooks:
-&gt; 1119             raise ValueError("Modules that have hooks assigned can't be compiled")
   1120 
   1121         for name, submodule in orig._modules.items():
</code></pre>
<p>I am using it for UNET MODEL from FASTAI. I was wondering if anyone else has tried it and would like to share their experiences if it worked. Also, does anyone if this new feature comes with a limitation of only working for very simple models?</p>
<p>For reference, the <a href="https://github.com/fastai/fastai/blob/master/courses/dl2/carvana-unet.ipynb" rel="nofollow noopener">FASTAI Notebook</a> with the relevant UNET definition.</p>

yf225: <p>Hi <a class="mention" href="/u/safi">@safi</a>, do you mind opening an issue in <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a> so that we can triage it to the right folks?</p>

Jeremy_Cochoy: <p>I tried to force the compilation by ignoring all the hooks. It compiled, but the inference does not work (pink squares on my output instead of the picture expected).<br>
I don’t have the time to dig into the DynamicUnet implementation for now saddly <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Just in case you want to try with different models, here is the SO thread ( <a href="https://stackoverflow.com/questions/56242857/how-can-i-force-torch-jit-trace-to-compule-my-module-by-ignoring-hooks/56242858#56242858" rel="nofollow noopener">https://stackoverflow.com/questions/56242857/how-can-i-force-torch-jit-trace-to-compule-my-module-by-ignoring-hooks/56242858#56242858</a> )</p>
Realtime capabilities of TorchScript / memory allocations

jviereck: <p>At the moment I am exploring the possibility of writing controllers for a real robot hardware using TorchScript. One of the requirements is to have low latency and constant execution times for the script. That is as the robot controller is running in a realtime environment with hard 1 kHz (=1 ms) time limits.</p>
<p>Does someone of you know how the JIT compiler of TorchScript handles memory allocations? Is there something like a “realtime” mode switch to avoid memory allocations or does this all depends on the implementation of the TorchScript C++ operators?</p>
<p>Best,<br>
julian</p>

Michael_Suo: <p>The JIT has no realtime mode. Allocation is controlled by operator implementations, which are generally (for CPU ops) tuned for server workloads.</p>
Create Dataset class with different input sizes

lorenzo_fabbri: <p>My PyTorch model contains two CNNs, the outputs of which are then merged and passed through a series of fully-connected layers. The inputs of the two CNNs are matrices: the problem is that for the first CNN the matrices have shape 128x100, while for the second it’s 128x1000. I’m now trying to create a <code>Dataset</code> class to generate the loaders. At the moment I wrote the following:</p>
<pre><code class="lang-auto">class Data(Dataset):
    
    def __init__(self, dataP, targetP, dataC, targetC, transform=None):
        self.dataP = [torch.from_numpy(X).int() for X in dataP]
        self.targetP = [torch.from_numpy(y).float() for y in targetP]
        
        self.dataC = [torch.from_numpy(X).int() for X in dataC]
        self.targetC = [torch.from_numpy(y).float() for y in targetC]
        
        self.transform = transform
    
    def __getitem__(self, index):
        Xp = self.dataP[index]
        yp = self.targetP[index]
        
        Xc = self.dataC[index]
        yc = self.targetC[index]
        
        if self.transform:
            Xp = self.transform(Xp)
            Xc = self.transform(Xc)
            
        return Xp, yp, Xc, yc
    
    def __len__(self):
        return len(self.dataP)
</code></pre>
<p>While the code seems to be running without any problem, I’m rather sure there is something wrong since in the <code>__len__</code> method I return the length of one of the inputs. Is it possible to take care of the different size of the inputs?</p>

LeviViana: <p>Maybe this would be safer:</p>
<pre><code class="lang-python">def __len__(self): 
    return min(len(self.dataP), len(self.dataC))
</code></pre>
<p>Otherwise you could have out of range errors if the <code>len(self.dataP) &gt; len(self.dataC)</code>. The drawback of this method is that it could skip a lot of training data. Are you datasets aligned ? i.e., the 10th sample of dataP must be trained together with the 10th sample of dataC ?</p>
Is it inefficient to apply nn.DataParallel to nn.Module which is composed of sub-module accelerated with nn.DataParallel?

toracorp: <p>We sometimes reuse existent models to build loss module like perceptual loss and GAN’s adversarial loss. I would like to know that if the existent models are accelerated with nn.DataParallel, it is inefficient to use nn.DataParallel one more time for the loss Module which use the existent models.</p>
<p>For example, model1 and model2 compose cycle loss module as follows. In that case, is the code, “cycle_loss = nn.DataParallel(CycleLoss(model1, model2)).cuda()”, inefficient?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
	def __init__(self):
		super().__init__()
		self.conv = nn.Conv2d(1,1,3,1,1)

	def forward(input):
		return F.relu(self.conv(input))

class CycleLoss(nn.Module):
	def __init__(self, model1, model2):
		super().__init__()
		self.model1 = model1
		self.model2 = model2
		self.l1 = nn.L1Loss()

	def forward(input1, input2):
		loss = self.l1(self.model2(self.model1(input1)), input1)
		loss += self.l1(self.model1(self.model2(input2)), input2)
		return loss[None] # expand dim=0 to concatnate

# make model
model1 = nn.DataParallel(Model()).cuda()
model2 = nn.DataParallel(Model()).cuda()

# loss module
cycle_loss = nn.DataParallel(CycleLoss(model1, model2)).cuda()

# opt
opt = torch.optim.Adam(list(model1.parameters()) + list(model2.parameters()))

# train
for input1, input2 in data_loader:
	opt.zero_grad()
	loss = cycle_loss(input1, input2)
	loss = loss.mean()
	loss.backward()
	opt.step()
</code></pre>
<p>I am afraid that nesting nn.DataParallel makes the code perform scatter and gather at each sub-module needlessly.</p>
<p>Thank you.</p>

pietern: <p>Yes, this is not great. The outer <code>nn.DataParallel</code> module will replicate N times. The inner modules will also be replicated N times. I think you’ll end up with N^2 replicas instead of just N. You can add some logging to the forward functions to confirm what really ends up happening.</p>
Can infiniband accelerate distributed training without GPUDirect?

GeoffreyChen777: <p>I have two 4x2080ti machines. I want to train my model by NCCL distributed backend. But the training is slow because these two machines are connected by a 1000M ethernet card.<br>
So I want to use two infiniband cards to connect these two machines.<br>
But my GPU is a GeForce not a Tesla. The question is, can infiniband accelerate the training if the GPU don’t support GPUDirect?</p>
<p>Thanks.</p>

pietern: <p>In theory, yes. As long as you get cards with a higher bandwidth than your Ethernet setup it should result in an improvement. But since NCCL is built for using GPUDirect, I’m not sure if it will work with NCCL out of the box. If it doesn’t, you could try and experiment with IPoIB and fall back to using NCCL’s TCP transport.</p>
<p>Good luck!</p>
5D audio data input to convolutional auto-encoder

fyjl: <p>In the context of audio processing, a 2D spectrogram is the most common input representation to the network such as CNN. Normally, one convert a 2D spectrogram into a 4D tensor as the input to the network, i.e., (n_frame, 1, n_freqbin, n_winsize), where <strong>n_frame</strong>, <strong>n_freqbin</strong> and <strong>n_winsize</strong> refers to the number of time frame, frequency bin, and size of context window*, respectively, and <strong>1</strong> refers to the channel analogously to image channels.</p>
<p>My problem is I would like to have an additional dimension of number of clip, so my input data is in size of (n_clip, n_frame, 1, n_freqbin, n_winsize). Notice that we are still batching the number of frame in this case.</p>
<p>And now, the input would go through an auto-encoder, which results in the code layer in size of (n_clip, n_frame, codesize), where codesize refers to the predefined dimension of the latent code. So my question is how do I pass a 5D tensor (n_clip, n_frame, 1, n_freqbin, n_winsize)  to a CNN auto-encoder, and generate latent codes (n_clip, n_frame, codesize)? The current workaround is to iterate through n_clip, and stack the output in every iteration.</p>
<p>Thanks for any tip!</p>
<p>*: a single frame of spectrogram is commonly represented as a context window, which includes prior and posterior frames (n_winsize), thereby convert a single time frame to a 2D representation.</p>

tom: <p>There are batch+channel+3d ops like conv3d (eg for video), but likely this isn’t exactly  what you want. Depending on the relation you want between the clips, you could also see if .view ing them in some 4d way makes sense - if you want to treat the clip dimension similar to batch (same weights for all clips, no interaction) or channel (per clip weights and full interaction).</p>
<p>Best regards</p>
<p>Thomas</p>

bdennyw: <p>I’m looking into a similar problem. What did you end up doing?</p>
How to load the optimizer for multi-GPU scenario?

fermat97: <p>Hi, as far as I know the correct way to build a model is:</p>
<pre><code class="lang-auto">model = Model() #build the model
model = nn.DataParallel(model)
model.to(device) #move the model to device 
optimizer = optim.Adam(model.parameters()) #build the optimizer
</code></pre>
<p>Now assume I want to load the parameters of the model and optimizer states from a pre-trained model (continue learning procedure) for a multi-GPU case. Then I am not sure where to load the optimizer:</p>
<pre><code class="lang-auto">model = Model() #build the model on cpu
checkpoint = torch.load(pretrainedModel) # load the pre-trained model
model.load_state_dict(checkpoint['model'])
model = nn.DataParallel(model)
model.to(device) #move the model to device 
optimizer = optim.Adam(model.parameters()) #build the optimizer
</code></pre>

alex.veuthey: <p>It’s pretty much the same, you would call:</p>
<pre><code class="lang-python">optimizer.load_state_dict(checkpoint['optim'])
</code></pre>
<p>Of course it requires having saved the optimizer state dict previously.</p>
<p>Also you should be aware that saving models and optimizers that were wrapped <code>nn.DataParallel</code> can result in errors when loading, because the wrapper adds a layer of abstraction, and weights will look like <code>model.module.conv1</code> instead of <code>model.conv1</code>, for example.</p>
<p><a href="https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/4">This answer</a> can help with the loading of the model (or you could call <code>nn.DataParallel</code> on your model before loading, but that only works when you actually want <code>nn.DataParallel</code> in your code).</p>
<p><a href="https://github.com/pytorch/pytorch/issues/9176#issuecomment-403570715" rel="nofollow noopener">This comment</a> can help when saving <code>nn.DataParallel</code> models, so that you don’t actually need the first solution. However I would recommend using both, so that you can handle every situation.</p>

fermat97: <p>Thank you. I have tried the following code snippet for the model according to <a href="https://github.com/pytorch/pytorch/issues/9176#issuecomment-403570715" rel="nofollow noopener">the link you proposed</a> and <a href="https://github.com/pytorch/pytorch/issues/4361#issuecomment-428443175" rel="nofollow noopener">another comment</a>:</p>
<pre><code class="lang-auto">model = Model() #build the model
model = nn.DataParallel(model)
model.to(device) #move the model to device 
optimizer = optim.Adam(model.parameters()) #build the optimizer
try:
    model.load_state_dict(checkpoint['model'])
except RuntimeError:
    model.module.load_state_dict(checkpoint['model'])
optimizer.load_state_dict(checkpoint['optimizer'])
</code></pre>
GPU utility usage is zero and CPU utility is very high even with Pytorch Cuda version of code

Asad_Munir: <p>I am training a network and I put my network parameters and data loader on Cuda but when I run training, nvidia-smi shows 0% GPU usage with memory occupied. All the computation is running on CPU as CPU usage is very high. I checked every variable and networks parameters with “variable_name.is_cuda” and  “next(net.parameters()).is_cuda”  respectively in my training loop. They are all true. Can anyone help me regarding this issue?</p>

gslaller: <p>You will have to share the code for this one, or an abstract of it.</p>

Asad_Munir: <p>I am sending an abstract of my code. I am sending data and network to cuda but they are running on cpu. Can you help?</p>
<pre><code class="lang-auto">def to_var(self, x):
        if torch.cuda.is_available():
            x = x.cuda()
        return Variable(x)

net = network.generator()
params = net.parameters()
optimizer = optim.Adam(params, lrG, [beta1, beta2], amsgrad=True)

if torch.cuda.is_available():
     net.cuda()

for i, data in enumerate(t_loader):

                source_img, target_img  = data[0], data[1]

                source_img = to_var(source_img)
                target_img = to_var(target_img)

                fake_generated_target = net(source_img)

                g_loss = torch.mean((fake_generated_target- 1) **2)

                optimizer.zero_grad()
                g_loss.backward()
                self.optimizer.step()```</code></pre>

gslaller: <p>You have to call “net = network.generator().cuda()” and give its paramters to the optimizer.</p>
Don't Understand Loss Calculation

beneyal: <p>Hi,</p>
<p>In the <code>evaluate</code> method <a href="https://github.com/pytorch/examples/blob/master/word_language_model/main.py" rel="nofollow noopener">here</a> the loss is calculated like so:</p>
<pre><code class="lang-auto">total_loss += len(data) * criterion(output_flat, targets).item()
</code></pre>
<p>Why is that? Why not just <code>criterion(output_flat, targets).item()</code>? Especially that during training, the loss is calculated</p>
<pre><code class="lang-auto">loss = criterion(output.view(-1, ntokens), targets)
</code></pre>
<p>One might say that I’m in a bit of a loss <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks for the help!</p>

ptrblck: <p>This is sometimes done during evaluation or testing to get the exact loss in case the last batch is smaller than the rest. If you just add the test losses together, remember that in the default case each one will be an average over the current batch. If all batches have the same length, you can just divide this accumulated loss by the number of batches (<code>len(test_loader)</code>) and you should get the average loss for this epoch.</p>
<p>However, if the dataset length is not divisible by the batch size without a remainder, the last batch will be smaller than the rest. Summing all the batch averages together and just dividing by the number of batches will give you a slightly wrong answer. To correct this you can multiply each batch loss with its length (alternatively use the summation option for your criterion) and divide this accumulated loss by the number of samples (<code>len(test_dataset)</code>) to get the exact test loss average.</p>

beneyal: <p>It’s driving me nuts, it seems I’m doing everything okay, including the loss calculation, but still the model and training specified <a href="https://discuss.pytorch.org/t/i-gave-up-on-trying-to-figure-out-what-im-doing-wrong-help/31090">here</a> aren’t working…</p>

K_Kk: <p>Hi, I would like to extend your answer. Suppose,<br>
running_loss += loss.item()<br>
epoch_loss = running_loss / len(dataset).<br>
I think this is also a way to calculate the loss. We don’t have problem with the batches. Am i right?</p>

ptrblck: <p>If your loss is being reduced (e.g. averaged over the batch dimension), <code>epoch_loss</code> won’t give you the true value.</p>
Save and load a model with extended classes from nn.Module

cherepanovic: <p>Hello community,</p>
<p>I have a question regarding models with extended classes.</p>
<p>How do I handle with models which are built by extended classes from nn.Module which have some specific features in itself?</p>
<p>Can I load it without any consideration? Or should I recreate the classes/some methods (if not, in which cases is a recreation of classes necessary; forward function and so on)?</p>
<p>And what should I consider If I load a model and want to extend it?</p>
<p>best regards!</p>
Libtoch issue in inference when loading through model.pt file

Zain_Ali_Zubair: <p>I have saved the model from python and successfully loaded in libtorch c++ but on inference, it always gives me the same output no matter what the input is?<br>
Here is my code for C++.</p>
<pre><code class="lang-auto">int main(int argc, const char* argv[]) {
  //torch::DeviceType device_type = at::kCUDA;
  torch::DeviceType device_type = torch::kCPU;
   torch::Device device(device_type);
  // Deserialize the ScriptModule from a file using torch::jit::load().
  std::shared_ptr&lt;torch::jit::script::Module&gt; module = torch::jit::load("../model.pt");
  module-&gt;to(device_type);
  assert(module != nullptr);
  std::cout &lt;&lt; "ok\n";

  //cv::Mat im;
  cv::Mat or_img = cv::imread("../1.png");
  cv::Mat res_img;
  cout&lt;&lt; or_img.cols &lt;&lt; or_img.rows &lt;&lt; or_img.channels() &lt;&lt;std::endl;
  cv::cvtColor(or_img, or_img, cv::COLOR_BGR2RGB);
  cv::resize(or_img, res_img, cv::Size(416, 416));
  //cout&lt;&lt; res_img &lt;&lt;std::endl;
  cv::Mat img;
  res_img.convertTo(img, CV_32F, 1.0 / 255);

  auto img_tensor = torch::from_blob(img.data, {1, 416, 416, 3}, torch::kFloat32);;
  img_tensor = img_tensor.permute({0, 3, 1, 2});
  //at::Tensor tensor_image = torch::from_blob(img.data, {1, 416, 416, 3}, at::kByte);
  //tensor_image = tensor_image.permute({0, 3, 1, 2});
  //tensor_image = tensor_image.to(at::kFloat);
  //auto img_var = torch::autograd::make_variable(tensor_image, false).to(device);
  //cout&lt;&lt;img_var&lt;&lt;endl;
  

  std::vector&lt;torch::jit::IValue&gt; inputs;
  inputs.push_back(img_tensor);
  //tensor_image.to(device_type);

  // Execute the model and turn its output into a tensor.
  at::Tensor output = module-&gt;forward({img_tensor}).toTensor();

  std::cout &lt;&lt; output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) &lt;&lt; '\n';
}
</code></pre>

Riddick_Gao: <p>You can try this:</p>
<p><code>torch::Tensor img_tensor = torch::from_blob(img.data, { 1, 416, 416, 3}, torch::kFloat32);</code></p>
<p>May be the reason is that you have not specify the data type.</p>

Zain_Ali_Zubair: <p>Thanks for your response I have changed the script but it remains the same, output of the model is the same no matter what the input image is?</p>
Torch.cross with a bigger dimension than 3?

Ingrid_Bernal: <p>Hi everybody,</p>
<p>Im trying to obtain the angle sin from two vectors, Im doing it by the formula</p>
<p>result1 / (result2 * result3)</p>
<p>where result1 is the norm of the cross multiplication from the two vectors, result 2 and 3 are the norm of each vector.</p>
<p>but when executing torch.cross between the two vectors its always out of range, except only if im multiplying a 3 dimension vectors, in my case I handle a 500 dimension vectors.</p>
<p>Any idea ?</p>
<p>Thanks.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8f986102ba5548068a2b2baab6e61c70c81bf2b5.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8f986102ba5548068a2b2baab6e61c70c81bf2b5" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8f986102ba5548068a2b2baab6e61c70c81bf2b5_2_690x246.jpeg" alt="Capture" data-base62-sha1="kuiKpRGgGvedirgTkjXvSpkkFaB" width="690" height="246" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8f986102ba5548068a2b2baab6e61c70c81bf2b5_2_690x246.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/8/8f986102ba5548068a2b2baab6e61c70c81bf2b5.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/8/8f986102ba5548068a2b2baab6e61c70c81bf2b5.jpeg 2x" data-dominant-color="441F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">931×332 51.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The error thrown in your line of code is because you are trying to calculate the cross product of two 1-dimensional tensors in dim4.<br>
However, even if you fix it by using <code>torch.cross(m1, m2, 0)</code>, you’ll get the following error:</p>
<pre><code class="lang-python">RuntimeError: dimension 0 does not have size 3
</code></pre>
<p>I’m not a mathematician, but as far as I know the cross product only exists in 3- and 7-dimensional Euclidean space. W. S. Massey wrote a <a href="https://pdfs.semanticscholar.org/1f6b/ff1e992f60eb87b35c3ceed04272fb5cc298.pdf" rel="nofollow noopener">paper</a> about it.</p>
<p>Would the cos formula for the angle calculation work for you or are you somehow bound to the sin formula?</p>
<pre><code class="lang-python">angle = torch.acos(torch.dot(m1, m2) / (m1.norm() * m2.norm()))
</code></pre>

Ingrid_Bernal: <p>Thanks, actually I could use the cos angle to find the sin.</p>
<p>cos = torch.dot(m1, m2) / (m1.norm() * m2.norm())<br>
sin = torch.sqrt(1-(cos*cos))</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5867e7b91230d09d68cf0b93ab599c6838a49580.jpeg" alt="cos" data-base62-sha1="cC4Dg00Niw49mjjshbSqVHSgxtS" width="149" height="60"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9ba81e0f2cfe199ee7af39a3c26dc5ede1fd602f.jpeg" alt="sin" data-base62-sha1="md0cgs8mkpDpHxA9DylOaB1q9or" width="206" height="59"></p>
Plot & save the image with PIL & numpy

lzozo95: <pre><code class="lang-auto">    plt.subplot(1, 3, 3)
    plt.title('Detected Face')
    plt.imshow(new_img)
    plt.axis('off')
    plt.show()

    # new_img = np.transpose(new_img,(1,0,2))

    new_img = Image.fromarray(new_img,"RGB")
    new_img.save(filename)
</code></pre>
<p>I used the code like above. I show the image with the variable called ‘new_img’. It shows me to the image perfectly. However, when I change the numpy array with ‘fromarray’ command, it may not work. I want to save the image same with which I printed at plt.imshow. How can I modify the code???</p>

Oli: <p>It looks good to me, perhaps your image has the wrong shape. Can you print the shape of the new_img? <code>print(new_img.shape)</code></p>
<p>What error are you getting with the current code?</p>
Easy VGG feature extraction module

mmcm: <p>If you ever wanted to do this:</p>
<pre><code class="lang-python">r11, r31, r51 = vgg_net.forward(targets=['relu1_1', 'relu3_1', 'relu5_1'])
</code></pre>
<p>then this module is for you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>This is something I made to scratch my own itch. I wanted to extract multiple features from (mostly VGG) models in a single forward pass, by addressing the layers in a nice (human readable and human memorable) way, without making a subclass for every combination of features I want to run.</p>
<p>After being unhappy with the solution I found <a href="https://discuss.pytorch.org/t/accessing-any-feature-of-pretrained-net-by-name/15568">here</a> I sat down and made a module that is pretty much a replacement for <code>torchvision.models.vgg</code>. And now I finally published it as a PyPI module, instead of copying the files from one project to the next <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue_closed_eyes.png?v=9" title=":stuck_out_tongue_closed_eyes:" class="emoji" alt=":stuck_out_tongue_closed_eyes:"></p>
<p>You can find it on github:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/black-puppydog/pytorch_vgg_named" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/189241?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/black-puppydog/pytorch_vgg_named" target="_blank" rel="nofollow noopener">black-puppydog/pytorch_vgg_named</a></h3>

<p>Provides VGG networks to do things like this: r11, r31, r51 = net.forward(targets=['relu1_1', 'relu3_1', 'relu5_1']). Nuff said. :) - black-puppydog/pytorch_vgg_named</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>or simply install it via pip:</p>
<pre><code class="lang-auto">pip install pytorch-vgg-named
</code></pre>
<p>I also added the <strong>normalized VGG</strong> weights provided by Leon Gatys. If you use these, pay attention to the note on normalization in the readme file.</p>
<p>If you need help, or have feedback, reach out to me here or on github, or twitter, or email, or or or… <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
Normalize gradient after unfolding

pmeier: <p>I’m extracting patches out of a <code>Tensor</code> with the <code>unfold</code> method. Consider the following snippet:</p>
<pre><code class="lang-python">import torch

torch.manual_seed(0)

numel = 10
length = 3
stride = 2

input = torch.ones(numel,).requires_grad_(True)
target = torch.zeros(numel,)

input_patches = input.unfold(0, length, stride)
target_patches = target.unfold(0, length, stride)
</code></pre>
<p>In the subsequent processing I calculate some loss between the <code>input_patches</code> and <code>target_patches</code>:</p>
<pre><code class="lang-python">loss = torch.sum((input_patches - target_patches) ** 2.0)
loss.backward()
print(input.grad)
</code></pre>
<p>With this some elements of <code>input</code> receive a higher <code>grad</code>ient than others, since they are included in multiple patches:</p>
<pre><code class="lang-auto">tensor([2., 2., 4., 2., 4., 2., 4., 2., 2., 0.])
</code></pre>
<p>Within my application this is unwanted. I can calculate the number of times each element was used as follows:</p>
<pre><code class="lang-python">counts = torch.zeros(numel)
count_patch = torch.ones(length)

for idx in range(0, numel - length + 1, stride):
    counts[idx:idx+length].add_(count_patch)

print(counts)
print(input.grad / torch.clamp(counts, min=1.0)) # clamp to avoid zero division
</code></pre>
<p>This results in ‘stable’ <code>grad</code>ients</p>
<pre><code class="lang-auto">tensor([1., 1., 2., 1., 2., 1., 2., 1., 1., 0.])
tensor([2., 2., 2., 2., 2., 2., 2., 2., 2., 0.])
</code></pre>
<p>which is what I need.</p>
<ol>
<li>While this example is in 1D I actually need 2D data. I have not tested it yet, but I imagine the needed double <code>for</code> loop is costly. Is there a better / faster / more efficient way to achieve the same result with builtins from <code>torch</code>?</li>
<li>My idea to implement this properly is to define an <code>autograd.Function</code>, as can be seen below, and apply this to the input before I extract the patches. Do you think this is a proper way to do this?</li>
</ol>
<pre><code class="lang-python">class _PatchNormalizer(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, length, stride):
        counts = _PatchNormalizer.get_patch_counts(input, length, stride)
        ctx.save_for_backward(counts)
        return input

    @staticmethod
    def backward(ctx, grad_output):
        counts, = ctx.saved_tensors
        grad_input = grad_output / counts
        return grad_input, None, None

    @staticmethod
    def get_patch_counts(input, length, stride):
        counts = torch.zeros_like(input)
        count_patch = torch.ones(length)

        for idx in range(0, input.size()[0] - length + 1, stride):
            counts[idx:idx + length].add_(count_patch)

        return torch.clamp(counts, min=1.0)
</code></pre>