akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can i select certain GPUs with DataParallel?

davidlee: <p>Hello, PyTorch!</p>
<p>Now i have totally 8 GPUs in my workstation and i want to train LSTM model with 2 GPUS out of 8.<br>
(Since 6 GPUs, the rest of total one is occupied.)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5c009898d46905b938ca491b66b3f150c64036a8.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c009898d46905b938ca491b66b3f150c64036a8" title="5"><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5c009898d46905b938ca491b66b3f150c64036a8.png" alt="5" data-base62-sha1="d7TcFDJOvbc9uvp1Zic5Q7j67AQ" width="690" height="91" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">5</span><span class="informations">1356×180 5.79 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please let me know how i can select two GPUs with DataParallel.</p>
<p>Thanks, PyTorch!</p>

alex.veuthey: <p>Have a look at <a href="https://discuss.pytorch.org/t/how-to-change-the-default-device-of-gpu-device-ids-0/1041/2">this solution</a>, it’s the preferred approach!</p>
<p>The other solution is to specify which GPUs to use with PyTorch internal functions like <code>torch.cuda.set_device()</code>, but it’s not recommended, I think it’s because there might be some index mismatch between <code>nvidia-smi</code> and the internal values… Most likely other reasons too.</p>

davidlee: <p>Hi alex.veuthey<br>
Thank you for your help.</p>
<p>Can i just solve this problem with “nn.DataParallel(model, device_ids=[0,1])”??</p>

alex.veuthey: <p>Yes, probably, but there might be the issue of indexing mismatch between PyTorch and <code>nvidia-smi</code>. Try and see!</p>
Load custom data pytirch

Rohan_Kumar: <p>How do i load custom csv data in pytorch. The dataset contains 2 text fields and one label field. I want to load the data, batch it, apply embedding, how do i go about doing it using torch.utils.data.dataset?  It’s very confusing.</p>

mmisiur: <p>There is extensive <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">tutorial</a> on <a href="http://pytorch.org" rel="nofollow noopener">pytorch.org</a> about creating your own <code>Dataset</code> and <code>DataLoader</code>. You shouldn’t have problems with adapting it into your case. However, if you’ll encounter some specific difficulties feel free to ask here.</p>
globalATenDispatch c++ memory exception after upgrading to 1.2 stable

Alex1: <p>Hi all,</p>
<p>I am using libtorch and I’ve just upgraded from Pytorch 1.1 to 1.2 stable. The (formerly working code) compiles ok, but I cannot even torch::cat two GPU tensors because of a c++ memory exception in</p>
<p>static inline Tensor cat(TensorList tensors, int64_t dim)</p>
<p>somewhere around globalATenDispatch.</p>
<p>From what I undestand, this dispatcher is something new, what am I missing here? Should the functions like cat be somehow registered to the dispatcher first?</p>
<p>Running Win 10.</p>
<p>Thanks!</p>
<p>Alex</p>

yf225: <p><a class="mention" href="/u/alex1">@Alex1</a> This seems like a bug (similar to <a href="https://github.com/pytorch/pytorch/issues/25298" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/25298</a>). Could you post an issue at <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a>? We will triage properly.</p>

Alex1: <p>Ok, I’ve created an issue, it’s easily reproducible.</p>
<p><a href="https://github.com/pytorch/pytorch/issues/25457" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/25457</a></p>
Pytorch distributed training on extremely large GPU fleet

arvindmohan: <p>I was wondering if anyone can point me to resources on how Pytorch scales on extremely large scale parallel jobs - I am researcher with a group working on ML for computational physics, with all my codes being in Pytorch.</p>
<p>Currently we are evaluating ML frameworks which can scale on super-massive distributed jobs i.e. &gt; 5000 GPUs.  Any idea on what is the largest distributed job using Pytorch so far (both in public domain and internally at Facebook) ?  Are there any fundamental limitations which prevent it from scaling at that level? If so, can I expect any improvements with 1.0 release?</p>
<p>I would like to avoid re-writing all my codes in Tensorflow for scaling, so I’d really appreciate some input. Thanks in advance…</p>

teng-li: <p>For imagenet, for converged model accuracy, it entirely depends on the total batch size of the model training. In other words, the largest number of ResNet batch size without model accuracy loss I personally have done is 4096 images, even though other people have published about going as large as 18-24K batch size.</p>
<p>So imagine, if you put 32 batch size per GPU, with 4096 images, you will be distributing your distributed job to 128 GPUs.  From my own experience, from pure HPC performance and scalability point of view, I have done 64 nodes * 8 GPU/node  (512 GPUs) training myself with pretty good linear scalability. Of course, the scalability depends upon your model size, your HPC cluster network setup.</p>
<p>So what is your network interconnect, Infiniband or Ethernet? at what speed and what model are you trying to train?</p>

arvindmohan: <p>Thanks for your input. Certainly interesting that you can see scalability for 512 GPUs - We are looking at really large scale and 5000 GPUs would be the lower bound. Some facts about the hardware:</p>
<ol>
<li>IBM Power9 with Nvidia Volta V100 GPUs</li>
<li>GPUs have NVLink (Gen 2) interconnect, with 4 GPUs/node.</li>
<li>Mellanox EDR Infiniband.</li>
</ol>
<p>I want to train a custom architecture which is similar to a convolutional recurrent network - we will be training on several terabytes of training data. Apart from the inherent issues of communication latency etc., is there any limitation in Pytorch as such when I scale up?</p>
<p>I am having difficulty finding resources to learn distributed GPU training with Pytorch (apart from documentation) - What approach  did you use? I am currently experimenting with Horovod on a much smaller scale, but I’d prefer using Pytorch’s native distributed module, if I can only get some nice tutorials to study. Thanks again for your insights.</p>

Lyken17: <p>I am using Horovod for distributed training. You only need to change few lines to make your code distributed.</p>

Shisho_Sama: <p>Its been several months since your last post. Can you lighten us up about what you ultimately did?<br>
Did you use Pytorch? How did you go about it? what did you do? was it good?<br>
or no ?<br>
I’d really appreciate it</p>
Connection reset by peer from torch.distributed.recv

xzhu1900: <p>Hi,</p>
<p>I am using torch distributed with gloo backend (because I need peer to peer communication). While running my test script, I got a ‘Connection reset by peer’ error while dist.recv is called. Any clue on what causes this?</p>
<p>I am using mpi to launch 2 processes and my script is pasted as below:</p>
<blockquote>
<p>def run(rank, size):<br>
tensor = torch.zeros(1).cuda()<br>
if rank == 0:<br>
tensor += 1<br>
# Send the tensor to process 1<br>
dist.send(tensor=tensor, dst=1)<br>
else:<br>
# Receive tensor from process 0<br>
dist.recv(tensor=tensor, src=0)<br>
print('Rank ', rank, ’ has data ', tensor[0])<br>
def init_processes(rank, size, addr, fn, backend):<br>
os.environ[‘MASTER_ADDR’] = addr<br>
os.environ[‘MASTER_PORT’] = ‘12345’<br>
my_rank = os.environ[‘OMPI_COMM_WORLD_RANK’]<br>
dist.init_process_group(backend, rank=my_rank, world_size=size)<br>
fn(rank, size)<br>
if <strong>name</strong> == “<strong>main</strong>”:<br>
hostname = socket.gethostname()<br>
addr = socket.gethostbyname(hostname)<br>
size = 2<br>
my_rank = os.environ[‘OMPI_COMM_WORLD_RANK’]<br>
init_processes(my_rank, size, addr, run, ‘gloo’)</p>
</blockquote>
<p>The error I got is as below:</p>
<blockquote>
<p>File “/home/xzhu1900/anaconda3/envs/test_py37/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 712, in recv<br>
pg.recv([tensor], src, tag).wait()<br>
RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:563] Read error [127.0.1.1]:48028: Connection reset by peer</p>
</blockquote>
How to dataloader delete memory

6fb4b64ff0a68bcbf8fa: <p>hello i’m james.</p>
<p>my model use three dataset.</p>
<p>and computer memory is 48gb.</p>
<p>batch size is 16.</p>
<p>each data load, and train model–&gt; three times.</p>
<p>ex)<br>
first: cifar10 train<br>
second : svhn train<br>
third : mnist train</p>
<p>my memory leaked.</p>
<p>so i want to know that when first train is finished, delete first dataloader’s memory.</p>
<p>how to delete dataloader’s memory?</p>
<p>thank you</p>

ptrblck: <p>When you see a memory leak, it won’t help to delete the <code>DataLoader</code> or the <code>Dataset</code>.<br>
Are you storing the computation graph somewhere, e.g. by appending the <code>loss</code> to a list without detaching?</p>
How to visualzie the output of an autoencoder trained on images?

n0obcoder: <p>i am training a Denoising AutoEncoder and i want to visualize its outputs. How do i do it<br>
One thing that i noticed is that the model takes in tensors of <strong>dtype</strong> <strong>torch.float32</strong></p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad():
    for batch_idx, (imgs, noisy_imgs) in enumerate(val_loader):
        clean_imgs = model(noisy_imgs)
        print(imgs.shape, imgs.dtype)
        print(noisy_imgs.shape, noisy_imgs.dtype)
        print(clean_imgs.shape, clean_imgs.dtype)
</code></pre>
<pre><code class="lang-auto">torch.Size([4, 1, 64, 256]) torch.float32
torch.Size([4, 1, 64, 256]) torch.float32
torch.Size([4, 1, 64, 256]) torch.float32
</code></pre>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>If you are using <code>matplotllib</code>, this code should work:</p>
<pre><code class="lang-python">fig, axarr = plt.subplots(clean_imgs.size(0))
for img, ax in zip(clean_imgs, axarr):
    img = img.squeeze(0).numpy()
    ax.imshow(img)
</code></pre>
How to generate a random displacement field followed by gaussian and trainable?

John1231983: <p>I am using grid_sample function, that</p>
<pre><code class="lang-auto">torch.nn.functional.grid_sample(input, grid, mode='bilinear', padding_mode='zeros')
</code></pre>
<p>I want to construct a random grid and it trained with the network. The scrip likes</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
          self.rnd_grid_tensor = ... (required_grad=True)

   def forward(self,input):
        output = torch.nn.functional.grid_sample(input, self.rnd_grid_tensor, mode='bilinear', padding_mode='zeros')
        return output
</code></pre>
<p>How to generate the rnd_grid_tensor so that the value in the grid followed by gaussian distribution. Thanks</p>
<p>This is my solution but I am not sure is it correct or not</p>
<pre><code class="lang-auto">from torch.autograd import Variable
def gaussian(ins, is_training, mean, stddev):
  if is_training:
      noise = Variable(ins.data.new(ins.size()).normal_(mean, stddev))
      return ins + noise
  return ins

grid = torch.rand((1,2,16,16), requires_grad=True)
grid = gaussian (grid, True, 1, 0.1)
</code></pre>

bnehoran: <p>Do you want a random displacement field where the displacements follow a Gaussian distribution?</p>
<p>If so, what you need to do is generate a tensor with the displacements</p>
<pre><code class="lang-auto">disp_field = torch.FloatTensor(1,2,16,16).normal_(mean, std).requires_grad_()
</code></pre>
<p>Then add it into an identity grid, which you can create using <a href="https://pytorch.org/docs/stable/nn.functional.html#affine-grid" rel="nofollow noopener"><code>affine_grid()</code></a>:</p>
<pre><code class="lang-auto">id_grid = torch.nn.functional.affine_grid(torch.FloatTensor([[[1, 0, 0],[0, 1, 0]]]), size=(1,2,16,16))
</code></pre>
<p>and then pass the sum to <code>grid_sample()</code>:</p>
<pre><code class="lang-auto">output = torch.nn.functional.grid_sample(input, id_grid + disp_field)
</code></pre>
<p>Just note that the mean and standard deviation will be in the grid units of a half-image (ie. a displacement of <code>1</code> will be equivalent to <code>(size - 1)/2</code> pixels).<br>
So if you have the mean and std in pixels, you have to divide them both by <code>(size - 1)/2</code>.</p>
<p>I hope this helps.</p>
How to obtain 3D grid from example of 2D grid?

John1231983: <p>I have a code to generate 2D grid</p>
<pre><code class="lang-auto">        B, C, H, W = x.size()
        # mesh grid 
        xx = torch.arange(0, W).view(1,-1).repeat(H,1)
        yy = torch.arange(0, H).view(-1,1).repeat(1,W)
        xx = xx.view(1,1,H,W).repeat(B,1,1,1)
        yy = yy.view(1,1,H,W).repeat(B,1,1,1)
        grid = torch.cat((xx,yy),1).float()
</code></pre>
<p>How to obtain 3D grid by adding zz in the code? This is what I tried</p>
<pre><code class="lang-auto">       B,C,D,H,W=input.size()
        xx = torch.arange(0, W).view(1, 1,-1).repeat(D, H, 1)
        yy = torch.arange(0, H).view(1, -1,1).repeat(D, 1, W)
        zz = torch.arange(0, D).view(-1, 1,1).repeat(1,H,W)
        xx = xx.view(1,1,D,H,W).repeat(B,1,1,1,1)
        yy = yy.view(1,1,D,H,W).repeat(B,1,1,1,1)
        zz = zz.view(1,1,D,H,W).repeat(B,1,1,1,1)
        grid = torch.cat((xx,yy,zz),1).float().to('cuda')
</code></pre>
<p>But i feel it wrong in somewhere. Could you verify it?</p>

ptrblck: <p>Was <a href="https://discuss.pytorch.org/t/how-to-obrain-3d-mesgrid-in-pytorch/54062/2">this suggestion</a> not working?</p>

John1231983: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: Yes. Because it is care about the order. I can use it to make grid but when I feed to grid_sample () function. The output looks rotated.</p>
<p>For example <code>xx = torch.arange(0, W).view(1, 1,-1).repeat(D, H, 1)</code> or <code>xx = torch.arange(0, D).view(-1, 1,1).repeat(1, H, W)</code></p>

bnehoran: <p>The easiest way to obtain a grid for use with the <code>grid_sample()</code> function is probably to use <a href="https://pytorch.org/docs/stable/nn.functional.html#affine-grid" rel="nofollow noopener"><code>torch.nn.functional.affine_grid()</code></a>.</p>
<p>You give it an affine matrix, and it returns a grid that you can then pass to <code>grid_sample()</code>. If you want an identity grid (no transformation), you can just pass it the identity affine matrix, which for 3D data is</p>
<pre><code class="lang-auto">aff = torch.FloatTensor([[[1, 0, 0, 0],[0, 1, 0, 0],[0, 0, 1, 0]]])
</code></pre>
<p>For example,</p>
<pre><code class="lang-auto">aff = aff.expand(B, 3, 4)  # expand to the number of batches you need
grid = torch.nn.functional.affine_grid(aff, size=(B,C,D,H,W))
torch.nn.functional.grid_sample(3d_data, grid)
</code></pre>
<p>So yeah, you could probably get the same effect using your code sample if you do it just right, but this should be much easier.<br>
I hope this helps.</p>
RuntimeError: Inplace operation and .backward()

Natasha_667: <p>Hello guys!</p>
<p>I’m getting the following error when calling .backward():</p>
<p>Encounter the RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
<pre><code class="lang-auto">for i, j, k in zip(X, Y, Z):
    A[:, i, j] = A[:, i, j] + k
</code></pre>
<p>X and Y are leaf tensors. Z is a non-leaf tensor resulting from the previous layer.<br>
X, Y pairs are non-unique: Ex. (3, 9, 0.5), (3, 9, 0.4) should result in (3, 9, 0.9)</p>
<p>I’ve tried .clone(), torch.add(), torch.sum(), and so on.</p>
<p>Please help! <img src="https://discuss.pytorch.org/images/emoji/apple/roll_eyes.png?v=9" title=":roll_eyes:" class="emoji" alt=":roll_eyes:"></p>

spanev: <p>Hi <a class="mention" href="/u/natasha_667">@Natasha_667</a> ,</p>
<p>If you don’t absolutely need these operations to be in-place, this should work:</p>
<pre><code class="lang-auto">B = A.clone()
for i, j, k in zip(X, Y, Z):
    B[:, i, j] = A[:, i, j] + k
</code></pre>
<p>or this what you are referring to when you say “I’ve tried .clone()”?</p>

Natasha_667: <p>Thanks Serge! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I was using it wrong <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
How to debug this unusual symbol look up error

NuttyNet: <p>Hi! I am trying to run a python script that uses torch. I get the following error. I have Python 3.6.8</p>
<blockquote>
<p>symbol lookup error: /home/workspace/Ubuntu1804/libtorch_python.so:: undefined symbol:PyThread_tss_alloc</p>
</blockquote>
<p>I am getting the same error when I <code>import torch</code> in IPython 3.6 in my terminal and haven’t been able to find much references to this specific type of symbol look up error.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/building-from-source-keeps-failing-ubuntu-18-04-02-lts-no-gpu/41637/8">This post</a> described the same undefined symbol.<br>
Could you try to create a new clean conda environment and reinstall PyTorch?</p>
To implement a variant of conv op in low-level APIs

nickkk: <p>My target is to implement a variant of convolution op, and I have implemented it in C++ and CUDA through CPP extension. However, the speed is not satisfactory, I guess I’ll have to implement it with lower level APIs. It seems “aten/src/ATen/native/Convolution.cpp” is the right place to do so, and it further calls “aten/src/THNN/generic/SpatialConvolutionMM.c”, so I modify “SpatialConvolutionMM.c”. The problem is after compiling, I cannot see any changes on the outputs. Maybe the files in THNN are wrapped in “ATen/native” in some way, I just cannot figure it out. Could anyone give me some suggestions? Thank you.</p>

ptrblck: <p>Are you sure the particular method is called in your current convolution?<br>
There are some switches in <code>Convolution.cpp</code>, so that maybe another algorithm might be used?</p>
Dueling Netowrk is getting worse over training episodes

lrpy_trainer: <p>Dear community, could you tell me, what is going wrong, if my reward is getting worse over time and the loss is increasing?</p>
<p>I am using a simple dueling network architecture with linear layers.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/53a7c57c6da979d7b626741a18d4dd249a016b1d.png" data-download-href="https://discuss.pytorch.org/uploads/default/53a7c57c6da979d7b626741a18d4dd249a016b1d" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/53a7c57c6da979d7b626741a18d4dd249a016b1d_2_690x193.png" alt="grafik" data-base62-sha1="bW34SraSZzCqtXSd5lXcrgNC9OZ" width="690" height="193" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/53a7c57c6da979d7b626741a18d4dd249a016b1d_2_690x193.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/53a7c57c6da979d7b626741a18d4dd249a016b1d_2_1035x289.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/53a7c57c6da979d7b626741a18d4dd249a016b1d.png 2x" data-dominant-color="ECF2F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">1230×345 28.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Compiling from source for compute capability 2.0 and Cuda 8.0

hxtk: <p>I’ve got a machine with a Tesla C2075 and a Quadro 5000. Both of these cards are compute capability of 2.0, and from this post, it appears I need to be using CUDA 8.0 or lower <a href="https://stackoverflow.com/a/28933055/1860372" rel="nofollow noopener">1</a>.</p>
<p>Additionally, my graphics cards do not support CuDNN. When I follow the instructions on the “Get Started Locally” page <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">2</a>, it installs perfectly and even returns true for <code>torch.cuda.is_available()</code>, but alas it errors out saying my GPU is too old to be supported when I try to move something into GPU memory. My guess is it has something to do with the fact it’s installing this version of PyTorch, which references CuDNN: <code>py3.7_cuda8.0.61_cudnn7.1.2_2</code>.</p>
<p>My next idea was to try building from source as it appears from this post <a href="https://discuss.pytorch.org/t/is-pytorch-fully-functional-without-cudnn/7175">3</a> that I don’t strictly need CuDNN support to use Cuda. However, when I try to build PyTorch from source with the steps specified in the getting started page <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">2</a> which specifically indicate Cuda 8.0, CMake encounters an error when it detects Cuda 8.0, saying “PyTorch requires CUDA 9.0 and above.”</p>
<p>Unless I’m missing something, it seems like that documentation is not in sync with the latest release. I think it’d be worth my while to get this working if it is feasible to do so, but now that a custom build has refused to compile, I’m not sure what to try next.</p>
Implement batch gradient descent by accumulate loss for multiple samples

Leo: <p>Hi,</p>
<p>So my training samples are of different sizes, for which reason I can’t stack them into one tensor and do in the traditional bgd way. So I created a custom <code>collate_fn()</code> to store all samples in a batch to a list, and during training, I go over each sample in the list and perform forward pass as usual. Since I want to implement the bgd in this way, I summed up the losses and backprop at the end. However, I found that the GPU memory usage is the same as the case where batch size = 1. I assume all input samples within a batch should be kept in GPU memory in order to retain all these graphs. So does this indicate this implementation is not working? If so, what’s the correct way to do so? Any help would be highly appreciated.</p>
<p>I also post my code script for training as follows:</p>
<pre><code class="lang-auto">for data_batch, label_batch in loader:
    loss = 0
    for (data, label) in zip(data_batch, label_batch):
        data = data.float().to(self.dev)
        label = label.long().to(self.dev)

        output = self.model(data)
    
        loss += self.loss(output, label)

    self.optimizer.zero_grad()
    loss.backward()
    self.optimizer.step()
</code></pre>
Cannot import torch in anaconda at python3 but works in python2

Shin_Yu_Wu: <p>I am using Mac and jupyter notebook(anaconda).</p>
<p>From the PyTorch website, I’ve tried to install PyTorch by:</p>
<pre><code class="lang-auto">conda install pytorch torchvision -c pytorch
</code></pre>
<p>and</p>
<pre><code class="lang-auto">pip3 install torch torchvision
</code></pre>
<p>When I  <code>import torch</code>  in jupyter notebook, the error shows:</p>
<pre><code class="lang-auto">ModuleNotFoundError: No module named 'torch'
</code></pre>
<p>However, in the terminal, if I enter python3 environment by:  <code>$ python3</code> , then  <code>import torch</code> , there is no error.</p>
<p>Then I tried:  <code>conda list pytorch</code>  in the terminal, and found:</p>
<pre><code class="lang-auto"># Name                    Version                   Build  Channel
pytorch                   1.2.0                   py2.7_0    pytorch
</code></pre>
<p>So I finally notice in the jupyter notebook, if I use python2 and import torch, there would be no error. But I don’t know how to install pytorch in python3 environment? I’ve already followed the PyTorch website  <code>pip3 install torch torchvision</code> , but it still not works.</p>
Custom batchnorm1d

ussamazahid96: <p>Hi,</p>
<p>I want to implement a batchnrom1d which should use a single (scalar) gamma, beta, mean and std rather than vectors. I want the layer to normalize the whole input mini-batch using its mean and variance and similarly  multiply the normalized input to scalar gamma and add beta. All parameters of this custom batchnorm1d  layer will be scalars rather than vectors.</p>
<p>Just like <a href="https://lasagne.readthedocs.io/en/latest/modules/layers/normalization.html#lasagne.layers.BatchNormLayer" rel="nofollow noopener">batchnorm layer of lasagne</a> offers a parameter named ‘axes’ which is use to achieve the above functionality. I was wondering if there is already such functionality available in Pytorch or do I have to write my own batchnorm layer for this?</p>
<p>Thanks</p>
InstanceNorm1D vs BatchNorm1D

abiro: <p>Hi,</p>
<p>I’m not sure if I should use <code>InstanceNorm1D</code> or <code>BatchNorm1D</code> in my network and I’d be grateful for some help.</p>
<p>I have an output  <code>x</code> of shape <code>(N, L)</code> where <code>N</code> is the number of elements in the batch and <code>L</code> is the number of activations. I’d like to perform normalization for each <code>l</code> in <code>L</code> where the statistics are computed across <code>x[:,l]</code> and there are separate parameters gamma and beta for each <code>l</code>. Based on the docs it seems to me that both of the following layers will achieve the desired effect:</p>
<ol>
<li><code>torch.nn.BatchNorm1d(L, affine=True)</code></li>
<li><code>torch.nn.InstanceNorm1d(L, affine=true)</code></li>
</ol>
<p>and there would only be a difference if I had an output <code>(N, C, L)</code>. Is this correct?</p>
<p>Thanks!</p>
Residual block conv2d

slavavs: <p>Residual for max_pool I did. But I don’t understand how to make alignment for layers.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cd2998ca7e7dbf74c02de761fdabb4864fecf0a8.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd2998ca7e7dbf74c02de761fdabb4864fecf0a8" title="resid"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cd2998ca7e7dbf74c02de761fdabb4864fecf0a8.png" alt="resid" data-base62-sha1="tgX08GHwbdrE723yx1P6YTmzzfi" width="609" height="500" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">resid</span><span class="informations">660×541 6.13 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

slavavs: <p>I solved this problem by concatenating the layers by a forced measurement</p>
A fast way to apply a function across an axis

IssamLaradji: <p>Is there an efficient way to apply a function such as <code>torch.inverse</code> for a matrix of size <code>(n, m, m)</code> where the function is applied to each of the <code>(m, m)</code> matrices ?</p>
<p>It seems that this does the job:</p>
<pre><code class="lang-auto">def apply(func, M):
    tList = [func(m) for m in torch.unbind(M, dim=0) ]
    res = torch.stack(tList, dim=0)

    return res 

apply(torch.inverse, torch.randn(100, 200, 200))
</code></pre>
<p>but I am wondering if there is a more efficient approach.</p>
<p>Tensorflow functions seem to generically achieve that, as explained here: <a href="https://www.tensorflow.org/api_docs/python/tf/matrix_inverse" rel="nofollow noopener">https://www.tensorflow.org/api_docs/python/tf/matrix_inverse</a><br>
but I am not sure whether their method uses a for loop or parallelizes the process.</p>
<p>It would be interesting to see a benchmark between the approach I see used in Pytorch and the one in Tensorflow. I will put the results of the benchmark later today.</p>

smth: <p>We dont support batch inverse right now, looks like TF does. I wonder if they do anything more sophisticated than a for-loop internally.</p>

IssamLaradji: <p>Does tensor comprehensions allow us to apply a function across an axis much faster ?</p>

sksenthilkumar: <p>I was also using unbind and stack as the equivalent of apply along axis in numpy. But the greatest problem was that it increased the processing by 2 times. The only way around this problem is to somehow convert the function as matrix operation. Luckily for me I was able replace the axis operation with a series of matrix multiplication.</p>
When use the loss.backward with L1 loss

hyuntae: <p>Hi.</p>
<p>I have a short question about “torch.nn.L1Loss()”.</p>
<p>As we know, it is not possible to differentiate |x| when x is zero (because the derivative of |x| is x/|x|).</p>
<p>However, when I used torch.nn.L1Loss() with network, loss.backward() worked…</p>
<p>How is it possible?</p>
<p>Also, would you explain how it is possible to calculate gradient of L1 Loss in pytorch when x is zero?</p>
<p>Thanks.</p>

spanev: <p>What is <code>x</code> in your example? Is it the input of the model?</p>
<p>Let’s say <code>x</code> your input (with <code>requires_grad=True</code>) and the first layer of the model in a Linear with no bias, parameters <code>w</code>, doing <code>y = w*x</code>.</p>
<p>The last step of the backprop would be <code>dL/dx = dy/dx * dL/dy</code><br>
and <code>dy/dx = w</code>, no matter what is the value of <code>x</code>.</p>

hyuntae: <p>Thanks for reply. And sorry for confusing.</p>
<p>x is just a variable of the equation ‘y = |x|’, not the input or output of network.</p>
<p>I just wonder when try to differentiate the y = |x|, how the torch handle when x is 0 although it is impossible to differentiate at that point.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6c3e7f7060ac0a36941a170e414df30ecfc038bb.png" alt="1" data-base62-sha1="frzt00SZkehJOYQZgosU5COsY4j" width="253" height="199"></p>

spanev: <p>Ok, my bad, I mistook the pipes in <code>|x|</code> for <code>l</code>.</p>
<p>I think it uses an approximate gradient on 0. And as you can see <a href="https://github.com/pytorch/pytorch/blob/e8acc2ebb17ec2ce80808f57bfcd68d7d6cbcb1f/aten/src/THCUNN/Abs.cu#L20" rel="nofollow noopener">here</a>, it considers 0 as a positive input.</p>

hyuntae: <p>Thank you very much!</p>
<p>Then, I need to find the implementation of approximate.</p>
<p>Thanks!</p>
PIL Image to FloatTensor (uint16 to float32)

fumoffu947: <p>I have tif images that have a data type of <strong>unsigned int 16 bit</strong>.<br>
PIL can read these images without problem and have the correct type.<br>
But as my custom data set reader reads this tif image and then tries to contert it to a tensor, for the normal normalization and then usage in the network, things goes wrong.</p>
<p>I read the image which have values from zero to the max value of <strong>uint16</strong>.<br>
then i use the standard way transformation:</p>
<pre><code class="lang-auto">image_transform = transfroms.Compose([
          transforms.ToTensor(),
          transforms.Normalize((value,), (value,))
)]
</code></pre>
<p>This code will result in errors.<br>
As the ToTensor function will convert images with value between [0, 255] and of some specific format to a tensor with values between [0,1] (thus a float tensor).<br>
But for other images it will keep the same datatype and just convert the values.</p>
<p>So for this case it will take the data type closes to a <strong>unsigned int16</strong> which is a <strong>signed int16</strong>…<br>
This will result in overflows and non correct data.</p>
<p><strong>So the question is how to do it an easy (using torch) and fast way?</strong></p>
<p>They way i do it is to first convert to a numpy array; then convert to a signed <strong>float 32</strong> then to a float tensor, that can be used as normal.</p>
<pre><code class="lang-auto">image_fp = open("filepath", "rb")
image = PIL.Image.open(image_fp)
im_arr = np.array(image)
im_arr32 = im_arr.astype(np.float32)
im_tensor = torch.tensor(im_arr32)
im_tensor = im_tensor.unsqueeze(0)
</code></pre>
<p>And this results in this ugly lambda:</p>
<pre><code class="lang-auto">image_transform = transfroms.Compose([
          transforms.Lambda(lambda image: torch.tensor(numpy.array(image).astype(numpy.float32)).unsqueeze(0)),
          transforms.Normalize((value,), (value,))
)]
</code></pre>
<p>(All these conversion will impact the loading time with large datasets)</p>
<p><strong>[edit]</strong><br>
So the differense that can save alot of time is to use:</p>
<pre><code class="lang-auto">transforms.Lambda(lambda image: torch.from_numpy(numpy.array(image).astype(numpy.float32)).unsqueeze(0))
</code></pre>
<p>instead of the torch.tensor function</p>

ptrblck: <p>Your approach seems valid.<br>
You could maybe save a unnecessary copy by using <code>torch.from_numpy</code> instead of creating a new tensor.</p>

fumoffu947: <p>Yes, i made a little script with a for loop where i read an tif image and then did the conversion. This was repeated 100 000 times (with nothing else in the loop) and timed in linux. I saved about 30 seconds by using from_numpy (torch.tensor ~ 2 min, torch.from_numpy ~1.5 min for 100K repetitions)</p>
Unable to run backward through a list of losses

Suchismita_Padhy: <p>In my setup, I have 2 networks and I am trying to run <code>backward</code> on both.</p>
<pre><code class="lang-auto">my_losses = [net1.get_loss(), net2.get_loss()]
for loss in my_losses:
    loss.backward()
</code></pre>
<p>When I run this, the first iteration of the <code>for</code> loop works fine, but the second call to <code>backward</code> throws the following error :</p>
<p><code>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</code></p>
<p>If I wrap the <code>loss</code> in a <code>torch.autograd.Variable</code> and set <code>required_grad=True</code> this works. But I am not sure if that is doing the right thing. Also, I am not able to understand why I am seeing the error.</p>

Oli: <p>Hi! Better to create one final loss and do the backward once like so</p>
<pre><code class="lang-auto">loss = net1.get_loss() + net2.get_loss()
loss.backward()
</code></pre>
<p>If you don’t want to do this, have a look at the retain_graph argument to the backward function <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Glgl</p>
Missing keys & unexpected keys when loading model

davidlee: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/2c0a0ad3eb96f7157d4cb9369480a97ad535010c.png" data-download-href="https://discuss.pytorch.org/uploads/default/2c0a0ad3eb96f7157d4cb9369480a97ad535010c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/2c0a0ad3eb96f7157d4cb9369480a97ad535010c.png" alt="image" data-base62-sha1="6hAwv6jdlETPHxzxWLRJGXpJXOY" width="690" height="251" data-dominant-color="F7F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1234×450 7.18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi guys,<br>
I’m having trouble with loading saved weights.<br>
I just trained with local cpu, and i didn’t use any DataParallel.<br>
When i see torch.load(‘./seq2seq.pth’), its in OrderedDict.</p>
<p>Please help me!!</p>

ptrblck: <p>This message might be a bit misleading, but no missing or unexpected keys were found, so your code works fine.<br>
The output was changed to <code>&lt;All keys matched successfully&gt;</code> in PyTorch 1.2.0.</p>

davidlee: <p>I found it works fine.<br>
Really appreciate with your help, ptrblck!<br>
Have a good day:)</p>
Backward for operations in torch.nn.functional

hgzmuliw: <p>The “torch.nn.grad” package (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/grad.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/grad.py</a>) contains functions to manually compute backwards through a convolution operation. It is unclear why it has this name but only contains this for convolutions; I was not able to find documentation for this package or these functions. My question is, is there any similar Python interface for other operations, for instance F.max_pool2d? (It is quite simple to compute the gradient backward through max pooling. But I believe I remember finding this exact backward function somewhere in PyTorch just a few days ago, but I cannot for the life of me remember where, which is why I ask about this specifically.)</p>

ethancaballero: <p>Bump.</p>
<p>There is a <code>max_pool2d_with_indices_backward</code> in L5188 of <code>native_functions.yaml</code> <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L5188" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L5188</a><br>
and L1145 &amp; L1282 of <code>derivatives.yaml</code> <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml#L1145-L1282" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml#L1145-L1282</a></p>
<p>“Life of a Tensor” pytorch wiki seems to imply that all the functions listed in <code>native_functions.yaml</code> could be used as python functions:<br>
<a href="https://github.com/pytorch/pytorch/wiki/Life-of-a-Tensor" rel="nofollow noopener">https://github.com/pytorch/pytorch/wiki/Life-of-a-Tensor</a></p>
<p>I have no idea how to directly call <code>max_pool2d_with_indices_backward</code> via pytorch’s python frontend though.</p>
<p>How can I call <code>max_pool2d_with_indices_backward</code> (and all the other <code>*_backward</code> functions) via pytorch’s python frontend (without using <code>torch.Tensor.backward</code> or <code>torch.autograd.backward</code>)?</p>
MC approximation with GPU. Adding values in the loop

Desdi: <p>Hello, I got a problem with memory explosion, when implementing MC approximation of a function in the loop, generating tensors on GPU.</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">       # self.post_mean and self.post_logvar are parameters to optimize

       sampler_normal = torch.distributions.Normal(
            loc=torch.zeros_like(self.post_mean),
            scale=torch.ones_like(self.post_logvar))

        kl_cross = 0 # initialization for average of MC approximation
        n_iter = 100
        for i in range(n_iter):
            weights = sampler_normal.sample((1,))
            weights = self.post_mean + \
                torch.sqrt(post_var) * weights / torch.norm(weights) * radius 

            kl_cross += (torch.sum(-1 / 2 * torch.log(2 * prior_var) +
                                   -1 / 2 / prior_var * (weights - self.prior_mean)**2) /\
                         n_iter)
</code></pre>
<p>In the code above the memory for gpu explodes. I believe it happens because kl_cross.grad_function collects pointers to n_iter objects.  Thus, if I have several such approximations, then memory increases very quick.</p>
<p>What would be your suggestions to implement MC approximation on GPU?</p>
Unable to get pytorch 1.2.0 working with cuda 9.0

MAsad: <p>Hi,<br>
I have cuda 9.0, and I am trying to install pytorch 1.2.0 using the following:<br>
<code>pip3 install torch torchvision</code></p>
<p>However when I check <code>torch.cuda.is_available()</code> it returns false.<br>
Checking <code>torch.version.cuda</code> returns <code>10.0.130</code> which tells me that it installed cuda10 version.</p>
<p>My question is, is there a way I can install pytorch 1.2.0 with cuda 9.0?<br>
I have tried to look around but couldn’t find a way to do this.</p>
<p>On the <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">https://pytorch.org/get-started/locally/</a> page it tells me to run the above pip3 command for cuda 9.0.</p>

Zayd: <p>Per <a href="https://github.com/pytorch/pytorch/issues/24842" rel="nofollow noopener">issue #24842</a>, torch 1.2.0 does not come with the binaries for cuda 9.0.  You would need to install it from source.</p>
Conflict error message: Only 3D, 4D and 5D input Tensors supported (got 3D) for the modes: nearest | linear | bilinear | bicubic | trilinear (got bicubic)

ThaiThien: <p>pytorch version ‘1.1.0’</p>
<p>Here is simple code that make error</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

img = torch.rand(1,128,128)
img2 = F.interpolate(img, scale_factor=(1,0.5,0.5), mode="bicubic")
</code></pre>
<p>and error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/tt/project/crowd_counting_framework/bug/interpolate_bug.py", line 8, in &lt;module&gt;
    img2 = F.interpolate(img, scale_factor=(1,0.5,0.5), mode="bicubic")
  File "/home/tt/anaconda3/envs/pytorch_gpu/lib/python3.7/site-packages/torch/nn/functional.py", line 2577, in interpolate
    " (got {})".format(input.dim(), mode))
NotImplementedError: Input Error: Only 3D, 4D and 5D input Tensors supported (got 3D) for the modes: nearest | linear | bilinear | bicubic | trilinear (got bicubic)
</code></pre>
<p>The error is conflict by it self.<br>
It said “Only 3D, 4D and 5D input Tensors supported”, but confirm got 3D.<br>
also, " nearest | linear | bilinear | bicubic | trilinear" =&gt;  but got bicubic</p>
<p>It should OK.</p>

andreaskoepf: <p>To rescale a 2D 1-channel image the input should be 4D: batch x chan x height x width<br>
The error message indeed is a bit unclear.</p>
<p>See the <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.interpolate" rel="nofollow noopener">interpolate()</a> in the docs:</p>
<p>"Currently temporal, spatial and volumetric sampling are supported, i.e. expected inputs are 3-D, 4-D or 5-D in shape.</p>
<p>The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width.</p>
<p>The modes available for resizing are: nearest, linear (3D-only), bilinear, bicubic (4D-only), trilinear (5D-only), area"</p>
<p>The following should work:</p>
<pre><code class="lang-auto">img = torch.rand(1,1,128,128)
img2 = F.interpolate(img, scale_factor=(0.5,0.5), mode="bicubic")
</code></pre>
Does PyTorch replicate data for processing convolutional layers?

MahdiNazemi: <p>I wonder how PyTorch implements convolutional layers to ensure reasonable performance.</p>
<ol>
<li>How are input feature maps stored in memory?</li>
<li>Is data replicated when applying filters to different patches of the input?</li>
</ol>

ptrblck: <p>You can have a look at <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp" rel="nofollow noopener">Convolution.cpp</a> which uses some switches to decide, which backend to call or if to use the ATen implementation.<br>
E.g. if you are using the GPU, in some cases cudnn will be used, which itself uses different algorithms depending on the data type and shape. If you have a static input shape you could use <code>torch.backends.cudnn.benchmark=True</code> to let cudnn chose the fastest algorithm for your work load.</p>
Laying out vector values into a matrix - LDL decomposition

Demplo: <p>Hello,</p>
<p>I am predicting a set of parameters that generate the decomposition of positive definite matrix using the LDL decomposition.</p>
<p>Given a NxN matrix, instead of estimating the full matrices I decided to estimate the set of elements <code>l</code> and <code>d</code>  necessary to re-compose the L and D matrices (e.g. 3 for <code>L</code> and 3 for <code>D</code> in the 3x3 case, note that this does not scale linearly for larger <code>N</code>).</p>
<p>While for <code>D</code> I just have to put <code>d</code> on the diagonal, the situation for <code>L</code> is a little more complex. In fact the size of <code>L</code> is a triangular number that can be obtained as <code>-0.5+(0.25+2*l.size(1))**(1/2)+1</code>. The matrix is a lower triangular matrix having ones on the main diagonal (see <a href="https://en.wikipedia.org/wiki/Cholesky_decomposition#LDL_decomposition" rel="noopener nofollow ugc">LDL decomposition</a> for more info).</p>
<p>Now the problem lies in putting the components of the <code>l</code> vector in <code>L</code>.</p>
<p>I made a generic function that does that:</p>
<blockquote>
<p>def batch_l_to_L(l_vec):</p>
<pre><code>L_size = -0.5+(0.25+2*l_vec.size(1))**(1/2)
L_size = L_size + 1 
L = torch.eye(int(L_size))
L = L.unsqueeze(0)
L = L.repeat(l_vec.size(0),1,1)

for b in range(l_vec.size(0)):    
    it_count = 0
    for j in range(1,L.size(1)):
        for k in range(0, j):
            L[b,j,k] = l_vec[b,it_count]
            it_count = it_count + 1
</code></pre>
<p>return L</p>
</blockquote>
<p>but I am not sure everything goes well wrt autograd.</p>
<p><em>Question 1: is this approach alright or do I do some operation that does not preserve the gradients?</em></p>
<p>I checked the graph using <a href="https://github.com/szagoruyko/pytorchviz" rel="noopener nofollow ugc">PyTorchViz</a> and it comes out as a huge graph (of course encorporating the network and other parts of the loss but this is function contributing for the biggest part)</p>
<p>Then I did the manual layout for the case where <code>N=3</code> which is a 3 lines code:</p>
<blockquote>
<p><code> L = torch.eye(int(-0.5+(0.25+2*l.size(1))**(1/2))+1)</code><br>
<code>L = L.repeat(l.size(0), 1,1)</code><br>
<code>L[:,1,0] = l[:,0]</code><br>
<code>L[:,2,0] = l[:,1]</code><br>
<code>L[:,2,1] = l[:,2]</code></p>
</blockquote>
<p>and in this case the graph is way smaller.</p>
<p><em>Question 2: why is it the case? Do I preserve gradients this time as well?</em></p>
<p>Edit: I think the difference in the graph comes from the fact that in the function I iterate over the batch dimension as well, while I use slices below. Still I would like to know if this the most efficient way and if this implementation causes any trouble to autograd.</p>
Share weights between the same instance of a model and stacking models

mina: <p>I have a encoder-decoder architecture. I want to do two things:</p>
<ol>
<li>I wanna stack the same network three times</li>
<li>and share the weights of some layers (the layers from green areas), and not all layers (not the pink ones)<br>
Some thing Iike this image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/5/50cf4ad08623f792951a4c2b4c4d51cea7f00b5d.png" alt="stacked%20net" data-base62-sha1="bwSjN5kxfIZrY26DGLEtsU2xcDz" width="525" height="342"></li>
</ol>
<p>I am new to pytorch and I am not shore how to do it.<br>
I appreciate your help for any reference or sudo code.</p>

JuanFMontesinos: <p>Hi, to do so the easiest way is to generate two nn.Modules. One for the green part and another one for the pink one.</p>
<p>To share weights you only have to instantiate the class once. To have different ways you need to instantiate classifierX and axiliary classifierX 3 times.</p>
<p>something like:</p>
<pre><code class="lang-auto">class GlobalNet(nn.Module):
init---
    self.net_in_green = instance()
    self.classifier0 = pinkinstance()
    self.classifier1 = pinkinstance()
    self.classifier2 = pinkinstance()
forward(inputs)----
   Here you have to squeeze everything those 3 inputs into  batch dimension, assuming you have a single intput it would be similar to
 inputs #shape--&gt; batch, 3, other dimensions.-..
inputs = inputs.view(-1,other dimensions)
inputs #shape (3*batch, other dimensions)

Then you apply green net
greenout = self.net_in_green(squeezed input)
Now unsqueeze them with view

output0 = self.classifier0(greenout_unsqueezed0)
output1 = self.classifier1(greenout_unsqueezed1)
output02= self.classifier2(greenout_unsqueezed2)

The same with auxiliar classifier

</code></pre>
<p>The only drawback is that batch normalization is also shared…<br>
As far as I know it’s not straight forward  to achieve different statistics.</p>

reham: <p>Hi,<br>
I need to design a similar network where modules in the green area should weights, but there is a feedback from the pink area to the green area. Any idea how to do that?</p>