akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Reshaping in custom layers

Xia_Zhengmi: <p>I’m implementing a custom layer that needs multiple reshaping of two input tensors, THE BATCH IS  OF THE SHAPE R^(N, a,b,c), where N is the number of tensors in the batch. Actually I need to reshape each 3D tensor with the size of R^(a,b,c) to be a matrix of the size (a*b,c) and then use torch.mm to multiply them. But I keep running into an error for the inconsistency of size (because torch.mm requires the inputs to be of 2D shape rather than 3D (in case of N&gt;1) while custom layer reads the input as a four dimensional tensor (N,a,b,c). Any help will be highly appreciated.</p>

Xia_Zhengmi: <p>Below you can find the code:<br>
class mmm(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(mmm,self).<strong>init</strong>()<br>
def forward(self, x, y):<br>
xp=torch.reshape(x,(x.shape[0]*x.shape[1],x.shape[2]))<br>
yp=torch.reshape(y,(y.shape[0]*y.shape[1],y.shape[2]))<br>
out=torch.mm(xp,yp.permute(1,0))<br>
return out</p>

InnovArul: <p>You can try using <code>torch.matmul</code>.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/generated/torch.matmul.html" target="_blank" rel="noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/generated/torch.matmul.html" target="_blank" rel="noopener">torch.matmul — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Good benchmarks for transfer learning?

iacolippo: <p>Hi everyone, I’m working on a new method to do transfer learning, but I can’t find good metrics to test if/how much my method improve on the existing one.</p>
<p>What are common benchmarks in transfer learning for models trained on ImageNet?</p>

lelouedec: <p>Time of training ? accuracy ?</p>

iacolippo: <p>I’m asking about possible datasets <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I’m going to benchmark training time, accuracy, memory utilization and inference time.</p>

lelouedec: <p>ohh my bad, mhh <a href="https://archive.ics.uci.edu/ml/datasets.html" rel="nofollow noopener">https://archive.ics.uci.edu/ml/datasets.html</a> some idea here <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"> Try some where there are objects or animals which is closer to imagenet first. that is what I would do.</p>

iacolippo: <p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> The issue with this problem is that I didn’t find a well defined benchmark in the literature and it is hard to design a meaningful one without being biased by my own method.</p>

enemis: <p><a class="mention" href="/u/iacolippo">@iacolippo</a> did you figure out anything on this? <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Query padding mask and key padding mask in Transformer encdoer

lisyuan: <p>Hi, I’m confusing in the padding masking of transformer.<br>
The following picture shows the self-attention weight of the query(row) and key(coloumn).<br>
As you can see, there are some tokens "&lt;PAD&gt;" and I have already mask it in key using <code>nn.MultiheadAttention</code>. Therefore the tokens will not calculate the attention weight.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fdda1641f3cb2c1e154592b5a8b1cf04b5070610.jpeg" alt="擷取" data-base62-sha1="AdG32UQ6pYvANLuylulZphutEZi" width="338" height="317"><br>
There are two questions:</p>
<ol>
<li>In query part, can I also mask them except for the red square part? Is this reasonable?</li>
<li>How can I mask "&lt;PAD&gt;" in the query?<br>
The attention weights use the <code>softmax</code> function along the row. If I set all the "&lt;PAD&gt;" row into <code>-inf</code>, the <code>softmax</code> will return <code>nan</code> and the loss with be <code>nan</code></li>
</ol>
How to flip random choosen samples in tensor?

hadaev8: <p>So, batch.flip(2) works fine, but how to flip only half of it?</p>

ptrblck: <p>You could index/slice the tensor, flip one part of it, and reconstruct it afterwards.</p>

hadaev8: <p>Is this a good way?</p>
<p>perm = torch.randperm(batch.size(0))<br>
idx = perm[:batch.size(0)//2]<br>
batch[idx] = batch[idx].flip(3)</p>

ptrblck: <p>Yes, your code looks good and seems to work fine:</p>
<pre><code class="lang-python">batch = torch.cat((torch.zeros(4, 2, 4, 2), torch.ones(4, 2, 4, 2)), dim=3)
print(batch)

perm = torch.randperm(batch.size(0))
idx = perm[:batch.size(0)//2]
batch[idx] = batch[idx].flip(3)

print(batch)
</code></pre>
How to select 1 sample but keep batch dimension in effiency way?

ThaiThien: <pre><code class="lang-auto">from torchvision.io.video import read_video, write_video
from torchvision.io import write_png
import os
v, a, info = read_video("/home/tt/Videos/VID_20201202_133703_090.mp4", pts_unit='sec')
print(v.shape)  # torch.Size([467, 1080, 1920, 3])
single_frame = v[100]
print(single_frame.shape)  # torch.Size([1080, 1920, 3])

batch_size_one = single_frame.unsqueeze(0)
print(batch_size_one.shape) # torch.Size([1, 1080, 1920, 3])
</code></pre>
<p>It required an unsqueeze.<br>
It might not efficiency.</p>
<p>I would like to get the frame and predict (model that train on single image).</p>

InnovArul: <p>I am not sure if <code>unsqueeze()</code> has a computation overhead. Does it?</p>
<p>Alternatively, you can use <code>single_frame = v[100:101]</code> to select single frame that contains a 1 in batch dimension.</p>
Using grad to calculate derivative of two outputs w.r.t. the same input

lvoet: <p>Hi,</p>
<p>I’m trying to use the autograd to get derivatives of two output values (y_a and y_b) w.r.t. the same input vector (x). I implemented two ways of doing this, which give me different results and I don’t understand why. Only the second method gives me the results.</p>
<p>Method 1: applying x.requires_grad_(True) only once</p>
<pre><code class="lang-auto">x.requires_grad_(True)
y_a = … # Calculation of y_a
y_a.backward(retain_graph=True)
dy_a/dx = x.grad
y_b = … # Calculation of y_b
y_b.backward(retain_graph=True)
dy_b/dx = x.grad
</code></pre>
<p>Method 2: applying x.requires_grad_(True) twice</p>
<pre><code class="lang-auto">x.requires_grad_(True)
y_a = … # Calculation of y_a
y_a.backward()
dy_a/dx = x.grad
x.requires_grad_(True)
y_b = … # Calculation of y_b
y_b.backward()
dy_b/dx = x.grad
</code></pre>
<p>Thanks for the help!</p>

InnovArul: <p>Make sure to zero the gradients (<code>if x.grad: x.grad.zero_()</code>) before calculating new <code>y</code>.</p>
<p>Some more info:</p><aside class="quote quote-modified" data-post="12" data-topic="4903">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/b77776/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/12">Why do we need to set the gradients manually to zero in pytorch?</a> 
  </div>
  <blockquote>
    y.backward() doesn’t just assign the value of y’(x) to  x.grad (say y depends on x). It actually adds y’(x) to the current value of x.grad (think it as x.grad += true_gradient). 
In the following example, y.backward() is called 5 times, so the final value of x.grad will be 5*cos(0)=5. 
import torch
from torch.autograd import Variable

x = Variable(torch.Tensor([[0]]), requires_grad=True)

for t in range(5):
    y = x.sin() 
    y.backward()
    
print(x.grad) # shows 5

Calling x.grad.data.zero_…
  </blockquote>
</aside>
Index tensor with list

eukaryote31: <p>I have a tensor <code>T</code> of shape (x, y) and a list <code>L</code> of shape (x), containing numbers [0, y). I want to have a tensor of shape (x), where each the <code>i</code>th element is <code>T[i, L[i]]</code>. I need this to be differentiable, and ideally with only pure torch operations and without loops. Apparently, T[:, L] gives a tensor of shape (x, x), where every element of L indexes every row.</p>
<p>Currently, I’m using this horrendous workaround, which feels extremely inefficient:</p>
<pre><code class="lang-auto">R = (T[:, L] * torch.eye(x)).sum(dim=0)
</code></pre>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">A, B = 10, 5
x = torch.randn(A, B)
idx = torch.randint(0, B, (A,))
x[torch.arange(A), idx]
</code></pre>

eukaryote31: <p>Thanks, that worked!</p>

Jaideep_Valani: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
<code>l=tensor([100,  12,  75,  55])</code><br>
i have this tensor list</p>
<p>and i want to use it his way<br>
X[:,- l:]<br>
but i will get an error saying only<br>
<code> only integer tensors of a single element can be converted to an index</code></p>
<p>basically i want to mask certain range of labels/outputs while computing the loss/metric.<br>
eg. for first tensor element in Batch i  want to use elements starting only from -100:,for second element  from -12: and like wise</p>

ptrblck: <p>I don’t think this would be possible via direct indexing as the result tensors would have a variable size, wouldn’t they?<br>
In that case you could loop all indices and append the result tensors to a <code>list</code>.</p>

Jaideep_Valani: <p>Actually that would add to epoch time my bs is 1280 .<br>
I would have loop over so many elements.</p>
<p>Not sure what can be other method of masking with variable no of elements to skip from every time</p>
GRU input as matrix

YuvalA: <p>Hi,<br>
I am using GRU model for many to one prediction.<br>
My input is represented at each time step as vector with length N, while using one feature, dim: (N, 1), The model is working as expected with dimension of (BATCH_SIZE, SEQ_LEN, N).<br>
I want to add new feature for each point in the N rows, dimension: (N, 2),  <strong>which will change the input dim to: (BATCH_SIZE, SEQ_LEN, N, 2) -How can I feed it to the GRU model? The model expected to (BATCH_SIZE, SEQ_LEN, N)</strong>, however, if I use  2*N it won’t be good by the problem’s logical.</p>
<p>Thanks</p>

googlebot: <aside class="quote no-group" data-username="YuvalA" data-post="1" data-topic="105880">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuvala/48/14996_2.png" class="avatar"> YuvalA:</div>
<blockquote>
<p>however, if I use 2*N it won’t be good by the problem’s logical.</p>
</blockquote>
</aside>
<p>If you have some additional input structure, it is best to learn some fixed size 1d “summaries” before RNN, as none of standard implementations supports input groups or structures beyond vectors.</p>
Custom Dataloader with few patients

LoiseauNicolas: <p>Hello everyone,</p>
<p>I will try to explain my problem.<br>
I am using a triplet network with triplet loss, and to train my model i use patches extracted around many key-points detected from 3D scanners.<br>
I have Few different patients (M), and few transformations per patients (N) with many keypoints per transformations (O keypoins). Then, i have approximately N*O patches per patients which i know exactly correspondences.<br>
I make a dataloader where I generate an array containing X triplets possible. I use a dataloader to load data from this array which contain my triplets.</p>
<p>Here is my problem. I don’t know inter-patients correspondances and I can’t mixed patients inside a mini-batch. I want to make a dataloader who can take into account of this, same patient for the actual minibatch and with the same patient for the the mini-batch at each epoch.</p>
<p>Tell me if i’m not clear,<br>
Thank you in advance !</p>
Preallocate memory for function outputs

Sobir_Bobiev: <p>This question is about efficiency and speed. Say, I need to run a function several times that returns a tensor and want to combine the results into a big tensor.<br>
Here are examples of two ways to do this:</p>
<pre><code class="lang-auto"># assume that funtion() returns (16,16) tensor
# first way
a = torch.empty(10, 16, 16)
for i in range(10):
    a[i,:,:] = function(i)

# second way
a = []
for i in range(10):
    a.append(function(i))
a = torch.stack(a, 0)

</code></pre>
<p>… and there are more ways, e.g. using <code>torch.cat</code></p>
<p>First natural question: which way is generally more efficient or faster for memory and/or computation?</p>
<p>Second: in both presented examples, to my knowledge, the function output is first <strong>stored as a separate tensor</strong>. Then, during stacking or assigning to a slice, it is <strong>copied</strong> from one memory location to another, which I believe is not efficient. So, is there any way to make the <code>function</code> output <strong>be directly stored into a preallocated memory location</strong> (in this case, the tensor <code>a</code>'s slice)?</p>
<p><strong>UPDATE:</strong> Ok, it turns out it is not much possible with custom python functions. So, the question becomes narrower, involving only some PyTorch operations which always output a new tensor, not sharing memory with anything else.</p>
<p>Thanks in advance.</p>

googlebot: <p>Most or all basic pytorch operations have optional “out” argument, that’s exactly preallocated memory mode. Unfortunately, errors are thrown when tensor arguments require gradients. One workaround is to wrap this in autograd.Function, but this also requires manually writing backward().</p>
U-net loss does not decrease

artykov: <p>Hello guys,<br>
I am trying to implement U-net for binary semantic segmentation task. The model input is rgb image with shape=(N, 3, 224, 224), output shape is (N, 1, 224, 224). I use Adam optimizer with default lr. And my loss function is nn.BCEWithLogitsLoss(). I am going to overfit 2-5 data samples but my loss does not decrease much. What should I do? Any help is appreciated. Thank you in regard.<br>
I hope you can help me <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
About create a random noise tensor with the same size of existing tensor in C++

Chen0729: <p>Hi, All</p>
<p>I have an inquiry about creating a random noise tensor with the same size of existing tensor.</p>
<p>the python code is:<br>
noise1=torch.randn(x.size())*0.3</p>
<p>but in C++, I cannot write like<br>
torch::Tensor noise = torch::randn({x.size()}) * 0.3; it does not allow to have x.size()</p>
<p>as the size of tensor x is varying, I cannot explicit write down all the dimensions of x, is there a better way to handle it please ?</p>
<p>Any comments are welcome.</p>
<p>Thanks.</p>

ptrblck: <p>I’m unsure what the right approach to pass the <code>size()</code> to the c’tor would be, but this should work as well:</p>
<pre><code class="lang-auto">auto noise = torch::empty_like(x)
noise.normal_(0., 0.3);
</code></pre>

Chen0729: <p>Thanks a lot.</p>
<p>That is very helpful.</p>
When is the closure called by the optimizer (in the optional and required cases)?

vkubicki: <p>On the Pytorch documentation <a href="https://pytorch.org/docs/stable/optim.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/optim.html</a>, the code to call <code>.step()</code> for an optimizer that needs a closure is:</p>
<pre><code class="lang-auto">for input, target in dataset:
    def closure():
        optimizer.zero_grad()
        output = model(input)
        loss = loss_fn(output, target)
        loss.backward()
        return loss
    optimizer.step(closure)
</code></pre>
<p>I have two similar questions:</p>
<ul>
<li>If I use an optimizer that needs a closure, would the closure function be called for the first evaluation ? Because the call to <code>zero_grad()</code>, <code>backward()</code>… does not appear outside the closure in the snippet.</li>
<li>If I use an optimizer that does not needs a closure and I provide one, do I still need to manually call <code>zero_grad()</code>, <code>backward()</code> before calling <code>.step(closure)</code> ?</li>
</ul>
<p>Those two question sum up as: if I provide a closure function, will it be called no matter the optimizer, therefore in all cases I do not need to call <code>zero_grad()</code>, <code>backward()</code>… before calling <code>.step(closure)</code> ?</p>
Is there a function to do this operation?

ooyamatakehisa: <p>When I have a tensor “x” (size: T), I want to get a tensor “dist” (size: (D, T)) like the following. (D &lt; T)</p>
<pre><code class="lang-auto">dist = torch.tensor(
    [
        [x[0], x[1], x[2], ..., x[T-1]],
        [x[-1], x[0], x[1], ...,x[T-2]],
        [x[-2], x[-1], x[0], ...,x[T-3]],
        ...
        [x[-D], x[1-D], x[2-D], ..., x[T-D]]
    ]
)
</code></pre>
<p>What is the best way to do this?<br>
Thank you!</p>

LeviViana: <p>This will do the job:</p>
<pre><code class="lang-python">import torch
D = 3
T = 4
x = torch.rand(T)

idx = torch.arange(T)
idx = (idx[None,:] - idx[:, None])%T

x.repeat(D).index_select(0, idx.view(-1)[:D * T]).view(D, T)
</code></pre>
<p>If <code>T</code> is very big, then you could have some memory issues, and a for loop would be better for constructing the idx tensor.</p>
<p>I’m not sure it is the best way, but it should be good enough for many contexts.</p>

ooyamatakehisa: <p>Thanks for the answer!<br>
That’s a very clever approach. I would never have thought of that.</p>
<p>I’m going to do this on a tensor of size batch_size×T with batch_size=20,T=6000.</p>
We looking for a PyTorch Graduate Machine Learning Engineer

lisajt: <p><strong>Responsibilities:</strong></p>
<ul>
<li>Producing high-quality code that delivers in accordance with the desired specification while also being<br>
readily extendable by other team members;</li>
<li>Being involved in the entire development life cycle, including the scope, design and implementation;</li>
<li>Demonstrating excellent attention to detail while taking pride in what is produced;</li>
<li>Work closely with Computer Vision Engineers, members of the Product team and the wider Hawk-Eye<br>
team as and when needed and communicate effectively to ensure the best solutions are built.</li>
</ul>
<p><strong>Requirements</strong></p>
<ul>
<li>Highly competent in Python and C++;</li>
<li>Competent in CUDA.</li>
</ul>
<p><strong>Nice to have:</strong></p>
<ul>
<li>Knowledge of traditional vision processing methods;</li>
<li>Experience with Pytorch and TensorRT;</li>
<li>Experience with real-time application development.</li>
</ul>
<p><strong>For application or more information please email us: lisa[]<a href="http://jobtensor.com" rel="noopener nofollow ugc">jobtensor.com</a>, <a href="https://jobtensor.com/uk" rel="noopener nofollow ugc">https://jobtensor.com/uk</a></strong></p>

Canopus: <p>Hey Lisa, I have dropped you an email, speak to you soon!</p>
<p>Have a great weekend!</p>
<p>–<br>
Regards</p>
Parameter/Weight Sharing in Bidirectional RNN

ryanleary: <p>I’m looking for some pointers as to how one might go about sharing input-hidden weights for both directions of a recurrence with Bidirectional RNNs. Does anyone have suggestions about how that might be done?</p>
<p>Any guidance greatly appreciated. Thanks!</p>

smth: <p>I think you would have to write your own RNNCell and do a bidirectional yourself.</p>

raraz15: <p>Hello! I had to implement a BiRNN with shared input matrices (W_ih) and I did it successfully. However, I can not write such a  class that uses the PackedSequence object.</p>
<p>Any recommendations?</p>
Network is not deterministic with random seed

spadel: <p>I am currently using the following random seeds:</p>
<pre><code class="lang-auto">torch.manual_seed(7)
torch.cuda.manual_seed(7)
np.random.seed(7)
</code></pre>
<p>I would have expected, that this should always produce the same results given identical hyperparameters. Unfortunately it is not, am I missing something?</p>

nairbv: <p>Do you mean that they aren’t identical to each other, or re-seeding doesn’t allow you to reproduce a set of values using the same seeding mechanism? It might help to provide a more concrete/complete example.</p>
<p>Random tensors created on cuda rely on the philox counter based RNG. CPU RNG uses Mersenne.</p>
<p>They use different RNGs because they are fundamentally different APIs. On a CPU we can sequentially choose subsequent values, but on CUDA we use “counter” based RNGs that can populate all values in parallel.</p>

spadel: <p>Thanks for your reply. Sorry for not being concrete - what I mean is, that I expect to always attain the same outputs/loss after each epoch, when restarting the training with the same hyperparameters. I.e. I expect the network to reproduce previous results.</p>
Use tensor.mean() method but ignore 0 values

jwillette: <p>I was calling <code>nonzero()</code> on a tensor and then getting the mean values, but it turns out that I will need to keep the shape of the original tensor, but just ignore the values that are 0 for the mean calculation, is there a way to do this?</p>

JuanFMontesinos: <p>I don’t understand what do you mean by “keep the shape…”. A mean is an scalar, thus, you can’t “keep the shape”.</p>
<p>You just can filter them out like mean = tensor[tensor !=0].mean()</p>

jwillette: <p>right, but the original tensor has a shape and I need to preserve the shape which would exist if I were to do…</p>
<pre><code class="lang-auto">x = torch.ones((2, 3, 4))
x.mean(dim=0)
</code></pre>
<p>which would be a <code>(3,4)</code> tensor of the means along dimension 0. I have found that I could do this by summing the elements along the dimension and then somehow dividing by the count of nonzero elements along the same dimension…</p>
<p>But then I realized my problem is a little bit harder than that because I am dealing with one tensor of values and then one probability distribution tensor with the same shape, which gives the probabilities of those values…</p>
<pre><code class="lang-auto">x = torch.rand((2, 3, 4))
y = torch.distributions.Normal(torch.randn((2, 3, 4), torch.randn((2, 3, 4)))
pr = y.log_prob(x)
...
</code></pre>
<p>at this point I need to get the mean of dimension 0 of the values in y which are 0 in <code>x</code> but they are not 0 in <code>y</code> because of the <code>log_prob</code> call… very confusing about how to go about it</p>

JuanFMontesinos: <p>So then use a masking<br>
If I got it you has a tensor (N,M,P)</p>
<pre><code class="lang-auto">mask = x!=0
y_mean = (y*mask).sum(dim=0)/mask.sum(dim=0)
</code></pre>
<p>I think this would fit what you want right?</p>

jwillette: <p>Yes I believe that will work. thanks for taking the time to show me the way</p>

Uwais_Iqbal: <p>You could use <code>.nonzero()</code> but just reshape back to the original dimensions</p>
<p><code>x[x.nonzero(as_tuple=True)].view(-1, x.shape[1])</code></p>
<p>and then do <code>torch.mean()</code> as normal</p>
Averaging non-zero tensors along an axis

sajastu: <p>Hello everybody!</p>
<p>I am wondering how we can average on the second dimension of a tensor without taking zero arrays into account. See below:</p>
<pre><code class="lang-auto">A = (tensor) [ [ [1, 2, 3], [0, 0 , 0], [3, 4, 5] ],
               [ [0, 0, 0], [5, 6, 7], [0, 1, 2] ]  
              ]

Avg(A) = (tensor) [  [ [2, 3, 4] ], 
                     [ [2.5, 3.5, 4.5] ]  
                  ]
</code></pre>

spanev: <p>Hi <a class="mention" href="/u/sajastu">@sajastu</a>,</p>
<p>Can you explain how do you obtain <code>3</code> for Avg(A[0][1]) when <code>A[0][1] == [0, 0, 0]</code>?<br>
I would expect this output:</p>
<pre><code class="lang-auto">Avg(A) = [[2, 0, 4  ],
          [0, 6, 1.5]]

</code></pre>
<p>There might be (I hope) a cleaner way, but you can do this:</p>
<pre><code class="lang-auto">avg = A.sum(axis=2) / (A != 0).sum(axis=2).clamp(min=1).float()
</code></pre>
<p>(clamp to avoid division by zero)</p>
<p>EDIT: whoops, I misread “without taking zero in arrays into account”, above solution does not count. Still I am curious to know how you get this output</p>

sajastu: <p>Thanks for the reply <a class="mention" href="/u/spanev">@spanev</a></p>
<p>I should edit the question, I want to calculate the mean along the <em>second</em> axis. The first array (first row) in tensor Avg(A) is calculated by averaging two <em>non-zero</em> arrays in tensor A. As such:</p>
<pre><code class="lang-auto">[2, 3, 4]  =  Avg( ([1, 2, 3], [3, 4, 5]) )
[2.5, 3.5, 4.5]  =  Avg( ([5, 6, 7], [0, 1, 2]) )
</code></pre>

spanev: <p>Oh right, second axis makes sense. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<p>Well, this should work:</p>
<pre><code class="lang-auto">avg = A.sum(axis=1) / (A.sum(axis=2) &gt; 0).sum(axis=1).view(-1,1).float()
</code></pre>
<p>In the denominator, we are counting the number of non-zero arrays on the second axis. We can then rely on <a href="https://pytorch.org/docs/stable/notes/broadcasting.html" rel="nofollow noopener">broadcasting</a> for the division.</p>

Uwais_Iqbal: <p>To get the non zero indices you could use <code>.nonzero(as_tuple=True)</code></p>
<pre><code class="lang-auto">x[x.nonzero(as_tuple=True)].view(-1, x.shape[1])
</code></pre>
<p>will give you back the original tensor with all the zero tensors removed. Then you could just use <code>.mean(dim=1)</code> as usual</p>
Confusion about image values when using tanh as an activation function

Li-Yun: <p>Hello,</p>
<p>I have seen in many GAN repositories, where tanh is used as a generator activation function, input images not be in the range [-1,1] but in [0,1].</p>
<p>I noticed the same thing when I tried to replicate some networks and train them. When using images normalized in range [-1,1] I get bad images in the first epoch whilst in the other case training, regarding losses and generated images “seems” to be more stable.</p>
<p>More specifically, I want to ask:</p>
<ol>
<li>
<p>Is it “valid” to train such a network, with images in the range [0,1] and tanh as an activation?</p>
</li>
<li>
<p>Is it valid to use instead (tanh+1)/2 in the outermost layer activation in the generator to enforce the values be in the range [0,1]</p>
</li>
</ol>

ptrblck: <p>I’m not sure what “valid” means in this case. Are you concerned about some theoretical claim and think this normalization approach is “invalid”?<br>
If so, I wouldn’t be too concerned about it (although I would still be interested to hear about your concerns), as I claim that a lot of explanations are written after the model training (finally) worked. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Li-Yun: <p>My main concern is about how would one decide the appropriate normalization values for the input images. Is the safest way, to run the experiments with all the available combinations, i.e tanh activation with values in [0,1] and [-1,1], and decide based on the crispness of the images?<br>
Furthermore, if some special losses are used (pre-trained VGG like style, perceptual), these networks were trained with images normalized in the range [0,1] so another confusion occurs.</p>
<p>In any case, as I understand from your answer, do you mean that the network’s “power” is too strong and alleviates all the “mismatches” and finally learns the appropriate weights regardless of the image scales?</p>
Get mask with multi index tensor

MehdiZouitine: <p>Hey !</p>
<p>I have a tensor of index called POS that contains 2D coordinates of agents.<br>
I would like to make mask using that coordinates. I tried with torch.scatter_   function but i didn’t succeed.</p>
<p>Input :</p>
<pre><code class="lang-auto"> pos  = tensor([[[5, 4],
         [9, 7]],

        [[5, 4],
         [9, 7]]])
pos.shape
&gt;&gt; (2,2,2)
mask = torch.zeros(2, 10, 10)
</code></pre>
<p>Expected output :</p>
<pre><code class="lang-auto">tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.]],

        [[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.]]])
</code></pre>
<p>Thanks you !</p>

ptrblck: <p>This code should work:</p>
<pre><code class="lang-python">pos = torch.tensor([[[5, 4],
                     [9, 7]],
                    [[6, 5],
                     [8, 6]]])
mask = torch.zeros(2, 10, 10)

mask[torch.arange(2).unsqueeze(1), pos[:, :, 0], pos[:, :, 1]] = 1.
print(mask)
&gt; tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.]],

          [[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]])
</code></pre>
<p>Note, I’ve changed the indices for the second batch tensor so check, if it would work without repeated indices.</p>
How to conduct convolution over part of a feature map?

wenbo_zhang: <p>I am gonna to conduct convolution over parts of a feature map instead of the whole feature map, which is somewhat like the operation in this paper[Image Inpainting for Irregular Holes Using Partial Convolutions(<a href="https://arxiv.org/abs/1804.07723" rel="noopener nofollow ugc">https://arxiv.org/abs/1804.07723</a>).</p>
<p>The difference between what I want to do and the paper is that I want to <strong>save the conputational cost to speed up inference</strong> by abandon convolution operation in non-target loaction of a feature map.</p>
<p>Specifically, suppose there is a mask with the same size of a feature map , which is used to annotate wether to take this location as the center of a convolution operation. 1 denotes <em>doing</em>, 0 denotes <em>not doing</em>. By this mask, convolution operation can be conducted only at the location marked 1, For those location marked 0, just preserve the value at like before. This idea is to save computation cost.</p>
<p>Is it able to achieve this idea in pytorch?</p>
ValueError: 1 is not in range (CustomDataSet)

akib62: <p>Hi, I have some images in a folder. I made my <code>CustomDataSet</code> get these images.</p>
<pre><code class="lang-auto">def __init__(self, main_dir, label_full, transform):
           #Reading path and doing some operations
          #sorting images based on the name
        self.all_imgs = sorted(os.listdir(main_dir), key= lambda x : int(x.split("_")[0]))

def __getitem__(self, idx):
         train_label_price = torch.tensor((self.labels[idx]))
         img_loc = os.path.join(self.main_dir, self.all_imgs[idx])
        image = Image.open(img_loc).convert("RGB")
        tensor_image = self.transform(image)

        return (tensor_image.type(torch.double), train_label_price.type(torch.double))
</code></pre>
<p>When I am running my <code>data_loader()</code> method I am getting an error</p>
<pre><code class="lang-auto">def data_loader():
    # Some path and transformation ...
    train_loader = DataLoader(train_data_tensor, batch_size= 10, num_workers= 0, shuffle= False)
    
    train_data_loader_iter = iter(train_loader)
    images, labels = train_data_loader_iter.next()
    print(labels) // Nothing is printing
</code></pre>
<p>The error I am getting</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/akib/.local/lib/python3.8/site-packages/pandas/core/indexes/range.py", line 355, in get_loc
    return self._range.index(new_key)
ValueError: 1 is not in range
</code></pre>
<p>Could you tell me, why I am getting this error and why my <code>CustomDataSet</code> is not working?</p>

ptrblck: <p>Your code snippet doesn’t seem to include all definitions, as e.g. <code>self.labels</code> is not defined in the <code>__init__</code> method. Based on the error message it seems <code>self.labels</code> (or any other missing code) is a <code>pandas.DataFrame</code>, which is raising this indexing issue, so you would have to make sure the length of this <code>DataFrame</code> matches the length of the dataset.</p>
How to distinguish between validation and train set when using random_split for custom dataset?

Anno: <p>I defined a simple custom data set like in the official <a href="https://pytorch.org/tutorials/recipes/recipes/custom_dataset_transforms_loader.html" rel="noopener nofollow ugc">example</a> and named the class “Dataset”.  Then in my training method I split the data set into training and validation like this:</p>
<pre><code class="lang-auto">    dataset = Dataset(opt)
    train_size = int(0.8*len(dataset))
    val_size = len(dataset) - train_size
    lengths = [train_size, val_size]

    train_dataset, val_dataset = torch.utils.data.dataset.random_split(dataset, lengths)
    trainloader = DataLoader(
        train_dataset,
        batch_size=opt.n_batches,
        shuffle=True,
        num_workers=opt.n_workers,
        pin_memory=True
        )
    valloader = DataLoader(
        val_dataset,
        batch_size=1,
        shuffle=True,
        num_workers=opt.n_workers,
        pin_memory=True
        )
</code></pre>
<p>In the <code>__getitem__(self, idx)</code> method of the data set I would like to have different functionality for validation set than for training set. How can I make a difference? Should I initialize the datasets independently?</p>
<p>I am basically looking for a variable like <code>self.training</code> in the nn.Module class (see <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module" rel="noopener nofollow ugc">here</a>).</p>

ptrblck: <p>Instead of using <code>dataset.random_split</code> I would create indices for both splits, create two separate datasets with the custom transformations etc. applied to the training and validation dataset, and wrap both datasets in <code>Subset</code>s using the indices.<br>
This would allow you to pass the different arguments or transformations without manipulating some internal flags.</p>
Hyperparameter tuning in multi-task learning

wqsgsutd: <p>HI, all,<br>
I am working on a multi-task learning project.<br>
I need to tuning hyperparameters in MTL model. I have try Gridserchcv in sk-learn. it cannot work.<br>
how do I  optimize hyperparameters?</p>
<pre><code>class MTLnet(nn.Module):
def __init__(self):
    super(MTLnet, self).__init__()
    self.sharedlayer = nn.Sequential(
        nn.Linear(feature_size, shared_layer_size),
        nn.ReLU(),
        nn.Dropout()
    )
    self.tower1 = nn.Sequential(
        nn.Linear(shared_layer_size, tower_h1),
        nn.ReLU(),
        nn.Dropout(),
        nn.Linear(tower_h1, tower_h2),
        nn.ReLU(),
        nn.Dropout(),
        nn.Linear(tower_h2, output_size)
    )
    self.tower2 = nn.Sequential(
        nn.Linear(shared_layer_size, tower_h1),
        nn.ReLU(),
        nn.Dropout(),
        nn.Linear(tower_h1, tower_h2),
        nn.ReLU(),
        nn.Dropout(),
        nn.Linear(tower_h2, output_size)
    )    

def forward(self, x1,x2):
    out1 = self.tower1(x1)
    out2 = self.tower2(x2)
    return out1, out2
</code></pre>
<p>Thank you!</p>