akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
2D Convolutional VAE loss getting stuck in a local minima

username: chaslie
cooked: <p>Ok, I give up.</p>
<p>I am trying to get a 2D convolutional autoencoder working with non image based data. The data is of size[-1,1,264,132].</p>
<p>The loss function get stuck at around 430, which means that some of the data is encoded, however some is not. I have tried playing with the hyper parameters on ADAM, and found that if i drop the LR to below 5e-4 the it gets a loss function Nan. I have tried different loss functions and nothing works (using the standard KLD value the error still gets stuck just at different non 0 values). The one i am currently using is:</p>
<pre><code class="lang-auto">def log_normal_pdf(sample,mean,logvar):
    log2pi=torch.tensor(np.log(2.*np.pi))
    return torch.sum(-0.5*((sample-mean)**2.*torch.exp(-logvar)+logvar+log2pi),1)
def loss_fn(Gen_model, data,lab):
    mean, logvar = Gen_model.encode(data,lab)
    z2=Gen_model.reparm(mean, logvar)
    out=Gen_model.decode(z2,lab)
    criterion = torch.nn.BCEWithLogitsLoss(size_average=True,reduce=False, reduction='mean')
    BCE=criterion(out,data)
    logpx_z=-torch.sum(BCE,(1,2,3),keepdim=False)
    logpz=log_normal_pdf(z2,torch.tensor(0.),torch.tensor(1.))
    logqz_x=log_normal_pdf(z2, mean, logvar)
    return -torch.mean(logpx_z+logpz-logqz_x),out

</code></pre>
<p>I have included the reparamatization</p>
<pre><code class="lang-auto">    def encode(self,data,lab):
        data=self.Encoder(data,lab)
        mean, logvar=torch.chunk(data,2,dim=1)
        return mean, logvar

    def reparm(self,  mean, logvar):
        epsilon = torch.randn(mean.size()).to(device)
        z = epsilon*torch.exp(logvar.mul(0.5))+mean
        #print("z=",z.shape)
        return z
</code></pre>
<p>I now i have got something wrong (obviously), but i cannot see it. I read somewhere that the the input data should be normalised with a mean of zero and an std of 1, however this cannot happen because the BCE function needs data between [0,1].</p>
<p>The raw data (before normalising) ranges from +1000 to -1000, normalising this data sets the pre normalised 0 to 0.5, the -1000 to 0 and +1000 to 1. the increments can therefore be very small, am i right in thinking that this maybe causing the issue??? if so, would it be better to create an input data set of [-1,3,264,132]?</p>
How to mark duplicates in a tensor

username: Beinan_Wang
cooked: <p>Given an int tensor of arbitrary shape, I would like to mark the first occurrence of every unique value as 0, and all the rest elements as 1, like below</p>
<blockquote>
<p>x = torch.tensor([1,2,3,1,3,4,5,2])<br>
y = mark_duplicates(x)</p>
</blockquote>
<p>y would be  [0,0,0,1,1,0,0,1], which has the same shape as x.</p>
<p>If torch.unique can return unique_indices like np.unique then the above can be achieved easily, but as far as I know it doesn’t. Is there an easy alternative?</p>
Optimizing and reverting nn.Embedding

username: catosphere
cooked: <p>Dear Pytorch community,</p>
<p>I am currently working on sparse/categorical data on which I am training auto-encoders like models.</p>
<p>For that purpose, it seems suited to have an embedding layer at the input of the encoder.<br>
The way I considered is passing my categorical data into an embedding layer to get float data into let’s say norm [-1,1] then encode-decode back to [-1,1].</p>
<p>At that point, I have a few questions about which I hardly find discussions, tutorials or guidelines … Maybe I am taking wrong solutions, please correct me if there is more suited processing for that case.</p>
<p><span class="hashtag">#1a</span> – reconstruction loss could be computed on the [-1,1] float data representation (like a L1 loss)<br>
<span class="hashtag">#2a</span> – reconstruction loss could be computed on the categorical data representation, then I need to invert the embedding transformation to reconstruct the categorical data from the [-1,1] decoder output … Is there a direct way to perform such a reverse transformation ??<br>
This inverted transformation would also be needed for <span class="hashtag">#1</span> as once my model trained, I would still need to reconstruct some categorical data from it.</p>
<p><span class="hashtag">#3a</span> – or I could define the decoder output to be of the size of the dictionary and output a logsoftmax prediction to be used with a classification loss. Then to recover the categorical data, I would take the categorical value corresponding to the bin of highest prediction probability in the dictionary. In my case, its size can go up to 7822 which which I am afraid could make the learning quite complicate ? Or does it seems reasonable ?</p>
<p>About optimizing such model, the nn.Embedding description points it should be used with a sparse optimizer (in case the embedding is trainable). Also, I set padding_idx=0 so that zeros are kept zero in the model, which I want. In such setting, what would be a best practice please ?</p>
<p><span class="hashtag">#1b</span> – not training on the embedding weight<br>
<span class="hashtag">#2b</span> – set a separate sparse optimizer for the embedding and a traditional optimizer for the auto-encoder<br>
<span class="hashtag">#3b</span> – optimize everything together with a sparse optimizer</p>
<p>Thank you very much for your recommendations !</p>

username: catosphere
cooked: <p>Up !</p>
<p>Anyone with recommendations on the right way to use nn.Embedding in that case ?</p>
<p>thanks</p>

username: catosphere
cooked: <p>By the way, if someone is passing-by here, there could be a typo on the torch.nn.functional.nll_loss documentation.</p>
<p>In case of a K-dimensional loss it should be a target shape of (N,d1,d2,…,dK) instead of (N,C,d1,d2,…,dK) (which is only applying to the input shape) ?</p>
All NaN output of TransformerEncoder with "normal" input

username: ataxias
cooked: <p>I am using a tansformer model (on the CPU) based on <code>nn.TransformerEncoder</code>. The first 2 layers before the transformer encoder layer are a nn.Linear projection layer and a fixed positional encoding layer (i.e. with no trainable parameters).</p>
<p>I have noticed that although the input to the model never includes NaN values or values very large in magnitude, the output of the model is always all-NaN. When using a simple linear layer instead of <code>nn.TransformerEncoder</code> the problem disappears. Gradients cannot be an issue, because I am evaluating the model immediately after initialization.</p>
<p>Now, I am monitoring the activations using hooks:</p>
<pre><code class="lang-auto">def recursively_hook(model, hook_fn):
    for name, module in model.named_children(): #model._modules.items():
        if len(list(module.children())) &gt; 0:  # if not leaf node
            for submodule in module.children():
                recursively_hook(submodule, hook_fn)
        else:
            module.register_forward_hook(hook_fn)

def hook_fn(m, inp, out):
    if torch.isnan(out).any() and not torch.isnan(inp).any():
        print(m)
        print("Input: [ Shape {} ]\n{}".format(inp.shape, inp))
        print("Output: [ Shape {} ]\n{}".format(out.shape, out))

model = create_model()
recursively_hook(model, hook_fn)
</code></pre>
<p>Based on the activations, I have observed that:</p>
<ul>
<li>the first 2 aforementioned layers give an output without NaN, and NaN first appears within the nn.TransformerEncoder.</li>
<li>supposedly, there is no module which gives NaN output without first having received NaN in the input - so the hook with the rule as above never prints anything!</li>
</ul>
<p>I am wondering how this second point can ever be possible (given that NaN is never in the input to the model). Am I doing something wrong when using the hooks?<br>
At the same time, model parameter values also appear to be okay (not NaN or “abnormally” large).</p>
<p>Weirdly enough, when trying to train the model, NaN now appears immediately in the output of the first nn.Linear layer, i.e. before the <code>nn.TransformerEncoder</code> (so the hook function above indeed only prints this first layer). Does this sound like an initialization problem?</p>

username: ataxias
cooked: <p>I am still not sure why the <code>hook_fn</code> as defined above doesn’t print anything, but I have identified the problem: I was not following the correct convention for <code>padding_mask</code> : <code>True</code> or <code>1</code> should mean “ignore/pad” and <code>False</code> or <code>0</code> should mean “consider”</p>
Class Imbalance in Pytorch CNN Segmentation

username: Danesh-A
cooked: <p>I have a model for segmentation using MRI images. Here is a graph of the dice loss during training. There are constant steps where the loss is equal to 1 where i am assuming the model is predicting the image to be all background. I am not sure how to improve these results. Any suggestions?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d22f2de20ce3ad5b5d286196ed4bd1ba145f5d5c.png" data-download-href="https://discuss.pytorch.org/uploads/default/d22f2de20ce3ad5b5d286196ed4bd1ba145f5d5c" title="Screen Shot 2020-08-21 at 1.55.02 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d22f2de20ce3ad5b5d286196ed4bd1ba145f5d5c_2_517x327.png" alt="Screen Shot 2020-08-21 at 1.55.02 PM" data-base62-sha1="tZnlGK9SZhf9NYdYuOFNQE39YBe" width="517" height="327" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d22f2de20ce3ad5b5d286196ed4bd1ba145f5d5c_2_517x327.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d22f2de20ce3ad5b5d286196ed4bd1ba145f5d5c_2_775x490.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d22f2de20ce3ad5b5d286196ed4bd1ba145f5d5c_2_1034x654.png 2x" data-dominant-color="EDD0C9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-08-21 at 1.55.02 PM</span><span class="informations">1402×886 63.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Cast tensor according to template type

username: MrHuff
cooked: <p>Dear all,</p>
<p>Assume this function:</p>
<pre><code class="lang-auto">template&lt;typename scalar_t&gt;
std::tuple&lt;torch::Tensor,torch::Tensor&gt; get_cheb_data(
        torch::Tensor &amp; cheb_nodes,
        int &amp;d,
        const std::string &amp; gpu_device
        ){
    int n = (int) pow(cheb_nodes.size(0),d);
    torch::Tensor cheb_idx = torch::zeros({n,d}).toType(torch::kInt32).to(gpu_device);
    torch::Tensor cheb_data = torch::zeros({n,d}).toType(torch::kFloat32).to(gpu_device);
    dim3 block,grid;
    int shared;
    std::tie(block,grid,shared) =  get_kernel_launch_params&lt;scalar_t&gt;(d,n);
    get_cheb_idx_data&lt;scalar_t&gt;&lt;&lt;&lt;grid,block,shared&gt;&gt;&gt;(
            cheb_nodes.packed_accessor32&lt;scalar_t,1,torch::RestrictPtrTraits&gt;(),
            cheb_data.packed_accessor32&lt;scalar_t,2,torch::RestrictPtrTraits&gt;(),
            cheb_idx.packed_accessor32&lt;int,2,torch::RestrictPtrTraits&gt;()
            );

    return std::make_tuple(cheb_idx,cheb_data);

}

</code></pre>
<p>The requirement is that</p>
<pre><code class="lang-auto">    torch::Tensor cheb_data = torch::zeros({n,d}).toType(torch::kFloat32).to(gpu_device);

</code></pre>
<p>has to have type “scalar_t”, where scalar_t could be anything from Half to Double. What is the best way to do this?</p>
<p>Thank you!</p>
<p>Best regards,<br>
Robert</p>
CUDA kernel that accumulates results

username: helange23
cooked: <p>Say I have a CUDA kernel that acts on two tensors and one of the tensors accumulates results.</p>
<pre><code class="lang-auto">
template &lt;typename scalar_t&gt;
__global__ void CUDA_kernel(
    torch::PackedTensorAccessor32&lt;scalar_t,2,torch::RestrictPtrTraits&gt; A,
    torch::PackedTensorAccessor32&lt;scalar_t,1&gt; accu) {

  const int index = blockIdx.x * blockDim.x + threadIdx.x;
  bool result = blabla(A[index]);

  if (result) accu[0] += 1;
  else accu[1] += 1;
  }
}
</code></pre>
<p>In this case, the following should be true: <code>sum(accu) == len(A)</code><br>
However, in reality <code>accu[0] == 1</code> and <code>accu[1] == 1</code>.</p>
<p>The cause for the error is obvious: every thread seems to overwrite <code>accu</code> thinking that <code>accu[x] == 0</code>. Is there a way to make this work?</p>
<p>Does this have to do with <code>torch::RestrictPtrTraits</code>? I feel stupid because I did not give <code>accu</code> RestrictPtrTraits.</p>

username: helange23
cooked: <p>For anyone who is having the same problem. The solution is described here: <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.85.1876&amp;rep=rep1&amp;type=pdf" rel="nofollow noopener">http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.85.1876&amp;rep=rep1&amp;type=pdf</a></p>
Forward hook containing trainable parameters

username: bask0
cooked: <p>I would like to modify an input to a forward call of a module I cannot modify using <code>register_forward_pre_hook</code>. The hook contains a forward call to another module with trainable parameters.</p>
<p>Let’s assume we have a module <code>FixModule</code> that we cannot modify. Also, we can’t change the input <code>x</code> to the forward call, as this is done somewhere deep down in a predefined PyTorch module.</p>
<pre><code>class FixModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(1, 1)

    def forward(self, x):
        return self.l1(x)
</code></pre>
<p>I define a forward pre-hook like this:</p>
<pre><code>class Linear(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(1, 1)

    def forward(self, x):
        return torch.sigmoid(self.l1(x))

linear = Linear()

def hook(self, input):
    return (linear(input[0]), )

model = FixModule()
model.register_forward_pre_hook(hook)
</code></pre>
<p>The hook takes input <code>x</code> and applies a linear transformation followed by a sigmoid activation. However, the parameters of <code>linear</code> are not part of the <code>model</code>. I could add the parameters of both <code>model</code> and <code>linear</code> manually to the optimizer, but I am sure there is a nicer way to do this.</p>
<p>Is there a clean way to achieve this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The way I would do this is to add the new layer directly on the Module:</p>
<pre><code class="lang-auto">def get_hook_for(mod):
    mod.my_linear = Linear()
    def hook_fn(self, input):
        # mod and self are the same here, so you can use any of them
        return (self.my_linear(input[0]), )
    return hook_fn

model = FixModule()
model.register_forward_pre_hook(get_hook_for(model))
</code></pre>

username: bask0
cooked: <p>Awesome, thanks! Exactly what I was looking for.</p>
Is there a way to make Autograd.grad behave element-wise?

username: tiwa
cooked: <p>I want to compute the gradient of a network with respect to its inputs. The network takes a 1-feature size-N batch of shape <code>(N,1)</code> and returns an output of the same shape.</p>
<p>If I pass in a <code>(1,1)</code> input <code>inp</code>, I can do it like this:</p>
<pre><code class="lang-auto">out = net(inp)
dNetdInp = torch.autograd.grad(out, inp, create_graph=True)[0]
</code></pre>
<p>But what if my input is <code>(N,1)</code>? If I try the same thing, I get a <code>RuntimeError: grad can be implicitly created only for scalar outputs.</code> What I want is a <code>(N,1)</code> tensor of the gradients, with the <code>i</code>th element in the tensor corresponding to the gradient of the net with respect to the <code>i</code>th input. How might I do this elegantly?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The error happens because <code>out</code> has ore than one element.<br>
If you have a scalar loss and want the gradient wrt that, you can just compute it and call <code>autograd.grad()</code> on that.<br>
If you want the gradient of the ith input wrt to the ith output. Then there are two cases:</p>
<ul>
<li>If your net compute the value for each input element independently (no batchnorm or layer norm), then you can simply call <code>autograd.grad()</code> with <code>out.sum()</code> or <code>grad_outputs=torch.ones_like(out)</code>.</li>
<li>If not, then you will have to do one backward for each element in your Tensor to make sure you can separate the gradient coming from each output element.</li>
</ul>

username: tiwa
cooked: <p>Thanks for the reply!</p>
<p>I want to apply a loss function to the gradients: in particular, I want to penalize negative gradients with respect to any input. If I got the <code>(N,1)</code> tensor <code>dNetdInp</code>, then I could simply take <code>nn.ReLU(-dNetdInp)</code> and <em>then</em> sum the result.</p>
<p>Is there a way to adapt your solution for this kind of use case? (Applying relu to the sum wouldn’t give me the right loss.)</p>
<p>Could <code>grad_outputs=torch.ones_like(out)</code> work for this?</p>
Test loss climbing along with accuracy

username: mcfn
cooked: <p>my train loss decreases and test accuracy climbing, all normal.<br>
my test loss is first decreasing and later climbing, so it looks like a second degree polynomial.<br>
i’ve read online some explanations that it could be a sign of overfitting or low confidence of the network.<br>
the thing is - the accuracy keeps climbing, so i’m not certain about the overfitting.<br>
i understand this can happen in general, but it also happens when i try the basic classification tutorial:<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></p>
<p>it uses cifar10 and resnet18, so it’s supposed to be very simple easy to learn and i wouldn’t expect this behaviour in that. does it make sense?</p>
Mixing trained and untrained models

username: Topsoil
cooked: <p>If i have a pretrained autoencoder/decoder that i want to use, and i want to train a model that would use that decoder as its output stage, how do i train them properly? I have several questions:</p>
<p>Should i <code>.detach()</code> tensor before feeding it to decoder, so:</p>
<pre><code class="lang-auto">def forward(self, input):
  out = self.main(input)
  out = out.detach() # &lt;- should i do this?
  out = self.decoder(out)
  return out
</code></pre>
<p>If not,should i set up optimizer to train only new model, or model+decoder to fit them together better?<br>
So if my <code>model</code> has two modules: <code>main</code> and <code>decoder</code> should i do this:<br>
<code>optimizer = optim.Adam(model.main.parameters());</code><br>
or this:<br>
<code>optimizer = optim.Adam(model.parameters());</code><br>
Thanks</p>

username: ayalaa2
cooked: <p>It depends on what your loss term is. Is the loss computed directly with the output of the decoder? If so: <strong>Don’t detach the tensor</strong></p>
<p>Detaching a tensor is the same as saying “I don’t want gradients to flow through here”. In this scenario, your loss is computed with the out of the decoder, gradients flow through to the start of the decoder and then stops there. Your main module will never see any gradients.</p>
<p>To answer your optimizer question; <em>it depends</em>. There’s no wrong answer really. You may want to train them jointly so that the decoder can learn to work better with your main module. You can also choose to just let the main module train on its own and adapt itself to the already trained decoder. I would personally suggest letting them train jointly (so <code>model.parameters()</code>) but you wouldn’t be wrong in only training the main module. Gradients have to flow through the decoder <em>anyways</em> (this would be a different story if you wanted to freeze the encoder instead), so I’m guessing speed-wise it’ll be the same.</p>

username: Topsoil
cooked: <p>Maybe train model first, keeping decoder unchanged, then enable decoder training, so they can better adapt to eachother?</p>

username: ayalaa2
cooked: <p>Seems viable to me. That’ll be following the idea of “<em>Learn to work with the decoder first, then we can train you both jointly</em>”. I don’t think there’s a wrong way to approach it; Maybe consider trying out the different methods and seeing if it even makes a difference</p>
Concatenation for all combinations for 4D Tensor

username: jpdevicente
cooked: <p>I have a 4D Tensor (batch, X, Y, Z). I would like to have all possible concatenations in the last axis ([z_1,1; z_1,2], [z_1,1;z_1,3], …) without permutations (I only want [z_1,1; z_1,2], I dont need [z_1,2; z_1,1]). The output shape of this should be (batch, (X<em>Y * (X</em>Y-1))/2 ,2Z) Is there a way to get this with vectorized operations without using FOR loops? I looked at <a href="https://discuss.pytorch.org/t/create-all-possible-combinations-of-a-3d-tensor-along-the-dimension-number-1/48155">link</a> but I cant see how to apply this to my solution.</p>
How to feed 2D images to conv3d layer?

username: praveen_r2998
cooked: <p>I planned to use a model made of conv3d layers for video action classification but struck on how to feed the frames which were extracted from videos to the conv3d layers.</p>

username: ayalaa2
cooked: <p>Conv3D wants in the form of <code>batch_size x channels x depth x height x width</code>. Your number of frames is the depth. Your channels can be RGB or whatever information you have on your frames. You may need to use a library to process raw videos into this format.</p>

username: praveen_r2998
cooked: <p>Tk u, can you suggest any libraries??</p>
I wrote some code to add a .tostring() function to Tensors

username: Megashot7727
cooked: <p>Put this above your code:</p>
<pre><code class="lang-auto">def tensor_to_string(self):
    return str(self.tolist())
setattr(torch.Tensor, 'tostring', tensor_to_string)
</code></pre>
<p>To use, just write</p>
<pre><code class="lang-auto">(tensor name).tostring()
</code></pre>
<p>Example:</p>
<pre><code class="lang-auto">test = torch.randn(4)

print(type(test.tostring()))
print(test.tostring())
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">&lt;class 'str'&gt;
[-0.6502066850662231, -1.1002898216247559, -0.7664051651954651, -0.4823172092437744]
</code></pre>
How can i implement the backward function of a backward function in Libtorch?

username: Carol_Ye_Liu
cooked: <p>In python, I can define the backward method for AFunctionBackward, but how can i do that in C++?</p>
<pre><code>class AFunctionBackward(Function):
@staticmethod
def forward(ctx, grad_output, out, negative_slope, scale):
    ctx.save_for_backward(out)
    ctx.negative_slope = negative_slope
    ctx.scale = scale

    empty = grad_output.new_empty(0)

    grad_input = myfunc(
        grad_output, empty, out, 3, 1, negative_slope, scale
    )

    dim = [0]

    if grad_input.ndim &gt; 2:
        dim += list(range(2, grad_input.ndim))

    grad_bias = grad_input.sum(dim).detach()

    return grad_input, grad_bias

@staticmethod
def backward(ctx, gradgrad_input, gradgrad_bias):
    out, = ctx.saved_tensors
    gradgrad_out = myfunc(
        gradgrad_input, gradgrad_bias, out, 3, 1, ctx.negative_slope, ctx.scale
    )

    return gradgrad_out, None, None, None


class AFunction(Function):
@staticmethod
def forward(ctx, input, bias, negative_slope, scale):
    empty = input.new_empty(0)
    out = myfunc(input, bias, empty, 3, 0, negative_slope, scale)
    ctx.save_for_backward(out)
    ctx.negative_slope = negative_slope
    ctx.scale = scale

    return out

@staticmethod
def backward(ctx, grad_output):
    out, = ctx.saved_tensors

    grad_input, grad_bias = AFunctionBackward.apply(
        grad_output, out, ctx.negative_slope, ctx.scale
    )

    return grad_input, grad_bias, None, None
</code></pre>
<p>In C++, I can define AFunctionForward and AFunctionBackward, but I have no idea how to implement the AFunctionBackwardBackward. Can someone please help?</p>
<pre><code>torch::Tensor AFunctionForward(const Tensor &amp; input,  const torch::Tensor&amp;  bias, double 
negative_slope, double scale ) {
...
}

struct  AFunctionBackward: public torch::autograd::Node{
...
}</code></pre>
Is reverse engineered Pytorch model a concern?

username: Arun_Kumar1
cooked: <p>Hi ,</p>
<p>One of the concern on found Android is that Pytorch model (.pt) file can be reverse engineered.<br>
Is the files extracted from .pt file is a concern? How can we secure if it is so?</p>
<p>Please let me know on an urgent note</p>
<p>Thanks,<br>
Arun</p>
Np.unwrap function in pytorch

username: 1c9d70faac66efabd051
cooked: <p>Is there same function like numpy.unwrap in pytorch?</p>
<p>I want to implement hilbert transform and extract envelope &amp; Temporal fine structure in pytorch with GPU.</p>
<p>I failed to find the same pytorch function like “scipy.hilbert”, “np.phase”, “np.abs(complex value)”. Now, I successfully made these three function myself.<br>
But, still, I cant realize “np.unwrap” function efficiently.<br>
Because of this, my model waste lots of time in data pre-processing.</p>
<p>Anyway I need a help. thanks</p>

username: richard
cooked: <p>I’m not sure if I understand the <code>np.unwrap</code> function correctly, but does something like</p>
<pre><code class="lang-auto">def unwrap(x):
    y = x % (2 * pi)
    return torch.where(y &gt; pi, 2*pi - y, y)
</code></pre>
<p>work?</p>

username: Past-Future-AI
cooked: <p>‘unwrap’ can be great to have when working with sin()'s and cos()'s. ‘diff’ can be useful in general. Both ideally should eventually be included in pytorch.</p>
<p>Here is a pytorch port of numpy’s implementation:</p>
<p>def unwrap(phi, dim=-1):<br>
assert dim is -1, ‘unwrap only supports dim=-1 for now’<br>
dphi = diff(phi, same_size=False)<br>
dphi_m = ((dphi+np.pi) % (2 * np.pi)) - np.pi<br>
dphi_m[(dphi_m==-np.pi)&amp;(dphi&gt;0)] = np.pi<br>
phi_adj = dphi_m-dphi<br>
phi_adj[dphi.abs()&lt;np.pi] = 0<br>
return phi + phi_adj.cumsum(dim)</p>
<p>def diff(x, dim=-1, same_size=False):<br>
assert dim is -1, ‘diff only supports dim=-1 for now’<br>
if same_size:<br>
return F.pad(x[…,1:]-x[…,:-1], (1,0))<br>
else:<br>
return x[…,1:]-x[…,:-1]</p>

username: Past-Future-AI
cooked: <p>NOTE: the call to ‘diff’ should have same_size=True</p>
RuntimeError: stack expects each tensor to be equal size, but got [3, 64, 68] at entry 0 and [1, 64, 64] at entry 18

username: nabil_nasifur_rahman
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/caab6cc01941151b5b833f52d8c9c090ee836b5f.png" data-download-href="https://discuss.pytorch.org/uploads/default/caab6cc01941151b5b833f52d8c9c090ee836b5f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/caab6cc01941151b5b833f52d8c9c090ee836b5f_2_690x373.png" alt="image" data-base62-sha1="sUTIWQ0VkxAKzojJK4NCAaAAa9F" width="690" height="373" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/caab6cc01941151b5b833f52d8c9c090ee836b5f_2_690x373.png, https://discuss.pytorch.org/uploads/default/original/3X/c/a/caab6cc01941151b5b833f52d8c9c090ee836b5f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/a/caab6cc01941151b5b833f52d8c9c090ee836b5f.png 2x" data-dominant-color="F3F2F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">933×505 54 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<p>In some part of your code which is not shown in the snapshot, you are using a stack function where you stack two elements with different sizes which are (1,48,48) and (3,48,48). If you share the full code snippet, we can have a look.</p>
<p>Thanks</p>

username: nabil_nasifur_rahman
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c384aa185a6a9160f3cd824650c18a746b3a9e3.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c384aa185a6a9160f3cd824650c18a746b3a9e3" title="image2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c384aa185a6a9160f3cd824650c18a746b3a9e3_2_690x315.png" alt="image2" data-base62-sha1="mhZ5xLQeDeBkRmq461tdH9jDRsL" width="690" height="315" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c384aa185a6a9160f3cd824650c18a746b3a9e3_2_690x315.png, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c384aa185a6a9160f3cd824650c18a746b3a9e3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c384aa185a6a9160f3cd824650c18a746b3a9e3.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image2</span><span class="informations">936×428 45.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to split tensors with overlap and average overlapping values?

username: whuang7000
cooked: <p>Hi. I am working on a medical image segmentation project, and I have a model that takes inputs of size 128x32x128 (X, Z, Y). However, my inputs are not all evenly divisible by these values, and not all inputs are the same size. For example, I have inputs that are divisible (512x96x512) and many that are not (480x68x480). How can I can split these inputs into overlapping tensors to feed into the network, and then take the outputs and reconstruct them into a tensor through averaging overlapping output values that would look as if I fed the entire image into the network? The output has shape. (B,C,X,Z,Y)</p>

username: whuang7000
cooked: <p>For clarification, the inputs are gray scale so they have 1 channel and are of shape (1, W, D, H) and the network takes in batches of these inputs and outputs a tensor of size (B, 10, W, D, H) where 10 is the number of classes (10). I have been looking into nn.Fold and nn.Unfold but I noticed that they only work on 4D tensors, so I could theoretically unfold the input volumes because they are 4D, but I cannot fold them back together because the output is a 5D tensor.</p>

username: vainaijr
cooked: <p>maybe consider reshaping the tensors after unfolding, I did something like this once,</p>
<pre><code class="lang-auto">x = torch.randn(5, 5, 5, 5)
a = nn.Fold((5, 5), (3, 3))
d = x.unfold(2, 3, 1).unfold(3, 3, 1).reshape(x.size(0), -1, 9)
a(d).shape
</code></pre>
<p>torch.Size([5, 5, 5, 5])<br>
to get the same shape, after unfolding then folding, pytorch version of fold, adds overlapping values.</p>

username: fepegar
cooked: <p>Hey, <a class="mention" href="/u/whuang7000">@whuang7000</a>. You can probably use the <a href="https://torchio.readthedocs.io/data/patch_based.html" rel="nofollow noopener">patch-based pipelines in TorchIO</a> for this.</p>
Compute loss on multiple targets task

username: nlplabsimon
cooked: <p>Hi,</p>
<p>I am working on a task that each example has multiple targets.<br>
My way to compute loss is mapping the example to an embedding and expand the embedding to the shape of (length of targets, hidden_size), then compute the cross entropy loss.<br>
My question is “Does each loss of targets affect the backward process?”</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>Could you give us a small code example, so that we could have a look if and how the backward pass could be affected? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: suraj.pt
cooked: <p>As <a class="mention" href="/u/ptrblck">@ptrblck</a> suggests, share a code example <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>If I understand correctly, your problem is of <a href="https://ruder.io/multi-task/" rel="nofollow noopener">multi-task learning</a>. You can create separate sub-tasks (i.e. separate classifiers) for each target, aggregate the losses (sum/avg) from each classifier, and backpropagate it. The prediction error from each of the tasks will update the network weights.</p>
Copy and assignment semantics of torch tensors in C++

username: Lee_Newberg
cooked: <p>In C++, if I construct a torch tensor from another tensor, <code>Tensor(Tensor &amp;)</code> (with one ampersand!) does that copy the element data from the source tensor into a new location for the new tensor?  Or is it more like a reference where a subsequent update to the elements of either tensor will be seen in the other?  Or is it more like a lazy copy, where they share element data until either is modified at which point a copy is made so that they no longer share data?</p>
<p>Is the behavior for an assignment of a tensor to a tensor different from the construction of a tensor from another?  If so, how?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not a specialist of the c++ API but a good mental model is <code>Tensor = shared_ptr&lt;ActualTensor&gt;</code>.<br>
So all the reference/const constructs from c++ will do the same thing as they would on a shared_ptr.</p>
<p>Note that you can also have Tensor being views of each other: <code>auto t2 = t1.alias();</code> for example. t1 and t2 are two independent ActualTensor objects. But they will share the same memory for their content. And change to the data of one will also change the other. But change to metadata <strong>won’t</strong> be reflected.</p>

username: Lee_Newberg
cooked: <p>Thanks!  Where you write “I’m not a specialist of the c++ API” are you being modest or should I seek out a second opinion?  If the latter … are there any second opinions out there?</p>

username: albanD
cooked: <p>I don’t think you will find a simpler answer than that I’m afraid.<br>
If you have specific questions, you can ask glaringlee for the c++ api or ezyang for more general backend questions. But they are quite busy and might be less responsive.</p>

username: Lee_Newberg
cooked: <p>I like your answer.  Thank you.</p>