akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom efficient Conv2d with intermediate calls to functions

username: vfinotti
cooked: <p>Hello everyone! I’ve been working on a customized Conv2d layer which allows me to do operations on the data immediately after every multiplication and sum (I’m trying to prove that we can quantize a CNN to use less bits while keeping the accuracy, it’s for my masters). My starting point is the code available <a href="https://discuss.pytorch.org/t/make-custom-conv2d-layer-efficient-wrt-speed-and-memory/70175">here</a>, where unfold and fold is used for that.<br>
The code is:</p>
<pre><code class="lang-auto"># Vanilla Convolution
def myconv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
    batch_size, in_channels, in_h, in_w = input.shape
    out_channels, in_channels, kh, kw =  weight.shape

    unfold = torch.nn.Unfold(kernel_size=(kh, kw), dilation=dilation, padding=padding, stride=stride)
    inp_unf = unfold(input)

    if bias is None:
        out_unf = inp_unf.transpose(1, 2).matmul(weight.view(weight.size(0), -1).t()).transpose(1, 2)
    else:
        out_unf = (inp_unf.transpose(1, 2).matmul(w_) + bias).transpose(1, 2)
    out = out_unf.view(batch_size, out_channels, out_h, out_w)
    return out
</code></pre>
<p>The problem is that I need to do call a function immediately after every multiplication and sum, so I could not use the <code>.matmul()</code> method since it does both for me. My first attempt was to do the operations independently, calling <code>myfunc()</code> after each one of them:</p>
<pre><code class="lang-auto"># Vanilla Convolution
def myconv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
    batch_size, in_channels, in_h, in_w = input.shape
    out_channels, in_channels, kh, kw =  weight.shape

    unfold = torch.nn.Unfold(kernel_size=(kh, kw), dilation=dilation, padding=padding, stride=stride)
    inp_unf = unfold(input)

    weight_t = weight.view(weight.size(0), -1).t()
    inp_unf_t = inp_unf.transpose(1, 2)
    inp_unf_t = inp_unf_t[:,:,:, None]
    inp_unf_t_exp = inp_unf_t.transpose(2,3).repeat((1,1,weight_t.t().size(0),1))
    weight_t_exp = weight_t.t().repeat((1,inp_unf_t.size(1),1,1))
    out_unf = myfunc(inp_unf_t_exp.mul(weight_t_exp))
    out_unf = myfunc(torch.sum(out_unf, dim=3).transpose(1, 2))
    if bias is not None:
        out_unf = out_unf + myfunc(bias.view(-1, 1))

    out = out_unf.view(batch_size, out_channels, out_h, out_w)
    return out
</code></pre>
<p>This approach however is <strong>terribly slower</strong> and (B) consumes a  <strong>lot of memory</strong>, to the point that a “RuntimeError: CUDA out of memory” error is thrown. Is there an alternative to that approch which would be less memory greedy?</p>
<p>Any help is much appreciated!</p>
How to design a custom layer with single weight and some masked nets

username: rnakhjahvani
cooked: <p>I am new to pytorch and I want to write a layer in which the input/output connecteion<br>
is defined through an adjacency matrix, and all existing edges should<br>
have same weight. My current implementation is as follows:</p>
<pre><code>class myFunc(Function):
    def __init__(self):
        super(myFunc, self).__init__()
    def forward(self, input, weight,adj):
        a = torch.zeros_like(adj,dtype=torch.float)
        a[adj] = weight
        return input.mm(a)

class MyLayer(nn.Module):
    def __init__(self,adj):
        super(MyLayer, self).__init__()
        self.adj = adj
        self.weight = nn.Parameter(data=torch.Tensor(1), requires_grad=True)
        nn.init.normal_(self.weight.data, mean=1, std=0.1)

    def forward(self, input):
        return myFunc().forward(input,self.weight,self.adj)
</code></pre>
<p>This approach does not use less memory than the case that I have a separate weight per net.<br>
Since I have a single parameter to learn (one weight for the entire layer), is there any way that I can implement this in a more efficient way?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You don’t need a custom Function here.<br>
You should just move that forward code into the nn.Module’s forward function.</p>

username: googlebot
cooked: <p>Don’t think so - here memory is consumed to preserve original input tensor for backprop (mm op), so right matrix “sparsity” makes no difference. Maybe take a look at torch.utils.checkpoint if memory is an issue.</p>
Faster RNNs with Haste 0.4.0

username: sharvil
cooked: <p>Hey folks, <a href="https://github.com/lmnt-com/haste" rel="nofollow noopener">Haste</a> is a fast RNN library with LSTM, GRU, and layer norm {LSTM,GRU} implementations. We just released version 0.4.0 with performance improvements, new RNN types, and CPU support. If you’re building sequence models, give it a try. You might find some performance wins.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/lmnt-com/haste" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:132/66;"><img src="https://repository-images.githubusercontent.com/237056148/28c05b80-4285-11ea-8725-6cc5f0fcf7c5" class="thumbnail" width="132" height="66"></div>

<h3><a href="https://github.com/lmnt-com/haste" target="_blank" rel="nofollow noopener">lmnt-com/haste</a></h3>

<p>Haste: a fast, simple, and open RNN library. Contribute to lmnt-com/haste development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Parallel loss using multi gpu

username: Ashish_Kumar2
cooked: <p>Hi, I’m trying to distribute the loss among 4 available GPUs. I’m getting below error.</p>
<p>Traceback (most recent call last):<br>
File “gg.py”, line 177, in <br>
logps = torch.mean(torch.stack(logps))<br>
RuntimeError: All input tensors must be on the same device. Received cuda:0 and cuda:3</p>
<p>Using parallel.py file from thomaswolf. Any suggestion/methods to achieve parallel loss?</p>
<h1>Code Snippet</h1>
<p>criterion = nn.NLLLoss(reduce=False)<br>
criterion = DataParallelCriterion(criterion, device_ids=[0, 1, 2, 3])</p>
<p>model =DataParallelModel(model, device_ids=[0, 1, 2, 3])<br>
model.to(device)</p>
<p>with torch.no_grad():<br>
for inputs, labels in testloader:<br>
inputs, labels = inputs.to(device), labels.to(device)<br>
#inputs=inputs.unsqueeze(1)<br>
logps = model.forward(inputs)<br>
batch_loss = criterion(logps, labels)<br>
batch_loss=batch_loss.mean()<br>
test_loss += batch_loss.item()<br>
logps = torch.mean(torch.stack(logps))<br>
print(logps)</p>
<pre><code>                # Calculate accuracy
                ps = torch.exp(logps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor)).item()</code></pre>

username: ptrblck
cooked: <p>Could you try to call the model directly instead of <code>model.forward</code>?<br>
I’m not sure, how <code>DataParallelModel</code> is implemented, but if you call the model directly, its internal <code>__call__</code> method will be used, which will properly register all hooks etc.<br>
Let me know, if that helps.</p>

username: Ashish_Kumar2
cooked: <p>Thank you for the reply. I’ll try that and let you know the results.</p>
<p>I have used <a href="https://github.com/zhanghang1989/PyTorch-Encoding" rel="nofollow noopener">https://github.com/zhanghang1989/PyTorch-Encoding</a> for custom parallelization of model and loss.</p>
Cast to long autograd

username: dizcza
cooked: <p>I want to make <code>tensor.long()</code> differentiable.</p>
<pre><code class="lang-python">import torch

class CastToLong(torch.autograd.Function):

    @staticmethod
    def forward(ctx, tensor: torch.Tensor):
        return tensor.long()

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output


torch.manual_seed(0)
x = torch.randn(3,3)
idx = torch.tensor([0,1], dtype=torch.float32, requires_grad=True)

idx = CastToLong.apply(idx)
y = x[idx]

y.sum().backward()
</code></pre>
<p>I receive the traceback</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/dizcza/PycharmProjects/EmbedderSDR/ignored/visible/idx_autograd.py", line 21, in &lt;module&gt;
    y.sum().backward()
  File "/home/dizcza/anaconda3/envs/embsdr/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/dizcza/anaconda3/envs/embsdr/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: range.second - range.first == t.size() INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1579022119164/work/torch/csrc/autograd/generated/Functions.cpp:55, please report a bug to PyTorch. inconsistent range for TensorList output
</code></pre>
<p>I was not sure about opening an issue on GitHub and decided firstly to post the problem here. Surely, this is not a typical usage of pytorch autograd. Nevertheless, for me, it makes sense because essentially the values after <code>.long()</code> are exactly the same (<code>idx</code> are already integers) and thus <code>CastToLong</code> should just broadcast the gradient backward.</p>
<p>Should I open an issue or let’s discuss it here first?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You won’t be able to make a long Tensor require gradients.<br>
And even if you trick pytorch into doing it, no differentiable op is implemented for integer types so you will have to reimplement everything.</p>
<p>Also, keep in mind that the indexing op is not differentiable. Even if you try to compute gradients for it, you will just get 0 everywhere which is not very usefull.</p>
Torch.cat doesn't notice dim=1

username: misha
cooked: <p>I am confused by the error I am receiving; it seems that torch is ignoring <code>dim</code>.</p>
<p>Output:<br>
<code>torch.Size([100, 10]) torch.Size([100, 32])</code></p>
<p>Traceback:</p>
<pre><code class="lang-auto">     10     print(y_sample.shape, z_sample.shape)
---&gt; 11     x_dist = ae.yz_dec(torch.cat([y_sample, z_sample]), dim=1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 10 and 32 in dimension 1 at ../aten/src/TH/generic/THTensor.cpp:612
</code></pre>
<p>For comparison, this piece of code runs without any errors and outputs <code>torch.Size([100, 42])</code> .</p>
<pre><code class="lang-auto">a, b = torch.ones(100, 10), torch.ones(100, 32)
c = torch.cat([a, b], dim=1)
print(a.shape, b.shape, c.shape)
</code></pre>
<p>As opposed to this one,</p>
<pre><code class="lang-auto">a, b = torch.ones(100, 10), torch.ones(100, 32)
c = torch.cat([a, b], dim=0)
print(a.shape, b.shape, c.shape)
</code></pre>
<p>which encounters the same runtime error as the code above.</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-12-d7fbede0f75e&gt; in &lt;module&gt;
      1 a, b = torch.ones(100, 10), torch.ones(100, 32)
----&gt; 2 c = torch.cat([a, b], dim=0)
      3 
      4 print(a.shape, b.shape, c.shape)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 10 and 32 in dimension 1 at ../aten/src/TH/generic/THTensor.cpp:612
</code></pre>
<p>I am on Python 3.7.5 (default, Nov 1 2019, 02:16:32) [Clang 11.0.0 (clang-1100.0.33.8)] and PyTorch 1.4.0.</p>

username: albanD
cooked: <p>I think this is expected. When you concatenate dimention 0, all the other ones must have the same size.<br>
When you cat on dim 1 it works because the other dimension is 100 for both inputs.</p>

username: misha
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Indeed! But in the broken code above, the first snippet, I also concatenate on dim=1.</p>
<p>ETA: to make it more visible</p>
<p>I do <code>torch.cat([y_sample, z_sample]), dim=1</code>, and just before that  <code> print(y_sample.shape, z_sample.shape)</code> outputs <code>torch.Size([100, 10]) torch.Size([100, 32])</code>.</p>

username: misha
cooked: <p>Oh… it is a typo on my part: <code>dim=1</code> is passed to the <code>ae.yz_dec</code>.</p>
LBFGS Optimizer is non-deterministic?

username: samuelsunarjo96
cooked: <p>LBFGS Optimizer:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/6be85603757a810be3d18b70279b104c0653e9b3/torch/optim/lbfgs.py#L183" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/6be85603757a810be3d18b70279b104c0653e9b3/torch/optim/lbfgs.py#L183" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/6be85603757a810be3d18b70279b104c0653e9b3/torch/optim/lbfgs.py#L183</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="173" style="counter-reset: li-counter 172 ;">
<li>        if abs(bracket[1] - bracket[0]) * d_norm &lt; tolerance_change:</li>
<li>            break</li>
<li>
</li>
<li>    # return stuff</li>
<li>    t = bracket[low_pos]</li>
<li>    f_new = bracket_f[low_pos]</li>
<li>    g_new = bracket_g[low_pos]</li>
<li>    return f_new, g_new, t, ls_func_evals</li>
<li>
</li>
<li>
</li>
<li class="selected">class LBFGS(Optimizer):</li>
<li>    """Implements L-BFGS algorithm, heavily inspired by `minFunc</li>
<li>    &lt;https://www.cs.ubc.ca/~schmidtm/Software/minFunc.html&gt;`.</li>
<li>
</li>
<li>    .. warning::</li>
<li>        This optimizer doesn't support per-parameter options and parameter</li>
<li>        groups (there can be only one).</li>
<li>
</li>
<li>    .. warning::</li>
<li>        Right now all parameters have to be on a single device. This will be</li>
<li>        improved in the future.</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I was wondering if this optimizer is supposed to produce non-deterministic results? Given that these results were produced under the same parameter initialization and optimization setting. Simply running the program multiple times without any changes would produce different results.</p>
<p>I have set the following to attempt disabling the non-deterministic outcome but no luck:<br>
<code>np.random.seed(0)</code><br>
<code>torch.manual_seed(0)</code><br>
<code>torch.backends.cudnn.deterministic = True</code><br>
<code>torch.backends.cudnn.benchmark = False</code></p>

username: albanD
cooked: <p>Hi,</p>
<p>You can check the note on reproducibility in the doc here: <a href="https://pytorch.org/docs/stable/notes/randomness.html">https://pytorch.org/docs/stable/notes/randomness.html</a></p>
<p>There are many reasons that could lead to it (or any other part of your code) to be non deterministic.</p>
What happens when MaxPool rounds a tensor?

username: alx
cooked: <p>I was looking at the tensor shape of the following cnn:</p>
<pre><code class="lang-auto">    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        
        self.conv1 = nn.Conv2d(1, 32, 5)
        self.conv2 = nn.Conv2d(32, 64, 5)
        self.conv3 = nn.Conv2d(64, 128, 5)

    def convs(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = F.max_pool2d(x, (2, 2))
       
        x = self.conv2(x)
        x = F.relu(x)
        # torch.Size([64, 19, 19])
        x = F.max_pool2d(x, (2, 2)) # torch.Size([64, 9, 9])
      
        x = self.conv3(x)
        x = F.relu(x)
        #torch.Size([128, 5, 5])
        x = F.max_pool2d(x, (2, 2)) # torch.Size([128, 2, 2])
</code></pre>
<p>I noticed that in the conv2 maxpooling we’re diving 19 by 2 = 8.5 ~ 9<br>
but in conv3 we’re diving 5 by 2 = 2.5 ~ 2</p>
<p>In the first scenario we round to the upper bound and in the second case the lower bound. Can anyone explain what happenng?</p>

username: ptrblck
cooked: <p>As explained in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.MaxPool2d">docs</a>, the output shape will be calculated by a final <code>floor</code> operation.<br>
Since <code>19/2 = 9.5</code>, the output shape of <code>9</code> is thus expected. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: alx
cooked: <p>Thanks Patrick! missed that one.</p>
Spectra data infrence

username: randomforest95
cooked: <p>Hi, I’m trying to utilize deep learning to analyze spectroscopy data. in the given photo below which belongs to XPS data. so basically imagine each peak represents an element from the periodic table and I have a dataset of all the elements for each element I have 3 samples with different background noise and peak intensity and from those, I augmented the dataset I used for training using Dirichlet distribution. then I tried converting the data to spectrograms and fed it to resnet 16, 34, 50, densenet… and it overfitted, I also tried converting the data to GAF (gramian angular field) it seems to be better but the problem I’m encountering I need to find an architecture or a way to train on elements only and be able to make inference on compound (multiple peaks)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="http://www.eagle-regpot.eu/info/images/Equipment_I_6a.jpg" title="Equipment_I_6a.jpg" rel="nofollow noopener"><img src="http://www.eagle-regpot.eu/info/images/Equipment_I_6a.jpg" alt width="690" height="497"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">Equipment_I_6a.jpg</span><span class="informations">3252×2343</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div></p>

username: vdm63
cooked: <p>Good afternoon, did you find a solution to your question?</p>
Changing model value based on epoch during training

username: mustgoplay
cooked: <p>Hi,</p>
<p>I want to change the forward behavior based on the epoch.  I can’t figure out how to pass either the epoch or a simple True/False to the network during training.</p>
<p>Should I make it a parameter?  If so can I change that in the training loop?</p>
<p>I did find a post that shows how to pass the value to forward, but if it’s a net module then I’m still defining the layers so I don’t see how to pass that value to the layers from the forward pass.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>I’m not sure, if this would fit your use case, but you could directly manipulate a model attribute via:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(1, 1)
        self.fc2 = nn.Linear(1, 1)
        self.switch = 0
        
    def forward(self, x):
        if self.switch == 0:
            print('using fc1')
            x = self.fc1(x)
        elif self.switch == 1:
            print('using fc2')
            x = self.fc2(x)
        else:
            print('returning x')
        return x

model = MyModel()

for epoch in range(3):
    x = torch.randn(1, 1)
    model.switch = epoch
    out = model(x)

&gt; using fc1
using fc2
returning x
</code></pre>

username: mustgoplay
cooked: <p>That works for now thanks!  It’s not ideal cause I have to share weights that way but it keeps me moving.</p>
How to extract features from intermediate layers of VGG16?

username: m.saber
cooked: <p>Hi, I would like to get outputs from multiple layers of a pretrained VGG-16 network. Following is what I have done:</p>
<pre><code class="lang-auto">model = torchvision.models.vgg16()

# make new models to extract features
layers = list(model.children())[0][:8]
model_conv22 = nn.Sequential(*layers)  

layers = list(model.children())[0][:15]
model_conv33 = nn.Sequential(*layers)   

layers = list(model.children())[0][:22]
model_conv43 = nn.Sequential(*layers) 

layers = list(model.children())[0][:29]
model_conv53 = nn.Sequential(*layers)  

# get outputs
x1 = model_conv22(input)   
x2 = model_conv33(input)   
x3 = model_conv43(input)   
x4 = model_conv53(input)
</code></pre>
<p>I know that this is not an intelligent way. Could you give me some ideas to make it efficient?<br>
Thank you very much.</p>

username: b02202050
cooked: <p>In your case, you can use <code>torchvision.models._utils.IntermediateLayerGetter</code>.<br>
For example:</p>
<pre><code class="lang-python">import torchvision
from torchvision.models._utils import IntermediateLayerGetter
model = torchvision.models.vgg16()
return_layers = {'8': 'out_layer8', '15': 'out_layer15'}
model_with_multuple_layer = IntermediateLayerGetter(model.features, return_layers=return_layers)
x = torch.rand(5,3,512,512)
y = model_with_multuple_layer(x)
</code></pre>
<p>You can see that <code>y</code> contains the output of layer 8 and layer 15 of model.features</p>
PyTorch no longer supports this GPU because it is too old.

username: Snuffy4
cooked: <p>have installed CUDA ToolKit 8.0<br>
<code>pip install torch==1.0.1 -f https://download.pytorch.org/whl/cu80/torch_stable.html</code><br>
Installed version 1.0.1 for CUDA 8.0 WINDOWS.</p>
<p>An error appears</p>
<pre><code class="lang-auto">Found GPU0 GeForce GT 630M which is of cuda capability 2.1.
    PyTorch no longer supports this GPU because it is too old.

  warnings.warn(old_gpu_warn % (d, name, major, capability[1]))
Found 1 GPUs available. Using GPU 0 (GeForce GT 630M) of compute capability 2.1 with 2.1Gb total memory.

Preparing the encoder, the synthesizer and the vocoder...
Traceback (most recent call last):
  File "demo_cli.py", line 61, in &lt;module&gt;
    encoder.load_model(args.enc_model_fpath)
  File "C:\Users\kisel\Desktop\Real-Time-Voice-Cloning-master\encoder\inference.py", line 32, in load_model
    _model = SpeakerEncoder(_device, torch.device("cpu"))
  File "C:\Users\kisel\Desktop\Real-Time-Voice-Cloning-master\encoder\model.py", line 21, in __init__
    batch_first=True).to(device)
  File "C:\Users\kisel\anaconda3\envs\TestVoice\lib\site-packages\torch\nn\modules\module.py", line 381, in to
    return self._apply(convert)
  File "C:\Users\kisel\anaconda3\envs\TestVoice\lib\site-packages\torch\nn\modules\rnn.py", line 117, in _apply
    self.flatten_parameters()
  File "C:\Users\kisel\anaconda3\envs\TestVoice\lib\site-packages\torch\nn\modules\rnn.py", line 113, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: cuDNN error: CUDNN_STATUS_ARCH_MISMATCH
</code></pre>
<p>I really need support for 2.1 compute capability and version &gt;= 1.0.1, please help!</p>

username: dpernes
cooked: <p>Consider using a cloud computing service (e.g. Google Colab, Google Cloud or AWS). Colab is free (with some limitations), so you may want to check if that meets your needs first.</p>
ReLU activation function cause Nan gradient

username: eprokopalo
cooked: <p>I’m trying to implement next NN:</p>
<p>class MyModel(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MyModel, self).<strong>init</strong>()<br>
self.conv0 = torch.nn.Conv1d(24,512,5)<br>
self.conv1 = torch.nn.Conv1d(512,512,3,dilation=2)<br>
self.conv2 = torch.nn.Conv1d(512,512,3,dilation=3)<br>
self.conv3 = torch.nn.Conv1d(512,512,1)<br>
self.conv4 = torch.nn.Conv1d(512,1500,1)<br>
self.lin1 = torch.nn.Linear(3000,512)<br>
self.lin2 = torch.nn.Linear(512,512)<br>
self.lin3 = torch.nn.Linear(512,1951)<br>
self.norm1 = torch.nn.BatchNorm1d(512)<br>
self.norm2 = torch.nn.BatchNorm1d(512)<br>
self.norm3 = torch.nn.BatchNorm1d(512)<br>
self.norm4 = torch.nn.BatchNorm1d(512)against<br>
self.norm5 = torch.nn.BatchNorm1d(1500)<br>
self.norm7 = torch.nn.BatchNorm1d(512)<br>
self.rel = torch.nn.ReLU()</p>
<pre><code>def forward(self,x):
    l1out = self.rel(self.conv0(x))
    l1norm = self.norm1(l1out)
    l2out = self.rel(self.conv1(l1norm))
    l2norm = self.norm2(l2out)
    l3out = self.rel(self.conv2(l2norm))
    l3norm = self.norm3(l3out)
    l4out = self.rel(self.conv3(l3norm))
    l4norm = self.norm4(l4out)
    l5out = self.rel(self.conv4(l4norm))
    l5norm = self.norm5(l5out)
    mean = torch.mean(l5norm,dim=2)
    std = torch.std(l5norm,dim=2)
    l6inp = torch.cat([mean,std],dim=1)
    l6out = self.rel(self.lin1(l6inp)) 
    l7out = self.rel(self.lin2(l6out))
    l7norm = self.norm7(l7out)
    l8out = torch.sigmoid(self.lin3(l7norm))
    result = torch.nn.functional.softmax(l8out,dim=1)
    return result
</code></pre>
<p>But the gradient of convolution layers, calculated by autograd contains Nans, and when i was using sigmoid instead ReLU, everything was ok. Can somebody explain me the reason of this problem?</p>

username: divyesh_rajpura
cooked: <p>Hello <a class="mention" href="/u/eprokopalo">@eprokopalo</a>, I have encountered same problem <a href="https://discuss.pytorch.org/t/batchnorm-and-relu/76563">https://discuss.pytorch.org/t/batchnorm-and-relu/76563</a>. I am using Conv1d --&gt; ReLU --&gt; BatchNorm, which gives me NaN. Have you found any reason or solution for this??</p>
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/THCTensorRandom.cu:25

username: Didem_Yaniktepe
cooked: <p>import sys, os<br>
sys.path.append(os.path.dirname(os.path.abspath(<strong>file</strong>)) + ‘/Data’)</p>
<p>CLASS_COUNT = 1<br>
BATCH_SIZE = 2<br>
how can i fix it please help?</p>

username: aplassard
cooked: <p>Is there more source code than that? Nothing in your code interacts with the GPU.</p>

username: Didem_Yaniktepe
cooked: <p>)<br>
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCTensorRandom.cu line=25 error=2 : out of memory<br>
Traceback (most recent call last):<br>
File “train.py”, line 110, in <br>
main()<br>
File “train.py”, line 56, in main<br>
dream_model.cuda()<br>
File “/mnt/home/dreamdd/newDream/DreamHU/venv/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 249, in cuda<br>
return self._apply(lambda t: t.cuda(device))<br>
File “/mnt/home/dreamdd/newDream/DreamHU/venv/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 176, in _apply<br>
module._apply(fn)<br>
File “/mnt/home/dreamdd/newDream/DreamHU/venv/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 176, in _apply<br>
module._apply(fn)<br>
File “/mnt/home/dreamdd/newDream/DreamHU/venv/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 176, in _apply<br>
module._apply(fn)<br>
File “/mnt/home/dreamdd/newDream/DreamHU/venv/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 182, in _apply<br>
param.data = fn(param.data)<br>
File “/mnt/home/dreamdd/newDream/DreamHU/venv/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 249, in <br>
return self._apply(lambda t: t.cuda(device))<br>
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/THCTensorRandom.cu:25</p>

username: aplassard
cooked: <p>Can you post your full source code? Or a minimum example that you have run that gets the error message? This is not enough information to help you.</p>

username: zuiluo
cooked: <p>can you help me solve my question?<br>
my question is similar like him ,<a href="https://discuss.pytorch.org/t/runtimeerror-couldnt-open-shared-event-error-code-2/76560">here</a> is my question,thanks</p>
Get string of tensor in libtorch

username: Maverobot
cooked: <p>I have tried to find a way to get the string reprensentation of a tensor for logging with <a href="https://github.com/gabime/spdlog" rel="nofollow noopener">spdlog</a>.</p>
<pre><code class="lang-auto">torch::Tensor tensor;

// works
std::cout &lt;&lt; tensor &lt;&lt; "\n";

// spdlog prints error: "basic_ios::clear: iostream error"
spdlog::info("tensor: {}", tensor);

</code></pre>
<p>I’m thinking to work around the error by getting the string of tensor first and pass it to <code>spdlog</code> but failed to find any API which does that. I really appreciate any help. Thanks in advance!</p>

username: LeviViana
cooked: <p>I think the best way is to transform the <code>std::ostringstream</code> version of your tensor into a string, you can do it this way:</p>
<pre><code class="lang-auto">std::ostringstream stream;
stream &lt;&lt; tensor;
std::string tensor_string = stream.str();
</code></pre>

username: Maverobot
cooked: <p>Thanks. I was actually hoping for something native. But if is does not exist, I guess your suggestion is the best I get <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Gabi_Melman
cooked: <p>spdlog provides an option for this.<br>
You can define a function that takes tensor and ostream and writes to it.<br>
See example in <a href="https://github.com/gabime/spdlog#user-defined-types" rel="nofollow noopener">https://github.com/gabime/spdlog#user-defined-types</a></p>

username: Maverobot
cooked: <p>Thanks for the awesome logging library. Based on my understanding, I wrote the following free function</p>
<pre><code class="lang-auto">template &lt;typename OStream&gt;
OStream&amp; operator&lt;&lt;(OStream&amp; os, const torch::Tensor&amp; t) {
  return os &lt;&lt; t;
}
</code></pre>
<p>which resulted in a pile of compile errors when combined with</p>
<pre><code class="lang-auto">spdlog::info("tensor: {}", tensor);
</code></pre>
<p>I am not sure where I am doing wrong here…</p>
Nvlink and Horovod

username: Rubeen_Mohammad
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , we generally use Nvlink for Parallelism and the Horovod framework for distributed training, where a common task will be executed on multiple processors. Do we have any connection between these two? What I mean is, can we connect the Horovod framework with Nvllink? Please, do help me if you know anything about this.<br>
Thank you</p>

username: mrshenli
cooked: <p>Not familiar with Horovod implementation. But if it internally uses PyTorch <code>ProcessGroup</code> or <code>DistributedDataParallel</code>, it would work with NVLink, if you specify the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.Backend" rel="nofollow noopener"><code>nccl</code> backend</a> when calling <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" rel="nofollow noopener"><code>init_process_group</code></a>.</p>

username: Rubeen_Mohammad
cooked: <p>Thank you so much <a class="mention" href="/u/mrshenli">@mrshenli</a> for your reply. Could you please share some of the code related links with me, if you have any.<br>
Thank you</p>

username: mrshenli
cooked: <p>Sure. Here two tutorials with code <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="nofollow noopener">c10d</a>, <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">ddp</a>, and a minimum <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="nofollow noopener">ddp example</a>.</p>

username: Rubeen_Mohammad
cooked: <p>Thank you so much <a class="mention" href="/u/mrshenli">@mrshenli</a> for your reply, I will work on it.</p>
Plotting the result of pytorch.stft

username: Kamil_Kwiatkowski
cooked: <p>Hi, how would I go about visualising the result of <a href="https://pytorch.org/docs/stable/torch.html?highlight=stft&amp;fbclid=IwAR1uokN-EyLZ61Lm4jcVrcxM5RrYGtDOSK5RdMvJKNwK55t3r1s0LhRww_Q#torch.stft" rel="noopener nofollow ugc">pytorch.stft</a> similiar to scipy?<br>
I don’t entirely understand how can I use that output to achieve a similiar plot:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bd49ca2f6f2165b1471ce28f978f858c5daf843.png" data-download-href="https://discuss.pytorch.org/uploads/default/1bd49ca2f6f2165b1471ce28f978f858c5daf843" title="Screenshot from 2020-04-12 15-01-28"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bd49ca2f6f2165b1471ce28f978f858c5daf843_2_690x329.png" alt="Screenshot from 2020-04-12 15-01-28" data-base62-sha1="3Ycps1bz7eWu9zrOyC1K8dBI5of" width="690" height="329" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bd49ca2f6f2165b1471ce28f978f858c5daf843_2_690x329.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bd49ca2f6f2165b1471ce28f978f858c5daf843_2_1035x493.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bd49ca2f6f2165b1471ce28f978f858c5daf843.png 2x" data-dominant-color="04040A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-04-12 15-01-28</span><span class="informations">1197×571 77.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Could you post a reference plot and highlight the difference?<br>
Based on your current output, it seems you might have some high outliers, which could squeeze the color map and thus result in a “dark” image.<br>
If that’s the case, you could <code>torch.clamp</code> the output before feeding it to <code>matplotlib</code>.</p>

username: Kamil_Kwiatkowski
cooked: <p>Perhaps I wasn’t clear - I’m just not sure how to use the output of pytorch.stft to get any plot. Could you please provide an example?</p>

username: ptrblck
cooked: <p>Ah OK, sure.<br>
Adapting the <code>scipy</code> example:</p>
<pre><code class="lang-python">fs = 10e3
N = 1e5
amp = 2 * np.sqrt(2)
noise_power = 0.01 * fs / 2
time = np.arange(N) / float(fs)
mod = 500*np.cos(2*np.pi*0.25*time)
carrier = amp * np.sin(2*np.pi*3e3*time + mod)
noise = np.random.normal(scale=np.sqrt(noise_power),
                         size=time.shape)
noise *= np.exp(-time/5)
x = carrier + noise

x = torch.from_numpy(x)
x = x.unsqueeze(0) # add batch dim


out = torch.stft(x, n_fft=1000)
out_real = out[:, :, :, 0]
out_imag = out[:, :, :, 1]

out_abs = torch.sqrt(out_real**2 + out_imag**2)

plt.pcolormesh(out_abs[0])
</code></pre>
Customized loss function - topk

username: marseille
cooked: <p>I am trying to implement a customized loss function in pytorch based on the formula below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54dedb9d58f1b99a34db28ce819ea18ac5345f25.png" data-download-href="https://discuss.pytorch.org/uploads/default/54dedb9d58f1b99a34db28ce819ea18ac5345f25" title="Screenshot 2020-04-13 at 10.13.24"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54dedb9d58f1b99a34db28ce819ea18ac5345f25_2_690x100.png" alt="Screenshot 2020-04-13 at 10.13.24" data-base62-sha1="c6NzQZqp2n1WgvDpMccMCTeogh7" width="690" height="100" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54dedb9d58f1b99a34db28ce819ea18ac5345f25_2_690x100.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54dedb9d58f1b99a34db28ce819ea18ac5345f25_2_1035x150.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/4/54dedb9d58f1b99a34db28ce819ea18ac5345f25.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-04-13 at 10.13.24</span><span class="informations">1074×156 9.16 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">def TopKLoss(pred, target, top_k=0.7):

    pred = F.log_softmax(pred)

    n = pred.size(0)
    c = pred.size(1)
    out_size = (n,) + pred.size()[2:]
    if target.size()[1:] != pred.size()[2:]:
        raise ValueError('Expected target size {}, got {}'.format(
            out_size, target.size()))

    pred = pred.contiguous()
    target = target.contiguous()

    for batch_no in range(n):
        for class_no in range(c):
            print(pred[batch_no][class_no])
            print(target[batch_no]==class_no)
            pixel_sum = (torch.numel(pred[batch_no][class_no]&lt;top_k and target[batch_no]==class_no))
            print(pixel_sum)
            topk_loss+= pixel_sum

    return topk_loss
</code></pre>
<p>This is what I have until now, but it throws an error when I try to sum over the elements<br>
which respect the condition.</p>
<pre><code class="lang-auto">RuntimeError: bool value of Tensor with more than one value is ambiguous 
</code></pre>
<p>Can somebody help with this?</p>
[Dataloader] ‘Shuffle=True’ makes IO slow

username: zhaominyiz
cooked: <p>Thanks everyone.<br>
My dataset contains 15 million images. I have convert them into lmdb format and concat them<br>
At first I set shuffle = False，envery iteration’s IO take no extra cost.<br>
Inorder to improve the performance , I set it into True and use num_workers.</p>
<pre><code class="lang-auto">train_data = ConcatDataset([train_data_1,train_data_2])
    train_loader = DataLoader(dataset=train_data, batch_size=64,num_workers=32, shuffle=True,pin_memory=False)
for i in range(epochs):
for j,data in enumerate(train_loader):
                print(i,"ITER",j,"IO END",datetime.now())
                continue
</code></pre>
<p>But IO takes too much time.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05742347516e42e24f42d6649721d596a964540b.png" alt="image" data-base62-sha1="Mfd6PUPEDqpR0KQ95Ze5qHpGaf" width="288" height="125"><br>
Is there something I can do to makes IO faster?</p>

username: JuanFMontesinos
cooked: <p>Hi,<br>
Have you evaluated the behaviour using both num_workers=0 in both cases?<br>
Is it possible your data is optimized for sequential reading (im not familiarized with lmdb  format)?<br>
Setting shuffle=True does nothing but replacing a sequential generator range(0,len) by a list of random indices.</p>

username: zhaominyiz
cooked: <p>Thanks very much.<br>
set num_workers = 0 doesnot work.<br>
It seems that the cloud sever cause the problem. I put the dataset in the SSD of my local machine and solve the problem.</p>
Type conversion in libtorch

username: Edwardmark
cooked: <p>Hi，<br>
Say we have two tensor priors, and loc_cat,<br>
I want to set the type of priors same as loc_cat,<br>
in python, we can use the following code:</p>
<p>priors.type(loc_cat.data.type)</p>
<p>So how to do that in cpp libtorch?<br>
Thanks.</p>
Loss decreases slowly with WeightedRandomSampler

username: KarthikR
cooked: <p>Hi,</p>
<p>I have an multi-label classification problem. There are 4 targets (500 observations each, for the first three labels and 50 observations for the fourth label). The loss function is cross entropy.</p>
<p>I use pretrained model (ResNet18) in Pytorch.</p>
<p><strong>Case 1</strong>:<br>
When I use train_test_split from sklearn (with stratify) and use it as usual (creating instance of Dataset class and then feeding to the DataLoader), my validation loss seems to reduce over epochs.</p>
<p><strong>Case 2</strong>:<br>
When I do train_test_split (with stratify) and use WeightedRandomSampler (to account for the imbalance), my validation loss reduces slowly (requiring more epochs) but never to the level seen in “Case 1”. There is lot of fluctuation in the loss too.</p>
<p>I saw the code for WeightedRandomSampler in GitHub. Using the WeightedRandomSampler, my dataset now contains more observations of the minor class, which is what I want. My questions are:</p>
<ol>
<li>Is “Case 2”, the expected behavior, when using the WeightedRandomSampler?</li>
<li>Any thoughts how I can reduce validation loss, when using the WeightedRandomSampler?</li>
</ol>

username: ptrblck
cooked: <p>In your first case, the loss might decrease fast, if your model simply overfits the majority class(es), thus ignoring the minority class.<br>
You could calculate the confusion matrix for both use cases and calculate the per-class accuracies to rate both approaches.<br>
Note that a simple accuracy calculation might be misleading for an imbalanced use case as explained by the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">Accuracy Paradox</a>.</p>

username: KarthikR
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="76443">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>confusion matrix</p>
</blockquote>
</aside>
<p>Thank you, will explore.</p>
Broadcasting the depthwise convolution kernel

username: sr_frost
cooked: <p>I am trying to do depthwise convolution with a fixed kernel. My kernel is 2D, and I want to use the same 2D kernel for all channles. This is what I am currently doing(code modified from <a href="https://github.com/adobe/antialiased-cnns" rel="nofollow noopener">antialiased-cnns</a>):</p>
<pre><code class="lang-auto">x = torch.randn([32, 256, 64, 64])

lpf = torch.tensor([[1., 2., 1.],
                    [2., 4., 2.],
                    [1., 2., 1.]])
lpf = lpf/torch.sum(lpf)
lpf = lpf.unsqueeze(dim=0).unsqueeze(dim=0)
lpf = lpf.repeat([self.channels, 1, 1, 1])
y = F.conv2d(input=x, weight=self.lpf, padding=1, groups=self.channels)
</code></pre>
<p>While this works correctly, I was wondering if I could somehow get rid of the <code>lpf.repeat()</code> command. Is there a way to broadcast the operation that doesn’t reqiure explicit <code>.repeat()</code> operation ?</p>

username: ptrblck
cooked: <p>You could use <code>expand(256, -1, -1, -1)</code> instead of <code>repeat</code>.<br>
Not however, that you would only save ~9kB of memory, since:</p>
<pre><code class="lang-python">print(lpf.nelement() * 4 / 1024)
&gt; 9.0
</code></pre>
Concatenating list of variable size tensors

username: Ashray_Aggarwal_14
cooked: <p>I have two python lists of tensors A and B each of size n.<br>
Each of n tensors is of size <code>(8* 1* Variable* Variable)</code><br>
I want to combine them into a single list of size n with tensors of size<code> (8* 2* Variable* Variable)</code> without the use of for loop. (Using for loop is breaking the computation graph).<br>
Any possible way out?</p>
Element-wise comparison of two tensor and take the maximum value

username: 19941116
cooked: <p>how to get element-wise comparison and take element-wise maximum? is there a function in pytorch？I think a for loop is too slow<br>
eg: tensor_1=[1,2,3,4] tensor_2=[-10,-20,10,20]<br>
I want to get a tensor=[1,2,10,20]</p>

username: Okan
cooked: <p>You can use torch.max(tensor_1, tensor_2). Check out <a href="http://pytorch.org/docs/stable/torch.html#torch.max" rel="nofollow noopener">http://pytorch.org/docs/stable/torch.html#torch.max</a></p>

username: shamoons
cooked: <p>How about more than 2?</p>

username: ptrblck
cooked: <p>You could concatenate all tensors and use <code>torch.max(tensor, dim=concat_dim)</code>.</p>

username: shamoons
cooked: <p>Would I have to <code>.cat</code> along a specific axis?</p>

username: ptrblck
cooked: <p>Yes and it depends on your use case. I.e. if you want to calculate the <code>max</code> in dim1 for all tensors, you should concatenate them in the same dimension.<br>
Let me know, if that would work for your use case.</p>
Get previous layer from batch normalization layer

username: Guilherme_Martins
cooked: <p>Hi guys,</p>
<p>I am using batch normalization weights as scores of the channel importance of previous convulotional layer, similarly to what is done in some network pruning methods.<br>
My objective is to single out the convulotional layers that have the lowest channel importance and get its weights. In essence, the problem is: How do I get the convulotional layer’s channel that is attatched to the lowest weight of the following batch normalization layer.</p>

username: ptrblck
cooked: <p>I’m not sure, if I understand the use case completely.<br>
If your batchnorm layer takes e.g. 6 input channels, and <code>bn.weight[1]</code> has the lowest value, it would correspond to <code>conv.weight[1]</code> of the preceding layer.</p>

username: Guilherme_Martins
cooked: <p>Thank you for the reply!<br>
Not always, there can be more then one convulotional layer connecting to the batch normalization layer and they can come from any previous layer. Dense net, for example does this. Where you have cnv1 and cnv2 connecting to bnorm1 and cnv1 connevting to cnv2 (If Im not mistaken).</p>

username: ptrblck
cooked: <p>That might be the case, so could you please post a code snippet for your use case?</p>
Using multiple losses for a model

username: _joker
cooked: <p>I am training two losses for multi-label autoencoder. Both losses aim at different segments in the attention module of the model. Initially, I was thinking of combining the two losses into a single loss like;</p>
<p><code>loss = loss1 - coef. * loss2 // having coef. to give less attention to loss2.</code></p>
<p>But in this case, the total loss starts going negative and keeps increasing in negative after some epochs. So I am thinking of updating both the losses separately for the model like;</p>
<pre><code class="lang-auto">loss1.backward(retain_graph=true)
loss2.backward()
optimizer.step()
</code></pre>
<p>I am looking for more suggestions and if I am missing something in my first approach (combined loss).</p>

username: zacharynew
cooked: <p>Clarifying question: Should both losses be minimized in your problem?</p>
<p>If so then it seems to me that you would want something like <code>loss = loss1 + coef. * loss2</code>. Seems to me that your coefficient setup is perfectly reasonable, it will no doubt need to be tuned to produce the right focus for your model though.</p>
<p>Mathematically speaking I think hitting both individual loss functions with <code>backward()</code> is equivalent to this loss: <code>loss = loss1 + loss2</code> as it will just accumulate the gradients together in the second backwards pass.</p>

username: _joker
cooked: <p>Thanks!<br>
It was surprise to me that the model’s accuracy increased a lot even if the over all loss was too extreme to negative. I tested the model and based on it’s so good performance I am confused in justifying the behavior of the loss below zero as I haven’t seen any such examples so far.</p>

username: zacharynew
cooked: <p><a class="mention" href="/u/_joker">@_joker</a> in your original example you show subtracting on loss function from another. Most loss functions are usually defined to be minimized and not maximized. If you are subtracting a loss function then you are in-effect asking the optimizer to make that loss function as big as it can. This might result in your large negative loss values.</p>

username: _joker
cooked: <p><a class="mention" href="/u/zacharynew">@zacharynew</a><br>
True! The reason to subtract the losses is –<br>
loss1 – encoded representation from the <em>i th</em> class label<br>
loss2 – encoded representation from the  <em>non i th</em> class label.</p>
<p>Here, I encourage the loss1 to be mapped to as close to hyper-ball center where each class is a p-norm ball so it has to be subtracted from loss1.</p>