akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Memory leak while exporting torchscript using torch.jit.trace

shengchunnan: <p><strong>I have found there is obvious memory leak when I try to export torchscript multiple times.</strong><br>
<strong>Here is the piece of code to reproduce this fault. The process may occupy over 20 GB RAM after exporting resnet50 40 times.</strong></p>
<pre><code class="lang-auto">import torch
import torchvision.models as vision_models
from torchvision.models import resnet

resnet50 = vision_models.resnet50()

for i in range(1000):
    jit_resnet50 = torch.jit.trace(resnet50, torch.randn(16, 3, 448, 448))
    print('{} times of jit trace done!'.format(i + 1))
    jit_resnet50 = None
</code></pre>
<h2>
<a name="pytorch-version-1100-1" class="anchor" href="#pytorch-version-1100-1"></a>Pytorch version: 1.10.0</h2>
<h2>
<a name="platform-ubuntu-1804-with-or-without-cuda-2" class="anchor" href="#platform-ubuntu-1804-with-or-without-cuda-2"></a>Platform: Ubuntu 18.04, with or without CUDA</h2>

tom: <p>Oh oh. I think this is half-expected because of how TorchScript’s caching works. It isn’t a good thing, though, and at least there should be a way to reset things.</p>
<p>Best regards</p>
<p>Thomas</p>

ydshieh: <p>Hope this could be fixed. It causes CI issue in <code>transformers</code> repository as we have more and more models added.</p>

ganler: <p>We also encountered this issue which makes it very hard for doing in-process neural architecture search or model generation since it will eventually cause OOM issue. <a href="https://github.com/pytorch/pytorch/issues/86537" class="inline-onebox" rel="noopener nofollow ugc">Running JIT trace for many times leads to OOM · Issue #86537 · pytorch/pytorch · GitHub</a></p>
<p>Hope this could be fixed or we have some mechanisms to reset the caches. <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
How to apply constraint to individual weights?

razvanc92: <p>Hello, I’m trying to implement Lotka Volterra system from <a href="https://dl.acm.org/doi/10.1145/2736277.2741092" rel="nofollow noopener">this paper</a>. Looking at eq 3. they are enforcing some constraints for each parameter. I was curios I how I achieve a similar behavior? For example some weights to be always in [0,1] interval, or another weight matrix, to always have 1. on the diagonal.</p>

anantguptadbl: <p>Lets say you have a constraint between 0 and 1, you can do it the following way</p>
<pre><code class="lang-auto"># Constraints
from torch import nn
from torch.autograd import Variable
import torch.nn.functional as F
import torch

torch.manual_seed(42)

class model1(nn.Module):
  def __init__(self):
    super(model1,self).__init__()
    self.l1=nn.Linear(100,10)
    self.l2=nn.Linear(10,5)
    self.l3=nn.Linear(5,1)

  def forward(self,x):
    out=self.l1(x)
    out=self.l2(out)
    out=self.l3(out)
    return(out)

model=model1()
criterion=nn.MSELoss()
optimizer=optim.Adam(model.parameters(),lr=0.001)

X=Variable(torch.from_numpy(np.random.rand(1000,100).astype(np.float32)))
Y=Variable(torch.from_numpy(np.random.randint(2,size=(1000)).astype(np.float32)))
epochs=1000

constraintHigh=1
constraintLow=0

for curEpoch in range(epochs):
  model.zero_grad()
  output=model(X)
  loss=criterion(output,Y)
  loss.backward()
  optimizer.step()
  # We will now apply the constraint and reset the weights
  model.l1.weight=torch.nn.Parameter(constraintLow + (constraintHigh-constraintLow)*(model.l1.weight - torch.min(model.l1.weight))/(torch.max(model.l1.weight) - torch.min(model.l1.weight)))
  if(curEpoch % 100==0):
    print("The loss is {0}".format(loss.item()))
    print("Weight Min is {0} and Max is {1}".format(torch.min(model.l1.weight),torch.max(model.l1.weight)))
</code></pre>
<p>Hope this helps</p>

kenzayy267: <p>Thank you for the solution.</p>
<p>I have tried this way, but I got bad regression results. I assume that the weight and bias are modified, and away from the origin value. I think the origin value is correct value by BP.</p>
<p>Do you have any solution of this issue?<br>
Thank you very much.</p>
How to measure model loading time layer by layer

oracal: <p>I wanna profile layer-wise loading time for DL models in pytorch. As shown below, I defined a timestamp list <em><strong>lTime</strong></em> and used it in forward_pre_hooks. However, I found that <em><strong>sum(iTime)</strong></em> is always not equal to <em><strong>duration(model.cuda())</strong></em> and there is such big difference, for example, for resnet18, layerwise loading time I measured was 2.8ms but 3.5ms by <em><strong>model.cuda()</strong></em>. Could you help me ?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c3321a191e9744dc30c4c53f6e4ba8efa36d78d.png" data-download-href="https://discuss.pytorch.org/uploads/default/4c3321a191e9744dc30c4c53f6e4ba8efa36d78d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c3321a191e9744dc30c4c53f6e4ba8efa36d78d_2_690x169.png" alt="image" data-base62-sha1="aS5PQoRYjcc62pv6eGwTgZ0mpoF" width="690" height="169" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c3321a191e9744dc30c4c53f6e4ba8efa36d78d_2_690x169.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c3321a191e9744dc30c4c53f6e4ba8efa36d78d_2_1035x253.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c3321a191e9744dc30c4c53f6e4ba8efa36d78d.png 2x" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1164×286 25.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to deal with nan in the target images

fdzoom: <p>I am trying to train a Unet. The inputs tensors is complete (withoout nan values). However, the target tensors contain numberous nan values, which is due to the detection methods, e.g., the surface properties masked by the clouds. How can I manage to train the Unet with these masked target images?</p>

KFrank: <p>Hi fdzoom!</p>
<aside class="quote no-group" data-username="fdzoom" data-post="1" data-topic="164420" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/a587f6/48.png" class="avatar"> fdzoom:</div>
<blockquote>
<p>However, the target tensors contain numberous nan values,</p>
</blockquote>
</aside>
<p>You don’t say much about your use case or what loss criterion you are using.</p>
<p>But, for example, if you were performing multi-class semantic segmentation<br>
(something for which U-Net could be appropriate), <code>CrossEntropyLoss</code><br>
would be a common choice of loss criterion.</p>
<p>In this case you could pre-process your targets, replacing the <code>nan</code>s with a<br>
non-<code>nan</code> sentinel value, and then use that value for <code>CrossEntropyLoss</code>’s<br>
<code>ignore_index</code> constructor argument.  (It would be convenient to use <code>-100</code><br>
for this as <code>-100</code> is <code>ignore_index</code>’s default value.)</p>
<p>Best.</p>
<p>K. Frank</p>

fdzoom: <p>Thanks for the reply.<br>
I’m dealing with a regression problem, using the input tensors (5, 256, 256) to predict the target image (256, 256). The values of target images varies from 0 to 1 and nan values are very common. Therefore, the current used loss criterion is MSEloss. From your proposal, I wonder if I can replace the nans with a non-nan values  and use a masked MSEloss?</p>

KFrank: <p>Hi fdzoom!</p>
<aside class="quote no-group" data-username="fdzoom" data-post="3" data-topic="164420" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/a587f6/48.png" class="avatar"> fdzoom:</div>
<blockquote>
<p>replace the nans with a non-nan values  and use a masked MSEloss?</p>
</blockquote>
</aside>
<p>Yes, that would seem like a sensible thing to try.</p>
<p>I’m not aware of pytorch having a built-in masked mse-loss, but it would be<br>
completely straightforward to write your own.  Or your could use <code>MSELoss</code><br>
with <code>reduction = 'none'</code> (after replacing your <code>nan</code>s with some placeholder<br>
value) and then perform your own masked average over the unreduced<br>
result.</p>
<p>Best.</p>
<p>K. Frank</p>
How to constrain weights in training?

kenzayy267: <p>I have tried like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, mask_1_2, mask_2_3, mask_3_4, mask_4_out):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1, 2)
        self.fc2 = nn.Linear(2, 3)
        self.fc3 = nn.Linear(3, 2)
        self.fc4 = nn.Linear(2, 1)
        
        self.mask_1_2 = mask_1_2
        self.mask_2_3 = mask_2_3
        self.mask_3_4 = mask_3_4
        self.mask_4_out = mask_4_out
    def forward(self, x, scale="no"):  
        if scale == "yes":
            self.fc1.weight.data = self.fc1.weight.data.mul_(self.mask_1_2)
            self.fc1.weight.data = self.fc1.weight.data.clamp(-1, 1)
            self.fc1.bias.data = self.fc1.bias.data.clamp(-1, 1)
            x = torch.sigmoid(self.fc1(x))
            
            self.fc2.weight.data = self.fc2.weight.data.mul_(self.mask_2_3)
            self.fc2.weight.data = self.fc2.weight.data.clamp(-1, 1)
            self.fc2.bias.data = self.fc2.bias.data.clamp(-1, 1)
            x = torch.sigmoid(self.fc2(x))
            
            self.fc3.weight.data = self.fc3.weight.data.mul_(self.mask_3_4)
            self.fc3.weight.data = self.fc3.weight.data.clamp(-1, 1)
            self.fc3.bias.data = self.fc3.bias.data.clamp(-1, 1)
            x = torch.sigmoid(self.fc3(x))
            
            self.fc4.weight.data = self.fc4.weight.data.mul_(self.mask_4_out)
            self.fc4.weight.data = self.fc4.weight.data.clamp(-1, 1)
            self.fc4.bias.data = self.fc4.bias.data.clamp(-1, 1)
            x = torch.sigmoid(self.fc4(x))
        
        elif scale == "no":
            self.fc1.weight.data = self.fc1.weight.data.mul_(self.mask_1_2)
            x = torch.sigmoid(self.fc1(x))
            
            self.fc2.weight.data = self.fc2.weight.data.mul_(self.mask_2_3)
            x = torch.sigmoid(self.fc2(x))
            
            self.fc3.weight.data = self.fc3.weight.data.mul_(self.mask_3_4)
            x = torch.sigmoid(self.fc3(x))
            
            self.fc4.weight.data = self.fc4.weight.data.mul_(self.mask_4_out)
            x = torch.sigmoid(self.fc4(x))
        return x
</code></pre>
<p>It’s ok to train when “scale” == no, the result for fitting a SIN func is good.<br>
But if I use “scale” == yes, the regression result will be very bad.<br>
I think the weights are modified, so they are away from the correct Backpropagation. Is there any way to train well with scale?</p>
Selecting n indices without replacement from dimension x

sachinruk: <p>Suppose I have the following embeddings <code>emb_user = torch.randn(64, 128, 256)</code>. From the second dimension (of length 128), I wish to pick out 16 at random at each instance. I was wondering if there was a more efficient way of doing the following:</p>
<pre><code class="lang-auto">idx = torch.multinomial(torch.ones(64, 128), 16)
sampled_emb_user = emb_user[torch.arange(len(emb_user)).unsqueeze(-1), idx]
</code></pre>
<p>What I also find curios is that the above multinomial would not work if the weight matrix (<code>torch.ones(64, 128)</code>) exceeded more than 2 dimensions.</p>
Is there a way to debug (evaluate the node parsed) during torch ONNX export?

Litchilitchy: <p>Hi I am trying to debug during torch.onnx.export, when the layer/node is being converted, a <code>def _parse_arg(value, desc, arg_name=None, node_name=None):</code> method is called (torch/onnx/symbolic_helper.py)</p>
<p>I tried to set a breakpoint and evaluate <code>value.node()</code> in debug console (in VS Code) and it said <code>RuntimeError: has been invalidated</code></p>
<p>Is there any way to check the nodes during debug?</p>

tom: <p>You need to hold on to the graph in order to look at nodes.</p>
Difference between torch.randn() and normal_()

f10w: <p>I would like to know if there is any difference in the end results between</p>
<pre><code class="lang-auto">self.register_buffer('my_layer', torch.randn(m, n))
</code></pre>
<p>and</p>
<pre><code class="lang-auto">self.register_buffer('my_layer', torch.Tensor(m, n))
self.my_layer.data.normal_()
</code></pre>
<p>Thank you in advance for your help.</p>

ptrblck: <p>The second approach will use uninitialized memory for <code>self.my_layer</code> while an embedding tensor is filled with values from a Gaussian distribution. Maybe you wanted to call <code>self.my_layer.normal_()</code>?<br>
In this case, both approaches would sample values from a Gaussian.</p>

f10w: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>!<br>
Sorry I forgot to change the variable name. I meant <code>self.my_layer.data.normal_()</code> in the second (I’ve updated the post).<br>
Now as you mention it, is there a difference between <code>self.my_layer.data.normal_()</code> and <code>self.my_layer.normal_()</code>?</p>

soulitzer: <p>Using <code>.data</code> bypasses certain correctness checks for in-place modification to tensor. You should avoid using that if possible.</p>

f10w: <p>Thanks for your reply.</p>
Error: ValueError: can't optimize a non-leaf Tensor

periklis17: <p>I get this error ValueError: can’t optimize a non-leaf Tensor. My variable is leaf (torch.Tensor([1.]).requires_grad_().to(device)) however when I pass it in optimizer I get this error.</p>

srishti-git1110: <p>Hi,<br>
<code>.to</code> is a differentiable operation and hence is recorded by autograd which makes your tensor as non-leaf.<br>
Please see if this helps:</p>
<pre><code class="lang-auto">import torch
a = torch.tensor([1.0], device='cuda', requires_grad=True)
print(a.is_leaf) # True - you might want to use this approach

b = torch.Tensor([1.]).requires_grad_().to('cuda')
print(b.is_leaf) # False
</code></pre>

periklis17: <aside class="quote no-group" data-username="srishti-git1110" data-post="2" data-topic="164472">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/srishti-git1110/48/52515_2.png" class="avatar"> srishti-git1110:</div>
<blockquote>
<p><code>torch.tensor([1.0], device='cuda', requires_grad=True)</code></p>
</blockquote>
</aside>
<p>Thanks for the reply, but again I get the same error.<br>
The problem is when<br>
torch.optim.SGD(list(a), lr=0.1). Here I get the error<br>
ValueError: can’t optimize a non-leaf Tensor<br>
If I set<br>
a = torch.Tensor([1.]).requires_grad_()<br>
then I can pass it through the optimizer but I can not optimize the “a”<br>
variable</p>

srishti-git1110: <aside class="quote no-group" data-username="periklis17" data-post="3" data-topic="164472">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ac91a4/48.png" class="avatar"> periklis17:</div>
<blockquote>
<p>torch.optim.SGD(list(a), lr=0.1). Here I get the error<br>
ValueError: can’t optimize a non-leaf Tensor</p>
</blockquote>
</aside>
<p>Use this instead:</p>
<pre><code class="lang-auto">opt = torch.optim.SGD([a], lr=0.1)
</code></pre>
<p>The reason why <code>list(a)</code> doesn’t work but <code>[a]</code> does:</p>
<pre><code class="lang-auto">print(list(a)) # [tensor(1., device='cuda:0', grad_fn=&lt;UnbindBackward0&gt;)]
print([a]) # [tensor([1.], device='cuda:0', requires_grad=True)]
</code></pre>
<p>In the first case, autograd was able to track the operation which is evident through the grad_fn attribute of <code>a</code> and so <code>a</code> doesn’t remain a leaf anymore.</p>

periklis17: <p>Great it is working now!<br>
Thank you so much!</p>
Quantization type error

ELIVATOR: <p><strong># CODE 1 - type = uint8</strong><br>
model.qconfig = torch.quantization.QConfig(<br>
activation=default_observer.with_args(dtype=torch.quint8),<br>
weight=default_float_qparams_observer.with_args(dtype=torch.quint8))<br>
model_prepared = torch.quantization.prepare(model, inplace=True)<br>
torch.quantization.convert(model_prepared, inplace=True)</p>
<p><strong>result : AssertionError: Weight observer must have dtype torch.qint8</strong><br>
(“/usr/local/lib/python3.9/dist-packages/torch/nn/quantized/modules/linear.py”, line 282, in from_float<br>
)</p>
<p><strong># CODE 2 - type = int8</strong><br>
model.qconfig = torch.quantization.QConfig(<br>
activation=default_observer.with_args(dtype=torch.qint8),<br>
weight=default_float_qparams_observer.with_args(dtype=torch.qint8))<br>
model_prepared = torch.quantization.prepare(model, inplace=True)<br>
torch.quantization.convert(model_prepared, inplace=True)</p>
<p><strong>result : AssertionError: The only supported weight dtype for nnq.Embedding is torch.quint8, got torch.qint8</strong><br>
(“/usr/local/lib/python3.9/dist-packages/torch/nn/quantized/modules/embedding_ops.py”, line 162, in from_float)</p>
<p>Each type make different error.<br>
If i change weight observer’s type to int8, i can watch unsupported type error.<br>
If i change weight observer’s type to uint8, i can watch weight observer error.</p>
<p>How can I solve this problems…</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/elivator">@ELIVATOR</a> , for embeddings the supported dtype for weight is quint8, and for other ops it’s usually qint8.  You can configure this by assigning the appropriate qconfigs to the right parts of the model.  For example:</p>
<pre><code class="lang-auto">qconfig_global = torch.quantization.QConfig(
    activation=default_observer,
    weight=default_weight_observer)
qconfig_emb = torch.quantization.QConfig(
    activation=default_observer,
    weight=default_float_qparams_observer)

# set global qconfig
model.qconfig = qconfig_global
# set custom qconfig for embedding layer
model.foo.bar.emb.qconfig = qconfig_emb
</code></pre>

Litchilitchy: <p>Sometimes there are multiple embedding layers in one model. Is there any way to set them once and for all?</p>

Vasiliy_Kuznetsov: <p>If you are using the eager mode workflow, you can do something like</p>
<pre><code class="lang-auto">for _, child in model.named_modules():
    if isinstance(child, torch.nn.Embedding):
        child.qconfig = emb_qconfig
</code></pre>
<p>If you are using the FX graph mode workflow, you can use <code>QConfigMapping</code> to map a module type to a qconfig.</p>
Cant convert model using qnnpack

CRPS: <p>I’m trying to convert model using this config</p>
<p>conf = QConfig(<br>
activation=Quantizers.FakeQuantize.with_args(<br>
observer=Observers.MovingAverageMinMaxObserver.with_args(<br>
dtype=torch.qint8),<br>
qscheme=torch.per_tensor_symmetric,<br>
quant_min=-127,<br>
quant_max=127,<br>
dtype=torch.qint8</p>
<pre><code>    ),
    weight=Quantizers.FakeQuantize.with_args(
        observer=Observers.MovingAveragePerChannelMinMaxObserver,
        quant_min=-127,
        quant_max=127,
        dtype=torch.qint8,
        qscheme=torch.per_channel_symmetric,
        reduce_range=False,
        ch_axis=0
    ))
</code></pre>
<p>It works for quint8 activations, but not for int8. I checked this readme <a href="https://github.com/pytorch/pytorch/tree/master/torch/ao/quantization/fx" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/ao/quantization/fx at master · pytorch/pytorch · GitHub</a> and found this description of observers insertion.</p>
<p>QuantDeQuantStubs are inserted based on the <code>qconfig_mapping</code> provided by users. Also we have a backend_config that specifies the configs that are supported by the backend. In this step, we will</p>
<ul>
<li>Check if <code>qconfig_mapping</code> is compatible with <code>backend_config</code> or not, if user requested a qconfig that is not compatible with <code>backend_config</code>, we’ll not insert observers for the operator, the config would just be ignored.</li>
<li>Insert observer for the input and output of the subgraph, based on the <code>qconfig_mapping</code> (what user requested) and the <code>backend_config</code> (how the operator should be observed in a backend).</li>
</ul>
<p>After i checked if default backends configs support qint8 data type. Qnnpack should support qint8 feature maps because of following configuration in <a href="https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/backend_config/qnnpack.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/qnnpack.py at master · pytorch/pytorch · GitHub</a> :</p>
<p>qnnpack_act_qint8_scale_min_2_neg_12 = DTypeWithConstraints(<br>
dtype=torch.qint8,<br>
scale_min_lower_bound=2 ** -12,<br>
)</p>
<p>qnnpack_weighted_op_qint8_symmetric_dtype_config = DTypeConfig(<br>
input_dtype=qnnpack_act_qint8_scale_min_2_neg_12,<br>
output_dtype=qnnpack_act_qint8_scale_min_2_neg_12,<br>
weight_dtype=qnnpack_weight_qint8_neg_127_to_127_scale_min_2_neg_12,<br>
bias_dtype=torch.float,<br>
)</p>
<p>qnnpack_default_op_qint8_symmetric_dtype_config = DTypeConfig(<br>
input_dtype=qnnpack_act_qint8_scale_min_2_neg_12,<br>
output_dtype=qnnpack_act_qint8_scale_min_2_neg_12,<br>
)</p>
<p>And this configs added into qnnpack backend config.</p>
<p>Also i set backend by command: torch.backends.quantized.engine=“qnnpack”</p>
<p>Why my model not converted by observers?</p>

Vasiliy_Kuznetsov: <p>Do you have a e2e reproducible example you could share?  High level, what you are trying to do should work, but it’s hard to say why it’s not working without some more context.</p>
MPS enabled libtorch builds

spiegelball: <p>Are there any MPS enabled libtorch builds available? It seems, homebrew on macOS is not building them.</p>

dfalbel: <p>I have set a small repository with pre-built binaries here: <a href="https://github.com/mlverse/libtorch-mac-m1" class="inline-onebox" rel="noopener nofollow ugc">GitHub - mlverse/libtorch-mac-m1: LibTorch builds for the M1 Macs</a><br>
See also discussion in <a href="https://discuss.pytorch.org/t/precompiled-libtorch-for-apple-silicon/162635" class="inline-onebox">Precompiled Libtorch for Apple Silicon</a></p>

spiegelball: <p>Thank you so much, this is very helpful!</p>
Nan Loss for instance segmentation with MASK -RCNN using custom dataset

Aaron_De_Rybel: <p>Hello Community,</p>
<p>I followed the instance segmentation tutorial of pytorch vision using mask-RCNN inside google colab notebook<br>
<a href="https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/torchvision_finetuning_instance_segmentation.ipynb#scrollTo=at-h4OWK0aoc" rel="noopener nofollow ugc">Tutorial notebook</a></p>
<p>I decided to finetune it to a different dataset.<br>
Segmentation masks are provided in Json format.<br>
I can convert these masks to correct format in dataloader<br>
You can find the notebook &amp; data in following github link:<br>
<a href="https://github.com/AaronDeRybelHowest/instance-segmentation" rel="noopener nofollow ugc">github link</a></p>
<p>My problem is that after initial epoch i get Nan loss functions. How do i fix this issue?</p>
C++: LoadLibrary failed with error 1114

xhuan8: <p>Hi,</p>
<p>I try to load torchvision.dll dynamically on windows, but it fails with error 1114.<br>
Torchvision runs good if call from Python, what should I do to make it avaliable for c++?<br>
cuda was turned on during the build.</p>
<p>I have a intel integrated graphics card and Nvidia 3060 on laptop.</p>
<p>Thanks.</p>

xhuan8: <p>Solved, some dependency missing.</p>
Convert to mel spectrogram while maintaining the generated gradient

skerlet_flandorle: <p>hai.<br>
I deal with VAE</p>
<pre><code class="lang-auto">sample_rate=16000
n_mels = 80
n_fft = 512
win_length = None
hop_length = 128
window_fn = torch.hann_window(80)

spectrogram = T.MelSpectrogram(
    sample_rate=sample_rate,
    n_fft=n_fft,
    n_mels=n_mels,
    win_length=win_length,
    hop_length=hop_length,
    win
</code></pre>
<p>And in the process, after creating the raw speech, it is necessary to convert it to a mel spectrogram while maintaining the gradient and remove the vAE reconstruction error</p>
<p>However, when inputting, an error occurs if you do it like when converting audio to a spectrum</p>
<pre><code class="lang-auto">    normalized, onesided, return_complex)
RuntimeError: stft input and window must be on the same device but got self on cuda:0 and window on cpu
</code></pre>
<p>I expected to move hann_window to cuda from the following error<br>
However</p>
<pre><code class="lang-auto">window_fn=torch.hann_window(80).to("cuda")
</code></pre>
<p>If you do like this, another problem will appear and I don’t know how to solve this</p>
<pre><code class="lang-auto">\cutevoice\lib\site-packages\torchaudio\transforms\_transforms.py", line 81, in __init__
    window = window_fn(self.win_length) if wkwargs is None else window_fn(self.win_length, **wkwargs)
TypeError: 'Tensor' object is not callable
</code></pre>
How to visualize the behavior of optimization algorithms?

BobKim: <p>Hello.<br>
I’m deep interested in optimization algorithm, such as Adam, SGD.<br>
I’ve been trying various things to improve Adam.<br>
Then i saw these animations. (<strong>Alec Radford’s animations</strong>)<br>
These are very famous animations.<br>
<strong>So, Are there any examples that can be implemented with PyTorch?</strong><br>
<strong>I want to apply my new algorithm to the example and visualize it like the animation below.</strong></p>
<p><img src="https://media.giphy.com/media/KXFOUfGczmL7MXkjIi/giphy.gif" alt="noisy-moons" width="400" height="399" class="animated"><br>
<img src="https://media.giphy.com/media/2iy2dxhzMGhFLqqIUA/giphy.gif" alt="beales-function" width="480" height="372" class="animated"><br>
<img src="https://media.giphy.com/media/dqHC8AIezoFKXnb56Z/giphy.gif" alt="long-valey" width="480" height="372" class="animated"><br>
<img src="https://media.giphy.com/media/cSRNLluRz69B93Zf5X/giphy.gif" alt="saddle-point" width="480" height="372" class="animated"></p>
How to deal with overlapping segmentations for chest scan images

ltmatrix85: <p>Hi all,</p>
<p>I am working on a chest scan image classification model on pneumonia related conditions but I came to understand from clinicians that some of the conditions they are looking for actually overlaps one another on the chest scan image. Am aware that UNet is widely used for image segmentation but does it really solve the problem of overlapping images? If not what are other methods to solve this issue? I have done up a multilabel classification model but results aren’t great, as I would score highly for my F1 in one condition but some overlapping conditions will get a low F1 score.</p>
<p>One alternative I had was to split each conditions up into an individual binary classification model, but am trying to explore if there are other better alternatives. Am currently training my images on densenet121</p>

tom: <aside class="quote no-group" data-username="ltmatrix85" data-post="1" data-topic="164431">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ltmatrix85/48/48446_2.png" class="avatar"> ltmatrix85:</div>
<blockquote>
<p>split each conditions up into an individual binary classification model</p>
</blockquote>
</aside>
<p>You can do this by just re-interpreting the output of the u-net as a “multi-class” classification output (i.e. one that is put through sigmoid to get a probability for each class individually).</p>
<p>Best regards</p>
<p>Thomas</p>
Is there any mismatch for hardswish_backend when input x equal to 3

i4oolish: <p>Hi, all<br>
From the define in the pytorch document link[1], it seems the d_in should equal to dout when x equal to 3. but from the implementation[2], we can see d_in equal to dout * (1+0.5), is there any mismatch between the implementation and the defination?</p>
<p>[1] <a href="https://pytorch.org/docs/stable/generated/torch.nn.Hardswish.html?highlight=hardswish#torch.nn.Hardswish" class="inline-onebox" rel="noopener nofollow ugc">Hardswish — PyTorch 1.12 documentation</a><br>
[2] <a href="https://github.com/pytorch/pytorch/pull/34747/files" class="inline-onebox" rel="noopener nofollow ugc">add hardswish FP operator by vkuzo · Pull Request #34747 · pytorch/pytorch · GitHub</a></p>

ptrblck: <p>I assume you are concerned about the derivative at exactly <code>x = 3</code>, which seems to use the derivative of the left part while the forward pass definition claims <code>f(x) = x</code> for <code>x &gt;= 3</code>.<br>
If so, then I think your concern is valid and you could create a GitHub issue so that the code owner could take a look at it. However, in practice you might not see any real difference at the derivative would only be different for exactly this value.</p>

i4oolish: <p>Hi, ptrblck, thanks for your reply, I have create an <a href="https://github.com/pytorch/pytorch/issues/87778" rel="noopener nofollow ugc">issue</a> based on your suggestion, thanks for your help.</p>
PartialConv2d in PyTorch

stan-hua: <p>Is it possible to have PartialConv2d (NVIDIA implementation) as part of the standard torch library?</p>
<p>Link: <a href="https://github.com/NVIDIA/partialconv/blob/master/models/partialconv2d.py" class="inline-onebox" rel="noopener nofollow ugc">partialconv/partialconv2d.py at master · NVIDIA/partialconv · GitHub</a></p>

ptrblck: <p>The <code>PartialConvXd</code> layers seem to be defined directly in Python, so copy/pasting them should work. What would be the benefit to add these layers to PyTorch?</p>