akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cumputing a batch of different gradients

pbsds: <p>The common way to compute multiple different gradients is to use the autograd.grad function multiple times. But it does accept multiple inputs, making me think this means it can compute batches.</p>
<p>Here is some example code:</p>
<pre><code class="lang-python">import torch
from torch.autograd import grad
torch.set_grad_enabled(True)

x = torch.rand(1, 3) # shape (..., 3)
x.requires_grad = True
y = x @ torch.rand(3, 3) # shape (..., 3)
assert y.grad_fn

def attempt_1(x, y):
    y0_dx, = grad([y[..., 0]], [x], [torch.ones_like(y[..., 0])], create_graph=True)
    y1_dx, = grad([y[..., 1]], [x], [torch.ones_like(y[..., 1])], create_graph=True)
    y2_dx, = grad([y[..., 2]], [x], [torch.ones_like(y[..., 2])], create_graph=True)
    return y0_dx, y2_dx, y1_dx


def attempt_2(x, y):
    y0_dx, y1_dx, y2_dx = grad(
        [y[..., 0], y[..., 1], y[..., 2]], 
        [x, x, x], 
        [torch.ones_like(y[..., 0])] * 3, 
        create_graph=True,
    )
    return y0_dx, y2_dx, y1_dx

print(attempt_1(x, y))
print(attempt_2(x, y))
</code></pre>
<p><code>attempt_1</code> works as intended, but is slow.<br>
<code>attempt_2</code> should to my understanding be equivalent, but is incorrect.<br>
Even worse: all its outputs are the same!</p>
<p>How can i batch the multiple <code>grad</code> calls as seen in <code>attempt_1</code> into something like in <code>attempt_2</code>?</p>

pbsds: <p>I just discovered that <code>grad</code> computes the gradient of the <em>sum</em> of the outputs. With that the observed behaviour makes sense.</p>
Cryptic error message from torch.jit.save

kunle12: <p>Hi All,</p>
<p>I am transforming a tts model into torchscript model. I was able to execute <code>torch.jit.script(tts_model)</code> call successfully without any errors. However, when I call <code>torch.jit.save</code> afterwards, I encounter a runtime error:</p>
<pre><code class="lang-auto">    712 See :func:`torch.jit.save &lt;torch.jit.save&gt;` for details.
    713 """
    714 print(f"got here {self.original_name}, {str(f)}, {kwargs}")
--&gt; 715 return self._c.save(str(f), **kwargs)

RuntimeError: Tensors of type UndefinedTensorImpl do not have strides
</code></pre>
<p>I am not sure how to interpret this error message correctly and figuring out where in my code may lead to this error. Any one can give me some clues? The model is big, I haven’t got a minimal example to reproduce this error yet.<br>
hmmm, perhaps my partial quantization of the model may have something to do with this.</p>

ptrblck: <p>Do you have a minimal, executable code snippet which would reproduce the error in the latest release, and could post it here, please?</p>

kunle12: <p>I have found the issue in one of the modules. It turned out that it has a forward method like<br>
<code>def forward(self, x, mask=None)</code><br>
The default value for the mask input triggered the runtime error. Replacing the method signature with<br>
<code>def forward(self, x, mask: Optional[torch.Tensor]=None)</code> resolves the issue.</p>
<p>torch.jit.script didn’t seem to be able to pick out this issue.</p>
Computing loss over ROI

Aonnghus: <p>I am trying to implement a loss that would be expressed as <em>L2(full_image) + weight * L2(ROI)</em>.<br>
I found the following <a href="https://discuss.pytorch.org/t/pixel-wise-loss-to-a-specific-area/28188">topic</a> suggesting I should create a mask over the ROI and zero out the loss.<br>
So I did the following:</p>
<pre><code class="lang-python">criterion = torch.nn.MSELoss(reduction="none")
...
for batch_idx, (X, y) in enumerate(train_loader):
    optimizer.zero_grad()
    X = X.to(device)
    y = y.to(device)
    out = model(X)
    loss = criterion(out, y)
    roi_mask = torch.where(y == 0, 0, 1)
    roi_loss = loss * roi_mask
    mean_roi_loss = torch.sum(roi_loss) / torch.sum(roi_mask)
    final_loss = torch.mean(loss) + weight * mean_roi_loss
    final_loss.backward()
</code></pre>
<p>However, I am facing 2 issues:</p>
<ul>
<li>In some cases, my images do not have any ROI, thus mask is empty resulting in <code>mean_roi_loss</code> being NaN (because <code>torch.sum(roi_mask) == 0</code>). I thought to enclose all this in a <code>if</code> statement checking if the mask is empty, and in that case, <code>final_loss = torch.mean(loss)</code>. But I am not sure whether backward computation would work with this condition, creating different losses depending on the images. I could also replace by <code>mean_roi_loss = torch.sum(roi_loss) / (torch.sum(roi_mask) + 1)</code>.</li>
<li>Should I compute <em>mean(loss) + weight * mean_over_ROI(loss * mask)</em> or <em>mean(loss + weight * mask * loss)</em> i.e. computing the mean over the whole image, not just the ROI?</li>
</ul>
Iterate over two dataloaders

papyrus: <p>Hi all,</p>
<p>How would one iterate over two different dataloaders?</p>
<pre><code class="lang-auto">auto data_set_0 = CustomDataset(...).map(torch::data::transforms::Stack&lt;&gt;());
auto data_set_1 = CustomDataset(...).map(torch::data::transforms::Stack&lt;&gt;());

auto data_loader_0 = torch::data::make_data_loader&lt;torch::data::samplers::SequentialSampler&gt;(
            std::move(data_set_0),
            torch::data::DataLoaderOptions().batch_size(...).workers(...));

auto data_loader_1 = torch::data::make_data_loader&lt;torch::data::samplers::SequentialSampler&gt;(
            std::move(data_set_1),
            torch::data::DataLoaderOptions().batch_size(...).workers(...));

// something like
for (auto&amp; batch_0, auto&amp; batch_1 : zip(*data_loader_0, *data_loader_1)) {

}
</code></pre>
<p>Thanks!</p>

ptrblck: <p>Maybe you could use <code>std::transform</code> as described <a href="https://stackoverflow.com/a/51032349">here</a> but it seems you might need to make sure both iterators have the same length.</p>

papyrus: <p>Thanks for the answer, I managed to get a working example using simple iterators with <code>data_loader_0-&gt;begin()</code>!</p>
Optimizing loss term coefficients

Asgard: <p>For my model, the total loss function is of the form <code>L = c1*L1 + c2*L2 + c3*L3 + ..</code> where <em>c1, c2, c3, …</em> are to be trainable coefficients of a loss function and <em>L1, L2, L3,…</em> are individual loss function terms.</p>
<p>Since these coefficient parameters are outside my model trainable weights, how can I train these coefficients so as to minimize the loss function <em>L</em>?</p>
<p>Thank you in advance.</p>

tom: <p>As you observe the coefficients aren’t part of the training, and for good reason: if your losses are all positive, 0 coefficients would give you zero L, and you could even move them to negative and then you would improve L if the L1 … increase.</p>
<p>This is why, indeed, the coefficients are not and cannot be part of the optimization of L itself and so are what is termed a hyper-parameter (as opposed to normal, trainable parameters), similar to learning rate etc.<br>
Now what is commonly done is to take some validation metric (e.g. accuracy for classification) and then try to find c1, … that lead to good results in that metric. Various approaches exist from manual search to randomly varying them to “learning to learn” approaches to Bayesian optimization schemes.</p>

Asgard: <p>Randomly varying them becomes an extremely difficult task when the coefficients are in the decimal numbers. Isn’t there a way to derive the coefficients so that there is at least one non-zero coefficient for min loss function?</p>

tom: <p>I don’t think there is a much better approach than treating them as hyper-parameters, at least not in this generality. In particular you won’t get around having some other metric of “was this a good pick”. You’re not required to vary them randomly, using intuition also works many times.</p>
<p>When you get to specific losses that have specific interpretations, things may improve (e.g. there might be probabilistic interpretatons or you in very specific cases it may be useful to try to achieve similar orders of magnitude in the loss contributions).</p>
<p>Best regards</p>
<p>Thomas</p>

FA_mn: <p>Is there any way to solve this problem?<br>
I need to define a loss <code>L=c1*L1 + c2*L2</code> where c1 and c2 are trainable coefficients. How can I define them in a parameter mode?</p>
<p>As mentioned in the question " Since these coefficient parameters are outside my model trainable weights, how can I train these coefficients so as to minimize the loss function <em>L</em> ?", is there any way?</p>
<p>Thank you in advance.</p>
Can't import torchvision, but I have it installed?

cq_1: <p>Hello I’m running python on a PC (via WSL2), and I can’t seem to figure out why I can’t import torchvision. When I try I get the following message:</p>
<p>/home/cq/mambaforge/lib/python3.9/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: libc10_cuda.so: cannot open shared object file: No such file or directory<br>
warn(f"Failed to load image Python extension: {e}")</p>
<p>I’m running<br>
Python 3.9.13<br>
pytorch : 1.12.1<br>
torchvision: 0.13.1</p>
<p>Any guidance you can give me would be appreciated.</p>

ptrblck: <p>The warning is raised from <a href="https://github.com/pytorch/vision/blob/a67cc87a33a3f713aebf5299bdeb2672c98e0bc5/torchvision/io/image.py#L10-L13">here</a> when <code>torchvision</code> tries to import the <code>"image"</code> library but fails. Most likely you won’t be able to use the serialization methods from <code>torchvision.io</code> but should be able to use other methods.</p>
Why modify the model parameter gives me different result?

torch_torch: <p>I’m trying to write some code that need to modify the model parameter.<br>
I understand that model.state_dict() returns a dictonary, if I modify that dictionary, the corrsponding model parameter also changes. But after I rewrite the code in a modularization way, such link disappearded.<br>
To show my problems more clearly, I write following example:</p>
<pre><code class="lang-auto">import torch
from torchvision.models import resnet18

class myclass(object):
    def __init__(self, model):
        self.model = model
        self.parameter = self.model.state_dict()
        
        
model = resnet18()
parameter = model.state_dict()
print(parameter['fc.weight']) # all of them is not 0
parameter['fc.weight'] = torch.zeros_like(parameter['fc.weight'])
parameter_new = model.state_dict()
print(parameter['fc.weight']) # all of them is 0

model2 = resnet18()
my_object = myclass(model2)
print(my_object.parameter['fc.weight']) # all of them is not 0
my_object.parameter['fc.weight'] = torch.zeros_like(my_object.parameter['fc.weight'])
my_object_parameter_new = my_object.model.state_dict()
print(my_object_parameter_new['fc.weight']) # all of them is not 0

</code></pre>
<p>The problem is why my_object_parameter_new[‘fc.weight’] is not zero after I manually set my_object.parameter[‘fc.weight’] to 0?</p>
<p>Thanks ! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
A question about high order derivative

ydc: <p>I want to use pytorch to calculate the following derivatives:∂((x+0.1*∂(xy)/∂x)^2)/∂y。It should be 0.1(x+0.1y) since ∂(xy)/∂y=y and ∂(x+0.1y)^2/∂y=(x+0.1y)*0.1<br>
I try to solve the above problem with the following code</p>
<blockquote>
<p>import torch<br>
torch.manual_seed(11037)<br>
x1 = torch.rand(1).requires_grad_(True)<br>
y1 = torch.rand(1).requires_grad_(True)<br>
print(x1)<br>
print(y1)<br>
loss1 = x1 * y1<br>
grad1 = torch.autograd.grad(loss1, x1, retain_graph=True)<br>
x2 = x1 + grad1[0] * 0.1<br>
loss2 = x2 * x2<br>
loss2.backward()<br>
print(y1.grad)<br>
print(x2 * 0.1)</p>
</blockquote>
<p>However, the results of y1.grad is None. I can somehow understand why this is the case, because y1 does not appear explicitly in the expression for loss2. If I change loss2=x2 * x2 to loss2=x2 * y1, the result will be true.<br>
So what should I do? I encounter this problem when implement this paper “Training Meta-Surrogate Model for Transferable Adversarial Attack”. The actual problem I’m trying to solve is to generate an adversarial sample based on model A, feed it to model B to get the loss, and pass the gradient back to model A.</p>

KFrank: <p>Hi Dingcheng!</p>
<aside class="quote no-group quote-modified" data-username="ydc" data-post="1" data-topic="161153" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/85e7bf/48.png" class="avatar"> ydc:</div>
<blockquote>
<blockquote>
<p>grad1 = torch.autograd.grad(loss1, x1, retain_graph=True)</p>
</blockquote>
</blockquote>
</aside>
<p>This line should be:</p>
<pre data-code-wrap="python"><code class="lang-python">grad1 = torch.autograd.grad (loss1, x1, create_graph = True)
</code></pre>
<p>Note the <code>create_graph</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

ydc: <p>Thanks!!! It works!!!</p>
Special treatment of "0/0"

mmoelle1: <p>I am implementing an algorithm that can lead to <code>0/0</code> by design with the convention that <code>0/0 -&gt; 1</code>. The relevant part of the algorithm is as follows</p>
<pre><code class="lang-auto">Tensor a = [a1, a2, ..., an]
Tensor b = [b1, b2, ..., bn]
Tensor c = (x - a) / (b - a)
</code></pre>
<p>Here, <code>/</code> denotes element wise division and <code>x</code> is a scalar. I can ensure that entries in b-a are equal to zero if and only if the relevant entries in x-a are zero as well. I also need to extend the above algorithm to <code>a</code>, <code>b</code>, and <code>c</code> being matrices and <code>x</code> being a vector of appropriate size.</p>
<p>What is the most efficient way to make the element wise division operator treat <code>0/0 -&gt; 1</code>?</p>

shivammehta007: <p>How about using <code>torch.where</code>?</p>
<pre><code class="lang-python">torch.where(a == b , 1 , (x - a) / (b - a) )
</code></pre>

J_Johnson: <p>That also will depend on how you want to treat x/0. Assuming you want both to be =1, here is what I would do:</p>
<pre><code class="lang-auto">x=6
a=torch.cat([torch.tensor([1, 2, 3, 4])]*5)
b=torch.cat([torch.tensor([1, 2, 3, 4,5])]*4)


c =torch.where(b-a==0, 1, (x - a) / (b - a))
</code></pre>

KFrank: <p>Hi Shivam (and Matthias)!</p>
<aside class="quote no-group" data-username="shivammehta007" data-post="2" data-topic="161107" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shivammehta007/48/31230_2.png" class="avatar"> shivammehta007:</div>
<blockquote>
<p>How about using <code>torch.where</code>?</p>
<pre data-code-wrap="python"><code class="lang-python">torch.where(a == b , 1 , (x - a) / (b - a) )
</code></pre>
</blockquote>
</aside>
<p>Two issues to be aware of:</p>
<p>First, if <code>a</code> and <code>b</code> are floating-point tensors (which I am guessing they<br>
might be), you have the issue that <code>a == b</code> is an <em>exact</em> floating-point<br>
comparison that won’t work (because of round-off error) for typical<br>
use cases.  You would probably need to substitute something like<br>
<code>close_enough_to_be_considered_equal_for_my_use_case (a, b)</code><br>
for the exact equality test.</p>
<p>Second, <code>torch.where()</code> doesn’t backpropagate nicely when <code>nan</code>s and<br>
<code>inf</code>s occur, even in the “branch not followed.”  (See for example this<br>
<a href="https://github.com/pytorch/pytorch/issues/23395" rel="noopener nofollow ugc">torch.where() github issue</a>.)  In instances where <code>a</code> and <code>b</code> are equal, you<br>
will get <code>nan</code>s or <code>inf</code>s.</p>
<p>You could do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">mask = (almost_equal (x, b)).float()   # 1.0 if b is "equal" to x, otherwise 0.0
a = a - mask
c = (x - a) / (b - a)   # 0/0 has now become 1.0 / 1.0
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Matthias_Moller: <p>This solution works perfectly. Thanks.</p>
CYCLICLR Plotting range of Learning Rate for Accuracy

KanZa: <p>I want to plot a graph for accuracy for  base_lr= 0.01 and max_lr=0.1, similar given in attached photo</p>
<pre><code class="lang-auto">accuracy = []
for cycle in range(CYCLES):            
            random.shuffle(unlabeled_set)
            subset = unlabeled_set[:SUBSET]
            # Model - create new instance for every cycle so that it resets
            with torch.cuda.device(CUDA_VISIBLE_DEVICES):            
                resnet18    = resnet.ResNet18(num_classes=NO_CLASSES).cuda()
            models = resnet18
            
            torch.backends.cudnn.benchmark = True
            models = torch.nn.DataParallel(models, device_ids=[0])
            
            # Loss, criterion and scheduler (re)initialization
            criterion      = nn.CrossEntropyLoss(reduction='none')
            optim_backbone = optim.SGD(models.parameters(), lr=LR, 
                momentum=MOMENTUM, weight_decay=WDECAY)
            schedulers = torch.optim.lr_scheduler.CyclicLR(optim_backbone, base_lr= 0.01, max_lr=0.1, step_size_up=2000, step_size_down=None, mode='triangular')
optimizers =  optim_backbone
            #schedulers = sched_backbone
            
            train(models, criterion, optimizers, schedulers, dataloaders, Epochs)
            acc = test(models, dataloaders, mode='test')
            print('Trial {}/{} || Cycle {}/{} || Label set size {}: Test acc {}'.format(trial+1, TRIALS, cycle+1, CYCLES, len(labeled_set), acc))
            #np.array([method, trial+1, TRIALS, cycle+1, CYCLES, len(labeled_set), acc]).tofile(results, sep=" ")
            
            accuracy.append(acc)
            #iterations.append(cycle)
            
            if cycle == (CYCLES-1):
                # Reached final training cycle
                print("Finished.")
                break
            # Get the indices of the unlabeled samples to train on next cycle
            arg = query_samples(models, data_unlabeled, subset, labeled_set, cycle)

            # Update the labeled dataset and the unlabeled dataset, respectively
            labeled_set += list(torch.tensor(subset)[arg][-ADDENDUM:].numpy())
            listd = list(torch.tensor(subset)[arg][:-ADDENDUM].numpy())
            unlabeled_set = listd + unlabeled_set[SUBSET:]
            # Create a new dataloader for the updated labeled dataset
            dataloaders['train'] = DataLoader(data_train, batch_size=BATCH, 
                                            sampler=SubsetRandomSampler(labeled_set), 
                                            pin_memory=True)
     
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d32f1f6af83bc6bfa365722c85af601a72e509df.png" alt="Screenshot from 2022-09-11 20-30-36" data-base62-sha1="u8dHMMEERSYi86MRau3vAeHIOt1" width="528" height="459"></p>
Link gradient of tensor to another before .backward()

sj.hwang: <p>Is there a way to link gradient of a tensor to another one?</p>
<pre><code class="lang-auto">import torch

fc = torch.nn.Linear(256, 256)
A = torch.nn.Embedding(10, 256)

a = A.weight
for _ in range(5):
    a = fc(a)

b = a.clone().detach()
b.requires_grad = True
# [1] is there a way to link gradient of b to A here:
# something like, A.grad &lt;- b.grad?

B = fc(b)
loss_B = B.sum()
loss_B.backward()

# [2] copying b's grad to A
A.grad = b.grad.clone()
</code></pre>
<p>I am hoping to use the gradient of <code>b</code> to update <code>A</code>. So what I can now think of is <code>[2]</code>, which copies the gradient of b to A after calculating the grad using the <code>backward</code> function.</p>
<p>But is there a way that I can link the grad of <code>b</code> to <code>A</code> before <code>backward</code>, something like in <code>[1]</code>?</p>
What should the input to DeepLabV3 be in training mode?

David1: <p>I am trying to train a <code>deeplabv3_resnet50</code> model on a custom dataset, but get the error <code>ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])</code> when trying to do the forward pass. The following minimal example produces this error:</p>
<pre><code class="lang-auto">import torch
import torchvision

model = torchvision.models.segmentation.deeplabv3_resnet50(weights="DEFAULT")
model.train()

batch_size = 1
nbr_of_channels = 3
img_height, img_width = (500, 500)
input = torch.rand((batch_size, nbr_of_channels, img_height, img_width))
model(input)
</code></pre>
<p>I do not understand this at all. What on earth is meant by <code>got input size torch.Size([1, 256, 1, 1])</code>?</p>

ptrblck: <p>The stacktrace points to a batchnorm layer, which gets an activation in the shape of <code>[1, 256, 1, 1]</code> and tries to normalize the input by calculating its <code>mean</code> and <code>std</code>. Since you are passing a single sample and the input activation to this batchnorm layer has a single pixel, the stats calculation will fail which this error message indicates.<br>
Increase the batch size and it will work.</p>
ImportError for modules of the project

papoo13: <p>I am using the code provided in this repo: <a href="https://github.com/igolan/bgd" class="inline-onebox" rel="noopener nofollow ugc">GitHub - igolan/bgd: Implementation of Bayesian Gradient Descent</a><br>
I have cloned the project and created a conda environment by pip install -r requirements.txt<br>
I run one the commands provided in the run_exps.sh but I get the following error:<br>
<strong>ImportError: No module named utils.utils</strong><br>
I am running the project in VSCode and I have no idea why I’m getting this error, so any help and hints is greatly appreciated. considering Python’s importing rules I have no idea why this is not working.</p>

ptrblck: <p>The repository seems to be abandoned so I’m unsure how much support you would get from the authors. In any case, did you check your Python environment and made sure the package is installed?<br>
Alternatively, try to build it locally (in case you need to actually build it) or just <code>cd</code> into their <code>dir</code> and try to execute the scripts locally.</p>
Torch.heaviside disables gradient

yegane: <p>I am trying to backpropagate through a torch.heaviside function, but it seems that it disables the gradient for its output. How can I have it in a backpropagation?</p>
<p>If it is is non-differentiable, is there any function that produces 1,0 only?</p>

KFrank: <p>Hi Yegane!</p>
<aside class="quote no-group" data-username="yegane" data-post="1" data-topic="136235" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yegane/48/10618_2.png" class="avatar"> yegane:</div>
<blockquote>
<p>How can I have it in a backpropagation?</p>
</blockquote>
</aside>
<p>You can’t use the actual Heaviside function in backpropagation.<br>
Use instead a “soft,” usefully-differentiable approximation to it<br>
such as <code>torch.sigmoid()</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>If it is is non-differentiable, is there any function that produces 1,0 only?</p>
</blockquote>
</aside>
<p>There is no such function that would be useful for backpropagation.<br>
Such a function would have zero derivative in regions where it was<br>
either zero or one and would have undefined derivative at those<br>
points where it changed from zero to one.  In neither case would<br>
you get any non-trivial backpropagation.</p>
<p>Best.</p>
<p>K. Frank</p>

abarbadan: <p>I’ve tried to use the heaviside function to generate piecewise differentiable functions. The gradient of the functions being stitched is what I want, and as such, having a constant gradient for heaviside be zero is useful. If the jump discontinuity has a value of 0.5 for example, this will set the gradient at the stitch point to be the average of the gradients of the left and right stitched functions. I’ll be using sigmoids for the time being, but I’d like to add to this discussion in the hopes that adding this functionality to heaviside will be reconsidered and implemented in the future.</p>
Insert some elements to tensor after some elements

jianyuedefeng: <p>How could I do like this in an efficitent way in PyTorch?<br>
Input X and Y:<br>
X = [1,2,3,4,5,6,7,8, 9, 10, 11, 12]<br>
Y = [77, 88, 99]<br>
I want to get a vector like this:<br>
Z = [1,2,3,4, 77, 5, 6,7,8, 88, 9, 10, 11, 12, 99]</p>
<p>I want to insert a elelemt  from Y into X after 4 elements of X, how to do it in an efficient way?</p>
<p>Thanks in advance.</p>
Mean Value Along Dim w Mask

J_Johnson: <p>Suppose we have a matrix as follows:</p>
<pre><code class="lang-auto">A=torch.arange(100).view(20,5)
</code></pre>
<p>And suppose we also have a mask to apply, such as:</p>
<pre><code class="lang-auto">B=torch.rand(100).view(20, 5)

mask=B&gt;0.1
</code></pre>
<p>Now suppose we also wish to get the mean along Dim=1 but with the mask applied:</p>
<p><code>print(torch.mean(A[mask], dim=1))</code></p>
<p>But this gives an error, because applying this type of mask causes the matrix to become a vector.</p>
<p>What is the best way to apply a mask and get dim-wise mean values?</p>

srishti-git1110: <p>Use this -</p>
<pre><code class="lang-auto">D = torch.where(mask, A, 0).type(torch.float32)
torch.mean(D, dim=1)
</code></pre>

KFrank: <p>Hi Srishti (and J)!</p>
<aside class="quote no-group" data-username="srishti-git1110" data-post="2" data-topic="161114" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/srishti-git1110/48/52515_2.png" class="avatar"> srishti-git1110:</div>
<blockquote>
<pre><code class="lang-auto">D = torch.where(mask, A, 0).type(torch.float32)
torch.mean(D, dim=1)
</code></pre>
</blockquote>
</aside>
<p>This replaces masked elements with 0.0 that then get mixed in with the<br>
<code>mean()</code> computation, diluting it.  Based on the pseudocode J posted, I don’t<br>
think this is what he wants.</p>
<p>Computing the mean in terms of a masked sum and a masked count gives<br>
what I think is the desired result:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.12.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; A = torch.arange (100).view (20, 5)
&gt;&gt;&gt; B = torch.rand (100).view (20, 5)
&gt;&gt;&gt; mask = B &gt; 0.1
&gt;&gt;&gt;
&gt;&gt;&gt; masked_mean = (mask * A).sum (dim = 1) / mask.sum (dim = 1)
&gt;&gt;&gt;
&gt;&gt;&gt; masked_mean
tensor([ 1.5000,  7.0000, 12.0000, 17.0000, 22.0000, 27.2500, 32.0000, 37.0000,
        42.0000, 47.0000, 52.5000, 57.0000, 62.0000, 67.0000, 71.7500, 76.7500,
        82.0000, 87.0000, 92.0000, 97.0000])
</code></pre>
<p>Note, if an entire row is masked out, the resulting mean will be <code>nan</code> which<br>
is as good an exceptional value as any.</p>
<p>Best.</p>
<p>K. Frank</p>

J_Johnson: <p>This is exactly what I was looking for. The actual implementation will not have any empty rows, so NaNs won’t be an issue. Thank you.</p>
Size mismatch (weights and biases)

Ascrypto: <p>Hi,</p>
<p>I have two linear-layer networks (one is a pre-trained model). The only differences between the two networks are the input and output sizes.  When I call: new_model.load_state_dict(torch.load(“model.pth”)), I get the size mismatch issues.</p>
<p>Solutions I have tried so far but none of them worked:</p>
<ol>
<li>strict=False when loading the state.</li>
<li>manually changing the parameters after loading the state dict.</li>
</ol>
<p>I am not sure how it can be solved.</p>
<p>Also, what are the consequences (in terms of predictive validity) of testing using different neural network architectures (as the one described above)?</p>
<p>Thank you.</p>

Ascrypto: <p>In case someone is looking for a solution:<br>
I loaded the model: chk=torch.load(“pathtofile”)<br>
chk.pop(‘head.weight’)<br>
chk.pop(‘head.bias’)<br>
chk.pop(‘hidden_layer1.weight’)<br>
and then load_state_dict(chk)</p>
<p>This solved the issue above, but I am not sure if it is a technically valid method since as far as I know higher layers’ weights and biases should not be discarded.</p>

ptrblck: <p>Your approach just skips loading the parameters which re causing a shape mismatch and is theoretically valid. Of course if depends on your use case as you are now using these parameters with their random init values and would need to retrain them. However, since you’ve changed the number of in and output features you won’t be able to directly load the pretrained parameters unless you can come up with a way to “use” the pretrained parameters and reshape them somehow (e.g. repeating the values etc.).</p>

Ascrypto: <p>Thank you for your input. I just want to clarify. So you suggest something similar to random projection (i.e., reducing the dimensions without losing the underlying characteristics), or is there a simpler approach? Could you please elaborate on your last point (by examples if possible)?</p>

ptrblck: <p>I’m not suggesting a specific approach as it depends on your actual use case and why the parameter shapes were changed.<br>
In a common fine-tuning approach you would e.g. just replace the classifier from the pre-trained model and fine-tune it afterwards, which doesn’t seem to fit your use case.</p>
Pretrained Googlenet

Samriddha_Majumdar: <p>How to remove the inception5b layer of the pretrained googlenet model ? It will be highly appreciated if anyone helps as soon as possible .</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57cee0f1fb1090570c0d6e905a333b792d5cd26a.png" data-download-href="https://discuss.pytorch.org/uploads/default/57cee0f1fb1090570c0d6e905a333b792d5cd26a" title="Screenshot 2022-09-10 at 3.39.48 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57cee0f1fb1090570c0d6e905a333b792d5cd26a.png" alt="Screenshot 2022-09-10 at 3.39.48 PM" data-base62-sha1="cwMM4XmtuhM9Hn6PbHQsHpaAAIi" width="690" height="408" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-09-10 at 3.39.48 PM</span><span class="informations">982×582 43 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You could try to replace it with an <code>nn.Identity</code> layer directly. Depending on the layer and in particular if the layer changes the size of the output activation compared to its input, this approach might not work since <code>nn.Identity</code> will just return the input directly without any manipulations.<br>
In this case you would need to change more of the model architecture to account for the shape change in the activation.</p>
Theoretical: Multimodal training with missing data

CarlosHernandezP: <p>Hello everyone:</p>
<p>I have a medical dataset with three different modalities:</p>
<ul>
<li>Images</li>
<li>Tabular data</li>
<li>Genetic data (needs to be handled separately from tabular)</li>
</ul>
<p>As much I would like to change this, I have a lot of missing data. The % patients with all the data is very small. For example, I only have images for about 10% of the patients.</p>
<p>On a normal multimodal approach, I would have 3 different “heads” of the network that would ingest each modality and somewhere in the middle I would concatenate it (or any other strategy) to send it again through a series of FC layers until the output layer.</p>
<p>If I were to have no missing data, I would just do a forward pass, compute the error and backpropagate it to update all the model’s weights. With so many missing values, if I were to send, for example, an image containing all black whenever I don’t have that modality, I would be biasing my model.</p>
<p>I came up with the idea of having different optimizers for the different “heads” of the network and maybe another that would update all the weights of the shared architecture. I would switch on/off the heads if I don’t have a specific modality and after the forward pass I would call <code>optimizer_n.step()</code> for all the modalities that I did have data. Thus updating only the weights of the nuerons that were used.</p>
<p>Is this a reasonable strategy? I have not got much experience working much with multiple optimizers nor multiple modalities so any advice is greatly welcomed.</p>

jasperhyp: <p>Missing modality in training is <strong>NOT</strong> a trivial problem. It is a cutting-edge research topic.  You might want to check out the recent multimodal VAE or meta-learning papers (e.g. Wu &amp; Goodman 2018; Ma et al. 2021; Joy et al. 2022) for possible directions.</p>

CarlosHernandezP: <p>Thank you very much for your answer. I was able to easily find the paper of <a href="https://proceedings.neurips.cc/paper/2018/file/1102a326d5f7c9e04fc3c89d0ede88c9-Paper.pdf" rel="noopener nofollow ugc">Wu &amp; Goodman 2018</a> and <a href="https://ojs.aaai.org/index.php/AAAI/article/view/16330" rel="noopener nofollow ugc">Ma et al. 2021</a>. But I am having some trouble finding the last one. Would you mind sharing the paper title?</p>

jasperhyp: <p>It’s named “LEARNING MULTIMODAL VAES THROUGH MUTUAL SUPERVISION”.  If you’re working with biomedical data I do believe it would be a lot noisier than the image-text or image-label or image-image data people usually benchmark with.</p>
What is the purpose of eps in the REINFORCE example?

jkking: <p>PyTorch’s github provides an example implementation of the REINFORCE algorithm. In calculating standardizing the rewards, it adds a term <code>eps</code>:</p>
<pre><code class="lang-auto">returns = (returns - returns.mean()) / (returns.std() + eps)
</code></pre>
<p>What is the purpose of this term? It can be found in the example here:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py#L50">
  <header class="source">

      <a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py#L50" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py#L50" target="_blank" rel="noopener nofollow ugc">pytorch/examples/blob/main/reinforcement_learning/reinforce.py#L50</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="40" style="counter-reset: li-counter 39 ;">
          <li>    def forward(self, x):</li>
          <li>        x = self.affine1(x)</li>
          <li>        x = self.dropout(x)</li>
          <li>        x = F.relu(x)</li>
          <li>        action_scores = self.affine2(x)</li>
          <li>        return F.softmax(action_scores, dim=1)</li>
          <li>
          </li>
<li>
          </li>
<li>policy = Policy()</li>
          <li>optimizer = optim.Adam(policy.parameters(), lr=1e-2)</li>
          <li class="selected">eps = np.finfo(np.float32).eps.item()</li>
          <li>
          </li>
<li>
          </li>
<li>def select_action(state):</li>
          <li>    state = torch.from_numpy(state).float().unsqueeze(0)</li>
          <li>    probs = policy(state)</li>
          <li>    m = Categorical(probs)</li>
          <li>    action = m.sample()</li>
          <li>    policy.saved_log_probs.append(m.log_prob(action))</li>
          <li>    return action.item()</li>
          <li>
      </li>
</ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>A small <code>eps</code> value is usually added to a division to avoid dividing by zero which would create invalid outputs and invalid gradients. Often it’s picked to be e.g. <code>eps = 1e-6</code>.</p>

jkking: <p>Is there any intuition to the very explicit definition given in the example? Setting it to something very small but nonzero makes sense, but that explicit a call (starting with <code>np.finfo</code>) seems prescriptive?</p>

vmoens: <p>The call to np.finfo is to link the value of eps to the dtype being used.<br>
If you’re using double precision or single precision, the numerical instability will be different for a given value of the denominator when it gets closer to 0.<br>
By using finfo you’re making sure that your eps is as big as it should be given the data type you’re using.</p>
How to fix gathering dim 0 warning in multi-gpu (DataParallel) setting?

AlexisW: <p>I am continuously facing this warning but have no idea what might be the issue, as it did not return which line of code triggers this. Does anyone have thoughts on what the common issue(s) could be?</p>
<pre><code class="lang-auto">python3.6/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
</code></pre>

alex.veuthey: <p>I just had the error this week myself, the cause is that you have a function/method (probably the <code>forward</code> method of one or your modules) that returns a scalar, like an accuracy or loss function for example.</p>
<p>See <a href="https://github.com/pytorch/pytorch/issues/9811" rel="nofollow noopener">this issue</a> for more details. This is just a warning, but it could mean that your code doesn’t work properly.</p>
<p>For example, my use case was a learnable loss which was shared across GPUs with <code>DataParallel</code>, but returned only a scalar in the <code>forward</code> function, which was then stacked in a vector and caused an error further on. The solution to this specific problem was to call <code>loss = loss.mean()</code>, which is what should happen when reassembling the values together anyway.</p>

AlexisW: <p>Thanks for sharing the insights. I did use <code>loss = loss.mean()</code> before calling <code>backward</code>. Do you know whether there is an easy way to check which line exactly causes the problem?</p>

alex.veuthey: <p>I don’t know if you can check the internals of <code>DataParallel</code>, but in my case I simply printed the loss value inside the <code>forward</code> method, which was a scalar, and the loss value outside the method (thus inside the training script), which was a vector.</p>
<p>Actually, just re-read the issue on github, and I realized that calling <code>.mean()</code> might be a weak solution, in case you have batch sizes that are not equally divided across the GPUs. The proper solution seems to be summing all outputs for the loss, and then dividing with the total batch size (after scaling the individual losses by their local batch sizes as well).</p>

Eric_Hedlin: <p>I found that doesn’t suppress the warning. Before returning the value from within <code>DataParallel</code> I unsqueezed the scalar along dim 0 and that did the trick.</p>
How to initialize nn.Parameter in an applied function?

Vedant_Roy: <p>I have the following function I’m using to initialize weights for testing purposes. (I need a deterministic function that is something non-trivial. I.e, not all 0s or 1s).</p>
<pre><code class="lang-auto">def init_weights(m):
    # Avoid random initializations
    if isinstance(m, nn.Linear) or isinstance(m, nn.Conv3d) or isinstance(m, nn.LayerNorm):
        print("initializing other layers")
	# Choose this b/c it's deterministic
        normal_(m.weight)
    elif isinstance(m, nn.Parameter):
        print("initializing parameters")
        normal_(m)
    else:
        raise Exception("unknown layer")
</code></pre>
<p>The problem is, this function does not seem to initialize parameters (i.e., <code>nn.Parameter</code>). I can tell b/c <code>initializing parameters</code> is never printed. Is there a good solution here?</p>

Vedant_Roy: <p>I suppose the correct follow-up question is: What function can I use to deterministically initialize weights non-trivially?</p>

Vedant_Roy: <p>My answer to the 2nd question (also indirectly solves my 1st question):</p>
<pre><code class="lang-auto">def deterministic_fill_(p):
    gen = torch.Generator()
    gen.manual_seed(42)

    shape = p.shape
    weights = torch.rand(shape, generator=gen)
    p.data = weights

def deterministic_init(m):
    params = 0
    for n, p in m.named_parameters():
        deterministic_fill_(p)
        params += 1
    print(f"{params} params deterministically initialized")
</code></pre>
Slow NNC cuda code for tensor expressions

namednil: <p>I’ve been playing a bit with tensor expressions and noticed that it can be very slow on cuda (I used Pytorch 1.12 with CUDA 11.3). For example, here I reimplemented some basic functionality of <code>torch.gather</code> and benchmarked it.</p>
<pre><code class="lang-auto">import torch
import torch.utils.benchmark as benchmark
import torch._C._te as te

def construct_gather(n: int, backend, dtype=torch.float32):
    A = te.BufHandle("A", [n, n], dtype)
    INDEX = te.BufHandle("INDEX", [n, n], torch.long)
    B = te.BufHandle("B", [n, n], dtype)
    i = te.VarHandle("i", torch.int)
    j = te.VarHandle("j", torch.int)

    store = te.Store.make(A, [i, j], B.load([INDEX.load([j + i*n]), j])) # flatten index manually for now
    for_j = te.For.make(j, te.ExprHandle.int(0), te.ExprHandle.int(n), store)
    for_i = te.For.make(i, te.ExprHandle.int(0), te.ExprHandle.int(n), for_j)

    loopnest = te.LoopNest(te.Block([for_i]), [A, B, INDEX])

    loopnest.prepare_for_codegen()
    stmt = te.simplify(loopnest.root_stmt())

    return te.construct_codegen(backend, stmt, [A, B, INDEX])

if __name__ == "__main__":
    n = 2000
    torch.manual_seed(42)
    d = torch.randn((n, n))
    index = (torch.rand((n, n)) * (n-2)).long()
    to = torch.zeros((n, n))

    print("Pytorch CPU")
    t0 = benchmark.Timer("torch.gather(d, 0, index, out=to)", globals={"d": d, "index": index, "to": to})
    print(t0.timeit(2))

    print("NNC/LLVM CPU")
    gather_llvm = construct_gather(n, "llvm")
    t0 = benchmark.Timer("gather.call([to, d, index])", globals={"d": d, "index": index, "to": to, "gather": gather_llvm})
    print(t0.timeit(2))

    d = d.to(0)
    index = index.to(0)
    to = to.to(0)

    print("Pytorch GPU")
    t1 = benchmark.Timer("torch.gather(d, 0, index, out=to)", globals={"d": d, "index": index, "to": to})
    print(t1.timeit(2))

    print("NNC/CUDA GPU")
    gather_cuda = construct_gather(n, "cuda")
    t0 = benchmark.Timer("gather.call([to, d, index])", globals={"d": d, "index": index, "to": to, "gather":gather_cuda})
    print(t0.timeit(2))
</code></pre>
<p>And I get an output like this:</p>
<pre><code class="lang-auto">Pytorch CPU
torch.gather(d, 0, index, out=to)
  50.85 ms
  1 measurement, 2 runs , 1 thread
  
NNC/LLVM CPU
gather.call([to, d, index])
  21.75 ms
  1 measurement, 2 runs , 1 thread
  
Pytorch GPU
torch.gather(d, 0, index, out=to)
  10.67 ms
  1 measurement, 2 runs , 1 thread
  
NNC/CUDA GPU
gather.call([to, d, index])
  444.69 ms
  1 measurement, 2 runs , 1 thread
</code></pre>
<p>Now, I wouldn’t expect my naive reimplementation to be faster than <code>torch.gather</code> but maybe not an order of magnitude off? I’ve played a bit with <code>loopnest.tile</code> to optimise it but nothing seemed to have a really noticable effect. Any ideas what I might be missing?</p>
Get features in vision::models::ResNet101 - help

Jesse_Stone: <p>python:<br>
resnet101 = torchvision.models.resnet101(pretrained=self._pretrained)<br>
children = list(resnet101.children())<br>
features = children[:-3]<br>
hidden = children[-3]<br>
features = nn.Sequential(*features)</p>
<p>c++:<br>
vision::models::ResNet101 model = vision::models::ResNet101();<br>
torch::OrderedDict&lt;std::string, std::shared_ptr&lt;<a>torch::nn::Module</a>&gt;&gt; children = model-&gt;named_children();<br>
std::shared_ptr&lt;<a>torch::nn::Module</a>&gt;  hidden_layer;<br>
nn::Sequential features;</p>
<pre><code>for (auto&amp; child : children)
{
    std::cout &lt;&lt; child.key() &lt;&lt; std::endl;
    if (child.key().find("layer4") != std::string::npos)
    {
        hidden_layer = child.value();
    }
}

both error: ‘forward’ is not a member of ‘std::remove_reference&lt;torch::nn::Module&gt;::type {aka torch::nn::Module}’
/*features-&gt;push_back( std::make_shared&lt;torch::nn::Module&gt;(children[0].value()) );
features-&gt;push_back( children[0].value() );*/
</code></pre>
<p>How to get features as python in libtorch c++ ?</p>

Jesse_Stone: <p>Done~</p>
<pre><code>https://fatalfeel.blogspot.com/2013/12/emnist-handwritten-dataset-on-libtorch-c.html
find -&gt; torch_resnet_cpp.tar.gz
</code></pre>
<p>can get every node and layer</p>
DistributedDataParallel and convert_sync_batchnorm

zzzf: <p>I find most people first convert <code>BatchNorm</code> to <code>SyncBatchNorm</code> and then wrap the <code>model</code> with <code>DistributedDataParallel</code>:</p>
<pre><code class="lang-auto">model = nn.SyncBatchNorm.convert_sync_batchnorm(model)
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank])        
</code></pre>
<p>If I reverse the order like the following, would I get the same results?</p>
<pre><code class="lang-auto">model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank]) 
model = nn.SyncBatchNorm.convert_sync_batchnorm(model)
</code></pre>

ptrblck: <p>I would generally stick to the docs and their usage, but based on <a href="https://github.com/pytorch/pytorch/blob/b136f3f310aa01a8b3c1e63dc0bfda8fd2234b06/torch/nn/modules/batchnorm.py#L820-L823">this code</a> I guess you might see the same result assuming the recursive <code>named_children()</code> call also sees the internal <code>.module</code> attribute.</p>
Some weird issue with operations inside "with no_grad"

fahd09: <p>Dear all.</p>
<p>I found this issue today and I am not sure if it is something expected or not. I am trying to code a linear regression with PyTorch.</p>
<pre><code class="lang-python">for epoch in range(n_epochs):
  # compute prediction, compute loss, loss.backward()
  with torch.no_grad():

    # option 1 (runtime error):
    # a = a - lr* a.grad
    # b = b - lr* b.grad 

    # option 2
    a -= lr* a.grad
    b -= lr* b.grad 
</code></pre>
<p>Both options are syntactically identical but the first one returns a runtime error<br>
<code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code></p>
<p><code> </code><br>
Can someone point me to the reason why PyTorch treats the shortened subtraction differently from the expanded version?</p>

srishti-git1110: <p>When you use the expanded version, that is -</p>
<pre><code class="lang-auto">a = a - lr* a.grad
b = b - lr* b.grad 
</code></pre>
<p>new tensors <code>a</code> and <code>b</code> get created each time.</p>
<p>and since these are getting created under <code>torch.no_grad()</code>, their <code>requires_grad</code> attribute will be False - this explains the error.</p>
<p>With this code -</p>
<pre><code class="lang-auto">a -= lr* a.grad
b -= lr* b.grad
</code></pre>
<p>tensors a and b get modified in place i.e. no new tensors are created, only the values of existing tensors are modified in place.</p>