akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Filtering out null rows of a 3-D Tensor

JulesBelveze: <p>Hi there,</p>
<p>I am having some problem trying to convert the following function into a function only manipulating tensors.</p>
<pre><code class="lang-auto">def valid_sequence_output(sequence_output, valid_mask, attention_mask):
    batch_size, max_len, feat_dim = sequence_output.shape
    valid_output = torch.zeros(batch_size, max_len, feat_dim, dtype=torch.float32,
                               device='cuda' if torch.cuda.is_available() else 'cpu')
    valid_attention_mask = torch.zeros(batch_size, max_len, dtype=torch.long,
                                       device='cuda' if torch.cuda.is_available() else 'cpu')
    for i in range(batch_size):
        jj = -1
        for j in range(max_len):
            if valid_mask[i][j].item() == 1:
                jj += 1
                valid_output[i][jj] = sequence_output[i][j]
                valid_attention_mask[i][jj] = attention_mask[i][j]
    return valid_output, valid_attention_mask
</code></pre>
<p>where the input tensors can be created as follow:</p>
<pre><code class="lang-auto">size = ((2,5,2))
sequence_output = torch.randint(0, 250, size=size)
valid_mask = torch.randint(0, 2, size=size[:2])
attention_mask = torch.randint(0, 2, size=size[:2])
</code></pre>
<p>I basically aim at “bubbling up” the non null rows of <code>sequence_output</code> and <code>attention_mask</code> to the top. The closest I could get was by doing something like:</p>
<pre><code class="lang-auto">torch.where(valid_mask.unsqueeze(-1) == 1, sequence_output, torch.zeros_like(sequence_output))
</code></pre>
<p>but then I couldn’t find a way to “bubble up” the non-null rows.</p>
<p>If somebody has a suggestion on how to do this, I would really appreciate <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>Cheers,<br>
Jules</p>
Loading custom audio dataset

Superklez: <p>When creating a custom dataset loader like that shown <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">here</a>. Is it advisable to do something like</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
    def __init__(self, csv_file, root_dir):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        audio_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
        target_path = os.path.join(self.root_dir, self.annotations.iloc[index, 1])
        audio, _ = torchaudio.load(audio_path)
        target, _ = torchaudio.load(target_path)
        return audio, target
</code></pre>
<p>when both the input and expected output are waveforms? In my case, I created a csv file that contains the filenames of both audios and I just load the path-like object into <code>torchaudio.load()</code>. Also, how do I ensure that all the <code>audio</code> and <code>target</code> in a batch are of the same length when being loaded?</p>

JuanFMontesinos: <p>Well there are two main possibilities.<br>
One is that you use a fix length of audio. You should make chunks of audio given a sample.<br>
Another option is that you use variable length (typically when using transformers)<br>
In that case you should rewrite dataloader’s collate function.</p>

Superklez: <p>What I implemented was</p>
<pre><code class="lang-auto">def batch_pad(batch):
    batch = [item.t() for item in batch]
    batch = torch.nn.utils.rnn.pad_sequence(batch, batch_first=True, padding_value=0.)
    return batch.permute(0, 2, 1)

def collate_fn(batch):
    tensors = []
    targets = []
    
    for waveform, target in batch:
        tensors += [waveform]
        targets += [target]

    tensors = batch_pad(tensors)
    targets = batch_pad(targets)

    return tensors, targets
</code></pre>
<p>Or is there some way I can use <code>batch_pad</code> on both <code>tensors</code> and <code>targets</code> together?</p>

JuanFMontesinos: <p>Batch pad just pads the tensors with zeros</p>
<p>What do you mean by together?</p>
AttributeError: 'float' object has no attribute 'detach'

sam101: <p>I am trying to do this :</p>
<p>my_file3 = x_mean[1].detach().cpu().numpy()</p>
<p>and it shows me this error :</p>
<p>AttributeError: ‘float’ object has no attribute ‘detach’</p>

Dwight_Foster: <p>Is the variable part of a graph? If it is not part of a graph or does not have a gradient you won’t be able to detach it from anything because it wasn’t attached in the first place.</p>

HimanshuArora: <p>Adding on to what <a class="mention" href="/u/dwight_foster">@Dwight_Foster</a> said, you can do <code>my_file3 = (x_mean[1].detach() if x_mean[1].requires_grad else x_mean[1]).cpu().numpy()</code> to make it work in any case.</p>

sam101: <aside class="quote no-group" data-username="HimanshuArora" data-post="3" data-topic="114461">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/himanshuarora/48/24914_2.png" class="avatar"> HimanshuArora:</div>
<blockquote>
<p>my_file3</p>
</blockquote>
</aside>
<p>Hi Himanshu,<br>
I tried what you have suggested but it is showing me this now<br>
AttributeError: ‘float’ object has no attribute ‘requires_grad’</p>

Manuel_Alejandro_Dia: <p>Are you sure that <code>x_mean[1]</code> is a <code>torch.Tensor</code> object?</p>
<p>Even if you take a member of a tensor that is a list, it will still have <code>.requires_grad</code> as <code>False</code> by default since it is of the <code>torch.Tensor</code> class:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x_mean = torch.ones((50))
&gt;&gt;&gt; x_mean.requires_grad
False
&gt;&gt;&gt; x_mean[1].requires_grad
False
&gt;&gt;&gt; type(x_mean[1])
&lt;class 'torch.Tensor'&gt;
</code></pre>
<p>Maybe you are appending the <code>.item()</code> of the original tensor somewhere, which changes the class to a normal Python float:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x_mean=[1.,2.,3.,4.,5.]
&gt;&gt;&gt; type(x_mean[1])
&lt;class 'float'&gt;
&gt;&gt;&gt; x_mean[1].requires_grad
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AttributeError: 'float' object has no attribute 'requires_grad'
</code></pre>
Hook function with more parameters

iacolippo: <p>Is there a way to <code>register_backward_hook</code> with a signature like</p>
<pre><code class="lang-auto">hook(module, grad_input, grad_output, parameter1, parameter2)
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">hook(module, grad_input, grad_output)
</code></pre>
<p>?</p>

fmassa: <p>You don’t need the extra signatures. You can use up-values to store it for you. Something like</p>
<pre><code class="lang-python">parameter1 = ...
def hook(module, grad_input, grad_output):
    return grad_input, grad_output, parameter1
</code></pre>
<p>You can even make it better, by slightly modifying the code in <a href="https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94/7?u=vabh">Why cant I see .grad of an intermediate variable?</a></p>

Keyv_Krmn: <p>You can use functools partial method</p>
<pre><code class="lang-auto">module.register_backward_hook(partial(hook, parameter1=p1, parameter2=p2))
</code></pre>
Densenet121 and GuidedGradCam

Pekka: <p>I’m a bit newbie with Captum, but I’ve trained a Pytorch densenet121 based classified for 224x224 medical images. Four output classes. Now I’d like to interpret results with Captum’s Guided GradCAM. Since Densenets have dense blocks and multiple interconnections, I’m a bit unsure what is the ultimately last conv layer that I should give as an argument. I’ve tried following:</p>
<pre><code class="lang-auto">gc = GuidedGradCam(net, net.features.denseblock4.denselayer16.layers.conv2)
attr_gc = attribute_image_features(gc, input, target=labels[ind])

viz.visualize_image_attr(attr_gc,
                         original_image,
                         method="heat_map",
                         sign="all",
                         show_colorbar=True, 
                         title="Guided GradCAM")
</code></pre>
<p>where the helper is similar to a Captum example:</p>
<pre><code class="lang-auto">def attribute_image_features(algorithm, input, target, **kwargs):
    net.zero_grad()    
    tensor_attributions = algorithm.attribute(input, target=target, **kwargs)
    
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
    
    # convert to numpy C,W,H
    attr = np.transpose(tensor_attributions.squeeze(0).cpu().detach().numpy(), (1, 2, 0))
    
    return attr
</code></pre>
<p>But for some reason I get an attribution heatmap that kind of only displays values in some blocks in the image (see attachment).</p>
<p>Why is that and what should I fix? Thanks!<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81fdfc6f2444f5cd295bffe3a5a51b24a4acc6d7.png" alt="Screenshot 2021-03-12 at 14.36.51" data-base62-sha1="ixXKTlQlz18YnE5kMVwWHvBU09V" width="445" height="448"></p>
Where can I find the doc of ARM64 BUILDS FOR LINUX

gemfield: <p>Pytorch team mentioned “ARM64 BUILDS FOR LINUX” in blog <a href="https://pytorch.org/blog/prototype-features-now-available-apis-for-hardware-accelerated-mobile-and-arm64-builds/" class="inline-onebox" rel="noopener nofollow ugc">Prototype Features Now Available - APIs for Hardware Accelerated Mobile and ARM64 Builds | PyTorch</a> , but where can I find the corresponding doc? Especially, how should I launch the build command on an ARM64 device, e.g. Rasparry PI 4b?</p>
Dynamic padding based on input shape

Andreas: <p>Hi,<br>
For my model my input (image) needs to be divisible by 32 and I would like to pad my input dynamically to fit this requirement. Meaning if the input would be for example 520x520x3, I want it to be padded to 544x544x3.</p>
<p>torch.nn.functional.pad() requires the pad to be list of ints, but my paddings would be a tensor computed dynamically depending on the shape of the input.<br>
The model (including the padding) is supposed to be exported to onnx so it has to be traceable.</p>
<p>Does anybody have a clue on how to approach this problem?<br>
Thanks in advance,<br>
Andreas</p>

LukasUz: <p>Hey, the reply is a little late, but I was struggling with the same problem. So I hope this will help somebody in the future. I solved my problem according to this example:</p>
<pre><code class="lang-auto">from torchvision import transforms
from torchvision.datasets.folder import pil_loader
from functools import partial
from math import ceil

p = "path/to/img.png"
i = pil_loader(p)

print("Shape:", i.size)
image_size = 544

def pad_to_minmum_size(min_size, image):
    h, w = image.size
    h_diff = h - min_size
    w_diff = w - min_size
    h_pad = ceil(abs(h_diff) / 2) if h_diff &lt; 0 else 0
    w_pad = ceil(abs(w_diff) / 2) if w_diff &lt; 0 else 0

    if h_pad == 0 and w_pad == 0:
        return image
    else:
        return transforms.functional.pad(image, [h_pad, w_pad])

transform = transforms.Compose([
    transforms.Lambda(partial(pad_to_minmum_size, image_size)),
    transforms.RandomCrop(image_size)
    ])
            

i_transformed = transform(i)
print("Shape after transform:", i_transformed.size)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Shape: (512, 383)
Shape after transform: (544, 544)
</code></pre>
<p>This should dynamically pad any image to the given image_size. Same code could be used for dynamic scaling as well.</p>
<p>Cheers!</p>
Dataloader for Weakly-Supervised Learning

NoobCoder: <p>Hello,</p>
<p>I need to have a data loader to be compatible with my model.</p>
<p>I want to run a model, say ResNet, on a dataset. In each epoch, I select for example 100 images randomly, find the top 20 images of those 100 selected images (lowest loss), and add these 20 to the next epoch and select 80 more.</p>
<p>In general, in each epoch, I need 100 images where 20 of them are top among the previous epoch and 80 of them are randomly chosen. Is there any way that I can implement this in the dataloder?<br>
What I am thinking is to writing the bacth_sampler that handle these stuffs, but I am not sure how to tell him the top 20 images?</p>
<p>Thanks,</p>

ptrblck: <p>You could create a custom <code>Subset</code> or <code>SubsetRandomSampler</code> for each training “iteration”.<br>
I.e. in the first iteration you would select 100 random indices and store the desired 20 indices after the training is done. Afterwards you would recreate the <code>Subset</code> or sampler as well as the <code>DataLoader</code> and repeat this workflow.</p>

rahulvigneswaran: <p>Hi <a class="mention" href="/u/noobcoder">@NoobCoder</a>,</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> is suggesting something like this,</p>
<pre><code class="lang-auto">class YourCustomSubset(torch.utils.data.Subset):
    r"""
    Same as torch.utils.data.Subset. This outputs the indices of each batch, you can choose which indices to keep.
    """

    def __getitem__(self, idx):
        return self.dataset[self.indices[idx]], idx
</code></pre>
<p>Check this <a href="https://gist.github.com/rahulvigneswaran/767e869ed4f0b410ffad4501d2252451" rel="noopener nofollow ugc">gist/Colab notebook</a> for the full implementation with dummy example.</p>
How to efficiently filter out classes

francisco-simoes: <p>Say I have a custom Dataset instance with 200 classes and a million samples.<br>
I want to create a custom Subset instance containing only the samples whose label is in a certain list of selected labels.</p>
<p>I tried to do this by looping through the indices as follows:</p>
<pre><code class="lang-auto">for idx in self.indices:
    datapoint_label: int = dataset[idx][1]
    if dataset.label_to_class[datapoint_label] in chosen_classes:
        self.filtered_chosen_indices += [idx]
</code></pre>
<p>This works but it is way too slow. I would expect Pytorch to have an efficient method to do something of this sort. Does it?</p>
<p>Note: the attribute <code>label_to_class</code> from the custom dataset is a dictionary Dict[int, str] that attaches the class name to the corresponding integer label.</p>
Which version should I download for libtorch?

Jelly123456: <p>I am using libtorch C++ to do model inference. I want to debug the libtorch code because I faced a “segmentation fault” error when running the model forward function{module_.forward(inputs)}</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d73eba5d533ef2aa056caa3f26dd9dab9f5ef7e4.png" alt="image" data-base62-sha1="uI93775AKQbLqaMiw956bFw6pJG" width="382" height="148"></p>
<p>[Compiler]: Gcc 10.2<br>
[CUDA]: 10.2</p>
<p>My question is which version of libtorch should I download for debugging? Pre-cxx ABI or cxx11 ABI? What are the differences for them?</p>
<p>Pre-cxx ABI:<br>
<a href="https://download.pytorch.org/libtorch/cu102/libtorch-shared-with-deps-1.8.0.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu102/libtorch-shared-with-deps-1.8.0.zip</a></p>
<p>cxx ABI:<br>
<a href="https://download.pytorch.org/libtorch/cu102/libtorch-cxx11-abi-shared-with-deps-1.8.0.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu102/libtorch-cxx11-abi-shared-with-deps-1.8.0.zip</a></p>
<p>Thanks.</p>
Topk extraction and reconstruction

Wang_Octavius: <p>Now I have a tensor ‘x’ with shape [b, c, d, h, w]. Now first I calculate the normalization of all the channels and the output tensor called ‘channel_norm’ is of size [b,1,d,h,w]. Then I sort the tensor ‘channel_norm’ along dimension ‘d’ and want to know the indices of the top k elements in each (b,h,w) position. Since ‘x’ and ‘channel_norm’ only differ in ‘c’, we can use the found top k ‘d’ indices in ‘channel_norm’ to find the corresponding ones in ‘x’. However, I do not how to do this.</p>
<p>After we find the top k elements in ‘x’, we will get a tensor ‘y’ of shape [b,c,k,h,w]. We do some convolutions on ‘y’, and output a tensor of shape [b,1,k,h,w]. Then what we want to do is to restore the k depths to the original position. In detail, suppose the top 2 elements are obtained in positions 1 and 5, and other elements are dropped, thus we have a generated vector of shape 2. Then after all the calculations, we want to reconstruct a vector of shape [b,1,d,h,w] by filling 0 in the positions we dropped before or interpolation. However, I do not know how to do this as well.</p>
<p>Hope I have clearly stated all the information. My code for the first part is attached below. Thank you so much!</p>
<pre><code class="lang-auto">    def forward(self, x):
        # x: [B,C,D,H,W]
        b, c, d, h, w = x.shape
        if self.hparams.top_depth != -1:
            channel_norm = torch.norm(x, p=float('inf'), dim=1, keepdim=True)  # channel_norm: [b,1,d,h,w]
            num_top_k = int(self.hparams.top_depth * d)
            _, indices_top_k = torch.topk(channel_norm, k=num_top_k, dim=2, largest=False, sorted=False)
            # _, indices_top_k: [b,1,num_top_k,h,w]
            # del _, channel_norm
            x = torch.index_select(x, dim=2, index=indices_top_k)
            x[indices_top_k]
</code></pre>
The high cpu usage in Pytorch, torchvision

k87974: <pre><code class="lang-python">from torchvision import transforms as T

trans = T.Compose([ T.ToTensor(),T.Normalize((.222,.222,.222),(.222,.222,.222))])

for img in img_fils:
    img = trans(img)

</code></pre>
<p>Suppose the img_files have loaded, I found that my CPU usage may higher than 1000% on the server.<br>
I thought the problem is that  F.normalize used multi-threads in CPU mode. However, the speed is not so fast by using so many resources.<br>
I wanna know if torch uses multi-threads in most of the functions in CPU mode by default?</p>

ptrblck: <p>Yes, some 3rd party CPU libs can and will use multi-threading by default.<br>
I think you could remove it via <code>torch.set_num_threads</code> as well as the env vars:</p>
<pre><code class="lang-auto">OMP_NUM_THREADS 
MKL_NUM_THREADS
</code></pre>
When does D2H happen?

111480: <p>Although to(at::kCUDA) was used for the model and input value, the output value printing was possible without DeviceToHost memcpy.<br>
How is this possible?<br>
Is that what’s happening internally? If so, where does it happen?</p>

ptrblck: <p>It should happen in <a href="https://github.com/pytorch/pytorch/blob/506f756a0a6027d74dfdc5d0602fd894586d6e41/aten/src/ATen/core/Formatting.cpp#L230-L249">Formatting.cpp</a> via the <code>to(kCPU, kDouble)</code> operation.</p>
Model Parallelism Across Multiple Devices

surya00060: <p>Is it possible to manually partition a DNN and run in parallel for inference alone across few CPUs/GPUs?</p>
<p>For Example consider a dummy example. there are four convolutions that can be run in parallel without any dependencies and there are four machines. I would like to run each convolution on a independent GPU. Is there way to do model parallel <strong>inference</strong> across GPUs?</p>
<pre><code class="lang-python">class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d()
        self.conv2 = nn.Conv2d()
        self.conv3 = nn.Conv2d()
        self.conv4 = nn.Conv2d()

    def forward(self, x):
        y1 = self.conv1(x)
        y2 = self.conv2(x)
        y3 = self.conv3(x)
        y4 = self.conv4(x)
        return [y1,y2,y3,y4]
</code></pre>

surya00060: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any solution?</p>

ptrblck: <p>Yes, you could push each module to the corresponding device and execute each forward pass using an input on the corresponding device.</p>
Inconsistent Outputs from BucketIterator

J_Johnson: <p>I’m getting some odd behavior from the <a href="https://torchtext.readthedocs.io/en/latest/data.html" rel="noopener nofollow ugc">BucketIterator</a>.</p>
<p>I have a dataset of size 2164 split into 136 batches of 16. But on some random occasions it sends a batch size of only 4 through(presumably the last batch) too early. I’ve run the script several times and so far it’s pretty random as to when it happens. Once it did so on iter <span class="hashtag">#103</span>, another time on <span class="hashtag">#118</span>, another time on <span class="hashtag">#124</span>. And the odd thing is I have shuffle set to None, so the error “should” at least be consistent. The data isn’t changing. It seems to be randomly skipping certain batches.</p>
<p>Any idea what could be causing this or is this a known issue? The size of each sequence is set to fix_length=1100.</p>

J_Johnson: <p>So I changed it to the Iterator class. Apparently BucketIterator implicitly randomizes batches via pool.</p>
Inconsistent results between forward and weighs usage

DL_C: <p>Hi, I’ve traind a super-simple linear regression model:</p>
<pre><code class="lang-python">class LinearRegressionNet(nn.Module):
    def __init__(self, in_size, out_size):
        super(LinearRegressionNet, self).__init__()
        self.layer = nn.Linear(in_size, out_size)

    def forward(self, x):
        return self.layer(x)
</code></pre>
<p>I’ve train this net using <code>nn.SmoothL1Loss</code>, for 200 epochs with <code>batch_size=32</code>.</p>
<p>The loss decreased to <code>1e-6</code>, and I tried to print the predictions.<br>
Consider my data is <code>X</code> and the labels are all 0, this is the prediction code:</p>
<pre><code class="lang-python">with torch.no_grad():
    X_torch = torch.from_numpy(X.astype(np.float64).to('cuda').float())
    print(model(X_torch).mean())
</code></pre>
<p>the output is: <code>0.023</code> which way far from <code>1e-6</code>. Then, I tried the straightforward way:</p>
<pre><code class="lang-python">w = model.weight.data.cpu().numpy()
print(np.dot(X, w.T).mean())
</code></pre>
<p>the output is -0.0015, which is more convincible.</p>
<p>How could it be? the output should be consistent within both code chunks</p>

crcrpar: <p>Hi,</p>
<p>Don’t you forget the bias term of <code>nn.Linear</code>?</p>
<p>To mimic pytorch with numpy, I think the code will be</p>
<pre><code class="lang-python">np.dot(x, w.T) + model.bias.data.numpy()
</code></pre>
'int' object is not callable in RNN

shark_test: <p>This is my code</p>
<p>import torch<br>
import torchvision<br>
from torch import nn<br>
import numpy as np<br>
from torch import optim</p>
<p>num_layers = 2<br>
batch_size = 1<br>
no_hidden_units = 128<br>
input_size = 1<br>
output_size = 1<br>
sequence_length = 21<br>
num_epochs = 2</p>
<p>x = np.linspace(0,20,21)<br>
y = np.linspace(1,21,21)<br>
print(x)<br>
print(y)</p>
<p>class Lstm(nn.Module):<br>
def <strong>init</strong>(self , input_size , no_hidden_units , output_size , num_layers):<br>
super(Lstm , self).<strong>init</strong>()<br>
self.no_hidden_units = no_hidden_units<br>
self.num_layers = num_layers<br>
self.lstm = nn.RNN(input_size , self.no_hidden_units ,<br>
self.num_layers , batch_first = True)<br>
self.fc = nn.Linear(self.no_hidden_units , output_size)</p>
<pre><code>def forward(self , x , h_state):
    out = self.lstm(x , h_state)
    out = out.reshape(-1,self.no_hidden_units)
    # or out = out[:,-1,:]
    output = self.fc(out)
    print(output.shape)
    output = output.reshape(-1,sequence_length,1)
    print(output.shape)
    return output , h_state
</code></pre>
<p>lst = Lstm(input_size , no_hidden_units , output_size , num_layers)<br>
h_state = torch.zeros(num_layers , batch_size , no_hidden_units)<br>
print(lst)</p>
<p>criterion = nn.MSELoss()<br>
optimizer = optim.Adam(lst.parameters() , lr = 0.001)</p>
<p>for i in range(num_epochs):<br>
xin = x.reshape(batch_size , sequence_length , input_size)<br>
y = y.reshape(batch_size , sequence_length , input_size)<br>
print(xin.shape)</p>
<pre><code>optimizer.zero_grad()
logits , _ = lst(xin ,h_state)
loss = criterion(logits , y)
print(loss)
loss.backward()
optimizer.step()
</code></pre>
<p>TypeError                                 Traceback (most recent call last)</p>
<p> in ()<br>
5<br>
6     optimizer.zero_grad()<br>
----&gt; 7     logits , _ = lst(xin ,h_state)<br>
8     loss = criterion(logits , y)<br>
9     print(loss)</p>
<p>3 frames</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)<br>
191         else:<br>
192             batch_sizes = None<br>
–&gt; 193             max_batch_size = input.size(0) if self.batch_first else input.size(1)<br>
194             sorted_indices = None<br>
195             unsorted_indices = None</p>
<p>TypeError: ‘int’ object is not callable</p>

Krasin_Georgiev: <p>May be <code>x</code> and <code>y</code> should be Pytorch tensors, not Numpy arrays</p>

Simin_Fan: <p>Hi, have you solve the problem? I have stuck with the same one…</p>
Necssity of repeating epochs

aswamy: <p>I saw a code like below</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
            batch = 0
            for _ in range(num_epoch_repeats):
                for data in self.train_data_loader:
                    losses = self.train_step(data, global_step=step_id)
                    ....................................................
                    ....................................................
</code></pre>
<p>I do not understand the necessity of this loop <code>"for _ in range(num_epoch_repeats):"</code> instead you can increase the num_epochs, right?</p>
<p>Are there any other reasons for this kind of implementation?</p>
<p>Thanks<br>
Anil</p>

googlebot: <p>You should see how authors use the outer loop - likely they do some periodic actions, and decided to change epoch numeration for that (e.g. there are validation epochs every 10 training epochs, and num_epochs counts the former). Not very good idea overall, IMHO.</p>

aswamy: <p>Yeah, I checked, It seems that learning rate is updated(<code>self.lr_scheduler.step()</code>) for every <code>self.num_epoch_repeats</code> But this would just be the same as counting on epochs and then performing <code>self.lr_scheduler.step()</code> for every required number count. Right?</p>
<pre><code class="lang-auto">        for epoch in range(self.num_epochs):
            self.writer.add_scalar(
                "lr", self.optim.param_groups[0]["lr"], global_step=step_id
            )

            batch = 0
            for _ in range(self.num_epoch_repeats):
                for data in self.train_data_loader:
                    losses = self.train_step(data, global_step=step_id)
                    loss_str = fmt_loss_str(losses) 
                    if (
                        batch == self.num_total_batches - 1
                        or batch % self.accu_grad == self.accu_grad - 1
                    ):
                        self.optim.step()
                        self.optim.zero_grad()

                    self.post_batch(epoch, batch)
                    step_id += 1
                    batch += 1
                    progress.update(1)
            if self.lr_scheduler is not None:
                self.lr_scheduler.step()
</code></pre>
<p>I am just wondering is there any other reason.</p>

googlebot: <p>seems that they also do it for gradient accumulation - as it interacts with the data loader a bit poorly (if dataset_size : batch_size*accum_batches ratio is low)</p>
How to calculate accuracy in pytorch?

twpann: <p>I want to calculate training accuracy and testing accuracy.In calculating in my code,training accuracy is tensor,not a number.Moreover,in converting numpy(),the accuracy is 2138.0 ,I used ypred and target in calculating accuracy.Why does the problem appear?Please answer how I solve.Thanks in advance!</p>

lycaenidae: <p>Could you please share the code where you calculate your accuracy?</p>
<p>As a general knowledge, you can calculate the accuracy on the training set based on your your metric defined beforehand. As an example, you can use the L1,L2 difference between two numpy arrays as a metric.</p>

twpann: <p>Thanks a lot for answering.Accuracy is calculated as seperate function,and it is called in train epoch in the following loop:<br>
for batch_idx,(input, target) in enumerate(loader):<br>
output = model(input)<br>
# measure accuracy and record loss<br>
batch_size = target.size(0)<br>
_, pred = output.data.cpu().topk(1, dim=1)<br>
pred = pred.t()<br>
y_pred = model(input)<br>
accuracy=binary_acc(y_pred,target)<br>
Please answer how can I calculate?Thanks in advance!</p>

chetan06: <pre><code class="lang-auto">train_acc = torch.sum(y_pred == target)
</code></pre>
<p>At the end</p>
<pre><code class="lang-auto">final_train_acc = train_acc/number_of_datapoints
</code></pre>

twpann: <p>Thanks a lot for answering! Is<strong>number_of_datapoints</strong>  total number of data samples?Thanks!</p>

Brando_Miranda: <p>read this answer: <a href="https://stackoverflow.com/a/63271002/1601580" rel="noopener nofollow ugc">https://stackoverflow.com/a/63271002/1601580</a></p>
Pytorch image normalization

nuhpiskin: <p>Pytorch image normalization is very slow in total code and my computer. But if only run normalization code is fast normally in my computer . My friend computer i9 9700K cpu and 2080 gpu. torch image normalization is very fast more than 25 times me in total code. My computer i7 9700K 2070 nvidia. Our code (torch.cuda.floattensor(image)/255-0.5)/0.5</p>

ptrblck: <p>Could you share the code snippet you’ve used to profile this operation?<br>
Note that CUDA operations are executed asynchronously, so you would have to synchronize the code before starting and stopping the timer (or use the <code>torch.utils.benchmark</code> utilities, which will synchronize the code for you).</p>

nuhpiskin: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b009f2b7ffb830caa1fddfccfe22e30020698ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/6b009f2b7ffb830caa1fddfccfe22e30020698ce" title="Screenshot from 2021-02-01 11-53-18"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b009f2b7ffb830caa1fddfccfe22e30020698ce.png" alt="Screenshot from 2021-02-01 11-53-18" data-base62-sha1="fgAq82HPvk2145AJ1PN7RtYhOb4" width="690" height="212" data-dominant-color="262626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-01 11-53-18</span><span class="informations">740×228 30.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Sorry for my late return. I am using torch.cuda.synchronize() also this code running fast alone. But if I use with other pipeline code timing is slow.I agree with you but i cant find where is slow. Actually this process should be very fast.</p>

dhiaeddineTounakti: <p>in the code snippet that you are providing, you are normalizing the image on the CPU then converting it to a CUDA tensor. While in the line of code that you have provided in your comment</p>
<aside class="quote no-group" data-username="nuhpiskin" data-post="1" data-topic="110273">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/b487fb/48.png" class="avatar"> nuhpiskin:</div>
<blockquote>
<p>code (torch.cuda.floattensor(image)/255-0.5)/0.5</p>
</blockquote>
</aside>
<p>you are converting to CUDA tensor and then doing the normalization on the GPU which is obviously faster.</p>

Vesnin: <p>Is it correct that you convert [0-255] range to [-0.5; 0.5] and then divide by 0.5 and get [-0.25; 0.25] range?</p>
Custom directed graph dataset using torch_geometric Data

sakh251: <p>Hello</p>
<p>I have directed graph and I want make a custom dataset to use with torch geometric.<br>
I have JSON file with link[‘source’], link[‘target’] data</p>
<pre><code class="lang-auto">from torch_geometric.data import Data
edge_index = torch.tensor([edge[0],
                                   edge[1]], dtype=torch.long)
x = torch.tensor(x_feats_train, dtype=torch.float)
 y = torch.tensor((y_labels_train))
data = Data(x=x, edge_index=edge_index, y=y)
</code></pre>
<p>But when a dataset using this data object is made, isdirected is False.<br>
How can I make it directed?</p>

engrodawood: <p>To create your own dataset from the data container.</p>
<pre><code class="lang-auto">from torch_geometric.data import Data, DataLoader

loader = DataLoader([data], batch_size=32)

The input to the DataLoader will be list. 
if you have more that one data container:

data_list = [Data(...), ..., Data(...)]
loader = DataLoader(data_list, batch_size=32)

https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html</code></pre>
Using grad on a Jacobian

mattief: <p>I have a function f(x;a) between two vector spaces parametrised by a. I’d like to take the derivative of the Jacobian D_x f(x;a) with respect to the parameters a, that is ∇_a (D_x f(x;a)). It’s obviously possible to code up the Jacobian by hand and call <code>backward()</code> on it like this (i’ve used a 1D space for simplicity):</p>
<pre><code class="lang-auto">import torch as th
from torch import nn

def sech2(y1_pre):
    return  1.0 / (th.cosh(y1_pre) * th.cosh(y1_pre)) 

# Forward Pass
M1 = nn.Parameter(th.tensor([[1.0, 2.0, 3.0]]), requires_grad=True)
x = th.tensor([1.0], requires_grad=True)
y1_pre = th.einsum('i,ij-&gt;j', x, M1)
y1 = th.tanh(th.einsum('i,ij-&gt;j', x, M1))
M2 = nn.Parameter(th.tensor([[1.0, 0.0, 1.0], [0.0, 1.0, 0.0], [1.0, 0.0, 1.0]]), requires_grad=True)
y2_pre = th.einsum('j,jj-&gt;j', y1, M2)
y2 = th.tanh(y2_pre)
M3 = nn.Parameter(th.tensor([[10.0, 10.0, 10.0]]), requires_grad=True)
y = th.einsum('j,ij-&gt;i', y2, M3)

# Backward pass
B1 = sech2(y1_pre) * M1
B2 = sech2(y2_pre) * M2
B3 = th.einsum('jj,ij-&gt;ij', B2 , B1)
jacob = th.einsum('ij,ij-&gt;i', M3 , B3)

jacob.backward()
</code></pre>
<p>but this seems rather clunky, especially as I’d ultimately like to be using PyTorch’s build-in network layers.</p>
<p>Does anybody have a more sophisticated way of doing this? I’ve tried taking backward on the grad of the output, but that doesn’t work.</p>

albanD: <p>Hi,</p>
<p>You need to make sure to pass <code>create_graph=True</code> the first time you run autograd.grad (or backward()) to be able to backward through that.<br>
Also you can use the <code>autograd.functional.jacobian(..., create_graph=True)</code> to get the full Jacobian directly.</p>

mattief: <p>Fab, thanks for this. I’ve played around and found <code>autograd.functional.jacobian(..., create_graph=True)</code> is an awful lot more expensive in comparison to my posted method - hopefully there’ll be a solution that is both elegant and cheap soon!</p>
How to release a Module from GPU ? (C++)

ZhiZe-ZG: <p>I tried cudaFree, it doesn’t work.</p>
<p>let model be a Module</p>
<p>I tried this:</p>
<p>for (auto pa:model.parameters()){<br>
cudaFree(pa.data_ptr());}</p>

copythatpasta: <p>How much memory is being occupied? If it is around 500mb it is the CUDA context cache.</p>
<p>You can simply move a model to the CPU using model.to(torch::kCPU) to release the GPU memory.  Just make sure to call c10::cuda::CUDACachingAllocator::empty_cache() afterwards. You may need to move the model to the CPU within a function and call empty_cache() afterwards.</p>
How to set hooks on intermediate variables?

mathDR: <p>Hello all!  I have a (hopefully) basic question regarding backward hooks.</p>
<p>I would like to set a hook on an intermediate variable in my model.</p>
<p>I try:</p>
<pre><code class="lang-auto">y.register_hook(lambda grad: print(grad.shape))
</code></pre>
<p>but that yields the error:</p>
<pre><code class="lang-auto">RuntimeError: cannot register a hook on a tensor that doesn't require gradient
</code></pre>
<p>So then I attempt to do:</p>
<pre><code class="lang-auto">y.requires_grad = True
y.register_hook(lambda grad: print(grad.shape))
</code></pre>
<p>but that yields the error</p>
<pre><code class="lang-auto">RuntimeError: you can only change requires_grad flags of leaf variables.
</code></pre>
<p>This seems circular?  I see a lot of examples where <code>register_hook</code> is used on intermediate variables.  What seems to be the error?</p>
<p>I am using Torch 1.6 on GPU</p>

googlebot: <p>y.requires_grad_() would work on tensors (though the need of a gradient of “external” values is a bit suspicious…).</p>
<p>I normally do:<br>
if x.requires_grad: x.register_hook(…)<br>
to avoid failures during validation / evaluation.</p>

mathDR: <p>Apologies, y was used as a generic place holder for an intermediate  layer output (I probably should have used layer or some other variable name).</p>
<p>The extension to this question is applying different lambdas in the hook (to zero out some masked gradient values for instance). I will try the method you suggested, thanks!</p>

gbilla: <p>From what I understand, <code>x.retain_grad()</code> should be preferred over using the register hook method. See this for a comparison of various approaches to saving gradients of intermediate variables:</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/rasbt/deeplearning-models/blob/master/pytorch_ipynb/mechanics/manual-gradients.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/rasbt/deeplearning-models/blob/master/pytorch_ipynb/mechanics/manual-gradients.ipynb" target="_blank" rel="noopener nofollow ugc">rasbt/deeplearning-models/blob/master/pytorch_ipynb/mechanics/manual-gradients.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
</code></pre>

  This file has been truncated. <a href="https://github.com/rasbt/deeplearning-models/blob/master/pytorch_ipynb/mechanics/manual-gradients.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to compute the variance or MSE between two losses?

en_tetteh: <p>I want to know if my implementations for computing the variance or MSE between two losses are correct:</p>
<pre><code class="lang-auto">import torch
import numpy as np
from sklearn.metrics import mean_squared_error 

loss1 = [0.3, 0.2, 0.8]
loss2 = [0.25, 0.3, 0.72]

### MSE ###
loss = torch.nn.MSELoss()

sk_mse = mean_squared_error(loss1, loss2)
n_mse = np.square(np.subtract(loss1, loss2)).mean()
t_mse = loss(torch.tensor(loss1), torch.tensor(loss2))   #only holds with the assumption one loss is a target
output: 0.0063   #output for all of the above code

### Var ###
t_var = torch.var(torch.tensor([[loss1, loss2]]), unbiased=False)
n_var = np.var([loss1, loss2], dtype=np.float64)

output: 0.056681 #output for all of the above code</code></pre>
Cost of nn.Identity

Carsten_Ditzel: <p>Here I am sitting clueless that nn.Identity actually takes up a considerable amount of GPU memory when I though, in fact, this should be a no-op should it not?</p>
<p>Can someone elaborate on this?</p>

googlebot: <p>it is a no-op, in a sense that only python bytecode of identity function is executed. your problem is likely elsewhere (note that cuda.memory_allocated() is not changed by nn.Identity, but you may be blocking tensor freeing somehow)</p>
Generalizability issues with deep learning models in medicine

WeNeedMore: <p>How well the deep learning models generalize to the medical images, like CT scans?<br>
According to this paper( <a href="https://arxiv.org/abs/2102.09616" rel="noopener nofollow ugc">Deep learning-based COVID-19 pneumonia classification using chest CT images: model generalizability</a>, haven’t passed the peer review yet), the model train by cnn do not generalize well for different dataset. The data set he prefer is not big, the largest one only has 1500~2000 CT scans per class.</p>
<p><a href="https://arxiv.org/abs/2102.04869" rel="noopener nofollow ugc">A Real-World Demonstration of Machine Learning Generalizability: Intracranial Hemorrhage Detection on Head CT</a> say the ML model can generalize well for different data set, but the “other data set” the test only come from their own emergency department in 2019.</p>
<p>New to the domain of medical images, do anyone  know the progress of this question? Thanks</p>
PyTorch copying to GPU is slow

Semir_Elezovikj: <p>The transfers to GPU seem to be slow - wondering whether I can do anything with the below snippet of code to speed things up.</p>
<p>Gpu.IDS is just an array. Example: <code>IDS = [4, 5, 6]</code></p>
<pre><code class="lang-auto">device = torch.device('cuda:' + str(Gpu.IDS[0]) if torch.cuda.is_available() else 'cpu')
print('running on ', device)
model = model.to(device)

if torch.cuda.device_count() &gt; 1:
        print("Let's use multiple GPUs:", len(Gpu.IDS), "out of ", torch.cuda.device_count(), "GPUs!")      
        model = nn.DataParallel(model, device_ids=Gpu.IDS)

tokens = train

dataset = NuscenesDataset(tokens, helper)
dataloader = DataLoader(dataset, batch_size=16*len(Gpu.IDS), num_workers=4 * len(Gpu.IDS))

n_iter = 0

minimum_loss = 0

loss_function = MTPLoss(NUM_MODES, 1, 5)
current_loss = 10000
model_dir = make_model_dir()
learning_rates = [0.1]


for lr in learning_rates:
    # optimizer = optim.SGD(model.parameters(), lr=lr, momentum = 0.9)
    optimizer = optim.Adam(model.parameters(), lr=lr)
    print('--------- LEARNING RATE ', lr, '--------------')
    for epoch in range(1, 10001): 
        # print below - only for large dataset
        # print('-&gt; epoch:', epoch)
        for img, agent_state_vector, ground_truth, _, _ in dataloader:            
    #         imshow(torchvision.utils.make_grid(img)) 
            img = img.to(device)
            agent_state_vector = agent_state_vector.to(device)    
            ground_truth = ground_truth.to(device)
            ground_truth = ground_truth.to(torch.float32)
            optimizer.zero_grad()

            prediction = model(img, agent_state_vector)            
    #         print('prediction.shape', prediction.shape, 'ground_truth.shape', ground_truth.shape)
            loss = loss_function(prediction, ground_truth)
            loss.backward()
            optimizer.step()

            current_loss = loss.cpu().detach().numpy()        
</code></pre>

albanD: <p>Hi,</p>
<p>Why do you think the transfer is slow?<br>
Keep in mind that the cuda API is asynchronous except when it needs to deal with CPU values. So if you measure without manual synchronization with torch.cuda.synchronize(), then it will appear that the copies are slow only because they wait for the rest of the computations before being able to execute as they deal with CPU values.</p>

Semir_Elezovikj: <p>Hi,</p>
<p>I guess, my question is more along the lines of whether something is an obvious red flag in my setup.</p>

albanD: <p>Hi,</p>
<p>No the code looks ok.<br>
The one thing is that you can collapse this:</p>
<pre><code class="lang-auto">            ground_truth = ground_truth.to(device)
            ground_truth = ground_truth.to(torch.float32)
</code></pre>
<p>into</p>
<pre><code class="lang-auto">            ground_truth = ground_truth.to(device=device, dtype=torch.float32)
</code></pre>

Semir_Elezovikj: <p>Thanks - much appreciated</p>
Preserving the gradient

danielstankw: <p>Hi all,<br>
I have an input to my NN consisting of the tensor(u) which stores the gradient needed for backpropagation in my example its <code>grad_fn&lt;AddBackward0&gt;</code> and numpy.array(y).<br>
The NN model takes an input that’s a tensor, so I convert the array to tensor and concatenate them together. z<br>
I feed this concatenated tensor into the NN model. It is important to note that the <code>u</code> part of the input stores gradient and the part corresponding to y doesn’t (it originates from numpy.array()).<br>
After I check the type of the output of the NN I see that it is a tensor with a <code>grad_fn=&lt;CatBackward&gt;</code></p>
<p>What is the meaning of the changed name of grad_fn?<br>
Does it mean that the gradient has been overwritten?<br>
If I run backpropagation through the model will it be able to use the gradient stored in u to do it successfully?<br>
If we want to backpropagate just through u (even though input is u and y)  is it possible?</p>
<p>Thanks in advance</p>

albanD: <blockquote>
<p>What is the meaning of the changed name of grad_fn?</p>
</blockquote>
<p>The grad_fn just tells you which op created this Tensor. In this case a concatenation</p>
<blockquote>
<p>Does it mean that the gradient has been overwritten?</p>
</blockquote>
<p>No, just that this concatenation op is taken into account when gradients will be computed (even if it is a no-op here because the other Tensor is a numpy array).</p>
<blockquote>
<p>If we want to backpropagate just through u (even though input is u and y) is it possible?</p>
</blockquote>
<p>If y does not require gradients (which it cannot if it is a numpy array), then the autograd will automatically ignore it when running backprop.</p>