akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Gradients for custom loss function using torch.where

ph14: <p>I have a custom loss function which uses the <code>torch.where</code> function. It takes my model’s output, <code>out</code> (which has two columns) and does the following:</p>
<pre><code class="lang-auto">out = torch.randn((8, 2), requires_grad=True)
loss = torch.where(out[:, 0] &gt; out[:, 1], 1., 0.).mean()
</code></pre>
<p>However, no gradients are carried through since <code>torch.where</code> uses the 1. and 0. values which do not have gradients. Is there any way to implement a loss function like this so that I can use <code>loss.backward()</code>?</p>

googlebot: <p>technically, following may work</p>
<pre><code class="lang-auto">d = out[:,0]-out[:,1]
loss = (d.abs() * torch.heaviside(d, torch.zeros(1))).mean()
</code></pre>
<p>but it still may be difficult to optimize with this formulation, as gradients are discrete in (-1,0,1). Adam + lr scheduler <em>may</em> get you somewhere, if not, reformulate your loss with d**2 and constraints/penalties somehow.</p>
Concatenation operation

lasmar: <p>Hi!</p>
<p>I am working with a model in which I would like to perform the following operation:</p>
<p>Z is an n x m matrix<br>
S is an k x l matrix</p>
<p>I am searching for a pytorch operation that would allow me to concatenate each pair of rows in Z and columns in S to yield a 3D tensor, where entry i,j contains row i in z concatenated with column j in S and hence has length (m + k). Hence, the tensor dimensions would be n x l x (m + k). Is there a convenient way to do this in pytorch?</p>
<p>Thx!</p>

ayhyap: <p>Here’s my attempt.<br>
I just repeated Z and S into (n x l x m) and (n x l x k) tensors respectively, then concatenated along the last dimension.</p>
<pre><code class="lang-auto">n = 2
m = 3
k = 4
l = 5
z = torch.rand((n,m)) # n x m
'''
tensor([[0.7424, 0.6298, 0.8356],
        [0.5601, 0.1725, 0.9190]])
'''
s = torch.rand((k,l)).T # l x k
'''
tensor([[0.5839, 0.9947, 0.7128, 0.2356],
        [0.4745, 0.8217, 0.4955, 0.7252],
        [0.7342, 0.0209, 0.8368, 0.1695],
        [0.4658, 0.3710, 0.6414, 0.1134],
        [0.7196, 0.9393, 0.5347, 0.1267]])
'''

z_repeated = z.repeat_interleave(l, dim=0).reshape(n,l,m) # n x l x m
'''
tensor([[[0.7424, 0.6298, 0.8356],
         [0.7424, 0.6298, 0.8356],
         [0.7424, 0.6298, 0.8356],
         [0.7424, 0.6298, 0.8356],
         [0.7424, 0.6298, 0.8356]],

        [[0.5601, 0.1725, 0.9190],
         [0.5601, 0.1725, 0.9190],
         [0.5601, 0.1725, 0.9190],
         [0.5601, 0.1725, 0.9190],
         [0.5601, 0.1725, 0.9190]]])
'''
s_repeated = s.repeat((n,1,1)) # n x l x k
'''
tensor([[[0.5839, 0.9947, 0.7128, 0.2356],
         [0.4745, 0.8217, 0.4955, 0.7252],
         [0.7342, 0.0209, 0.8368, 0.1695],
         [0.4658, 0.3710, 0.6414, 0.1134],
         [0.7196, 0.9393, 0.5347, 0.1267]],

        [[0.5839, 0.9947, 0.7128, 0.2356],
         [0.4745, 0.8217, 0.4955, 0.7252],
         [0.7342, 0.0209, 0.8368, 0.1695],
         [0.4658, 0.3710, 0.6414, 0.1134],
         [0.7196, 0.9393, 0.5347, 0.1267]]])
'''
out = torch.cat((z_repeated, s_repeated), dim=-1) # n x l x (m+k)
'''
tensor([[[0.7424, 0.6298, 0.8356, 0.5839, 0.9947, 0.7128, 0.2356],
         [0.7424, 0.6298, 0.8356, 0.4745, 0.8217, 0.4955, 0.7252],
         [0.7424, 0.6298, 0.8356, 0.7342, 0.0209, 0.8368, 0.1695],
         [0.7424, 0.6298, 0.8356, 0.4658, 0.3710, 0.6414, 0.1134],
         [0.7424, 0.6298, 0.8356, 0.7196, 0.9393, 0.5347, 0.1267]],

        [[0.5601, 0.1725, 0.9190, 0.5839, 0.9947, 0.7128, 0.2356],
         [0.5601, 0.1725, 0.9190, 0.4745, 0.8217, 0.4955, 0.7252],
         [0.5601, 0.1725, 0.9190, 0.7342, 0.0209, 0.8368, 0.1695],
         [0.5601, 0.1725, 0.9190, 0.4658, 0.3710, 0.6414, 0.1134],
         [0.5601, 0.1725, 0.9190, 0.7196, 0.9393, 0.5347, 0.1267]]])
'''
</code></pre>
Vector times matrix simple question

tommaso-green: <p>Hello,</p>
<p>I can’t seem to find a way to do the following operation with one simple matrix multiplication. Given</p>
<pre><code class="lang-auto">a = torch.randn((2,768))
b= torch.randn((2,4,768))
</code></pre>
<p>I want to multiply the first row of a, a[0], with the matrix b[0]. Similarly, I would like to get the product a[1]*b[1]. The total result should be a (2,4) tensor.</p>
<p>I know how to get this results using einsum, however I need to do this operation also with nn.Bilinear (aWb, with W being the weight matrix): how should I shape the nn.Bilinear weight matrix to get the result that I want?</p>
<p>Thanks for your time.</p>
From pytorch to tensorflow

111495: <p>I have fine-tuned PEGASUS model for abstractive  summarization using <a href="https://gist.github.com/jiahao87/50cec29725824da7ff6dd9314b53c4b3" rel="noopener nofollow ugc">this script</a> but it is in pytorch.<br>
Is there a way to transorm it into tensorflow model so I can use it in a javascript backend?</p>
Is MKL a must-have to run torch.fft() on a non-GPU environment?

erogol: <p>I try the official conda release of PyTorch (1.8.0) on Mac with M1 chip. Everything runs smoothly except the <code>torch.fft()</code>module that complains about MKL.</p>
<p>I was just wondering if there is some way to run torch.fft() without MKL or MKL<br>
is a prerequisite for it.</p>
How to check if an object is an LR scheduler?

timothygebhard: <p>Checking if something is a PyTorch optimizer is pretty straightforward:</p>
<pre><code class="lang-auto">isinstance(something, torch.optim.Optimizer)
</code></pre>
<p>I was wondering what is the best / preferred way to check if something is a <em>LR scheduler</em>?</p>
<p>There is a (protected) class <code>_LR_scheduler</code> in the <code>lr_scheduler.py</code>, but first of all not every type of LR scheduler inherits from it (e.g., <code>ReduceLROnPlateau</code>), and secondly it is kind of against Python conventions to type-check against a protected member of a module…</p>
<p>Anybody got some advice what is the best practice here? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<hr>
<p><strong>Edit:</strong> I gave it some more thought, and the best I came up with is to check if:</p>
<pre><code class="lang-auto">something.__module__ == 'torch.optim.lr_scheduler'
</code></pre>
<p>I guess this works, but it still feels a bit hacky, so if anyone’s got a better idea let me know!</p>

Kushaj: <p><code>torch.optim.lr_scheduler</code></p>

nico: <pre><code class="lang-python">
from torch.optim.lr_scheduler import _LRScheduler, ReduceLROnPlateau
# _LRScheduler can be imported, just ignore the error hint of your IDE
# The only exception is that ReduceLROnPlateau not inherits from _LRScheduler


isinstance(something, (_LRScheduler, ReduceLROnPlateau))
</code></pre>
Loss calculation within batch iteration

guijun_ren: <p>You can see there are two different loss calculation methods in following code snippet. The first approach is quite common and easy to understand, while the second one is a bit obscure to me. Can someone shed any lights on how to interpret this? Much appreciated.</p>
<pre><code>for batch_index, batch_dict in enumerate(batch_generator):
    ........
    y_pred = self.model(batch_dict['train'])
    loss = self.loss_func(y_pred, target)
    ## Approach One:
    running_loss += loss.item()  
    ## Approach Two:
    running_loss += (loss.item() - running_loss) / (batch_index + 1)
    ........</code></pre>

guijun_ren: <p>This problem has been resolved.</p>
<p>I derived a bit and figured those two loss calculation approaches are essentially the same.</p>

B_S: <p>can you explain how they are the same?</p>
`apply` function in custom extension not supported by TorchScript

moderato: <p>Hello! I’m trying to run compile my model that contains a custom op with TorchScript using the method in the following link:<br>
<a href="https://pytorch.org/tutorials/advanced/torch-script-parallelism.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/torch-script-parallelism.html</a></p>
<p>However, it failed with the following error</p>
<pre><code class="lang-auto">Python builtin &lt;built-in method apply of FunctionMeta object at 0x55ad125f0e18&gt; is currently not supported in TorchScript.
</code></pre>
<p>indicating the <code>apply</code> function in the custom extension code is not supported by TorchScript. I wonder if there’s any way to fix this? Also is there any plan to support <code>apply</code> in TorchScript recently?</p>
<p>Thanks in advance!</p>
Transformerencoderlayer init error

111412: <p>hello. I’m in trouble using TransformerEncoderLayer.</p>
<p>my torch version is 1.8.1+cu102</p>
<p>when I use batch_first, It shows error.</p>
<p>torch.nn.TransformerEncoderLayer(d_model=time_step, nhead=4, dropout=0.2, batch_first=True)</p>
<p>TypeError: <strong>init</strong>() got an unexpected keyword argument ‘batch_first’</p>

Shisho_Sama: <aside class="quote no-group" data-username="111412" data-post="1" data-topic="125805">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111412/48/31053_2.png" class="avatar"> 111412:</div>
<blockquote>
<p>TransformerEncoderLayer</p>
</blockquote>
</aside>
<p>1.8.1’s version does not take any <code>batch_first</code> argument (ref <a href="https://pytorch.org/docs/1.8.1/generated/torch.nn.TransformerEncoderLayer.html#transformerencoderlayer" class="inline-onebox" rel="noopener nofollow ugc">TransformerEncoderLayer — PyTorch 1.8.1 documentation</a> ), if you want that you need to upgrate to 1.9.0</p>
Requiers_grad is list in state_dict?

Roman_Girin: <p>Is it correct behavior that tensors with model’s weights in state_dict have requiers_grad value False even though in ‘original’ model the tensor’s requiers_grad value is True?<br>
Does the state_dict contain references to the tensors that in model or in state_dict just copies of the model’s tensors?</p>
<p>Here is code to recreate the case (I use pytorch 1.9.0 with CUDA 11.1 Win 10):</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F

# define some model
class TheModelClass(nn.Module):
    def __init__(self):
        super(TheModelClass, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return x

model = TheModelClass()
print(model.conv1.weight.requires_grad)
st = model.state_dict()
print(st['conv1.weight'].requires_grad)
</code></pre>
<p>This code outputs:<br>
True<br>
False</p>

ptrblck: <p>Yes, this is expected, as the model defines, if the parameters are frozen or not, while the <code>state_dict</code> stores their values.<br>
You can thus create a new model instance, freeze/unfreeze parameters as you want, and load the <code>state_dict</code> afterwards.</p>
Dynamically set Conv2d based on input channels

andrejankas: <p>I am currently doing a module that initializes a Conv2d and BatchNorm2d <strong>outside</strong> the <code>__init__</code>. Code works fine in cpu, but when moving module to cuda these <em>conv</em> and <em>bn</em> aren’t moved properly. Code:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

class ExpandChannels(nn.Module):
    def __init__(self, num_classes: int = None):
        super(ExpandChannels, self).__init__()
        self.num_classes = num_classes
        self.conv = None
        self.bn = None

    def reset_parameters(self, x):
        self.conv = nn.Conv2d(x.size(1), self.num_classes, kernel_size=1)
        self.bn = nn.BatchNorm2d(self.num_classes)

    def forward(self, x):
        if self.conv is None:
            self.reset_parameters(x)
        x = self.conv(x)
        x = self.bn(x)
        return x

m = ExpandChannels(100).cuda()
m(torch.randn(4, 3, 28, 28).cuda())
</code></pre>
<p><code>Error: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same</code></p>

ptrblck: <p>You are creating the new <code>self.conv</code> and <code>self.bn</code> layers inside the <code>forward</code> pass without specifying the device, so they will be created on the CPU by default.<br>
To properly push them to the GPU, you could use:</p>
<pre><code class="lang-python">    def reset_parameters(self, x):
        self.conv = nn.Conv2d(x.size(1), self.num_classes, kernel_size=1).to(x.device)
        self.bn = nn.BatchNorm2d(self.num_classes).to(x.device)
</code></pre>
<p>Additionally, you could also check the lazy modules, e.g. <a href="https://pytorch.org/docs/stable/generated/torch.nn.LazyConv2d.html"><code>LazyConv2d</code></a>, which perform a similar approach.</p>
Getting nan value loss with DCT transform

afsaneh_ebrahimi: <p>I want to replace average pooling in one architecture with DC part of DCT transformation, but when I replace this with each other I got nan values for loss.<br>
This is my DCT transform over all channels:</p>
<pre><code>def dct3(self, x):
    X1 = dct.dct_3d(x[0])
    X2 = dct.dct_3d(x[1])
    a = torch.stack([X1,X2], 3)
    for i in range((x.shape[0])-2):
        X = dct.dct_3d(x[i+2])
        X = X.unsqueeze(3)
        a = torch.cat([a,X], 3)
    a = a.reshape(x.shape[0],x.shape[1],7,7)
    a = a[:,:,1,1]
    a = a.reshape(x.shape[0],x.shape[1],1,1)
    return a   
</code></pre>
<p>And this is my forward path:</p>
<pre><code>def forward(self, x):
    x = self.first_conv(x)
    x = self.second(x)
    x = self.features(x)
    x = self.conv_last(x)
    #x = self.globalpool(x)
    #x = self.dct3(x)
    if self.model_size == '2.0x':
        x = self.dropout(x)
    x = x.contiguous().view(-1, self.stage_out_channels[-1])
    x = self.classifier(x)
    return x</code></pre>

ptrblck: <p>Could you post the code of <code>dct.dct_3d</code> or check, if you might be dividing by zero or any other operation which could cause invalid outputs?</p>
How to convert this pytorch neural net into nn.Sequential format?

torch_user_92: <p>Hi how’s it going? I have a simple neural net that takes in a convolutional layer, flattens the output, concats another observation to the output, and then passes it through two linear layers. I know how to do everything in nn.Sequential except for the concatenation. How would I structure this?</p>
<p>Here is the original neural net:</p>
<pre><code class="lang-auto">        self.conv1 = nn.Conv2d(in_channels=in_dim, out_channels=1, kernel_size=(1, 1), stride=1)
        self.fc1 = nn.Linear(flattened_dim+obs_dim,20)
        self.fc2 = nn.Linear(20,output_dim)

        def network(x,obs):
            x = self.conv1(x)
            x = torch.flatten(x)
            x = torch.cat((x,obs))
            x = self.fc1(x)
            x = self.fc2(x)

            return x
</code></pre>
<p>And I’d like to convert it to something like this:</p>
<pre><code class="lang-auto">        self.network = nn.Sequential(
                        nn.Conv2d(in_channels=in_dim, out_channels=1, kernel_size=(1, 1), stride=1),
                        nn.Flatten(start_dim=0),

                       '''Here I need to concatenate an observation with 
                         the output of nn.Flatten before the next layer'''

                        nn.Linear(flattened_dim+obs_dim,20),
                        nn.Linear(20,output_dim),
                    )
</code></pre>
<p>How would you go about implementing this?</p>

ptrblck: <p>While there might be a way to write a custom <code>nn.Module</code> accepting two inputs and somehow make sure the <code>nn.Sequential</code> contain won’t break, I think the best approach would still be to use your original model, which uses <code>torch.cat</code> in its <code>forward</code>.<br>
<code>nn.Sequential</code> containers are used for simple sequential models, while custom modules allow more flexibility, which seems to perfectly fit your use case.<br>
What’s the reason you want to transform it into an <code>nn.Sequential</code>?</p>
GaussianBlur transform causing error

Kaustubh_Kulkarni: <p>No other transform is giving me an error but using the GaussianBlur transform gives me this error:</p>
<pre><code class="lang-auto"> File "/home/kaustubh/test/basic_augment.py", line 41, in &lt;module&gt;
    main()
  File "/home/kaustubh/test/basic_augment.py", line 34, in main
    transformed_image = data_transforms(image)
  File "/home/kaustubh/Envs/ml/lib/python3.9/site-packages/torchvision/transforms/transforms.py", line 60, in __call__
    img = t(img)
  File "/home/kaustubh/Envs/ml/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/kaustubh/Envs/ml/lib/python3.9/site-packages/torchvision/transforms/transforms.py", line 1730, in forward
    return F.gaussian_blur(img, self.kernel_size, [sigma, sigma])
  File "/home/kaustubh/Envs/ml/lib/python3.9/site-packages/torchvision/transforms/functional.py", line 1238, in gaussian_blur
    t_img = to_tensor(img)
  File "/home/kaustubh/Envs/ml/lib/python3.9/site-packages/torchvision/transforms/functional.py", line 129, in to_tensor
    np.array(pic, mode_to_nptype.get(pic.mode, np.uint8), copy=True)
TypeError: __array__() takes 1 positional argument but 2 were given

</code></pre>
<p>Here is the code:</p>
<pre><code class="lang-auto">def main():
    data_transforms = transforms.Compose([
        transforms.ToPILImage(),
        transforms.GaussianBlur(kernel_size=(11, 11), sigma=(0.1, 2.0)),
    ])

    im_dir = './test/'
    im_save_dir = './test/'

    for img in tqdm(sorted(os.listdir(im_dir))):
        ext = img.split('.')[1]
        im_name = img.split('.')[0]

        if ext == 'jpg':
            im_path = os.path.join(im_dir, img)

            image = cv2.imread(im_path)
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

            transformed_image = data_transforms(image)

if __name__ == "__main__":
    main()
</code></pre>

ptrblck: <p>This seems to be the <code>PIL/Pillow</code> issue as described <a href="https://discuss.pytorch.org/t/quickstart-tutorial-array-takes-1-positional-argument-but-2-were-given/125550/2">here </a>.</p>
Model keeps changing results for evaluation

Sohaib: <p>I used a pretrained squeezenet model and retrained it on my data. While training I get a good training and validation accuracy.<br>
Now for inference, either I run my model on one file at a time or I use dataloader, the output changes completely (I use model.eval). One time is very good, and 4 or 5 times (almost) random results.<br>
I do the following:</p>
<pre><code class="lang-auto">model = torch.load('./models/SN_all.pth')
model.eval()

data_transforms = {
    'predict': transforms.Compose([
        transforms.RandomResizedCrop(input_size),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])
    }
dataset = {'predict' : datasets.ImageFolder('./data_test/val/', data_transforms['predict'])}
dataloader = {'predict': torch.utils.data.DataLoader(dataset['predict'], batch_size = 1, shuffle=False, num_workers=4)}
device = torch.device("cuda:0")

for inputs, labels in dataloader['predict']:
    inputs = inputs.to(device)
    output = model(inputs)
    output = output.to(torch.device('cpu'))
    index = output.data.numpy().argmax()
</code></pre>
<p>Do you see any issue in my code?<br>
Thank you in advance.</p>

Kapil_Rana: <p>Please check the results just with centercrop or resize</p>
<p>I feel this is the effect of RandomResizedCrop().<br>
because will be random, so results will also be random.</p>

Sohaib: <p>That’s right!<br>
I have completely missed that while it was clear!<br>
Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How can i use .pth file?

samm: <p>i have a classification model and need to extract features map from it by freezing some layers and i did it already and got</p>
<blockquote>
<p>checkpoints.pth</p>
</blockquote>
<p>now i need the features in a pickle file</p>
<p>the file started with</p>
<pre><code class="lang-auto">OrderedDict([('pos_embed1', tensor([[[-0.0053, -0.0039,  0.0004,  ...,  0.0012,  0.0061, -0.0006],
         [ 0.0091, -0.00.....
</code></pre>

ptrblck: <p><a href="https://pytorch.org/docs/stable/generated/torch.save.html"><code>torch.save</code></a> would also use <code>pickle</code> under the hood and would also zip it by default. You can specify the <code>pickle_module</code> and <code>pickle_protocol</code> via the arguments.</p>

samm: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="125739">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.save</p>
</blockquote>
</aside>
<p>Thanks a lot for replying , excuse me can you give me an example</p>
<p>i mean the file i have which is checkpoint.pth is supposed holding the features after freezing the layers … right ?<br>
i need to get a pickle file for those features , i searched a lot but couldn’t find the right structure</p>

ptrblck: <aside class="quote no-group" data-username="samm" data-post="3" data-topic="125739">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samm/48/36169_2.png" class="avatar"> samm:</div>
<blockquote>
<p>i mean the file i have which is checkpoint.pth is supposed holding the features after freezing the layers … right ?</p>
</blockquote>
</aside>
<p>I don’t know what this file is supposed to contain or how it was created, so you could check with the author of this file.</p>
<aside class="quote no-group" data-username="samm" data-post="3" data-topic="125739">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samm/48/36169_2.png" class="avatar"> samm:</div>
<blockquote>
<p>i need to get a pickle file for those features , i searched a lot but couldn’t find the right structure</p>
</blockquote>
</aside>
<p>Could you give an example, what a “pickle file” would mean in this context?<br>
Based on the previous output, the file contains an <code>OrderedDict</code> and you would be able to access its content by directly indexing it. However, I’m unsure what the necessity of a “pickle file” means and why the direct indexing isn’t working for you.</p>
Wrong solution by lstsq()?

Babaki: <p>The solution that I obtain from <code>torch.linalg.lstsq(A, B)</code> is different from what I obtain using <code>numpy.linalg.lstsq(A, B)</code>. More importantly, the value of the objective function for the former solution is significantly larger than the latter.</p>
<p>Moreover, in numpy I obtain the same solution using <code>numpy.linalg.lstsq(A, B)</code> and <code>numpy.linalg.pinv(A) @ B</code>. The solution obtained by <code>torch.pinverse(A) @ B</code> is closer to the numpy solution, but also different from the one obtained by <code>torch.linalg.lstsq(A, B)</code>.</p>
<p>Does this mean that <code>torch.linalg.lstsq()</code> is providing wrong solutions?</p>
<p>This <a href="https://gist.github.com/Behrouz-Babaki/c3db95c00ca61d9ad3a92a9528c8930b" rel="noopener nofollow ugc">gist</a> contains the data and scripts that demonstrate this inconcictency.</p>
Loading LSUN object images

quasar: <p>In PyTorch, when loading LSUN images, only a few hardcoded classes are accepted:<br>
<a href="https://pytorch.org/vision/stable/datasets.html#lsun" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/datasets.html#lsun</a></p>
<p>The accepted classes are:<br>
[‘bedroom’, ‘bridge’, ‘church_outdoor’, ‘classroom’,<br>
‘conference_room’, ‘dining_room’, ‘kitchen’,<br>
‘living_room’, ‘restaurant’, ‘tower’]</p>
<p>However, there are also 20 object categories like cars and horses, as shown on the official LSUN website: <a href="https://www.yf.io/p/lsun" class="inline-onebox" rel="noopener nofollow ugc">LSUN</a></p>
<p>How can we load these object classes instead?</p>
Transformers and language modeling tutorial questions

majkelpl: <p>Hello everyone!</p>
<p>I was following a tutorial on transformers in language modelling ( <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Language Modeling with nn.Transformer and TorchText — PyTorch Tutorials 1.9.0+cu102 documentation</a> ) and I came across a bunch of questions.</p>
<ol>
<li>
<p>What exactly does the particular model in this tutorial return? When I feed it with a sequence of N length (in one batch), it returns always a N x B x V array (where B is batch dim, and V vocab dim). When I do argmax on the V dimension and decode it back to words, I get an array of N words. Why N? If the last layer is Linear, shouldn’t I get just one word? After all, given a sequence, I care about words that are extension to that sequence.<br>
What exactly each word in the generated sequence mean? Are the first generated words based on first words in the input sequence and the last are based on whole input sequence? So if I want a next word prediction for input sequence, I should access only the last element in N dimension?</p>
</li>
<li>
<p>Also, if I’d like to get M words following N sequence, am I supposed to run the model M times, feeding it each time with a sequence N with attached newly generated words?</p>
</li>
<li>
<p>The model asks for mask in the forward function. If I understand it correctly, whether I skip it or not, when predicting with this model, the last word in the generated sequence should be the same, because generating the last word in the attention mechanism should have access to all tokens. However, it is not true for all cases. Why?<br>
Also, if I’d teach the model not by a sequence shifted by 1 right, but for example by 10 right, and the bptt also was 10, so there was no way for attention mechanism to access following words, do I understand it correctly that then it would be no point in masking?</p>
</li>
</ol>
<p>Thanks in advance for answers.</p>

majkelpl: <p>Really no one can help?</p>
Torchvision Semantic Segmentation

Anshumaan_Dash: <p>Hi,</p>
<p>I am trying to use the scripts in <code>https://github.com/pytorch/vision/blob/master/references/segmentation/train.py</code> to train a Semantic Segmentation model using a custom dataset of 6 classes (background-red, face-yellow, Eyes-blue, nose-green, hair-brown, lips-pink). I ned the background class as it does facial matting.</p>
<p>I have created a mask with 0 as default, 1-bg, 2-face,…, 6-lips. In the code, the cross entropy uses <code>ignore_index=255</code> which I changed to <code>ignore_index=0</code>.</p>
<p>It should run fine, but I get error as : <code>Target 255 is out of bounds</code> with <code>num_classes = 7</code></p>
<p>What am I doing wrong? I need help in understanding what the code base expects/how it works…</p>
<p>Thanks.</p>

ptrblck: <p>Based on the error message it seems your target tensor contains the class label 255, which is neither being ignored (via <code>ignore_index</code>) nor is a valid class index, as it’s out of bounds.<br>
It seems you want to map the class index 255 to 0 and ignore it and if that’s the case, the mapping seems to fail since values of 255 are still found.</p>

Anshumaan_Dash: <p>I found the issue. It was a faulty dataset.</p>
<p>My aim here is to have an understanding as to how the segmentation code works in the torchvision library. Now, there are 23 classes (0 being the unlabeled). So, I put <code>ignore_index=0</code> and <code>num_classess=23</code></p>
<p>The original images are 4000x6000. The network sees random crops of 480x480. The mask is a single channel of target class values (0-22). Obviously, some of the targets, because of cropping, do not have max value as 22.</p>
<p>Now, I get error : <code>target 22 is out of bounds</code></p>
<p>What am I missing?</p>

ptrblck: <p>The output of the model for a segmentation use case (assuming you are using <code>nn.CrossEntropyLoss</code>) should have the shape <code>[batch_size, nb_classes, height, width]</code>, even if not all classes are present in the current crop.<br>
In your case it would thus be <code>[batch_size, 23, height, width]</code>, where each channel would represent the logits for the corresponding class.<br>
Based on the error message it seems your channel dimension in the output is smaller than 23.</p>

Anshumaan_Dash: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thanks a lot for all the help. Really appreciate it.</p>
K-BS/breathsound-use-stethoscope/blob/main/torch/torch_pilot_model_1.ipynb

K-BS: <aside class="onebox githubblob" data-onebox-src="https://github.com/K-BS/breathsound-use-stethoscope/blob/main/torch/torch_pilot_model_1.ipynb">
  <header class="source">

      <a href="https://github.com/K-BS/breathsound-use-stethoscope/blob/main/torch/torch_pilot_model_1.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/K-BS/breathsound-use-stethoscope/blob/main/torch/torch_pilot_model_1.ipynb" target="_blank" rel="noopener nofollow ugc">K-BS/breathsound-use-stethoscope/blob/main/torch/torch_pilot_model_1.ipynb</a></h4>


    <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4c2fc57b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import librosa\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import librosa.display\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from tqdm import tqdm_notebook as tqdm\n",
</code></pre>


  This file has been truncated. <a href="https://github.com/K-BS/breathsound-use-stethoscope/blob/main/torch/torch_pilot_model_1.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
PreTrained Model has no attribute 'copy'

krronk: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/ddbd2524194e55c0e4a1de51c382e45c00565c82.png" data-download-href="https://discuss.pytorch.org/uploads/default/ddbd2524194e55c0e4a1de51c382e45c00565c82" title="Screenshot 2021-06-25 024222"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/ddbd2524194e55c0e4a1de51c382e45c00565c82_2_690x310.png" alt="Screenshot 2021-06-25 024222" data-base62-sha1="vDALgOtpoBIQu29b0Lv7HzuqPw6" width="690" height="310" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/ddbd2524194e55c0e4a1de51c382e45c00565c82_2_690x310.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/ddbd2524194e55c0e4a1de51c382e45c00565c82_2_1035x465.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/ddbd2524194e55c0e4a1de51c382e45c00565c82_2_1380x620.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-06-25 024222</span><span class="informations">1914×861 73.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>the mobilenet model was trained in an earlier session and downloaded to local machine , it shows this error when inference is carried out at another instance<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9c51416a20625d1f7f063dbb09ba8bd3a7323db.png" data-download-href="https://discuss.pytorch.org/uploads/default/f9c51416a20625d1f7f063dbb09ba8bd3a7323db" title="Screenshot 2021-06-25 030645"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9c51416a20625d1f7f063dbb09ba8bd3a7323db.png" alt="Screenshot 2021-06-25 030645" data-base62-sha1="zDz7ZaKbh4uhvoNch9N4k5PRGKD" width="690" height="90" data-dominant-color="F8F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-06-25 030645</span><span class="informations">1589×209 5.23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i trained and saved another instance and the error message is the same</p>

ptrblck: <p>You are currently saving the string <code>'model.state_dict()'</code>, not the real <code>state_dict</code> object.<br>
Use <code>torch.save(model.state_dict(), 'mobilenet_68.pth')</code> instead.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier and allows the forum search to index the post. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

krronk: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> , that was really stupid from me</p>
Simultaneously multiple block indexing in a torch tensor

Amine08: <p>Hi,</p>
<p>what is the fastest method to index simultaneously multiple blocks in a torch tensor to modify them?</p>
<p>Here an example: i take the first two rows after every 4 rows and modify them:</p>
<pre><code class="lang-auto">a=torch.rand(8,8)
a[0:2,:]=2*a[0:2,:]
a[4:6,:]=2*a[4:6,:]
</code></pre>
<p>I want to avoid the for loop to reduce the execution time.</p>
<p>Thanks!</p>

onomatopoeia: <aside class="quote no-group" data-username="Amine08" data-post="1" data-topic="125725">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amine08/48/30330_2.png" class="avatar"> Amine08:</div>
<blockquote>
<pre><code class="lang-auto">a=torch.rand(8,8)
a[0:2,:]=2*a[0:2,:]
a[4:6,:]=2*a[4:6,:]
</code></pre>
</blockquote>
</aside>
<p>Your solution works fine.</p>
<pre data-code-wrap="In"><code class="lang-In">
In [2]: a=torch.ones(8,8)
   ...: a[0:2,:]=2*a[0:2,:]
   ...: a[4:6,:]=2*a[4:6,:]

In [3]: a
Out[3]: 
tensor([[2., 2., 2., 2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2., 2., 2., 2.],
        [1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1.],
        [2., 2., 2., 2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2., 2., 2., 2.],
        [1., 1., 1., 1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1., 1., 1., 1.]])

In [4]: 
</code></pre>
<p>Using a simple timing function (e.g. importing <code>time</code> ) you can measure the speed both ways.</p>

Amine08: <p>right, i have a huge matrix and i want the modify the rows in parallel and to avoid the for loop. So i am looking for a better solution.</p>

Amine08: <p>i got it!</p>
<p>Instead of using the for loop for the whole matrix, i just have to use it for the block_matrix size</p>
<pre><code class="lang-auto">a=torch.ones(8,8)
print(a)
block_size=2
for i in range(block_size):
    a[i::4]=2*a[i::4]
print(a)
</code></pre>
Creating tensor directly from bytes?

John_Carmack: <p>The tensor constructor doesn’t accept the ‘bytes’ data type, so when I read raw image data from a file, I wind up going through numpy frombuffer just to get it into an acceptable format.</p>
<p>frameBytes = rgbFile.read(frameSize)<br>
frameTensor = torch.tensor(np.frombuffer(frameBytes, dtype=np.uint8), device=dev)</p>
<p>Is there a better way to do this, or should torch.tensor() get modified to accept bytes as a uint8 data source?</p>

tom: <p>If you use <code>torch.from_numpy</code>, this will be without copy and should be good enough. On the C++ side there is <code>torch::from_blob</code> but I it is not yet mapped to a Python frombuffer, <a href="https://github.com/pytorch/pytorch/pull/59077" rel="noopener nofollow ugc">this is ongoing work in PR 59077</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Pytorch C++ for raspberry pi 3

dlam0002: <p>Hi,</p>
<p>I’m trying to install pytorch c++ for my raspberry pi 3 B+. I’ve been unsuccessful with trying to get it installed. Would anyone know where to start with this?</p>
<p>Cheers</p>

tom: <p>If you’re on 64 bit (I would recommend that), you could take <a href="http://mathinf.com/pytorch/arm64/2021-01/" rel="noopener nofollow ugc">my Python distribution</a> and just ignore the Python bits. Or you could grab the arm64 package from <a href="http://Pytorch.org" rel="noopener nofollow ugc">Pytorch.org</a> and then ignore the Python bits.<br>
I still need to update to 1.9.</p>
<p>Best regards</p>
<p>Thomas</p>