akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Accuracy of nn.NLLLoss

shell-escape: <p>I am trying to use nn.NLLLoss, so the code is:</p>
<pre><code class="lang-auto">loss = nn.NLLLoss()
loss_by_torch = loss(predictions_logp, actual_tokens)
</code></pre>
<p>There is another method to compute it:</p>
<pre><code class="lang-auto">loss_by_gather = -torch.mean(torch.gather(predictions_logp, dim=1, index=actual_tokens[:,None]))
</code></pre>
<p>And another one by using a function like this:</p>
<pre><code class="lang-auto">def compute_NLLLoss(logs, targets):
    out = torch.zeros_like(targets, dtype=torch.float)
    for i in range(len(targets)):
        out[i] = logs[i][targets[i]]
    return -torch.mean(out)

loss_by_func = compute_NLLLoss(predictions_logp, actual_tokens)
</code></pre>
<p>So, loss_by gather and loss_by_func have absolutely the same values, while loss_by_torch differs.<br>
In one minibatch the shape of predictions_logp is [87312, 85], and the difference between loss_by_gather (or loss_by_func) and loss_by_torch varies from -0.001 to 0.001.<br>
Does torch implementation use different formulae?</p>
Pretrained model used for same test data

Hala_Hamdoun: <p>I create a LSTM model consist of 2 layers and output layer. ten i predict test data on it and compute mape error.<br>
I saved this model to be used as pre-trained model and freeze the two layer then reuse the dense layer to predict same test data but got different mape error.<br>
Is not logic that i have same value of error.</p>
<p>trainx3, trainy3, testx3 are the same for pretained  saved model</p>
<pre><code class="lang-auto">new_model = tf.keras.models.load_model('saved_model/my_model')
new_model.summary()


#new_model.trainable=False

new_model.get_layer('lstm').trainable = False
new_model.get_layer('lstm_1').trainable = False


#yhat3 = new_model.predict(test_X3, batch_size=1)
base_inputs=new_model.layers[0].input

base_outputs=new_model.layers[-2].output

final_outputs=layers.Dense(56)(base_outputs)

now_model=keras.Model(inputs=base_inputs, outputs=final_outputs)
now_model.compile(loss='mean_squared_error', optimizer='adam')
now_model.fit(train_X3, train_y3, epochs=5, batch_size=1,verbose=2)


yhat3 = now_model.predict(test_X3, batch_size=1)
</code></pre>

InnovArul: <p>This seems like a question about a Keras model.<br>
Are you sure you are asking it in the correct forum?</p>

Hala_Hamdoun: <p>oh … i should not ask in pytorch<br>
so sorry <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Network inference with multiple inputs

cRis: <p>Hi,<br>
I have an application for a network which must receives multiple inputs (N &gt; 3). So, I have an input tensor of (batch size, no channels, N, w, h). In my network, I have to extract features from each image from those N and to feed them into an LSTM. It is correct to write the network as below (especially the forward method) ?</p>
<p>I suspect that the training process is not performed well. Do you think the forward method is ok?</p>
<pre><code class="lang-auto">from networks.ResNet import resnet18
import torch.nn as nn
import torch


class ReccurentNet(nn.Module):

    def __init__(self, seq_len, n_features, hidden_dim=64):
        super(ReccurentNet, self).__init__()

        self.seq_len, self.n_features = seq_len, n_features

        self.rnn1 = nn.LSTM(
            input_size=n_features,
            hidden_size=hidden_dim,
            num_layers=1,
            batch_first=True
        )

    def forward(self, x):
        x, (_, _) = self.rnn1(x)
        x, (hidden_n, _) = self.rnn2(x)

        return hidden_n


class ConvRecNet(nn.Module):
    def __init__(self, hidden_dim=128):
        super(ConvRecNet, self).__init__()
        self.backbone = resnet18() # backbone features extractor, output 512 features

        self.rnn = nn.LSTM(
            input_size=512,
            hidden_size=hidden_dim,
            num_layers=5,
            batch_first=True
        )

        self.activation = nn.LeakyReLU()
        self.fc = nn.Linear(128, 24)

    def forward(self, x):
        windows = []
        for i in range(0, x.shape[2]):
            windows.append(self.backbone(x[:, :, i, :, :]))

        x = torch.stack(windows)
        x, _ = self.rnn(x)
        x = x[-1] # Take the last output from rnn, is it  ok ?

        x = self.activation(x)
        x = self.fc(x)
        return x
</code></pre>
<p>Thank you!</p>
How to set inequality constraints in my loss function

amidher: <p>Suppose I have a neural network with 5 nodes in the output layers.</p>
<p>I have a custom loss function but I want to add another term to the loss function that satisfies the following condition:</p>
<p><strong>0 &lt;= out_Node1 &lt;= out_Node2 &lt;= out_Node3 &lt;= out_Node4 &lt;= out_Node5 &lt;= 100</strong></p>
<p>any thoughts on how to achieve this?</p>
Cyclic Loss (distance between classes)

Bues: <p>Hi there,</p>
<p>I am getting started with PyTorch, and I attempting to fine-tune a network to classify containing orientation data (pictures of pedestrians mainly). In particular, I clustered orientation into 24 bins, each of these belonging to a portion of 15º (from 0 to 360), and I have the true labels for all pedestrians to train the classifier. In particular, the last layer of my model is a FC layer with a 1x24 output.</p>
<p>By using a cross-entropy loss the obtained results are acceptable, but atm I am not taking into account the distance between classes: i.e. the bin number 0 (from 0-15º) is close to the bin number 1 (15-30º), but it is also close to the 23rd bin (345-360º).</p>
<p>My question is: is there any kind of implemented loss function that can take into account this cyclic feature and uses the distance between classes?<br>
Thanks,</p>

suraj.pt: <p>Hello,</p>
<p>You could use modulo arithmetic to calculate distance between classes</p>
<pre><code class="lang-auto">c1 = (x - y)%23
c2 = (y-x)%23
min_dist = min(c1,c2)
</code></pre>
<p>Since you’re comparing orientations, you <em>might</em> be able to use <a href="https://pytorch.org/docs/stable/generated/torch.nn.CosineSimilarity.html" rel="noopener nofollow ugc">cosine similarity</a> directly without binning.</p>
How to use pre-trained weights for 5 channels?

honuact: <pre><code>if args.init_weights:
    model_dict = model.state_dict()
    snapshot = torch.load(args.init_weights, map_location='cpu')
    snapshot = snapshot['model']
    model_dict.update(snapshot)
    model.load_state_dict(model_dict)
</code></pre>
<p>RuntimeError: Error(s) in loading state_dict for ModelSpatioTemporalTCN:<br>
size mismatch for conv1_scene.weight: copying a param of torch.Size([64, 5, 7, 7]) from checkpoint, where the shape is torch.Size([64, 4, 7, 7]) in current model.</p>
<pre><code>    self.conv1_scene = nn.Conv2d(**5**, 64, kernel_size=7, stride=2, padding=3, bias=False)
    self.bn1_scene = nn.BatchNorm2d(64)
    self.layer1_scene = self._make_layer_scene(block, 64, layers_scene[0])
    self.layer2_scene = self._make_layer_scene(block, 128, layers_scene[1], stride=2)
    self.layer3_scene = self._make_layer_scene(block, 256, layers_scene[2], stride=2)
    self.layer4_scene = self._make_layer_scene(block, 512, layers_scene[3], stride=2)
    self.layer5_scene = self._make_layer_scene(block, 256, layers_scene[4], stride=1)
</code></pre>
<p>I need to use 5 channels, how to solve it?</p>
Replace part value of matrix with new value

zhihao_xing: <p>For simple cases, Given a [2,2] tensor，<br>
[[1,1],<br>
[1,1]]<br>
we can use “scatter” function to replace index [0,1] and [0,0] with new value (such as zero),<br>
[[0,0],<br>
[1,1]];<br>
but if given a [2,2,2] tensor<br>
[[[1., 1.],<br>
[1., 1.]],<br>
[[1., 1.],<br>
[1., 1.]]]<br>
, how can I achieve the same operation as above with scatter funtion?<br>
[[[0, 0],<br>
[1, 1]],<br>
[[0, 0],<br>
[1, 1]]]</p>

ptrblck: <p>You can directly index the tensor as seen here:</p>
<pre><code class="lang-python">x = torch.ones(2, 2, 2)
idx = torch.tensor([0])
x[:, idx] = 0.
print(x)
&gt; tensor([[[0., 0.],
           [1., 1.]],

          [[0., 0.],
           [1., 1.]]])
</code></pre>
Looking for advice regarding schedulers: learning rate and milestones

TenenteColombian: <p>Hello everyone!</p>
<p>I am quite new at PyTorch, still trying to figure out all the <em>ins and outs</em>. I am currently working on my M.Sc. Mathematics Thesis and I was wondering if anyone could give me some advice on setting milestones for my scheduler: what are your <em>go-to solutions</em> when trying to set your scheduler (e.g., how many and which milestones, the learning rate, etc.)? I know that there’s no perfect recipe when it comes to Neural Networks, but I was wondering if there is a smart way (besides brute force) to understand what your data might need. Right now I am working with medical images (PET and CT) and I started <em>playing</em> with the MultiStepLR scheduler.</p>
<p>Any advice is very much appreciated. Thank you very much in advance.</p>
<p>Sincerely,</p>
<p>Nicolas Destefanis</p>

ptrblck: <p>I don’t know if there are good approaches of setting milestones besides running a lot of experiments.<br>
As the default I would try to reduce the learning rate once the validation loss doesn’t decrease significantly anymore via <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau"><code>lr_scheduler.ReduceLROnPlateau</code></a>.</p>
Indexing a Tensor's Second Dimension

ptrmcl: <p>So I have this tensor <strong>t</strong> of size a x b x c</p>
<p>I have another tensor <strong>u</strong> of size a. Each element of <strong>u</strong> is an index of the element I want from <strong>t</strong>’s second dimension. So I want to apply the mask and get something that’s a x c.</p>
<p>How can I do this? Please let me know if I can provide any clarifications.</p>

ptrblck: <p>This code should work:</p>
<pre><code class="lang-python">a, b, c = 2, 3, 4
t = torch.arange(a*b*c).view(a, b, c)
print(t)
&gt; tensor([[[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11]],

          [[12, 13, 14, 15],
           [16, 17, 18, 19],
           [20, 21, 22, 23]]])

u = torch.tensor([0, 2])

res = t[torch.arange(a), u]
print(res)
&gt; tensor([[ 0,  1,  2,  3],
          [20, 21, 22, 23]])
</code></pre>
Torchaudio.functional.vad()

Kla: <p>Hi everyone, I’m trying to implement torchaudio.functional.vad() but I’m having problems with it. First it takes to much time to work. Is there any method how to filter all the data and then save them? I mean to filter the data once and every time I want to train my modell just to use the filtered one. <a href="https://pytorch.org/audio/stable/functional.html#vad" rel="noopener nofollow ugc">Here</a> stays: The effect can trim only from the front of the audio, so in order to trim from the back, the reverse effect must also be used.<br>
I’m not understanding how to trim from the back. Can someone please help me?</p>
What is the correct way to use torchvision.io.video.read_video with transforms.Normalize?

ThaiThien: <p>I load video frame by frame with</p>
<pre><code class="lang-auto">from torchvision.io.video import read_video
v, _, _ = read_video(video_path, pts_unit='sec')
</code></pre>
<p>Because I use each frame of video to predict on model train with image, I need to normalize.</p>
<pre><code class="lang-auto">transform = transforms.Compose([
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225]),
            ])
frame = transform(frame)
</code></pre>
<p>However, as video return uint8, it cannot normalize</p>
<p><code>ValueError: std evaluated to zero after conversion to torch.uint8, leading to division by zero.</code></p>
<p>I need to cast to float.</p>
<pre><code class="lang-auto">transform = transforms.Compose([
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225]),
            ])
frame = frame.float()
frame = transform(frame)
</code></pre>
<p>do the type cast here is the correct way to do so in this video case ?</p>
<p>For image case, I load with PIL and use <code>transforms.ToTensor()</code> so I don’t have to worry about int.</p>

ptrblck: <p>Wouldn’t the <code>ToTensor()</code> transformation also work for your <code>uint8</code> video frames or are you seeing another error?<br>
Just transforming to <code>float()</code> sounds wrong, as the values would still be in the range <code>[0, 255]</code>, while you are using <code>mean</code> and <code>std</code> stats for inputs in the range <code>[0, 1]</code>, which will be the value range after using <code>ToTensor()</code> in <code>unit8</code> inputs.</p>
How send extracted features from CNN to an LSTM as an input

rubijade: <p>Hello guys.<br>
I am new to PyTorch and I am implementing a two-stream action recognition architecture.<br>
For each of them, I have a CNN and at the top an Lstm.<br>
My question is how can I pass the feature extracted from the CNN to the LSTM as input. I am confused a little bit.<br>
Any guidance from you guys will be so helpful for me. Thank you</p>

ptrblck: <p>Or course it depends on your actual use case, but you could treat the height and width of the CNN output as the temporal dimension and the channels as the features and pass it to the RNN.<br>
By default the RNN modules expect an input in the shape <code>[seq_len, batch_size, features]</code> and you could use <code>batch_first=True</code> to pass it as <code>[batch_size, seq_len, features]</code> (the docs give you all expected shapes).</p>
In CBOW code, why has 2 linear

zhangsunsuochang: <pre><code class="lang-auto">class CBOWModeler(nn.Module):
    def __init__(self, vocab_size, embedding_dim, context_size):
        super(CBOWModeler, self).__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear1 = nn.Linear(context_size*embedding_dim, 128)
        self.linear2 = nn.Linear(128, vocab_size)
        
    def forward(self, x):
        embeds = self.embeddings(x).view(1,-1)
        output = self.linear1(embeds)
        output = F.relu(output)
        output = self.linear2(output)
        log_probs = F.log_softmax(output, dim=1)
        return log_probs
</code></pre>
<p>According to the paper, I think there should be 1 linear, but the code has 2, can anyone help answer??</p>
How i can find best learning rate?

utkuumetin: <p>I waste so much time to find best learning rate for model. Is there any fast way other than random vars, i usually look loss graph to adjust learning rate.</p>

Toby: <p>You can:<br>
1/ Use the adaptive learning rate optimization method such as ADAM family<br>
2/ A dynamic way to fine-tune learning rate while training (example: cycle learning rate: <a href="https://arxiv.org/abs/1506.01186" rel="noopener nofollow ugc">https://arxiv.org/abs/1506.01186</a>)<br>
3/ Switch the optimization while training with the best learning rate (<a href="https://arxiv.org/abs/1712.07628" rel="noopener nofollow ugc">https://arxiv.org/abs/1712.07628</a>)</p>

utkuumetin: <p>Thanks for help. I’m going to read papers.</p>
How to compress [B,C,H,W] to [B,H,W]

akcaymert: <p>i have b,c,h,w image and i want to b,h,w size for segmentation  without loss of information.i try to squeeze func but not working.<br>
thanks for comments</p>

KFrank: <p>Hi Akcaymert!</p>
<aside class="quote no-group" data-username="akcaymert" data-post="1" data-topic="105960" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7c8e57/48.png" class="avatar"> akcaymert:</div>
<blockquote>
<p>i have b,c,h,w image and i want to b,h,w size for segmentation  without loss of information.</p>
</blockquote>
</aside>
<p>You can’t.  <code>[B, C, H, W]</code> consists of <code>B * C * H * W</code> numbers, while<br>
<code>[B, H, W]</code> consists of <code>B * H * W</code>, which is fewer (assuming that <code>C &gt; 1</code>).</p>
<p>So, unless your original <code>[B, C, H, W]</code> data have some substantive<br>
redundancy that you haven’t told us about, your desired compression<br>
will lose information.</p>
<p>Best.</p>
<p>K. Frank</p>

akcaymert: <p>I think I expressed it wrong. actually, i have masking image (3xwxh).  tried combining all the masked pictures for example<br>
(tab is not important)</p>
<pre><code class="lang-auto">image = np.zeros((predicted.size()[0],224,224))
            for i in range(predicted.size()[0]): # i is a batch 
              for j in range(predicted.size()[1]): # j is a masking dim 
                image[i] += predicted[i,j].detach().numpy()
            predicted = torch.tensor(image,requires_grad=True)
</code></pre>
How to work with large training set when dealing with auto-encoders on google colaboratory?

OuisYasser: <p>hello, I dont know I f I can post random questions rather than pytorch ones.</p>
<p>I am training an auto-encoder (keras) on google colab. however, I have 25000 input image and 25000 output image. I tried to: 1- copy the large file from google drive to colab each time (takes 5-6 hours). 2- convert the set to numpy array but when normalizing the images, the size get a lot bigger (from 7GB to 24GB for example) and then I can not fit it into the ram memory. 3- I can not zip and unzip my data. So please, if anyone knows how to convert it into numpy array( and normalize it) without having large file(24GB).</p>

ptrblck: <p>The question seems to be rather Keras/TF-specific and I think you would get the best support on e.g. StackOverflow. I’m unfortunately not deeply familiar with Keras and don’t know, how the data loading (lazy loading) can be implemented.</p>
Validation loss oscillates a lot, validation accuracy > learning accuracy, but test accuracy is high. Is my model overfitting?

duygusar: <ol>
<li>I am training a model, and using the original learning rate of the author (I use their github too), I get a validation loss that keeps oscillating a lot, it will decrease but then suddenly jump to a large value and then decrease again, but never really converges as the lowest it gets is 2 (while training loss converges to 0.0 something - much below 1)</li>
</ol>
<p>At each epoch I get the training accuracy and at the end, the validation accuracy. Validation accuracy is always greater than the training accuracy.</p>
<p>When I test on real test data, I get good results, but I wonder if my model is overfitting. I expect a good model’s val loss to converge in a similar fashion with training loss, but this doesn’t happen and the fact that the val loss oscillates to very large values at times worries me.</p>
<ol start="2">
<li>Adjusting the learning rate and scheduler etc etc, I got the val loss and training loss to a downward fashion with less oscilliation, but this time my test accuracy remains low (as well as training and validation accuracies)</li>
</ol>
<p>I did try a couple of optimizers (adam, sgd, adagrad) with step scheduler and also the pleateu one, I played with step sizes etc. but it didn’t really help, neither did clipping gradients.</p>
<ol>
<li>Is my model overfitting?</li>
<li>If so, how can I reduce the overfitting besides data augmentation?</li>
<li>If not (I read some people commented it is nothing to worry about to similar cases, but I think ideally the model should converge), If it is really okay, how can I justify it? Even if I would get similar results for a k-fold experiment, would it be good enough? I don’t feel it would justify the oscillating. How should I proceed?</li>
</ol>

moreshud: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Your input will be appreciated</p>

ptrblck: <p>I think the oscillating behavior of the validation accuracy might depend on its size. I.e. if the validation dataset is “small”, it would yield noisy estimates.<br>
However, it’s hard to quantify what a “small” dataset size would be and it would be interesting to see your training, validation, and test sizes. As a quick debugging step, you could also swap the training and validation datasets and check how the losses behave.</p>
Row-wise or of shifted boolean 1-D tensors

seewoo5: <p>I have a 1-D boolean tensor, let’s say, <code>a = [True, False, False, False, True, False, True, False, False, False]</code>. What I want to do is, for given <code>N</code>, I want to find bitwise_or of the tensors <code>a, a[1:] + [False], a[2:] + [False, False], ..., a[N-1:] + [False, ..., False]</code> (with abuse of notations). For example, when <code>N=1</code>, then the answer would be bitwise_or of two tensors<br>
<code>[True, False, False, False, True, False, True, False, False, False]</code><br>
<code>[False, False, False, True, False, True, False, False, False, False]</code>,<br>
so is<br>
<code>[True, False, False, True, True, True, True, False, False, False]</code>.<br>
I can do this with using a for loop over <code>N</code>, by</p>
<pre><code class="lang-auto">for i in range(1, N):
    a = a | torch.cat(a[i:], torch.BoolTensor([False] * i))
</code></pre>
<p>but I wonder if there’s more efficient way to do this. Also, I want if there’s a way to do the same thing for numpy array.</p>

Usama_Hasan: <p>hy <a class="mention" href="/u/seewoo5">@seewoo5</a>, you can do something like this</p>
<pre><code class="lang-auto"> b = torch.tensor((True,False))
 c = torch.tensor((True,True))
j = torch.tensor(b.clone().detach() | c.clone().detach() )
print(j)
tensor([True, True])

</code></pre>
Error building from source on windows

Kareem_Shaik: <p>How to resolve this error:</p>
<p>error C2039: ‘FuncPtr’: is not a member of ‘c10::impl::detail::with_scattere<br>
d_tensor_options&lt;FuncPtr,void&gt;’</p>
<p>pytorch\aten\src\ATen/core/op_registration/hacky_wrapper_for_legacy_signatures.h(109): error C2039: ‘FuncPtr’: is not a member of ‘c10::impl::detail::with_scattere<br>
d_tensor_options&lt;FuncPtr,void&gt;’</p>

peterjc123: <p>Please describe some details about your specs and the steps.</p>

Kareem_Shaik: <p><a class="mention" href="/u/peterjc123">@peterjc123</a> thanks for your reply. I stopped building  source using msvc2017. Recently in last week I got new msvc2019 licensed version with this I am not facing any issue building pytorch source on windows.</p>
<p>But there is one prb, after build and installing pytorch on windows in my local virtual env, I tried to run simple python visual studio project.  I am seeing one error while doing " import torch"<br>
Error:<br>
Could not convert to integer: 3221225477. path exitcode.<br>
value was either too large or too small for an Int32.</p>
<p>any reason why is this error?</p>

Kareem_Shaik: <p>Any comments on this issue?</p>
Docker Image 1.7.1

zerotosingularity: <p>Is there an official Docker image for the current stable 1.7.1 on Docker hub?</p>
<p>Last update was 2 months ago for 1.7.0:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://hub.docker.com/r/pytorch/pytorch/tags?page=1&amp;ordering=last_updated" target="_blank" rel="noopener nofollow ugc">hub.docker.com</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://hub.docker.com/r/pytorch/pytorch/tags?page=1&amp;ordering=last_updated" target="_blank" rel="noopener nofollow ugc">Docker Hub</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
RuntimeError: expected scalar type Double but found Float

Tomash: <p>Hi,<br>
I got an error like in the tittle and it sounds easy (I wonder that I just need to change type of loaded data) but I have a problem with that… I changed types but error still appears.</p>
<p>In my main function:</p>
<pre><code class="lang-auto">inputs = load_images(glob.glob(args.input))
outputs = predict(model, inputs)
</code></pre>
<pre><code class="lang-auto">

def load_images(image_files):
    loaded_images = []
    for file in image_files:
        img = Image.open(file)
        new_size = (640, 480)
        img = img.resize(new_size)
        x = np.clip(np.asarray(img) / 255, 0, 1).astype(np.float64)
        loaded_images.append(torch.DoubleTensor(torch.from_numpy(x)))
    return np.stack(loaded_images, axis=0)
</code></pre>
<pre><code class="lang-auto">def predict(model, images, minDepth=10, maxDepth=1000):
    # Support multiple RGBs, one RGB image, even grayscale
    if len(images.shape) &lt; 3:
        images = np.stack((images, images, images), axis=2)
    if len(images.shape) &lt; 4:
        images = images.reshape((1, images.shape[0], images.shape[1], images.shape[2]))
    # Compute predictions
    images_tensor = []
    for i in images:
        i = torch.DoubleTensor(i)
        i = i.permute(2, 0, 1)
        images_tensor.append(i)
    images_tensor = torch.stack(images_tensor)
    predictions = model(images_tensor.double())
    return (
        np.clip(DepthNorm(predictions, maxDepth=maxDepth), minDepth, maxDepth)
        / maxDepth
    )
</code></pre>
<p>Ideas?</p>

ptrblck: <p>This error might be raised, if your model parameters are in <code>float32</code> while the input is in <code>float64</code>.<br>
Based on your code snippet you are indeed passing the input as <code>DoubleTensor</code>s (<code>float64</code>) so you would need to make sure the model parameter have the same <code>dtype</code> via <code>model.double()</code>.<br>
Alternatively, you could also cast the input to <code>FloatTensor</code>s via <code>input = input.float()</code>.</p>
How does strided convolution work?

conrad: <p>This is an extremely basic question, but I can’t find an answer anywhere. Let’s say I take Conv2d(nin, nout, kernel_size=1, stride=2). What are the coordinates of the pixels that get the convolution operation applied? Does it start at the top left with coordinate (0, 0) and then move to (0, 2)? Or does it start at coordinate (1, 1) and then move to (1, 3)?</p>

JamesDickens: <p>I believe it would start at (0,0) and then move to (0,2). You can check this out by creating your own conv2d layer, inputting a simple array like [[1,2,3], [4,5,6], [7,8,9] to torch.tensor as your input image (so 1 channel in and 1 channel out), and then setting the weight of the conv layer to be all 1s and its bias to be 0, and then observing the result, which I believe should be [[1,3], [7, 9]].</p>

conrad: <p>That works. I confirmed that it does start at (0, 0).</p>
<p>Thanks for the help!</p>
An inquiry about computing classification accuracy with output in C++

Chen0729: <p>Hi, All</p>
<p>I have an inquiry about computing the classification accuracy with output and ground truth in C++,<br>
in python it is:<br>
acc = torch.mean((output_label.max(1)[1] == y).float())</p>
<p>where output_label is of the size 64 by 10, here 64 is the batch size and 10 is the number of classes, and y is of the size 64.</p>
<p>In C++, I convert it to be<br>
auto labelmax=output_label.max(1);</p>
<pre><code>    auto acc=torch::mean(get&lt;1&gt;(labelmax)==y);
</code></pre>
<p>but I get a running error when computing acc as labelmax is a tuple, saying :</p>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  Can only calculate the mean of floating types. Got Bool instead.<br>
Exception raised from mean_out_cpu_gpu at /pytorch/aten/src/ATen/native/ReduceOps.cpp:507</p>
<p>Any suggestions to fix that please ?</p>
<p>Thanks.</p>

glaringlee: <p><code>get&lt;1&gt;(labelmax)==y</code> returns a boolean tensor, but torch::mean support floating type tensor only, you have to convert this boolean tensor into float type, just like what you did in python.</p>
<p>Try this:<br>
auto acc=torch::mean((get&lt;1&gt;(labelmax)==y)).to(torch::kFloat))</p>

Chen0729: <p>Thanks.</p>
<p>I revised with your comments and it works. Very helpful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Using transformers (BERT, RoBERTa) without embedding layer

tueboesen: <p>I’m looking to train a RoBERTa model on protein sequences, which is in many ways similar to normal nlp training, but in others quite different.</p>
<p>In the language of proteins, I have 20 characters instead of the normal 26 characters used in english (it is 26 right? :D), so that is rather similar. The big difference is that you don’t really combine the characters in proteins to actual words, but rather just keep each character as a distinct token or class.</p>
<p>Hence essentially my input to the transformer model could just be a list of numbers ranging from 0-19. However that would mean that my input would only have 1 feature if I did that, and I’m not sure a transformer could work with that?</p>
<p>I’m thinking of just doing a onehot encoding of these characters, which would give me 20 input features. However this is of course still very low in comparison to how normal transformers are trained, where d_model is somewhere in the range of 128-512 if I understand correctly.<br>
Does anyone have any experience with anything like this? any good advice for how it is most likely to work?</p>
Run a network twice before backpropagating

Valerio_Biscione: <p>I am trying to understand better how autograd works. Let’s say that I have 2 different batch of images, each of size <code>b</code>: <code>x1</code> and <code>x2</code>, and a general model that outputs a a tensor <code>b x 64</code>. Let’s say that I also have another general model <code>r</code> which output a tensor <code>b x 1</code>. Let’s say that I do something like</p>
<pre><code class="lang-auto">o1 = m(x1)
o2 = m(x2)
output = r(o2 - o1)
loss = loss_fn(output, labels)
loss.backward()
</code></pre>
<p>Notice how m is used twice, sequentially, on two different batches. My question is: when the gradient is backpropagated, is the derivative going to be computed with respect to o1, o2, or both? In the latter case, how would it work?</p>

yoad.tewel: <p>The derivative will compute with respect to both o1 and o2.<br>
Lets call d = o2 - o1.<br>
The derivative of o1 will be minus the gradient of the loss with respect to d,<br>
And the derivative  of o2 will be the gradient of the loss with respect to d.</p>

Valerio_Biscione: <p>are these gradients summed up when performing the update step?</p>

InnovArul: <p>Yes the gradients will be summed up accordingly.</p>
Access/replace layer using its name (string)

SEBDZ: <p>I want to access a layer in a model by its name (string) to change it.<br>
In my case, I want to change a layer inside bn_vgg16 model (model.output.fc3)<br>
I didn’t a way to access to the layer via its name or replace it</p>
<pre><code class="lang-auto">model = ptcv_get_model('bn_vgg16', pretrained=True)
model.load_state_dict(torch.load('bn_vgg16_vx.pth')
layer_name = 'output.fc3'
###  ###
model.(layer_name ) = nn.Linear(in_features, 5)
</code></pre>
<p>I know I can change the last layer easily using:</p>
<pre><code class="lang-auto">model = torch.nn.Sequential(*(list(model.children())[:-1]))
model = nn.Sequential(model, nn.Linear(in_features, 5))
</code></pre>
<p>but I am interested on accessing any layer by its name</p>
<p>Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

InnovArul: <p>You could try some trick as mentioned here:<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/31174295/getattr-and-setattr-on-nested-subobjects-chained-properties/31174427#31174427" target="_blank" rel="noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/190597/unutbu" target="_blank" rel="noopener">
    <img alt="unutbu" src="https://www.gravatar.com/avatar/aabc98d5c6482ca0e1405ec97710f30a?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/31174295/getattr-and-setattr-on-nested-subobjects-chained-properties/31174427#31174427" target="_blank" rel="noopener">getattr and setattr on nested subobjects / chained properties?</a>
</h4>

<div class="tags">
  <strong>python, python-2.7, recursion, attributes</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/190597/unutbu" target="_blank" rel="noopener">
    unutbu
  </a>
  on <a href="https://stackoverflow.com/questions/31174295/getattr-and-setattr-on-nested-subobjects-chained-properties/31174427#31174427" target="_blank" rel="noopener">01:49AM - 02 Jul 15 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Dataloader with random mini-batch selection, but each mini-batch keeps holding same item

Minseok: <p>Hello guys, I would like to select mini-batches randomly using DataLoader where each mini-batches keep holding the same item. Can I do this via DataLoader or do I just have to manually iter dataset? Although it’s possible to use just manual iteration, I just wonder about this since I still want to exploit the fancy features provided by DataLoader(e.g., collate_fn, pin_memory, num_workers…)</p>
<p>For example, suppose there is a dataset with item indices from 1 to n,  chronologically sorted.<br>
If the mini-batch size is k, then I want the dataset is divided into (1 ~ k), (k+1 ~ 2k), (2k+1 ~ 3k), …, where each mini-batch is randomly selected for every epoch, but never changes the items inside the mini-batch.</p>

nivesh_gadipudi: <pre><code class="lang-auto">data = DataLoader(dataset, batch_size = 8, num_workers = 4, pin_memory = True, shuffle = True, drop_last = False)
</code></pre>
<p><a class="mention" href="/u/minseok">@Minseok</a> for instance lets consider the above DataLoader, in order to enable batch shuffling you need to pass <strong>shuffle = True</strong>…num_workers, batch_size, pin_memory depends on your hardware, optimal selection of these parameters reduce the overheads and loads the data from CPU to GPU faster.</p>

Minseok: <p>Thanks for your answer, but using only dataloader parameters won’t fix items within batch,</p>
<p>I rather came up with an idea, which is</p>
<ol>
<li>receive the ‘random’ minibatch index from a dataloader with parameter batch_size=1</li>
<li>
<strong>getitem</strong> function in my custom Dataset class delivers consecutive items of mini-batch size (suppose k) once called.</li>
</ol>
<pre><code class="lang-auto">def __getitem__(self, index):
    batch_item = torch.zeros(self.batch_size, self.data_size)
    for i in range(index*self.batch_size, (index+1)*self.batch_size):
        batch_item[i] = self._get_data_function(i) # custom single data getter function for given index
    return batch_item
</code></pre>
<p>However, I’m worried about the ‘for loop’ which might harm speed.</p>

InnovArul: <p>Strange. Are you using a custom dataset written by you or a dataset from <code>torchvision</code>?<br>
If you are using a custom dataset, can you verify <code>__getitem__</code> method if you are handling the indices correctly?</p>
<p>Would you be able to provide a toy example to reproduce the issue?</p>
TypeError: __call__() takes 2 positional arguments but 3 were given

Lucius: <p>Hello. I am trying to work with the VOC dataset and want to display some images to see if I loaded it properly, however I am getting this error and I don’t understand what is causing it. Here is my code:</p>
<pre><code class="lang-auto">transform = transforms.Compose([
    transforms.Resize((224,224)),
    transforms.ToTensor(), 
    transforms.Normalize([0.5], [0.5])
])

data = torchvision.datasets.VOCSegmentation('./data', download=True, transforms = transform)
trainloader = torch.utils.data.DataLoader(data, batch_size=4, shuffle=True, num_workers=2)

import matplotlib.pyplot as plt
import numpy as np

dataiter = iter(trainloader)
(images, labels) = dataiter.next()
</code></pre>
Why memory formats are mixed?

t_naumenko: <p>I’m running the following code snippet on PyTorch version 1.6.0:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Conv2d(3, 16, 1),
            nn.ReLU(),
        )

    def forward(self, model_inputs):
        return self.layers(model_inputs)


device = torch.device('cuda:0')
torch.cuda.set_device(device)

model = Model()
model = model.to(device=device, memory_format=torch.channels_last)

x = torch.zeros((1, 3, 32, 32), dtype=torch.float, device=device)
x = x.contiguous(memory_format=torch.channels_last)

loss = model(x).mean()
loss.backward()
</code></pre>
<p>During backward, it generates the following message:</p>
<pre><code class="lang-auto">[W TensorIterator.cpp:924] Warning: Mixed memory format inputs detected while calling the operator. The operator will output channels_last tensor even if some of the inputs are not in channels_last format. (function operator())
</code></pre>
<p>If I remove backward() call, then no warning is raised, as well as if I remove ReLU function from the model.</p>
<p>Where may formats not match? Can someone provide a deeper understanding of what’s happening here?</p>

ptrblck: <p>I don’t get this warning in the nightly release. Could you update to <code>1.7.1</code> or the nightly, as this warning might have been wrong and seems to be fixed?</p>
Minimalistic RoBERTa example?

tueboesen: <p>I’m trying to find a minimalistic RoBERTa example that I can modify and use for some research on protein folding.</p>
<p>So far all I can find is fairseq:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md" target="_blank" rel="noopener nofollow ugc">pytorch/fairseq/blob/master/examples/roberta/README.md</a></h4>
<pre><code class="lang-md"># RoBERTa: A Robustly Optimized BERT Pretraining Approach

https://arxiv.org/abs/1907.11692

## Introduction

RoBERTa iterates on BERT's pretraining procedure, including training the model longer, with bigger batches over more data; removing the next sentence prediction objective; training on longer sequences; and dynamically changing the masking pattern applied to the training data. See the associated paper for more details.

### What's New:

- December 2020: German model (GottBERT) is available: [GottBERT](https://github.com/pytorch/fairseq/tree/master/examples/gottbert).
- January 2020: Italian model (UmBERTo) is available from Musixmatch Research: [UmBERTo](https://github.com/musixmatchresearch/umberto).
- November 2019: French model (CamemBERT) is available: [CamemBERT](https://github.com/pytorch/fairseq/tree/master/examples/camembert).
- November 2019: Multilingual encoder (XLM-RoBERTa) is available: [XLM-R](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).
- September 2019: TensorFlow and TPU support via the [transformers library](https://github.com/huggingface/transformers).
- August 2019: RoBERTa is now supported in the [pytorch-transformers library](https://github.com/huggingface/pytorch-transformers).
- August 2019: Added [tutorial for finetuning on WinoGrande](https://github.com/pytorch/fairseq/tree/master/examples/roberta/wsc#roberta-training-on-winogrande-dataset).
- August 2019: Added [tutorial for pretraining RoBERTa using your own data](README.pretraining.md).

## Pre-trained models
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Which definitely have the model I’m looking for, but it also looks like there is tons of other stuff that makes just extracting the model quite complicated.
<p>Does anyone have or knows of a simpler place to get a base RoBERTa model, it shouldn’t be pretrained at all, I just want the base framework.</p>
From torchsummary import summary

PytorchBeginner: <p>Hi, I just used summary to output the information about my model, but it did not work. Then, I tested it with an official example, and it did not work too. For example,</p>
<p>from torchsummary import summary<br>
model=torchvision.models.vgg16()<br>
model=model.cuda()<br>
summary(model,(3,224,224))</p>
<p>The error is “can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first”</p>
<p>In a word, the function “summary” seems does not work on my machine. I do not know why it happened. Please help me. Thanks!</p>

ptrblck: <p>You could try to keep the model on the CPU, as the error points to a device mismatch and I guess <code>summary</code> creates a <code>CPUTensor</code> using the passed input shapes.</p>

PytorchBeginner: <p>I want to see how much GPU memory is required by the model. If I keep the model on the CPU, is the memory requirement same as that on GPU? I am a little confused.</p>

PytorchBeginner: <p>More importantly, if I just keep the model on the CPU, it also gives the same error information “can’t convert CUDA tensor to numpy…” I do not know why it happens. I understand the meaning of the error information, but I do not know how to solve it. It is very strange.</p>

ptrblck: <p>This sounds like an issue in <code>torchsummary</code>. Note that there seems to be a new library, which can be foudn <a href="https://github.com/TylerYep/torch-summary">here</a>, as the old one was apparently abandoned.</p>

PytorchBeginner: <p>That works! Just uninstall the original ‘pytorchsummary’ and re-install ‘pytorch-summary’. Thanks very much!</p>
Calculate mutual information loss

111429: <pre><code class="lang-auto">out1=model1(input)
out2=model2(input)
</code></pre>
<p>I want to minimize the mutual information between <code>out1</code> and <code>out2</code><br>
Any recommended api or implementation?<br>
Thanks.</p>

pascal_notsawo: <p>Can you reformulate your problem to make it more understandable?</p>
<p>I don’t understand what you want to do.</p>

111429: <p>same question as <a href="https://discuss.pytorch.org/t/mutual-information-as-custom-loss-function/3794" class="inline-onebox">Mutual information as custom loss function</a></p>
Is there a gamma function in pytorch?

yxchng: <p>I am intending to calculate factorial like in this post</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/47141359/how-to-calculate-factorial-in-tensorflow" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1577800/ainthet" target="_blank" rel="nofollow noopener">
    <img alt="AintheT" src="https://www.gravatar.com/avatar/1efe5a533a74e52020f2ad0fe0a7d547?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/47141359/how-to-calculate-factorial-in-tensorflow" target="_blank" rel="nofollow noopener">How to calculate factorial in tensorflow?</a>
</h4>

<div class="tags">
  <strong>python, tensorflow, factorial</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1577800/ainthet" target="_blank" rel="nofollow noopener">
    AintheT
  </a>
  on <a href="https://stackoverflow.com/questions/47141359/how-to-calculate-factorial-in-tensorflow" target="_blank" rel="nofollow noopener">04:20PM - 06 Nov 17</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I am just wondering if I can not use math.factorial?</p>

simonp: <p>I know this conversation is old, but maybe it still helps someone: Just like in TensorFlow, you can use <code>lgamma</code>, the log of the gamma function, to calculate the factorial, as <code>n! == gamma(n+1) == exp(lgamma(n+1))</code> for integers <code>n</code>.</p>
<p>Compare</p>
<pre><code class="lang-python">from math import factorial
print([factorial(n) for n in range(5)])
&gt;&gt;&gt; [1, 1, 2, 6, 24]
</code></pre>
<p>and</p>
<pre><code class="lang-python">import torch
print((torch.arange(5) + 1).to(torch.float).lgamma().exp())
&gt;&gt;&gt; tensor([ 1.,  1.,  2.,  6., 24.])
</code></pre>

jasha: <p>Scipy has a solution:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import scipy.special
&gt;&gt;&gt; import torch
&gt;&gt;&gt; arr = torch.arange(5)
&gt;&gt;&gt; arr
tensor([0, 1, 2, 3, 4])
&gt;&gt;&gt; scipy.special.factorial(arr)
tensor([ 1.,  1.,  2.,  6., 24.], dtype=torch.float64)
</code></pre>
<p>This does not work with backprop / autograd, however.</p>
How to use embedding with images for RNN encoder-decoder?

zimmer550: <p>Hi all. I plan to use RNN-Encoder-Decoder to predict some actions in an image sequence. My idea is to first do some convolutions and feature extraction with fully connected layers to get a final vector of n elements. Then, I plan on sending this vector of n elements to the RNN encoder cell. Now, I heard that when dealing with words, the one-hot vector is converted to a lower-dimensional vector through embedding.</p>
<p>So, my question is, given that I have already done feature extraction from an image and obtained a lower-dimensional vector (n is really small), does it make sense to do embedding here? Can my convolution and fully-connected layers be treated as “embedding” the image?</p>

exploding-gradients: <p>Did you managed to get answer to this question? I am also interested in a similar problem.</p>
AttributeError: 'Tensor' object has no attribute 'copy'

Hao_Meng: <p>I want to deep copy like numpy.copy().<br>
But I met such problem, how can I solve it?</p>
<p>Thanks!</p>

InnovArul: <p>you can use  <code>tensor.clone()</code>.</p>

pevogam: <p>I am not sure though why doesn’t PyTorch just reuse standard method names which are easier to anticipate and lookup. Anyway, just my 5 cents.</p>
Turning off batch_size of DataLoader

Wesley_Neill: <p>This is counter-intuitive but there is a reason. I know parallel processing through batches is what makes DataLoaders great.</p>
<p>I designed a general purpose (I thought) method that accepts an autoencoder and a <code>DataLoader</code> and trains it.</p>
<p>Now I’m working on a new model that exclusively uses <code>LSTMCells</code>, which it looks like don’t accept batches. So instead of re-writing my <code>train_encoder(model, DataLoader)</code> method with a huge <code>if-else</code> block, is there a simple way to turn off batching completely?</p>

ptrblck: <p>I’m not sure I understand the use case completely, but you could set <code>batch_size=1</code> to get single samples in each batch (and remove the batch dimension if needed).<br>
Would this work for you? If not, could you explain what “removing batching” would mean?</p>

Wesley_Neill: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, I just used your suggestion of removing batch size. Thank you!</p>