akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch 1.2 linker error on Windows related to THP_CORE preprocessor definition

leowalkling: <p>Hi everyone,<br>
I’m trying to update an extension module of mine.<br>
However, I am getting linker errors when building with msvc15: About THPVariableClass not being defined.<br>
I could resolve it by removing the THP_CORE preprocessor definition, which seems to be used to activate __declspec(dllexport) as opposed to __declspec(dllimport).<br>
I think that the flag came in from the caffe2 CMake target.</p>
<p>My question is whether it is the right thing to do, simply undefining THP_CORE.</p>

peterjc123: <p>No, actually, you should link against <code>_C.lib</code> instead of changing the code.</p>

leowalkling: <p>I forgot to mention, that I used the CMake package in site-packages/torch/share/cmake to specify the PyTorch dependency:<br>
<code>target_link_libraries(torch_plus_ops ${TORCH_LIBRARIES})</code>.</p>
<p>_C.lib does not define the symbol either. Also, when using __declspec(dllexport), the linker would not look for the symbol in imported libraries, right?</p>
<p>I checked and succeeded building without the “Torch” CMake package and instead used the compiler/linker flags from <code>torch.utils.cpp_extension</code>.</p>
Cuda and conv layer

slavavs: <p>when i add cuda to the conv network, my network stops working. If you make a network on Linear, then everything works … Who knows why?<br>
The forward pass works, but the return does not.</p>

ptrblck: <p>Could you post a code snippet, which reproduces this issue?</p>

slavavs: <p>The problem was that I used cuda 9.2. When I installed cuda 10, the problem was resolved.</p>
Selecting data based on label

Lam_Nguyen: <p>Hello,</p>
<p>I have a vector of values</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/47cc4c0806a744a833d3b1de275462f87d29b435.png" alt="image" data-base62-sha1="af9But0ABqwEvnwlJSQv9MAXXO5" width="215" height="37"></p>
<p>And a vector of labels</p>
<p>(image1)</p>
<p>And a set of set of values, that I will call ‘results’</p>
<p>(image2)</p>
<p>Is there anyway I can fill all of the results s_i such that</p>
<p>(image3)</p>
<p>Because I’m a new user I can only post one picture. Here’s the rest : <a href="https://imgur.com/a/MUJ5tcw" rel="noopener nofollow ugc">https://imgur.com/a/MUJ5tcw</a></p>
<p>Here’s an example → </p>
<p>v = [1,2,3,4]<br>
u = [1,1,2,3]</p>
<p>f(u,v) = S</p>
<p>S_1 = [1,2]<br>
S_2 = [3]<br>
S_3 = [4]</p>
<p>I know I can do this using for loops, but I’d like to work with greater amount of data, for which their references are calculated on the spot (not done by a human beforehand) and it is taking too much time with for loops…</p>
<p>Thank. If there’s anything that is unclear about my query, please feel free to ask.</p>
<p>Best regards,</p>
Ubuntu : Not a supported wheel on this platform

Kim_KA: <p>pip3 install --user <a href="http://download.pytorch.org/whl/cu90/torch-0.3.1-cp36-cp36m-linux_x86_64.whl" rel="nofollow noopener">http://download.pytorch.org/whl/cu90/torch-0.3.1-cp36-cp36m-linux_x86_64.whl</a><br>
ERROR: torch-0.3.1-cp36-cp36m-linux_x86_64.whl is not a supported wheel on this platform.</p>
<p>Python 3.6.9<br>
pip 19.2.2</p>
<p>Thanks in advance.</p>

ptrblck: <p>I’m not sure about this particular error, but why are you trying to install PyTorch <code>0.3.0</code>, which is quite old by now.<br>
Have a look <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">here</a> for install instructions for the latest stable release (<code>1.2.0</code>).</p>

Kim_KA: <p>Thanks for the response.</p>
Where should I use with torch.no_grad()?

Shisho_Sama: <p>Hello every one.<br>
I’m trying to write a function that does both training and validation since like 90% of the code for them is the same .  But I’m facing a slight problem here. I don’t know where this should be placed so the gradient is not calculated for the tensors.<br>
Should I place <code>with torch.no_grad()</code> before the operations such as <code>model(imgs)</code>, <code>criterion(preds, labels)</code> or<br>
Should I be using it Like this e.g.</p>
<pre><code class="lang-auto">for imgs, labels in dataloader:
    imgs = imgs.to(device) 
    labels = labels.to(device)
    with torch._nograd():
            model.eval()
            preds = mode(imgs)
   # the rest 
   loss = criterion(preds, labels) 
</code></pre>
<p>or</p>
<pre><code class="lang-auto">for imgs, labels in dataloader:
    with torch._nograd():
           imgs = imgs.to(device) 
           labels = labels.to(device)
           model.eval()
           preds = mode(imgs)
   # the rest 
   loss = criterion(preds, labels)
   # acc, etc  
</code></pre>

ptrblck: <p>Both codes would work the same, if you just want to run inference and if your input doesn’t require gradients.</p>

Shisho_Sama: <p>Thank you very much.<br>
So you mean simply doing sth like :</p>
<pre><code class="lang-auto">def train_validation_loop(model, dataloader, optimizer, criterion, is_training,
                          device, topk=1, interval=1000 ):
    
    preds = None 
    loss = 0.0
    loss_total = 0.0
    accuracy_total = 0.0
    total_batches = len(dataloader)
    status = 'training' if is_training else 'validation' 
    
    for i, (imgs, labels) in enumerate(dataloader):
        imgs = imgs.to(device)
        labels = labels.to(device)
        
        if is_training:
            model.train()
            preds = model(imgs)
            loss = criterion(preds, labels)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        else:
            model.eval()
            with torch.no_grad():
                preds = model(imgs)
                loss = criterion(preds, labels)

        loss_total += loss.item()
        _, class_idxs = preds.topk(k, dim=1)
        results = (class_idxs.view(*labels.shape) == labels).float()
        accuracy_total += torch.mean(results)

        if i % interval == 0: 
            _, class_idxs = preds.topk(k, dim=1)
            results = (class_idxs.view(*labels.shape) == labels).float()
            accuracy_per_batch = torch.mean(results)
            print(f'{status} loss/accuracy(per batch): {loss:.6f} / {accuracy_per_batch:.4f}')
    # calculate the loss/accuracy after each epoch        
    print(f'{status} loss/accuracy: {loss_total/total_batches:.6f} / {accuracy_total/total_batches:.4f}')
</code></pre>
<p>will consider gradients when in training mode and wont do this when in validation mode?<br>
So basically</p>
Why does the same python model code behave differently in scripting and not scripting?

kevinhaoliu: <p>When I want to script a model, it comes a warning message:  <strong>RuntimeWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().</strong> But if I don’t script it and just run it the warning message won’t appear. I wonder whether it is a BUG? And Could it be solved?</p>

kevinhaoliu: <p>I have the same problem with this <a href="https://github.com/pytorch/pytorch/issues/10234" rel="nofollow noopener">Github issue</a> when run the model  in C++ on GPU. If I don’t run it on GPU it will work well.</p>

kevinhaoliu: <p>My pytorch version is 1.1. CUDA version is 9.0, CUDNN is 7.3.0. Can anyone give me a help?</p>
ValueError: optimizer got an empty parameter list when define FM module

abtion: <p>I have some problems when I train the model, it raise a ValueError(“optimizer got an empty parameter list”)，I am searching for resolution and need your help, thanks very much!</p>
<p>here is my module code:</p>
<pre><code class="lang-python">class TorchFM(nn.Module):
    def __init__(self, input_dim):
        super(TorchFM, self).__init__()
        self.w0 = torch.tensor(torch.zeros(1), requires_grad=True)
        self.w = torch.tensor(torch.zeros(input_dim), requires_grad=True)
        self.v = nn.init.kaiming_normal_(torch.tensor(torch.zeros((input_dim, 10)), requires_grad=True))

    def forward(self, x):
        torch.set_grad_enabled(True)
        linear_terms = torch.add(self.w0, torch.sum(torch.mul(self.w, x)))
        pair_interactions = 0.5 * torch.sum(torch.sub(torch.pow(torch.matmul(x, self.v), 2),
                                                      torch.matmul(torch.pow(x, 2), torch.pow(self.v, 2))))

        out = torch.add(linear_terms, pair_interactions)
        return out
</code></pre>
<p>And here is the code which I train the model</p>
<pre><code class="lang-python">loss_object = torch.nn.modules.MSELoss()
model = TorchFM(x_train.shape[1])
optimizer = torch.optim.Adam(model.parameters())

EPOCH = 100

for e in tqdm(range(EPOCH)):
    for i, (x, y) in enumerate(train_loader):
        optimizer.zero_grad()
        predictions = model(x)

        writer.add_graph(model, x)
        loss = loss_object(predictions, y)
        writer.add_scalar("train_loss", torch.mean(loss.values()))
        loss.backward()
        optimizer.step()

    for i, (x, y) in enumerate(test_loader):
        test_pred = model(x)
        loss = loss_object(test_pred, y)
        writer.add_scalar("test_loss", torch.mean(loss.values()))
        # writer.add_graph(model, x)
writer.close()
</code></pre>

abtion: <p>I have solved this problem by using nn.Parameter like follow:</p>
<pre><code class="lang-python">class TorchFM(nn.Module):
    def __init__(self, input_dim):
        super(TorchFM, self).__init__()
        w0 = torch.zeros(1).double()
        w = torch.zeros(input_dim).double()
        v = torch.empty(input_dim, 10).double()
        nn.init.xavier_normal_(v)

        self.w0 = nn.Parameter(w0)
        self.w = nn.Parameter(w)
        self.v = nn.Parameter(v)

    def forward(self, x):
        torch.set_grad_enabled(True)
        linear_terms = torch.add(self.w0, torch.sum(torch.mul(self.w, x)))
        pair_interactions = 0.5 * torch.sum(torch.sub(torch.pow(torch.matmul(x, self.v), 2),
                                                      torch.matmul(torch.pow(x, 2), torch.pow(self.v, 2))))

        out = torch.add(linear_terms, pair_interactions)
        return out

</code></pre>
Is there any case of training RNN without packed sequence?

tere3927: <p>Hi,<br>
As title, Is there any real use case of training RNN without using packed_sequence?<br>
I wonder that if batch size &gt; 1, then there will be training samples with different sequence length. Therefore, padding is required. Is there any application of training RNN by simply feeding the padded tensor into the RNN without using pack_padded_sequence still work?</p>
<p>Thanks.</p>

vdw: <p>Using <code>pack_padded_sequence</code> when having batches with sequences of different lengths is not absolutely mandatory. Just try it with and without. In many cases – I talking mainly about classification here!!! – I don’t think you will see major differences in the results.</p>
<p>Ideally, the network will learn to kind of ignore the special padding word (e.g., <code>&lt;pad&gt;</code>). Some affects are probably always gonna be in practice.</p>
<p>How important using <code>packed_padded_sequence</code> is will most likely depend on the training data. Most notably, batches where the lengths of the sequences vary greatly and/or are very skewed (99 sequences with around 10 items and 1 sequence with 100 items, so the other 99 sequences have to be padded a lot) will arguably have the most issues. That’s why existing solutions try to ensure that batches are relatively homogeneous:</p>
<ul>
<li>
<a href="https://torchtext.readthedocs.io/en/latest/data.html" rel="nofollow noopener">The <code>BucketIterator</code> of <code>torchtext</code></a> defines an iterator that batches sequences of similar lengths together. This minimizes amount of padding needed while producing freshly shuffled batches for each new epoch.</li>
<li>So this <a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284">older post</a> where we came up with an iterator that ensures that each batch contains sequences of the same length. While this might yield batches that are not full, for large datasets this issues is absolutely negligible.</li>
</ul>
<p>Summing up:</p>
<ul>
<li>Padding is not necessarily bad (i.e., packing is not necessarily needed)</li>
<li>Simply try with or without padding to see the effects.</li>
<li>Try approaches to automatically minimize the required padding.</li>
</ul>
<p>I hope that helps.</p>
Batch control during training

suke: <p>Hello, i have a question about pytorch training, in current case. i can set batch size and provide image size for every images by reload data.Dataset class. but that means i must set same shape for all batches.<br>
can i decide data shape for different batch?, for example 1 batch 1x3x25x25, 2nd batch 1x3x50x50, 3rd batch 1x3x100x100. Plz help me! thank you!</p>

Oli: <p>Hello, yes this is possible if you write a custom collate function for your dataloader. Have a look for an example below, but you might have to change it a little bit <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Let me know if this is unclear, I’m just heading to sleep</p>
<pre><code class="lang-auto">def collate(batch):
    im_sizes = [25, 50, 100]
    im_size = # select size from im_sizes
    uniform_size = transforms.Compose([
      transforms.Resize((im_size, im_size)),
      transforms.ToTensor(),
      transforms.Normalize(mean=[0.485, 0.456, 0.406],
                           std=[0.229, 0.224, 0.225])
    ])

    ims = [uniform_size(b) for b in batch]
    return torch.stack(ims)
</code></pre>

suke: <p>ok, i reload dataloader and reslove this problem. thank you!</p>
Renaming nn.Sequential Layers?

ajhanwar: <p>I’ve trained on a <code>nn.Sequential</code> model and would like to load the previously saved weights into another instance of the model. However, I’ve been trying to add names to the layers in this new model definition (which I didn’t previously do) and am running into an error with missing/unexpected keys.<br>
Any ideas for how to circumvent such an issue?</p>

ptrblck: <p>You could load the <code>state_dict</code> and rename all keys to the newly given names.<br>
The fastest way would probably be to use the <code>state_dict</code> of your new model definition and just create a loop over the keys of both dicts changing one to the other.</p>
Index into tensor with indices from another?

mraggi: <p>Hello,</p>
<p>I’ve been struggling for hours with this.</p>
<p>Say I have a 2d tensor:<br>
<code>A = torch.randn(10,4)</code></p>
<p>And also a 1d LongTensor of with A.shape[0] elements (e.g. 10)<br>
<code>I = torch.randint(0,4,(10))</code></p>
<p>I’d like an easier but equivalent way of doing this:<br>
<code>torch.take(A,I+torch.arange(0,I.shape[0])*A.shape[1])</code></p>
<p>That is, I want to obtain a 1-d tensor like this: from the 0th row of A, the <code>I[0]</code>-th value, from the first row of A the <code>I[1]</code>-th value, etc.</p>
<p>If I do <code>A[I]</code> it gives me a matrix (I want a 1d tensor) that selects the corresponding rows.<br>
I also tried index_select, but I’m also getting a matrix.</p>
<p>Thank you!</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">A[torch.arange(A.size(0)), I]
</code></pre>

mraggi: <p>Thank you! This works!</p>
Where are the train/eval scripts for action recognition? (torchvision video)

Niklas_Donges: <p>In the release notes of torchvision 0.4 and also in your blog post about the release, you talk about “Training and evaluation scripts for reproducing the training results” of action recognition in videos, but I can’t find them anywhere.</p>

ptrblck: <p>Not sure, if you are looking for this, but <a href="https://github.com/pytorch/vision/blob/master/references/video_classification/train.py" rel="nofollow noopener">this training script</a> contains the training using the kinetics dataset.</p>
How do I set --input_dims option in caffe_translator.py

e86b307d59a4c2c780c5: <p>Hello,</p>
<p>I want to translate caffe model to caffe2 model using caffe_translator.py.<br>
but I cannot figure out how to specify the --input_dims options.</p>
<pre><code class="lang-auto">usage: caffe_translator.py [-h] [--init_net INIT_NET]
                           [--predict_net PREDICT_NET] [--remove_legacy_pad]
                           [--input_dims INPUT_DIMS [INPUT_DIMS ...]]
                           prototext caffemodel
</code></pre>
<p>Is there anybody explain the usage of --input_dims with caffe’ mnist tutorial<br>
at <a href="http://caffe.berkeleyvision.org/gathered/examples/mnist.html" rel="nofollow noopener">http://caffe.berkeleyvision.org/gathered/examples/mnist.html</a>.</p>
<p>Thanks.</p>

farhan: <p>This question is worth a Billion Dollars.</p>

farhan: <p>python caffe_translator.py --remove_legacy_pad new.prototxt new.caffemodel --input_dims 1 3 210 280</p>
<p>putting it at the end<br>
this worked for me</p>
Which import is better?

vainaijr: <p>%%timeit -n 1000000<br>
import torch.nn as nn<br>
1000000 loops, best of 3: 357 ns per loop</p>
<p>%%timeit -n 1000000<br>
import torch.nn<br>
1000000 loops, best of 3: 307 ns per loop</p>
<p>%%timeit -n 1000000<br>
from torch import nn<br>
1000000 loops, best of 3: 864 ns per loop</p>

ptrblck: <p>I’ve never seen someone profiling the import. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>My personal and biased choice is <code>import torch.nn as nn</code>.</p>
Different behavior in stochastic forward pass

gaurav71531: <p>How can I have different behavior for stochastic layers in the forward passes? For example, with bith batch normalization and dropout present, how can I prevent BN to update its mean and variance while dropout to behave as if in the training phase? Or in other words, use BN in eval mode while dropout in train mode in a forward pass.<br>
Thanks a lot</p>

ecdrid: <p>Don’t use BN then, simple.</p>

spanev: <p>You could also do:</p>
<pre><code class="lang-auto">net = net.eval()
for m in net.modules():               
    if 'Dropout' in str(type(m)):     
        m.train()         
</code></pre>

gaurav71531: <p>Exactly what I was looking for, thanks a lot.</p>
Mapping tensor based on corresponding index tensor

IjlalBaig: <p>Hi,</p>
<p>I have two tensors of size [b, c, h, w, dim]; one contains values, while other contains indices that these values should map to. I would like to create a third tensor of this dimension and perform the map:<br>
<strong>out = value[idx]</strong></p>
<p>How can I do this? I believe index operations like index_copy don’t work with dimensions greater than 1.</p>
<p>Thanks</p>

ptrblck: <p>Could you post an example of the input and index tensors, which shows, how the indexing should work?</p>

IjlalBaig: <pre><code class="lang-auto">value = [[[[[0.4196, 0.4720, 0.4312], [0.7560, 0.2302, 0.0757]],
          [[0.4395, 0.1510, 0.0983], [0.7749, 0.4382, 0.3548]]]]]
idx = [[[[[2, 1, 0], [2, 0, 1]],
          [[0, 1, 2], [0, 1, 2]]]]]
</code></pre>
<p>So here I would want a result like:</p>
<pre><code class="lang-auto">out = [[[[[0.4312, 0.4720, 0.4196 ], [0.0757, 0.7560, 0.2302]],
          [[0.4395, 0.1510, 0.0983], [0.7749, 0.4382, 0.3548]]]]]
</code></pre>
How does one make a bidirectional RNN if one is processing a sequence token by token?

pinocchio: <p>How does one make a bidirectional RNN if one is processing a sequence token by token? Do I have to use a hardcoded for loop or is the bidirectional flag essentially useless?</p>
<p>Does things change if my RNN is to some degree custom (but has LSTM/GRU in it)?</p>
<hr>
<p>Some example code of how I am doing one direction:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from pdb import set_trace as st

torch.manual_seed(1)

def step_by_step(net,sequence,hidden):
    '''
    an example of an LSTM processing all the sequence one token at a time (one time step at a time)
    '''
    ## process sequence one element at a time
    print()
    print('start processing sequence')
    for i, token in enumerate(sequence):
        print(f'-- i = {i}')
        #print(f'token.size() = {token.size()}')
        ## to add fake batch_size and fake seq_len
        h_n, c_n = hidden # hidden states, cell state
        processed_token = token.view(1, 1, -1) # torch.Size([1, 1, 3])
        print(f'processed_token.size() = {processed_token.size()}')
        print(f'h_n.size() = {h_n.size()}')
        #print(f'processed_token = {processed_token}')
        #print(f'h_n = {h_n}')
        # after each step, hidden contains the hidden state.
        out, hidden = lstm(processed_token, hidden)
    ## print results
    print()
    print(out)
    print(hidden)

if __name__ == '__main__':
    ## model params
    hidden_size = 6
    input_size = 3
    lstm = nn.LSTM(input_size=input_size,hidden_size=hidden_size)
    ## make a sequence of length Tx (list of Tx tensors)
    Tx = 5
    sequence = [torch.randn(1, input_size) for _ in range(Tx)]  # make a sequence of length 5
    ## initialize the hidden state.
    hidden = (torch.randn(1, 1, hidden_size), torch.randn(1, 1, hidden_size))
    step_by_step(lstm,sequence,hidden)
    print('DONE \a')
</code></pre>
Is it possible to extract automatically all the arguments to Pytorch layers/functions/modules?

pinocchio: <p>I asked a similar question where I want to download all the layers into pytorch:</p>
<aside class="quote" data-post="1" data-topic="53007">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-does-one-download-all-possible-layers-allowable-in-pytorch-into-one-data-structure/53007">How does one download all possible layers allowable in Pytorch into one data structure?</a> 
  </div>
  <blockquote>
    I want to have say, a list of all the allowable layers in pytorch. How do I get that info? Is it possible? 
I am fine as a started for only sequential models…
  </blockquote>
</aside>

<p>but I also want to have a list of all the hyperparameters/arguments for each of them. Is it possible to extract all of those automatically? (at the very least as string representations or something like that)</p>
[SOLVED] Build the C++ frontend using g++ or any other compiler (no cmake)

pim: <p>Hi All,</p>
<p>I was trying to add the libtorch library to our project which is being build by using g++ or visual studio (it’s a multiplatform project) now I was trying to include the specified files in the g++ command but it does not seem to work, or I’m doing something wrong or the files are not meant to be included this way into a compliler.</p>
<p>I used the standard (most simple) example taken from: <a href="https://pytorch.org/cppdocs/installing.html#minimal-example" rel="nofollow noopener">https://pytorch.org/cppdocs/installing.html#minimal-example</a></p>
<p>then I tried to use the command: g++ -std=c++17 -o test *.cpp -I /home/user/libtorch/include/ -L /home/user/libtorch/lib/</p>
<p>but I got the error:<br>
example-app.cpp:1:10: fatal error: torch/torch.h: No such file or directory<br>
<span class="hashtag">#include</span> &lt;torch/torch.h&gt;</p>
<p>The cmake example worked great, but writing the cmake for the entire project will take quite some time (especially since I have never done it before), so my question is if there is any way to use the libtorch in a way described above, with a normal compiler?</p>
<p>Any help would be greatly appreciated!</p>
<p><strong>EDIT:</strong><br>
When I just posted this I found the other include directory in the api folder, and I managed to compile it perfectly after that <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
If other people might be interested this is the command I used:</p>
<p>g++ -std=c++17 -o test *.cpp -I /home/user/libtorch/include/torch/csrc/api/include/ -I /home/user/libtorch/include/ -L /home/user/libtorch/lib/ -lc10 -lcaffe2 -ltorch -D_GLIBCXX_USE_CXX11_ABI=0</p>

grantrostig: <p><a class="mention" href="/u/pim">@pim</a> Thank you very much for documenting this!  It would have taken me years to figure out, if ever, especially the -D parameter.</p>
<p>++Regards,<br>
Grant Rostig</p>

thekchang: <p>For future wanderers, I finally got it to work using the following (referencing the cmake tutorial and the commands it invokes)</p>
<pre><code class="lang-auto"> g++ main.cpp -I/libtorch/include -I/libtorch/include/torch/csrc/api/include -D_GLIBCXX_USE_CXX11_ABI=0 -std=gnu++11 -L/libtorch/lib/ -ltorch -lc10 -o output
</code></pre>
<p>Don’t forget to add /libtorch/lib/ to LD_LIBRARY_PATH</p>
Model fails after saving and loading state_dict from earlier epochs

Accelerator: <p>I was using this code to ensemble the model from several epochs, but as you can see from the kappa score printed out below, except for the  last epoch, all other epochs have very strange scores:</p>
<pre><code class="lang-auto">global labels_v 
val_preds = np.zeros(len(val19_df))
for i in range(len(Ws)):
    if Ws[i] &gt; 0:
        model.load_state_dict(torch.load(f'weight_best{i * 4 + 5}.pt'))
        model.cuda()
        preds_i, labels_v = test_model(return_p=True)  # test_model does inference for validation set in model.eval() mode, and returns the kappa score
# def test_model(val_loader=val_loader, val_df=val19_df, return_p=False)
        val_preds += Ws[i] * preds_i
val_preds /= w_total
</code></pre>
<p>kappa score: -0.007528865258592532<br>
kappa score: 0.0<br>
kappa score: 0.0<br>
kappa score: 0.49560576349219887<br>
kappa score: 0.9044690071390352</p>
<p>The exact same test_model function was used at training time and they behaved normally before saving. Model in these epoches were saved this way:</p>
<pre><code class="lang-auto">if epoch % 4 == 1:
    print(f'model{epoch} saved') 
    torch.save(model.state_dict(), f'weight_best{epoch}.pt')
</code></pre>
<p>And also seems that if I use model.train() mode for inference it behaves more normally. So what’s wrong?</p>

ptrblck: <p>I’m not sure, what <code>Ws</code> contains, did you make sure to call <code>model.eval()</code> inside <code>test_model</code>?</p>
"PyTorch on Azure" vs "Azure CustomVision AI"

Purna_Chandra_Rao_Ye: <p>Hi,</p>
<ol>
<li>I used Microsoft CustomVision AI to do Image classification - damn easy.</li>
<li>Now this “PyTorch on Azure” look similar to me.</li>
</ol>
<p>Which is best in terms of</p>
<ol>
<li>Cost</li>
<li>Easiness</li>
<li>Any other parameters</li>
</ol>
<p>Thank you in advance…</p>
How to update the weight by an old grad?

JTiC: <p>Hi!<img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"> I am new in PyTorch. I want to update the weight by a old grad:<br>
Here is a pseudocode example:</p>
<pre><code class="lang-auto">output = model(input)
loss1 = criterion1(output, target)
loss2 = criterion2(output, target)
grad1 = loss1.compute_grad()
grad2 = loss2.compute_grad()
parameters.update(grad1)
... # other operation will be done here.
parameters.update(grad2)
</code></pre>
<p>The key point is that the grad2 is calculated w.r.t the old weight, before the <code>parameters.update(grad1)</code>.<br>
I have no ideas to do this in an efficient way. Can <code>create_graph=True</code> or other method in Pytorch can help me? An easy example is better. Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>

ptrblck: <p>It’s probably not the most efficient way, but you could clone all <code>.grad</code> attributes, zero them out in the model, perform your custom operations, copy them back into the model parameters, and call <code>optimizer.step()</code> when you are done.</p>

JTiC: <p>Thank you! It helps!</p>
How to use custom label with image data set?

zimmer550: <p>Hi. This is my first post here. I am a PyTorch newbie. I have written code for a convolution neural network that gives me an output. The forward pass works fine. I have loaded the dataset correctly and made the loader properly as well. Now, each image gives an output of 4 elements and this output is compared with a label which also consists of 4 elements (the label elements represent the bounding box coordinates).</p>
<p>The output of my forward pass gives a tensor of size 10 by 4 (I am using batch size of 10). To compute the loss I made a tensor of size 10 by 4. Each row of this tensor consists of the bounding box coordinate/ label for each image. I am trying to compute the loss using the following line</p>
<p><code>loss = criterion(outputs, all_bbox_tensor)</code></p>
<p>However, I get the following error:<br>
Traceback (most recent call last):<br>
File “C:/Users/sarim/PycharmProjects/thesis/pytorch_learning.py”, line 127, in <br>
loss = criterion(outputs, all_bbox_tensor)<br>
File “C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\loss.py”, line 942, in forward<br>
ignore_index=self.ignore_index, reduction=self.reduction)<br>
File “C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\functional.py”, line 2056, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\functional.py”, line 1871, in nll_loss<br>
ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: <strong>multi-target not supported at …\aten\src\THNN/generic/ClassNLLCriterion.c:20</strong></p>
<p>I don’t understand this error. What can I do to fix this?</p>

kl_divergence: <aside class="quote no-group" data-username="zimmer550" data-post="1" data-topic="53229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/bbe5ce/48.png" class="avatar"> zimmer550:</div>
<blockquote>
<p>multi-target not supported</p>
</blockquote>
</aside>
<p>Are you using Cross Entropy loss ? <code>CrossEntropyLoss</code> does not expect a one-hot encoded vector as the target, but class indices</p>

zimmer550: <p>Hello. Thank you very much. I solved the problem by using MSE to predict bounding box from image data. Thank you.</p>
Element-wise function evaluation

IjlalBaig: <p>Hi,</p>
<p>I have a tensor where I would like to evaluate a function on each element, and based on its output, fill corresponding values in another tensor. Lets say if the function returns value lesser than 0 or greater than 1, then we set value in the output tensor to 0, else we set it to function output. How would I go about doing that?</p>
<p>Thanks!</p>

zhl515: <p>You can do it like this:</p>
<pre><code class="lang-auto"> def f2(x):
        return x*2

def f1(x):
    # evaluate a function on each element
    tmp = f2(x)
    #based on its output, fill corresponding values in another tensor
    if tmp &lt; 0 or tmp &gt; 1:
        return 0
    else:
        return tmp

x = torch.randn(10)
x.apply_(f1)
</code></pre>
<p>But apply_() function only works with <strong>CPU</strong> tensors and should <strong>not</strong> be used in code  sections that require high performance(becase it’s slow).</p>

IjlalBaig: <p>Thanks! this is just what I was looking for.</p>
Facing problem detection using pre-trained Faster R-CNN

Kim_KA: <p>model = models.detection.fasterrcnn_resnet50_fpn(pretrained=True)<br>
AttributeError: module ‘torchvision.models’ has no attribute ‘detection’</p>
<p>Tried with Pytorch 0.4.1 and 1.1.0 as well.<br>
Thanks in advance.</p>

spanev: <p>Did you try this after upgrading <code>torch</code> to 1.2.0:</p>
<pre><code class="lang-auto">pip install --upgrade torchvision
</code></pre>
<p>Related issue:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/vision/issues/988" target="_blank" rel="nofollow noopener">github.com/pytorch/vision</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/goyalsaransh97" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/26386379?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/vision/issues/988" target="_blank" rel="nofollow noopener">Issue: Importing Mask RCNN in Torchvision 0.2.2</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/goyalsaransh97" target="_blank" rel="nofollow noopener">goyalsaransh97</a>
	on <a href="https://github.com/pytorch/vision/issues/988" target="_blank" rel="nofollow noopener">2019-06-04</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/fmassa" target="_blank" rel="nofollow noopener">fmassa</a>
	on <a href="https://github.com/pytorch/vision/issues/988" target="_blank" rel="nofollow noopener">2019-06-04</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">On running "model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)", I get the error
"AttributeError: module 'torchvision.models' has no attribute 'detection' ". How can I solve this?
I'm...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: models</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">question</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">topic: object detection</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

kl_divergence: <p>This is because you seem to using the version before torchvision 0.3. Detection modules were added in 0.3.</p>

Kim_KA: <p>Thanks.</p>
<pre><code class="lang-auto">pip install --upgrade torchvision
</code></pre>
<p>Resolved it.</p>

Kim_KA: <p>Thanks for the response.</p>
Why not giving the whole model to DataParallel in the imagenet example?

fanq15: <p>I have seen this question many times in different questions about DataParallel, but no one can give an explicit answer. So I question it again in a new topic, and hope anyone could answer it. Does this operation have any special purposes?<br>
This expression is borrowed from <a class="mention" href="/u/trypag">@trypag</a>. <a href="https://discuss.pytorch.org/t/dataparallel-and-cuda-with-multiple-inputs/272/3">https://discuss.pytorch.org/t/dataparallel-and-cuda-with-multiple-inputs/272/3</a></p>
<blockquote>
<p>Could anyone explain this code extracted from the imagenet example :</p>
</blockquote>
<blockquote>
<pre><code>if args.arch.startswith('alexnet') or args.arch.startswith('vgg'):
    model.features = torch.nn.DataParallel(model.features)
    model.cuda()
else:
    model = torch.nn.DataParallel(model).cuda()
</code></pre>
<p>Is there a specific reason to separate the classifier and the features in the alexnet and vgg models ?<br>
Why not giving the whole model to DataParallel, like in the resnet model ?</p>
</blockquote>

apaszke: <p>The answer is in <a href="https://arxiv.org/abs/1404.5997">One weird trick for parallelizing convolutional neural networks</a> by Alex Krizhevsky.</p>

fanq15: <p>Oh! Thank you very much!</p>

lusofa: <p>I read the paper yet, but still feel confused.<br>
Is that means “model.features = torch.nn.DataParallel(model.features)” corresponding to Model parallelism, and “torch.nn.DataParallel(model)” corresonding to Data parallelism?<br>
But why “args.arch.startswith(‘alexnet’) or args.arch.startswith(‘vgg’)” ,then do Model Parallelism, else do Data parallelism?</p>
Implement my reduce function

saisai96: <p>I try to write a reduce function by myself. If it’s just execute a sum of dimensions , it will perform well. But when i try to input a variance it is broken(reduce(tensorx) = pow((tensorx.data - mean), 2) / n). It will overflow with the datatype of __half(16-bits). How is the pytorch function do to this phenomenon(I tried to find it, if you kown where the examples are, please tell me the exact location, thanks!!!)</p>
Fast write Pytorch CUDA tensors to memory mapped files on GPU

Santosh-Gupta: <p>I saw that is possible to use CUDA to write to memory mapped files (reference <a href="https://stackoverflow.com/questions/29518875/cuda-zero-copy-memory-memory-mapped-file" rel="nofollow noopener">https://stackoverflow.com/questions/29518875/cuda-zero-copy-memory-memory-mapped-file</a> )</p>
<p>I am wonder if it is somehow possible in Pytorch to write a cuda mounted tensor directory to a mem mapped stored on GPU.</p>
<p>The purpose of this is to speed up writing tensors after each training step. Currently,</p>
<pre><code class="lang-auto">with torch.no_grad():
    numpyMemmap[arrayOfRandomIndexes] = u_embeddings.weight.data.detach().cpu().numpy()
</code></pre>
<p>takes 6 seconds. I think it’s because the numpy memory map is stored on CPU. I need something that would write in a fraction of a second since I will be storing the tensors after each training step, and there will be hundreds of thousands of training steps.</p>

ptrblck: <p>How are you profiling the code?<br>
Make sure to synchronize the code before starting and stopping the timer using <code>toch.cuda.synchronize()</code>.<br>
Since CUDA operations are asynchronous, the <code>cpu()</code> call will create a synchronization point, so that you might be in fact profiling some other workload which is still executed on the GPU.</p>
What does pack_padded_sequence do?

Rafael_R: <p>Hi,</p>
<p>I have this code:</p>
<pre><code class="lang-auto">embedded = self.embedding(input)
packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths)
output, hidden = self.LSTM(packed, None)
</code></pre>
<p>Here, input has the shape = [20, 32] , embedded = [20, 32, 300] , input lengths = [32],</p>
<p>and packed.data.shape is [368, 300].</p>
<ol>
<li>
<p>What does it do? Why does it require input lengths?</p>
</li>
<li>
<p>Why is the shape [368, 300]? Why should the LSTM accept this as input, LSTM should take [seq_length, batch_size, feature_size] right, how can then LSTM figure out the correct shape from the [368, 300] dimensional tensor? What attributes in the packed object allow this? The output.data has the shape [368, 512]? How does the LSTM process, since the output is per time step?</p>
</li>
<li>
<p>What does the below operation do? It is clearly unpacking the packed input to LSTM, but how is it doing it?</p>
</li>
</ol>
<p><code>output, _ = torch.nn.utils.rnn.pad_packed_sequence(output)</code></p>
<p>Any help would be great <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Can we normalize the time series data

nikitha_kona: <p>I have a time series data with each column(feature) with different scale and the output is a a forecasting problem for the next time step and the output dimension is same as the input dimension.I am using a RNN and is it a good idea to standardize the data?<br>
In case if I do at test time should I stack the bpttt and then apply standardization?</p>
<p>Thanks</p>
Reshaping image

LazySleeper: <p>I used to use keras and the image format it followed is [Height x Width x Channels x Samples]. i decided to switch to PyTorch. But i didn’t switch out my data loading schemes. So now i have numpy arrays of shape HxWxCxS, instead of  SxCxHxW which is required for PyTorch. Does anyone have any idea to convert this ?</p>

spanev: <p>What you want to achieve here sounds more like permutation of dimensions rather than reshaping?<br>
If that is the case, this should do:</p>
<pre><code class="lang-auto">t = torch.Tensor(np_arr)
t = t.permute(3, 2, 0, 1)
</code></pre>
<p>But if you know that the underlying data is in <code>SxCxHxW</code> but your array dims are  <code>HxWxCxS</code>, you can use</p>
<pre><code class="lang-auto">t = t.reshape(S, C, H, W)
</code></pre>
Running Model inside Dataset's __getitem__()

av093: <p>My training steps consist of running a model twice, with some numpy computation in between.<br>
So, the basic workflow is:</p>
<ol>
<li>Fetch a batch of data samples using dataloader which uses a custom dataset class</li>
<li>Run the model:  output = model(input)</li>
<li>Perform some processing on each data point in the output, I am converting tensors to numpy arrays here.</li>
<li>Request new samples from the custom dataset class instance based on the results of previous step</li>
<li>Run inference again and backpropagate based on the last inference only.</li>
</ol>
<p>Now steps 3 and 4 are currently sequential, i.e. I loop through the batch one at a time. This is quite time consuming.</p>
<p>Are there any ways to cleanly parallelize the steps 3 and 4 together.</p>
<p>I could parallelize step 3. But, fetching data again from the custom dataset class is sequential (step 4), which is time consuming due to heavy computations required for each data point.</p>
<p>So, later i tried another idea: pushed my first inference (step 1) into the <em>getitem</em> of the dataset class, hoping that the dataloader can do steps 1 to 3 (4 is not needed anymore) in batches.</p>
<p>But the issue is i can not have a GPU model now. Because the way multiprocessing works in Python, it doesnt allow to process CUDA tensors in the subprocess. Python interpreter complains about CUDA initialization error. Hence, one way now is to keep two model copies one in the CPU for the first inference which is injected into the dataset and another a GPU one for 2nd inference and back propagation, but that means somehow the GPU and CPU models needs to be synchronized after every back propagation step? Am I thinking the right solution here?</p>
<p>Can anyone suggest something in this regard? any hints and ideas would be appreciated.</p>
<p>I also have tried to force the python multiprocessing lib to only ‘spawn’ iso. ‘fork’/‘forkserver’ so that the CUDA context error can be mitigated, but the issue is then a large amount of data need to be serialized and deserialized, which is not possible.</p>
My Linear Regression Model predicting falsely

surfsamson: <p>Please everyone, i am very new to Machine learning and i just tried out linear regression and i’m not getting accurate result from my model.</p>
<p>Aim:<br>
Predict the salary of an employee based on his/her number of experience.</p>
<p>Excel sheet content:</p>
<p>Years of Experience  |   Salaries<br>
10                             |    300000<br>
2                              |    20000<br>
8                              |   599000<br>
9                              |    290000<br>
1                              |    20000<br>
7                              |    80000<br>
4                              |    700000</p>
<pre><code class="lang-auto"># -*- coding: utf-8 -*-
import torch
import torch.nn as nn
import pandas as pd

datasets = pd.read_csv("./salaries.csv")

features_X = datasets.iloc[:, :-1].values 
features_Y = datasets.iloc[:, 1:].values

features_X = torch.from_numpy(features_X).float()
features_Y = torch.from_numpy(features_Y).float()

print(datasets, "\n\n", features_X, "\n\n", features_Y, "\n\n", features_X.type(), "\n\n", features_Y.type())


x_train = torch.FloatTensor(features_X)
y_train = torch.FloatTensor(features_Y)

""" Define out Hyper-parameters """
learning_rate = 0.0001
epochs = 500



""" Building the Network """

class Model(nn.Module):
    def __init__(self, input_layer, output_layer):
        super(Model, self).__init__()
        self.linear = nn.Linear(input_layer, output_layer)
        
    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

model = Model(input_layer=1, output_layer=1)
cost_func = nn.MSELoss(size_average=False)
optim = torch.optim.SGD(model.parameters(), lr=learning_rate)

for epoch in range(epochs):
    y_pred = model(x_train)
    loss = cost_func(y_pred, y_train)
    print("traning: ", loss.data)
    optim.zero_grad()
    loss.backward()
    optim.step()
    
test_data = torch.FloatTensor([[2.0]])
print("predicted output: ", model(test_data).data[0][0].item())
</code></pre>

Oli: <p>Your model only has one node. It is to simple to compute anything. But either way, the code looks clean! What is the output and why do you have so little data?</p>

surfsamson: <p>Thank you Oli for your swift response.<br>
I am very new to pytorch and i’m just using this as an example to test neural network in pytorch.<br>
However, when i eject 2.0 as the years of experience, hoping to get something similar to 20,000 as in the dataset, but i’m getting way above it and when i tweak the hyper-parameters i get different values.<br>
Please any idea? what am i doing wrong.</p>
Batch_norm the derivative for 'running_mean' is not implemented

tueboesen: <p>I’m trying to reproduce the Wide residual network 28-2 for a semi supervised learning article I’m creating. But I’m having trouble using the Batch_norm.</p>
<p>I keep getting this error:<br>
File “C:\Anaconda3\lib\site-packages\torch\nn\functional.py”, line 1708, in batch_norm<br>
training, momentum, eps, torch.backends.cudnn.enabled<br>
RuntimeError: the derivative for ‘running_mean’ is not implemented</p>
<p>Currently I’m using it like this:<br>
F.batch_norm(z,weight=bnW0,bias=bnB0,running_mean=bnM0,running_var=bnV0,training=training)</p>
<p>where weight,bias,running_mean, and running_var all have been instatiated as:<br>
nn.Parameter((torch.rand(16) - 0.5) * 1e-1)</p>
<p>Is batch_norm currently not working in training mode, or am I just doing something wrong here?</p>

Mazhar_Shaikh: <p>Hi tueboesen,<br>
The running_mean and running_var should be registered as buffers and not parameters. The update to these tensors happens inside the function call. And gradients are not available. Hence, the error.</p>

tueboesen: <p>Okay thank you for clarifying.</p>
<p>If there is no plan of implementing that at some point I would say that the error should probably be changed to be more informative, because right now it just sounds like batch_norm isn’t working.</p>