akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Extract feature using pytorch

Sammo1: <p>I need to write a code using PyTorch to extract features like this one</p>
<pre><code class="lang-auto">def extract_features(image):
    feature_extractor = models[config.base_model_name]["model"](
        include_top=False, weights="imagenet"
    )

    print(feature_extractor.summary())
    if config.pooling != "avg":
        feature_extractor = tf.keras.Model(
            feature_extractor.input,
            tf.keras.layers.AveragePooling2D(int(config.pooling[0]))(
                feature_extractor.output
            ),
        )
</code></pre>
<p>I tried this one</p>
<pre><code class="lang-auto">
model.head = nn.Identity()
model.eval()
def extract_features(directory):
    results = []
    preprocess = T.Compose([
        T.Resize(256, interpolation=3),
        T.CenterCrop(224),
        T.ToTensor(),
        T.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
    ])
    for name in listdir(directory):
        filename=directory + '/' + name
        image= load_img(filename)
        vector=model(preprocess(image))
</code></pre>
<p>but the <code>averaging pool</code> I don’t use, how can I apply it with PyTorch, or does the Keras function extract features from the layer before classification only?</p>

Michal_Bogacz: <p>Could you show which exact model are you using and which layers do you want to remove? According to classic implementation of the Style Transferring (which i believe is very similar to your problem), features are extracted from various depths of the model. But you might require another approach.</p>
<p>Please do, <code>print(model)</code> and describe your problem better.</p>

Sammo1: <p>I’m using the ResNet50 model and I need to remove the last layer which is for classification as I need to extract the features for the images before classifying them. but i found this code for extract features of medical images</p>
<pre><code class="lang-auto">def extract_features(image):
    feature_extractor = models[config.base_model_name]["model"](
        include_top=False, weights="imagenet"
    )

    print(feature_extractor.summary())
    if config.pooling != "avg":
        feature_extractor = tf.keras.Model(
            feature_extractor.input,
            tf.keras.layers.AveragePooling2D(int(config.pooling[0]))(
                feature_extractor.output
            ),
        )
</code></pre>
<p>so I compared what I’m using for the code I saw and found that difference, does that make changes in the features I extract or not</p>
Load_state_dict returns 'object is not subscriptable'

Marco_Perin: <p>Hi everyone,<br>
I have the following model (from transformers):</p>
<pre><code class="lang-auto">class ROBERTAClass(torch.nn.Module):
    def __init__(self):
        super(ROBERTAClass, self).__init__()
        self.l1 = RobertaForTokenClassification.from_pretrained('roberta-base', num_labels= number_of_outputs)
      
    
    def forward(self, ids, mask, label):
        outputs= self.l1(ids, attention_mask = mask, labels=label)


        #return torch.nn.Softmax(1)( outputs.logits)
        # return torch.sigmoid(outputs.logits)
        return outputs
</code></pre>
<p>and the following save and load functions</p>
<pre><code class="lang-auto">
def save_ckp(state, checkpoint_path):

    f_path = checkpoint_path

    torch.save(state, f_path)

def load_ckp(checkpoint_fpath, model, optimizer):

    checkpoint = torch.load(checkpoint_fpath)
  
    model.load_state_dict(checkpoint['state_dict'])

    optimizer.load_state_dict(checkpoint['optimizer'])
 
    valid_loss_min = checkpoint['valid_loss_min']
   
    return model
   
   
</code></pre>
<p>The issue is that if I save the model and then load it I get the following error on   the call of model.load_state_dict(checkpoint[‘state_dict’]): <strong>ROBERTAClass object is not is not subscriptable</strong>.<br>
Do you know the reason for this?<br>
Thanks!</p>

ptrblck: <p>Based on the error I guess you are not saving the <code>state_dict</code> (as would be the recommended way) but the model directly while assuming that <code>checkpoint</code> is a <code>dict</code> containing several internal <code>state_dict</code>s.<br>
Here is a small example showing the issue:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)

# save model directly
torch.save(model, "tmp.pt")
checkpoint = torch.load("tmp.pt")
checkpoint['state_dict']
# TypeError: 'Linear' object is not subscriptable
</code></pre>

Marco_Perin: <p>Thanks so much, that’s it. Such a silly mistake <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
Calling F.relu() in forward() of custom activation function

zfzhang: <p>Hi,<br>
I am defining a custom activation function, which inherits from Function class and has static methods forward() &amp; backward().  If I call F.relu() inside forward(), will it incur autograd (which is undesirable)?</p>
<p>Thank you!</p>

eqy: <p>Could you check if wrapping your call to <code>F.relu()</code> inside a <code>with torch.no_grad()</code>: would work?</p>

ptrblck: <p><a class="mention" href="/u/eqy">@eqy</a>’s suggestion would also work, but by default gradient calculation is disabled in custom <code>autograd.Functions</code> and you could check it via <code>print(torch.is_grad_enabled())</code> which will return <code>False</code> inside the <code>forward</code> and <code>backward</code> methods.</p>

zfzhang: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> &amp; <a class="mention" href="/u/eqy">@eqy</a> , thanks for the prompt &amp; helpful reply !</p>
Where and how does prefetch_factor work?

yzholdfish: <p>a simple trainning<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/8/28cd5eb6f19195c0aaf757e8a022c8fcfd1957ed.png" alt="image" data-base62-sha1="5OX6jXZFxW2XRvQSj8MzsIQyhit" width="503" height="277"><br>
During the training, i found that there will be a long wait every other period of time, which corresponds to the value of num_workers.In dataloader, prefetch_factor is 2, i think the cycle should be prefetch_factor * num_workers</p>
<p>I commented out the calculation process in the picture 1, and the phenomenon is more obvious<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/4634a22542fe8df38e95af66ed0737d7c27a1894.png" data-download-href="https://discuss.pytorch.org/uploads/default/4634a22542fe8df38e95af66ed0737d7c27a1894" title="图片11"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/4634a22542fe8df38e95af66ed0737d7c27a1894_2_690x347.png" alt="图片11" data-base62-sha1="a14bKaxKrGhagfSa3nzpQUUEwdK" width="690" height="347" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/4634a22542fe8df38e95af66ed0737d7c27a1894_2_690x347.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/4634a22542fe8df38e95af66ed0737d7c27a1894_2_1035x520.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/4634a22542fe8df38e95af66ed0737d7c27a1894_2_1380x694.png 2x" data-dominant-color="F6F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">图片11</span><span class="informations">1625×818 131 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i find the part of source code of dataloader<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc23f242f3ccec869a9b638413fb512351729da4.jpeg" alt="image" data-base62-sha1="zYxkUgGprqlLiVFI9EmoXnqeUfi" width="554" height="406"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2cf0c0ff368c8ddaeb15bc1ee4d58e1e69a8477.jpeg" alt="image" data-base62-sha1="u4TRDPi6x1poh01PKnQhY8ynf5t" width="527" height="367"><br>
im confused<br>
where does prefetch_factor work?<br>
work in the whole time or just in the initial of the dataloader as the picture above shows</p>

eqy: <p>What is the number of workers here and have you tried increasing it? I’m curious if the number of workers isn’t enough to keep up with the device’s training throughput; note that if the selected number of workers cannot keep up in terms of throughput, prefetching alone cannot address the underlying issue as the workers will always be busy.</p>
Advantages of TorchScript vs. Porting a Model to C++

JamesDickens: <p>Hello. I have a relatively simple model I’m interested in using in production, in particular it is a graph convolutional network CTR-GCN, used for skeletal action recognition, with not much variation in<br>
control statements (if statements).</p>
<p>I’m wondering given the two choices of:</p>
<ol>
<li>
<p>Using TorchScript to scipt the model, save a checkpoint, and then load it in C++ as in this<br>
tutorial :<br>
<a href="https://pytorch.org/tutorials/advanced/cpp_export.html" class="inline-onebox" rel="noopener nofollow ugc">Loading a TorchScript Model in C++ — PyTorch Tutorials 1.11.0+cu102 documentation</a></p>
</li>
<li>
<p>Manually port the code to LibTorch C++ and train a new model (the training process is sufficiently fast, this is a minor concern)</p>
</li>
</ol>
<p>At inference time, will there be a significant advantage to either of these approaches in terms of<br>
performance?  Any insights much appreciated.</p>

eqy: <p>In theory torchscipt can be substantially faster at inference depending on the number of fusable operations in your model (more likely if you have custom layers/activations that have many pointwise operations). If the computation time in your model skews more heavily towards compute-heavy ops like convs and matmuls then the gains may not be as dramatic, but I would still consider a quick experiment using the Python frontend of jit scripting to see if there is a substantial speedup. <a href="https://pytorch.org/docs/stable/jit.html" class="inline-onebox" rel="noopener nofollow ugc">TorchScript — PyTorch 1.11.0 documentation</a></p>
Libtorch with OpenACC doesnt work

estojoverde: <p>I’ve tried a simple code in C++ to test libtorch:</p>
<pre><code class="lang-auto">int main() {
  torch::Tensor out = torch::eye( { 3, 3 }, torch::device(at::kCUDA).dtype(at::kFloat) );
  std::cout &lt;&lt; out &lt;&lt; std::endl;
  return 0;
}
</code></pre>
<p>After some tinkering, it worked ok:</p>
<blockquote>
<p>1  0  0<br>
0  1  0<br>
0  0  1<br>
[ CUDAFloatType{3,3} ]</p>
</blockquote>
<p>However, after trying to compile it with OpenACC, it doesnt work anymore…</p>
<pre><code class="lang-auto">#define SIZE 10000
float a[SIZE][SIZE];
float b[SIZE][SIZE];
float c[SIZE][SIZE];
int main() {
  int i,j;
  for (i = 0; i &lt; SIZE; ++i) {
    for (j = 0; j &lt; SIZE; ++j) {
        a[i][j] = (float)i + j;
        b[i][j] = (float)i - j;
        c[i][j] = 0.0f;
    }
  }
  #pragma acc kernels copyin(a,b) copy(c)
  for (i = 0; i &lt; SIZE; ++i) {
    for (j = 0; j &lt; SIZE; ++j) {
      c[i][j] = a[i][j] + b[i][j];
    }
  }
  torch::Tensor out = torch::eye( { 3, 3 }, torch::device(at::kCUDA).dtype(at::kFloat) );
  std::cout &lt;&lt; out &lt;&lt; std::endl;
  return 0;
}
</code></pre>
<p>It only evaluates at CPU now.</p>
<blockquote>
<p>1  0  0<br>
0  1  0<br>
0  0  1<br>
[ CPUFloatType{3,3} ]</p>
</blockquote>
<p>My cmake is as minimal as I could do it:</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.10)
set (LANGUAGES "CXX")
project(main LANGUAGES ${LANGUAGES})
add_definitions("-DENABLE_SSE")
find_package(Torch REQUIRED)
find_package(OpenACC REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${OpenACC_CXX_FLAGS} ${TORCH_CXX_FLAGS} ")
add_executable(main main.cpp)
target_link_libraries(main "${TORCH_LIBRARIES}")
target_link_libraries(main "${TORCH_CUDA_LIBRARIES}")
target_link_libraries(main ${OpenACC_CXX_OPTIONS})
set_property(TARGET main PROPERTY CXX_STANDARD 14)
</code></pre>
<p>I’m trying to expose memory from torch via cuda array interface to openacc, that’s why I’m compiling both frameworks in the same code.</p>
Location of dropout in CycleGAN

fllci: <p>Hello everyone!</p>
<p>I’ve been training a CycleWGAN-GP model for some time on 1-D data. But, my model overfits really bad. To avoid overfitting, I have tried many solutions:</p>
<ol>
<li>Data augmentation (as a result I have now twice size of the original data)</li>
<li>Tried adding dropout in different layers but this time model did not learn anything at all. Actually, at this point I might have ended the training earlier because even the training results sucked. But I actually waited for enough time to see maybe it improves but no <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> . I am actually very confused where exactly I should add the dropout in my network.</li>
<li>Added decaying white noise in my both domains. This helped me a little but not much. It slightly improved the test results.</li>
<li>Simplified the model. Basically at first I had total of 34 layers (generator and critic) also including residual blocks. I simplified to total of 16 layers. The training results were so bad. Then, gradually increased the layers up back to 34 layers. As I added more layers, the training improved. So does the test, but test results are still so bad.</li>
</ol>
<p>At this point I really don’t know what else I should do. I especially need help on where to put the dropout layers since in practice they really help to prevent overfitting. Any help would be appreciated.<br>
Thanks!</p>
Trying to get new list of indices using custom batch sampler

skarah00: <p>I have several Datasets merged into one large dataset. I am attempting to create a batch sampler such that I receive a list of indices corresponding to the separate datasets within my merged dataset. I need the first index from the list to always come from the 1st dataset, 2nd always come from the 2nd dataset and so on.</p>
<pre><code class="lang-auto">class BatchSamplerIndicies(Sampler):
    def __init__(self):
        self.dataset1 = list(range(0,1505))
        self.dataset2 = list(range(1505,2328))
        self.dataset3 = list(range(2328,3442))
        self.dataset4 = list(range(3442,8293))
      
    def __iter__(self):

        index1 = chunk(self.dataset1)
        index2 = chunk(self.dataset2)
        index3 = chunk(self.dataset3)
        index4 = chunk(self.dataset4)
       
        combined = [index1,index2, index3, index4] 
        return iter(combined)
    def __len__(self):
        return(len(self.dataset1) + len(self.dataset2) + len(self.dataset3) +len(self.dataset4) )
</code></pre>
<p>The code above does provide the shuffled list of indices, however when reading it into a DataLoader, and attempting to view the chosen indices, I recieve the error</p>
<pre><code class="lang-auto">dl=DataLoader(merged_datasets, batch_sampler=BatchSamplerIndicies())
next(iter(dl))
</code></pre>
<pre><code class="lang-auto">TypeError: 'int' object is not iterable
</code></pre>
<p>I assume the <code>combined</code> list isn’t the proper way to get the batch of indices in the strict order that I need them. How would I go about fixing this?</p>
How to have Residual Network using only Sequential Blocks?

pinocchio: <p>I want to implement a ResNet network but I really want it to be in the sequential network form.</p>
<p>What I mean by sequential network form is the following:</p>
<pre><code class="lang-auto">    ## mdl5, from cifar10 tutorial
    mdl5 = nn.Sequential(OrderedDict([
        ('pool1', nn.MaxPool2d(2, 2)),
        ('relu1', nn.ReLU()),
        ('conv1', nn.Conv2d(3, 6, 5)),
        ('pool1', nn.MaxPool2d(2, 2)),
        ('relu2', nn.ReLU()),
        ('conv2', nn.Conv2d(6, 16, 5)),
        ('relu2', nn.ReLU()),
        ('Flatten', Flatten()),
        ('fc1', nn.Linear(1024, 120)), # figure out equation properly
        ('relu4', nn.ReLU()),
        ('fc2', nn.Linear(120, 84)),
        ('relu5', nn.ReLU()),
        ('fc3', nn.Linear(84, 10))
    ]))
</code></pre>
<p>but of course with the NN lego blocks being “ResNet”.</p>
<p>Thanks!</p>

pinocchio: <ul>
<li>cross posted: <a href="https://stackoverflow.com/questions/57229054/how-does-one-implement-my-own-resnet-with-torch-nn-sequential-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/57229054/how-does-one-implement-my-own-resnet-with-torch-nn-sequential-in-pytorch</a>
</li>
</ul>

Brando_Miranda: <p>related: <a href="https://www.reddit.com/r/pytorch/comments/uyyr28/how_to_implement_my_own_resnet_with/" rel="noopener nofollow ugc">https://www.reddit.com/r/pytorch/comments/uyyr28/how_to_implement_my_own_resnet_with/</a></p>
Matrix Multiplication Question

cjohns: <p>I’m working on an embedding problem.  I want to multiply each row of a tensor with another matrix and sum the result.  I’m struggling with how to do this without using a loop. The following example works via indexing (row 0 for tensor x).  How do I do this without having to use a loop?</p>
<pre><code class="lang-auto">import torch
x = torch.rand(100,120)
y = torch.rand(514, 120)
result = (x[0]*y).sum(0)
print(result)
</code></pre>

Matias_Vasquez: <p>You could use <code>einsum</code>.</p>
<pre><code class="lang-python">import torch
x = torch.rand(100,120)
y = torch.rand(514, 120)

# Yours
result_1 = (x[0]*y).sum(0)

# Einsum
result_2 = torch.einsum("ij, kj -&gt; ij", [x, y])

# Select only the first row of result_2 to compare to the indexed version
print(torch.allclose(result_1, result_2[0]))
</code></pre>
<pre><code class="lang-python"># Output:
True
</code></pre>

cjohns: <p>I ran across the einsum when doing some searching but wasn’t sure if it was the right solution or not.  Thanks for taking the time for pointing me in the right direction, thank you!</p>
Add an attention meachanism to LSTM

joekid: <p>Hi, guys,</p>
<p>I am studying LSTM, and in <a href="https://discuss.pytorch.org/t/lstm-input-size-hidden-size-and-sequence-lenght/151817" class="inline-onebox">Lstm input size, hidden size and sequence lenght</a>, I found a good illustration how LSTM process input to outputs, so I implemented a sequence-to-label classifier.</p>
<p>However, I like to add a temporal attention, but reading in the forum I don’t  find basis ideas to implement it, and some papers were confused for me.</p>
<p>If someone can share your experience or some useful tutorial, I could implement it.</p>
How to skip quantizing specific layers in fx quantization QAT

heng: <p>Hi there,</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = nn.Sequential(
                nn.Conv2d(3, 128, 3, 1, 1), 
                nn.BatchNorm2d(128),
                nn.ReLU(inplace=True),
                nn.Conv2d(128, 128, 3, 1, 1), 
                nn.BatchNorm2d(128),
                nn.ReLU(inplace=True),
        )
    
    def forward(self, x): 
        return self.conv(x)


class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.backbone = Net()
    
        self.avg_pool = nn.AdaptiveAvgPool2d((1, 1)) 
    
        self.fc = nn.Linear(128, 32) 
        self.relu = nn.ReLU(inplace=True)
    
    def forward(self, x): 
        x = self.backbone(x)

        x = self.avg_pool(x)
        x = x.view(x.size(0), -1) 
        x = self.fc(x)
        x = self.relu(x)
    
        return x
</code></pre>
<p>If I want to  attach observers (weight+activation) to backbone layers only,  and keep the rest layers (self.avg_pool, self.fc, self.relu)   unquantized, how should I define  the qconfig_dict?</p>
Increasing of model size leads to poor backward performance

konstantin-m: <p>I built ResNet - like model with Pytorch 1.11.<br>
I use a machine with 2 CPU (12 cores), 72 GB memory, no GPU.<br>
While trainig the following results were obtained:</p>
<p>|params        |forward, s|backward, s|<br>
|76 929 892  |2,81         |7,1              |<br>
|80 075 818  |3,14         |7,76            |<br>
|83 322 861  |3,32         |7,95            |<br>
|86 634 044  |3,45         |8,47            |<br>
|90 009 367  |3,58         |8,86            |<br>
|93 516 104  |3,47         |8,92            |<br>
|96 979 046  |3,90         |9,23            |<br>
|100 547 209|4,01         |9,93            |<br>
|104 179 512|4,27         |10,19          |<br>
|107 875 955|4,31         |244,46        |<br>
|111 710 028|4,17         |255,14         |</p>
<p>Seems like I achieved limitation of some resource, L3 cache?</p>
<p>Any ideas how to resolve it?</p>
<p>Thank you.</p>
Adding in the weight parameter for PyTorch's cross-entropy loss causes datatype RuntimeError

seankala: <p>I’m currently using PyTorch to train a neural network. The dataset that I’m using is a binary classification dataset with a large number of 0’s.</p>
<p>I decided to try and use the <code>weight</code> parameter of PyTorch’s cross-entropy loss. I calculated the weights via <code>sklearn.utils.class_weight.compute_class_weight</code> and got weight values of <code>[0.58479532, 3.44827586]</code>.</p>
<p>When I added this <code>class_weights</code> tensor into the <code>weight</code> parameter of my loss (i.e., <code>criterion = nn.CrossEntropyLoss(weight=class_weights)</code>, I’m suddenly getting a <code>RuntimeError: expected scalar type Float but found Double</code>. The outputs and labels that I’m feeding into my loss are of types <code>float32</code> and <code>int64</code>, respectively. The loss was working fine but when I add the weight parameter I’m getting this error. Attempting to cast my data via <code>outputs.float()</code> doesn’t seem to work either.</p>
<p>Does anybody know why this error may be occurring and how I might fix this? Thanks.</p>

ptrblck: <p>Most likely you are passing the <code>class_weights</code> as <code>float64</code> as this is the default <code>dtype</code> in <code>numpy</code> which raises the error as seen here:</p>
<pre><code class="lang-python">x = torch.randn(2, 2, requires_grad=True)
y = torch.randint(0, 2, (2,))

criterion = nn.CrossEntropyLoss()
loss = criterion(x, y) # works

weight = torch.tensor([0.1, 0.2], dtype=torch.float32) # float32 is the default so specifying it is just for clarity
criterion = nn.CrossEntropyLoss(weight=weight)
loss = criterion(x, y) # works

weight = weight.double()
criterion = nn.CrossEntropyLoss(weight=weight)
loss = criterion(x, y)
# RuntimeError: expected scalar type Float but found Double
</code></pre>
<p>Transform the <code>weight</code> to <code>float32</code> and it should work.</p>
C++ libtorch library not found for -lmkl_intel_ilp64

peterbaile: <p>I followed the <a href="https://pytorch.org/cppdocs/" rel="noopener nofollow ugc">documentation</a> here to use pytorch in C++. I was able to compile everything until the building stage where I bumped into an issue of <code>ld: library not found for -lmkl_intel_ilp64</code>. I’m wondering what’s the solution to it, thanks!</p>
<p>I’m using Mac (OS is Catalina) and installed 1.11.0 libtorch</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f053415b530f55742f5505257971e8b5d534d769.png" data-download-href="https://discuss.pytorch.org/uploads/default/f053415b530f55742f5505257971e8b5d534d769" title="Screen Shot 2022-05-26 at 10.23.04 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f053415b530f55742f5505257971e8b5d534d769_2_690x430.png" alt="Screen Shot 2022-05-26 at 10.23.04 PM" data-base62-sha1="yi0XVtsO9lK0Uh2THKw05ntCOM1" width="690" height="430" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f053415b530f55742f5505257971e8b5d534d769_2_690x430.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f053415b530f55742f5505257971e8b5d534d769_2_1035x645.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f053415b530f55742f5505257971e8b5d534d769_2_1380x860.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-05-26 at 10.23.04 PM</span><span class="informations">2108×1316 348 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You might want to add the folder containing <code>libmkl_core.so</code> etc. to <code>LIBRARY_PATH</code> so that the build stage find these libraries.</p>
Is it possible to create a branch in a torch::nn::Sequential model?

pollo_loco: <p>I have a single torch::nn::Sequential object to which I am adding each layer of ResNet50 model. For example, the third module that I need to add to Sequential object should take the output of the first layer and not the second. Is this even possible? In essence can we create branches in the model? If yes, how?</p>
<p>Thank you</p>

leoil: <p>I tried to split one sequential model into two seperate Sequential, and add branch after the first one. And merge the two branch at the end. It worked for me.</p>
<p>However, I find the training time nearly doubled, while the <span class="hashtag">#params</span> is almost the same.</p>
<p>I wonder if there are any better solutions.</p>
Given transposed=1, weight of size [3, 1, 2, 2], expected input[8, 1024, 8, 10] to have 3 channels, but got 1024 channels instead

SiYeon-Kwon: <p>I am merging the two by referring to the fast-depth and sparse-to-dense of girhub. (I want to use sparse-to-dense on tx2)<br>
However, if insert fast-depth’s model.py into sparse-to-dense’s model.py and run it, the following error occurs:<br>
Is there a way to reduce the channel?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19bd463b1ace5146b13dd35d47fbac23ace9b4a5.png" data-download-href="https://discuss.pytorch.org/uploads/default/19bd463b1ace5146b13dd35d47fbac23ace9b4a5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19bd463b1ace5146b13dd35d47fbac23ace9b4a5_2_690x468.png" alt="image" data-base62-sha1="3FHsdE6IGSwEVBHxXmCBLUcFafz" width="690" height="468" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19bd463b1ace5146b13dd35d47fbac23ace9b4a5_2_690x468.png, https://discuss.pytorch.org/uploads/default/original/3X/1/9/19bd463b1ace5146b13dd35d47fbac23ace9b4a5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/9/19bd463b1ace5146b13dd35d47fbac23ace9b4a5.png 2x" data-dominant-color="351F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">917×623 54 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

SiYeon-Kwon: <p>I’ve got a solution, thank you everyone!!</p>
Is there anyway to avoid oom?

wayne1226: <p>I do an element-wise multiplication as:</p>
<pre><code class="lang-auto">    a = a.unfold(1, val*2, 1).unfold(2, val*2, 1)
    b=b.unsqueeze(dim=0).unfold(1, val*2, 1).unfold(2, val*2, 1)
    a=((a*(a&gt;c)).sum((3,4)))/((a&gt;c).sum((3,4)))
</code></pre>
<p>here, a is a [3,256,256,256,256] matrix, b is [1,256,256,256,256], and I got an OOM when I run this code, I also try the in-place operation like a*=(a&gt;c), although the memory consumption is reduced, I still got an OOM, even the (a&gt;c).sum((3,4) consume a lot of memory. Is there any way to avoid OOM and implement the same functional?</p>

ptrblck: <aside class="quote no-group" data-username="wayne1226" data-post="1" data-topic="151658">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wayne1226/48/15986_2.png" class="avatar"> wayne1226:</div>
<blockquote>
<p>a is a [3,256,256,256,256] matrix, b is [1,256,256,256,256]</p>
</blockquote>
</aside>
<p><code>a</code> will already allocate <code>48GB</code> in <code>float32</code> and <code>b</code> <code>16GB</code>. Additionally <code>a&gt;c</code> will create a temp. tensor, which should also consume <code>16GB</code>, so you would be at <code>80GB</code> to store these tensors alone.<br>
Since the CUDA context and potentially other tensors would also need some memory, I don’t think you could fit it into a GPU without scaling down the problem.</p>

wayne1226: <p>Thank you for your reply.<br>
What I want to do is similar to a ‘2-d convolution with a different kernel on each element location’, now I apply this by for-loop like this</p>
<pre><code class="lang-auto">d = torch.zeros(1,3,256,256).cuda()
for idx in range(val,color.shape[1]-val):
        for idy in range(val,color.shape[2]-val):
            b_block=b[idx-val:idx+val,idy-val:idy+val].clone() 
            a_block = a[...,idx - val:idx + val, idy - val:idy + val].clone()
            c_1=b_block&gt;c
            rendered_img[...,idx-val, idy-val]=(c_1*a_block).sum((1,2))/c_1.sum()
</code></pre>
<p>Although this implementation can avoid OOM in this image scale, it is time-consuming and cannot handle images on a larger scale. Can you give me some advice on this? Or I have to trade-off between timing and memory? If so, what if don’t care about the timing?</p>

ptrblck: <p>If you don’t care about the timing, then the nested loop should work as it should use the least necessary memory.</p>

wayne1226: <p>I’m trying to add more constraints to scaling down the problem, thanks again for your reply.</p>

wayne1226: <p>I’m now scaling down the problem but the oom still occurs times, can I chunk the tensor ‘a’ and ‘b’ to multi-GPU? I have tried but it seems it doesn’t work.</p>
What recompute_scale_factor actually does in interpolate?

abhisek: <p>I understand that the <code>torch.nn.functional.interpolate</code> first computes the out size(s) using <code>round(size * scale)</code>. According to documentation:</p>
<blockquote>
<p>If <code>recompute_scale_factor</code> is <code>True</code>, then scale_factor must be passed in and scale_factor is used to compute the output size. The computed output size will be used to infer new scales for the interpolation. Note that when scale_factor is floating-point, it may differ from the recomputed scale_factor due to rounding and precision issues.</p>
</blockquote>
<p>How does this recomputed scale factor helps in interpolation when the output size has already been calculated?</p>
<p>When and what difference does it make if I toggle the <code>recompute_scale_factor</code> for the exact same input?</p>
Cannot reproduce results given by torchvision.ops.nms

CorsBlady: <p>Hello,</p>
<p>I’m struggling to understand why results of torchvision.ops.nms differ so much when compared with some of the implementations that I’ve tried to recreate. I’m doing some sample projects in cpp and I need to code up NMS for object detection. In my PyTorch code I’m using <code>torchvision.ops.nms</code>, I’ve been trying to reimplement it in Python in the first place so that I can easily test my new code against my current one that is using <code>torchvision.ops.nms</code>. I’ve tried the following 3 implementations:</p>
<ol>
<li><a href="https://github.com/aladdinpersson/Machine-Learning-Collection/blob/master/ML/Pytorch/object_detection/metrics/nms.py" class="inline-onebox" rel="noopener nofollow ugc">Machine-Learning-Collection/nms.py at master · aladdinpersson/Machine-Learning-Collection · GitHub</a></li>
<li><a href="https://learnopencv.com/non-maximum-suppression-theory-and-implementation-in-pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Non Maximum Suppression: Theory and Implementation in PyTorch</a></li>
<li>Last one I’ve used (can’t paste the link as new user is limited to 2 links) was from PyTorch github, more specifically from <code>pytorch/vision/torchvision/csrc/cpu/nms_cpu.cpp</code>
</li>
</ol>
<p>As to my surprise, all 3 implementations give me exactly the same <code>mAP</code> results, however, none of them give me the same results as the one obtained from <code>torchvision.ops.nms</code>. Is this something that anyone noticed before, and if so, what would be the solution? I was hoping that the 3rd implementation which is literally a Python version of <code>nms_cpu</code> would solve it but to my surprise it didn’t.</p>
<p>Appreciate any feedback.</p>
<p>Cheers</p>
Deem the lower part of an Image in PyTorch

hustine316: <p>I wish to deem a part of an image. Not completely black but just creating shadows in the lower regions.</p>

Andrei_Cristea: <p>You just have to bring the pixel values of the tensor that represents the image closer to zero. Something like this:</p>
<pre><code class="lang-auto">import torchvision
img = Image.open("cat.jpg").resize((224, 224))
img_tensor = torchvision.transforms.ToTensor()(img)
img_tensor[:, 180:, :] *= 0.75  # reduce the brightness of everything below pixel 180 by 25%
img_shaded = torchvision.transforms.ToPILImage()(img_tensor)
</code></pre>
<p>original:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67ecb21f7fbd11d69d4ebea886b0cdc8b595e6d8.png" alt="image" data-base62-sha1="ePmisX0mcLggjL1c4IO0YZzsJ1S" width="168" height="168"></p>
<p>after shading:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c83d5e2bd61bfdfc8381960fa4ab6c1c8dec748.png" alt="image" data-base62-sha1="1MIbwwWlAINN5gKVT70UD0M5HNC" width="168" height="168"></p>

hustine316: <p>Wow, thanks a lot. This is exactly what I wanted.</p>

hustine316: <p>What if I don’t always intend to use a straight line. E.g a wavy line or some sort of randomized line going from one point to another endpoint but not a straight line?</p>

Andrei_Cristea: <p>In that case you need to replace this with some fancier, custom logic:</p>
<pre><code class="lang-auto">img_tensor[:, 180:, :] *= 0.75  # reduce the brightness of everything below pixel 180 by 25%
</code></pre>
<p>It will come down to finding clever ways to parameterize the behavior you’d like without too much work. For example for a wavy line you might do:</p>
<pre><code class="lang-auto">img = Image.open("cat.jpg").resize((224, 224))
img_tensor = torchvision.transforms.ToTensor()(img)

period = 80
amplitude = 20

for n in range(224):
    i = amplitude * math.sin(n / (period / (2 * math.pi)))
    img_tensor[:, (180 + int(round(i))):, n] *= 0.75
img_shaded = torchvision.transforms.ToPILImage()(img_tensor)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f576b44b49a3949d0c9075799dd670bd0640b060.png" alt="image" data-base62-sha1="z1tiMHSP1WEH5EF8oBLMSn6UPyo" width="168" height="168"></p>

hustine316: <p>Thanks so very much Andrei. You’re such a gem. Everything is perfect the way I want it now. Thanks for your reply.</p>
Building a CNN class

Techloverguy: <p>I need help to build a CNN class with the following architecture:</p>
<p>(1) Input: 26</p>
<p>(2) 1D convolution: 256 output channels and a configurable kernel size(specified as an argument to <strong>init</strong>)</p>
<p>(3) Relu</p>
<p>(4) 1D convolution: 256 output channels and the same kernel size as the 1D convolution at position 1</p>
<p>(5) fully connected layer: 25 output features</p>
<p>(6) output:  -</p>
ANN Immune Noise

Andre_Amaral_IST: <p>Hello,</p>
<p>I am training an ANN to model the behaviour of a circuit. My model is capable to follow the amplitude and frequency variations with good accuracy.</p>
<p>However, the label curve presents noise. Why does not my model capture the noise in the input data and produces an output with noise, it seems that my network filters the noise.</p>
<p>Any theoretical explanation?</p>
Gradient question

deepfailure: <p>Hello pytorch community,</p>
<p>Consider a neural network N_\theta(x) which outputs y. Then consider an iterative method F applied on y that outputs z.</p>
<p>I need to calculate a loss which is L(\theta) = |z_real - z_prediction|. However, the gradients can not flow through F as it is an iterative method (as far as my limited understanding goes).</p>
<p>If I would represent the gradient flow from z to x I would have:</p>
<p>d z/d_\theta z = d_z/d_y d_y/d_\theta</p>
<p>What should I do with the term d_z/d_y? Is it possible to put it to 1 somehow?</p>
<p>Thanks in advance.</p>

JuanFMontesinos: <p>If F is backpropagable then they will go through F.</p>
<p>Think that your big graph is:<br>
y --&gt;F_iter(y)–&gt;z<br>
But it can be decomposed in<br>
y --&gt;F(y)–&gt;z_1–&gt;F(z_1)–&gt;z_2–&gt;F(z_2)–&gt;…–&gt;z_n–&gt;F(z_n)–&gt;z<br>
Where you have all upstream and downstream gradients.</p>
<p>And so Dz/Dy will be computed via chain rule as any other op in pytorch. You can in fact check it by calling your iterative function with a random vector, suming the output to obtain a scalar and then backprop it up to the random tensor and check wheter it has a gradient or not.</p>

deepfailure: <p>Unfortunatedly, F is not backpropagable in this case. I wonder:</p>
<ol>
<li>What would happen if I just set Dz/Dy just to one (in a theoretical sense)?</li>
<li>Is it possible to do that in pytorch? Perhaps, I could just get Dy/Dtheta and then multiply by z_target-z_prediction.</li>
</ol>

AlphaBetaGamma96: <p>Can you share a minimal reproducible error of this? It’ll be easier for people to help if they have a code snippet they can visualize and correct themselves!</p>

deepfailure: <p>It should be something like this. The <strong>calculate_z</strong> function works inside a simulation and it’s not backpropagable. Therefore I would like to set d_z/d_y = 1 however I would like to still include the difference between z_real and z_prediction in the update part.</p>
<p>x: input<br>
y: target 1<br>
z: target 2<br>
<a href="http://self.net" rel="noopener nofollow ugc">self.net</a>: neural network to optimise</p>
<pre><code>def example(self, x, y_real, z_real):
    y_predicted = self.net(x)
    z_predicted = self.calculate_z(y_predicted)
    self.loss = self.criterionL1(z_predicted, z_real) + self.criterionL1(y_real, y_predicted)
    self.loss.backward()
</code></pre>
<p>Any ideas?</p>

AlphaBetaGamma96: <p>Well, if your function isn’t differentiable you can’t learn from gradient descent. If you want to set <code>d_z/d_y</code> to 1 you can do that via using a <code>torch.autograd.Function</code> object with its forward being your <code>calculate_z</code> function and the backward being defined as something along the lines of <code>grad_in = grad_output @ torch.ones_like(size)</code></p>
Multiple Successive Backwards to Train each sample

DatDat8: <p>Hello everyone, I am currently involved in a problem of Missing data imputation  using Autoencoder for corrupted sample (randomly zeroed) and its ground_truth, which requires retraining each sample batch multiple times. So I try to use optimizer and backpropagation like this on each training times of the sample. It raised a Runtime error like this:</p>
<blockquote>
<p>Blockquote “Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.”</p>
</blockquote>
<p>I followed and added the retain_graph=True in the backward() but it still produced an error:</p>
<blockquote>
<p>Blockquote “one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [5, 9]], which is output 0 of TBackward, is at version 9; expected version 8 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).”</p>
</blockquote>
<p>I also tried to add the hint but it actually did not work and I am being confused now. My competencies are still not enough to understand thoroughly the new situation here. Hopefully I can receive some advices from your sides, many thanks!</p>
<pre><code class="lang-auto">import torch.optim as optim
model=AutoEncoder1D(n_bus=3, latent_dim=5)
criterion = nn.MSELoss()
n_epochs=100
epsilon=1e-4
optimizer=optim.Adam(model.parameters(), lr=1e-3 ) 
err_all=[]

for i, (X_corr, X_gr) in enumerate(zip(X_corrupted, X_groundtr)):

    missing_idx = np.where(X_corr==0)[1]

    err_sample=[]  

    for j in range(n_epochs):

        # loss calculation and saving

        X_pred = model(X_corr)

        loss_epoch = criterion(X_pred, X_gr)

        err_sample.append(loss_epoch.item())

       

        # back propagation

        #with torch.autograd.set_detect_anomaly(True):

        optimizer.zero_grad()

        loss_epoch.backward()#retain_graph=True)

        optimizer.step()

       

        # updating the predicted values for missing positions

        X_corr[0][missing_idx] = X_pred[0][missing_idx]

       

        # tracking the training error

        if j % 20 == 0:

            print("MSE Training Error at {}_th epoch of {}_th sample: {}"

                  .format(j, i, loss_epoch))

    err_all.append(np.max(err_sample))


</code></pre>

ptrblck: <p>I guess you are keeping the computation graph alive via:</p>
<pre><code class="lang-python">X_corr[0][missing_idx] = X_pred[0][missing_idx]
</code></pre>
<p>Assuming you want to replace some values in the input without tracking the gradient history you could <code>.detach()</code> the right-hand side and try to assign it to <code>X_corr</code>.</p>

DatDat8: <p>Many thanks sir, it actually helped me a lot!<br>
If there is anything else, it would be my honor to receive your support! I hope you have a good day!</p>

ptrblck: <p>Assuming the error is gone I don’t see anything obvious or are you still hitting the issue?</p>
When and why do I need to use detach() in loss calculating and backpropagation?

Emanul_Yang: <p>Hi,</p>
<p>I am running a project where each agent has its own networks, so I need to use backward() multi times.<br>
I have posted an issue to illustrate it: <a href="https://discuss.pytorch.org/t/i-am-training-my-multi-agents-reinforcement-learning-project-and-i-got-an-error-trying-to-backward-through-the-graph-a-second-time/152352">I am training my multi agents reinforcement learning project, and I got an error “Trying to backward through the graph a second time…” - reinforcement-learning - PyTorch Forums</a>.</p>
<p>Some modified codes are here:</p>
<pre><code class="lang-auto"> for agent_idx in range(self.n_agents):
        """ Critic Part """
        # current Q estimate
        current_Q1, current_Q2 = agent.critic.forward(states, old_actions)
        # target Q value
        with T.no_grad():
           target_Q1, target_Q2 = agent.target_critic.forward(states_, new_actions)
           target_Q_min = T.min(target_Q1, target_Q2)
           # target_Q[dones[:, 0]] = 0.0
           target_Q = rewards[:, agent_idx] + (agent.gamma * target_Q_min)
        # critic loss
        # critic loss calculation
        self.agents[agent_idx].critic_loss = F.mse_loss(current_Q1.float(), target_Q.float()) +\
                                             F.mse_loss(current_Q2.float(), target_Q.float())

        # critic optimization
        self.agents[agent_idx].critic.optimizer.zero_grad()
        self.agents[agent_idx].critic_loss.backward()
        self.agents[agent_idx].critic.optimizer.step()

        """ Actor Part """
        if steps_total % self.freq == 0 and steps_total &gt; 0:
            # actor loss calculation, here I use detach() function for variables
            self.agents[agent_idx].actor_loss = self.agents[agent_idx].critic.Q1(states.detach(), mu.detach())
            self.agents[agent_idx].actor_loss = -T.mean(self.agents[agent_idx].actor_loss)
            # actor optimization
            self.agents[agent_idx].actor.optimizer.zero_grad()
            self.agents[agent_idx].actor_loss.backward()
            self.agents[agent_idx].actor.optimizer.step()
            self.agents[agent_idx].update_network_parameters()
</code></pre>
<p>After adding some detach() functions, my codes could run. However, I have no idea when and why I need to use this function.</p>
<p>Another issue is that both critic and actor need some variables(states, old_actions, states_, new_actions, mu) to calculate losses. In the critic part, I do not need to detach variables(states, old_actions, states_, new_actions) and the backward() function could be called multi times. However, if the variables(states, mu) are not detached in actor part, there is an error about ‘backward through the graph a second time’. I have no idea about this issue.</p>
High PCIE traffic

Sharpy: <p>I’m getting a very high PCIE bandwidth usage training a UNet-like network, in the viscinity of 50% of PCIe 3.0 x16 for no apperent reason. Training on a single GPU.</p>
<p>Did you ever experience this sort of problem and, maybe, know how to fix this? Or at least, how to check what’s causing this, debug.</p>

Sharpy: <p>Backwards pass seems to be much more PCIE-intensive - few times so - than forward pass.</p>

taehyunzzz: <p>Hi, can you elaborate on how you are profiling PCIe bandwidth during training?</p>
Status of CUDA Quantization

jonahclarsen: <p>Hi there,</p>
<p>I have been facing a frustrating issue over the past few months.</p>
<p>I am creating a CUDA-accelerated neural style transfer plugin in LibTorch, but as it is, it takes up far too much VRAM because the model I’m using (VGG-19) is so large. The only viable solution I can see for adequately (3-4x) reducing VRAM is through int8 quantization. However, this isn’t currently implemented in PyTorch.</p>
<p>I tried using Torch-TensorRT, and even contributed to getting it working on Windows, but that won’t support back-propagation (a requirement for neural style transfer), plus I’m not even sure it will substantially (if at all) reduce VRAM usage; I am planning on testing this soon.</p>
<p>The only viable solution seems to be PyTorch/LibTorch supporting CUDA int8 quantization. I’ve seen it mentioned across Github and this forum for a few years, but there doesn’t seem to be any clear indication on its current status.</p>
<p>When can we expect this feature to be released? Will it be a part of 1.12? Is there currently a fork of PyTorch that already has it implemented, at least with enough supported ops to run VGG-19?</p>
<p>Also, if you have any other suggestions that might be helpful for this problem, that would be greatly appreciated; I am open to almost any solution here.</p>
<p>Thank you!<br>
Jonah</p>

Vasiliy_Kuznetsov: <p>Just checking, have you tried NVIDIA amp?</p>
<p>We are working on PyTorch quantization → TensorRT as well as eager mode int8 CUDA kernels, but both of those are not releasing in v1.12 (no committed release date at this moment) and are also targeting inference first, without support for backpropagation.</p>
<p>For unofficial status of eager mode CUDA int8 ops, you could check out <a href="https://github.com/pytorch/pytorch/blob/master/test/quantization/core/test_quantized_op.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_quantized_op.py at master · pytorch/pytorch · GitHub</a>, specifically the tests marked with <code>@unittest.skipIf(not TEST_CUDNN, "cudnn is not enabled.")</code></p>
Tensorflow Extended with Pytorch

bing: <p>Hi Everyone,<br>
I have trained a model on pytorch.<br>
Now, I want to design a complete pipeline around it. Has anyone tried Tensorflow extended with it and can comment on the compatibility issues with it(if any).<br>
Also, any other suggestions would also be welcomed regarding the production and the complete pipeline.</p>

EvanZ: <p>Would be interested to hear an answer to the OP’s question or if anyone knows if the PyTorch team is working on a similar end-to-end ML ops solution like TFX?</p>

ptrblck: <p>I don’t know the exact use case or support of Tensorflow Extended, but <a href="https://github.com/pytorch/torchx">torchx</a> might target a similar use case.</p>

EvanZ: <p>Thank you! That looks like the bullseye.</p>

dbish: <p>Did you end up using torchx, and if so, did it meet your needs similar to what you were looking for from TFX?</p>

imayachita: <p>I’m having a similar problem and I have just checked <code>torchx</code>. What I want to do is to do data preprocessing (e.g. normalise, embed text), save the transformation as a layer (or anything basically) that I can load back during inference time to ensure the same transformation is applied during training and inference. Unfortunately, I didn’t see <code>torchx</code> is able to do that. Can anyone please advise? Thanks.</p>