akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
"mat1 and mat2 shapes cannot be multiplied (448x4 and 576x10)" while doing Layer Wise relevance Propagation

Ani_H: <p>Hello,<br>
I am getting the following error message: “mat1 and mat2 shapes cannot be multiplied (448x4 and 576x10)”.</p>
<p>I am trying to do Layer Wise Relevance Propagation according to this tutorial: <a href="https://giorgiomorales.github.io/Layer-wise-Relevance-Propagation-in-Pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Layer Wise Relevance Propagation In Pytorch – GiorgioML – Ph.D. Student in Computer Science, MSU</a></p>
<p>Here is the NN class:</p>
<pre><code>class Cnn(nn.Module):
    def __init__(self):
       super(Cnn,self).__init__()
    
        self.layer1 = nn.Sequential(
         nn.Conv2d(3,16,kernel_size=3, padding=0,stride=2),
         nn.BatchNorm2d(16),
         nn.ReLU(),
         nn.MaxPool2d(2)
       )
    
        self.layer2 = nn.Sequential(
          nn.Conv2d(16,32, kernel_size=3, padding=0, stride=2),
          nn.BatchNorm2d(32),
          nn.ReLU(),
          nn.MaxPool2d(2)
        )
    
        self.layer3 = nn.Sequential(
          nn.Conv2d(32,64, kernel_size=3, padding=0, stride=2),
          nn.BatchNorm2d(64),
          nn.ReLU(),
          nn.MaxPool2d(2)
       )
    
    
        self.fc1 = nn.Linear(3*3*64,10)
        self.fc2 = nn.Linear(10,2)
        self.relu = nn.ReLU()
    
    
    def forward(self,x):
    
       out = self.layer1(x)
       out = self.layer2(out)
       out = self.layer3(out)
       out = out.view(out.size(0),-1)
       out = self.relu(self.fc1(out))
       out = self.fc2(out)
       return out
</code></pre>
<p>I have used this to train a network to do binary classification, and am now trying to run Layer Wise Relevance Propagation on it. Here is the part of the LRP code where I am getting the error:</p>
<pre><code>def LRP_individual(model, X, device):
   # Get the list of layers of the network
   layers = [module for module in model.modules() if not isinstance(module, torch.nn.Sequential)][1:]

   # Propagate the input
   L = len(layers)
   A = [X] + [X] * L # Create a list to store the activation produced by each layer

   for layer in range(L):
       A[layer + 1] = layers[layer].forward(A[layer])
    
  # Rest of the LRP function
</code></pre>
<p>I would be very grateful for an explanation or solution to this problem.</p>
How to Exit in DDP

zyc: <p>If I raise a <code>SystemExit</code>, only the process encountered while exit, while rest are waiting infinitely.</p>

cbalioglu: <p>Processes participating in a distributed data parallel job communicate with each other using collective communication calls (e.g. all_reduce, all_gather). If one of your processes fails, those calls will block the rest of your processes. Fault tolerance and job retries are not part of the DDP framework. I would suggest checking <a href="https://pytorch.org/docs/stable/elastic/quickstart.html" rel="noopener nofollow ugc">TorchElastic</a>, and optionally Slurm and other OSS job schedulers for what you want to achieve.</p>

zyc: <p>If one process <em>fails</em>, it will exit. For example, if I raise a <code>SystemError</code>, the whole process exits as expected.</p>
NodeSplitting SegFault

bucky: <p>splitNode method throws segfault. I am in “Debug” mode, and using “resnet-runtime” network as the input.</p>
<p>In my case, the segfault occured when dealing Maxpool and AvgPool nodes.</p>
<pre><code class="lang-auto">for (auto &amp;node : F-&gt;getNodes())
    {
      if (node.getUsers().size() == 0 || static_cast&lt;std::string&gt;(node.getKindName()) == "Convolution")
      {
        continue;
      }
      std::vector&lt;glow::Node *&gt; splitNodes;
      auto dims = node.dims(0);
      if(dims.size() == 4)
      {
        if(dims[3]/VW &gt; 1)
        {
          std::cout &lt;&lt; static_cast&lt;std::string&gt;(node.getName()) &lt;&lt; "\n";
          ASSIGN_VALUE_OR_FATAL(splitNodes, glow::splitNode(&amp;node, SplitNodeByNumChunks({ShapeNHWC::DimC},{dims[3]/VW})));
        }
      }       
    }```</code></pre>

jfix: <p>Thanks for reporting this. Can you file an issue on Github with what you have here, and tag mciprian13, who is the original author of much of the <code>splitNode</code> infra?</p>

bucky: <p><a href="https://github.com/pytorch/glow/issues/5789" rel="noopener nofollow ugc">https://github.com/pytorch/glow/issues/5789</a></p>
One of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [12, 4096]], which is output 0 of EuclideanDistBackward, is at version 4096; expected version 0 instead

111558: <p>There is a bug when I run total_loss.backward() in my main.py<br>
What can I do to fix the bug?<br>
I do not use any -= or += but I still got this</p>
<p>this is my code</p>
<pre><code class="lang-auto">    def forward(self):
        point_cls = torch.reshape(self.point_cls, (4, 4096, 2))
        point_regression = torch.reshape(self.point_regression, (4, 4096, 2))
        batchb = point_cls.shape[0]
        point_num = point_cls.shape[1]
        total_loss = 0
        for b in range(0, batchb):
            l2 = torch.cdist(self.targets[b], point_regression[b, :, :], p=2)
            gt_num = self.targets[b].shape[0]
            for pn in range(0, point_num):
                l2[:, pn] = 0.05 * l2[:, pn] - point_cls[b, pn, 1]
            row_ind, match_res = match(l2.cpu().detach().numpy())
            point_cls_temp = point_cls[b, :, 1]
            regression_loss = l2[row_ind, match_res].sum()/gt_num

            for gt in range(0, gt_num):
                if gt == 0:
                    matched_cls = torch.unsqueeze(point_cls[b, match_res[gt], 1], 0)
                    point_cls_temp = torch.cat([point_cls_temp[0:match_res[gt]], point_cls_temp[match_res[gt]:-1]])
                else:
                    matched_cls = torch.cat((matched_cls, point_cls[b, match_res[gt], 1].unsqueeze(0)), dim=0)
                    point_cls_temp = torch.cat([point_cls_temp[0:match_res[gt]], point_cls_temp[match_res[gt] :-1]])

            matched_cls = torch.cat((matched_cls, point_cls_temp))
            class_loss = torch.div(torch.sum(torch.log(matched_cls[:gt_num])) + 0.5*torch.sum(torch.log(matched_cls[gt_num:])), point_num) * -1
            print(class_loss)
            print(regression_loss)
            total_loss = total_loss + class_loss + regression_loss * 2 * 0.0001
        total_loss = total_loss / batchb
        print(total_loss)
        print('a')
        return total_loss

</code></pre>

ptrblck: <p>I guess this line of code:</p>
<pre><code class="lang-python">l2[:, pn] = 0.05 * l2[:, pn] - point_cls[b, pn, 1]
</code></pre>
<p>might be causing the issue, as you are assigning the new values inplace to <code>l2</code>.<br>
As a workaround you could append the results to a new <code>list</code> and create the tensor afterwards.</p>
How do you run transformers example scripts (e.g. run_wwm_mlm.py) with a saved model?)

jungminc88: <p>I have trained BertForSequenceClassifications using one of the Huggingface-provided pretrained models (cl-tohoku/bert-base-japanese-whole-word-masking)<br>
and now I want to train it on MLM task using run_mlm_wwm.py.</p>
<p>I have saved the model after the classification task:<br>
<code> torch.save(model.state_dict(), my_model_name)</code></p>
<p>Now when I try to run</p>
<pre><code class="lang-auto">python run_mlm_wwm.py \
    --model_name_or_path my_model_name \
    --train_file my_data.txt \
    --do_train \
    --output_dir /output_dir
</code></pre>
<p>it returns this error message</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “run_mlm_wwm.py”, line 408, in <br>
main()<br>
File “run_mlm_wwm.py”, line 274, in main<br>
config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs)<br>
File “/home/cl/jungmin-c/.pyenv/versions/anaconda3-5.1.0/envs/jp/lib/python3.7/site-packages/transformers/models/auto/configuration_auto.py”, line 446, in from_pretrained<br>
config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)<br>
File “/home/cl/jungmin-c/.pyenv/versions/anaconda3-5.1.0/envs/jp/lib/python3.7/site-packages/transformers/configuration_utils.py”, line 495, in get_config_dict<br>
config_dict = cls._dict_from_json_file(resolved_config_file)<br>
File “/home/cl/jungmin-c/.pyenv/versions/anaconda3-5.1.0/envs/jp/lib/python3.7/site-packages/transformers/configuration_utils.py”, line 578, in _dict_from_json_file<br>
text = reader.read()<br>
File “/home/cl/jungmin-c/.pyenv/versions/anaconda3-5.1.0/envs/jp/lib/python3.7/codecs.py”, line 322, in decode<br>
(result, consumed) = self._buffer_decode(data, self.errors, final)<br>
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x80 in position 64: invalid start byte</p>
</blockquote>
<p>After some searching, I gathered that I have to somehow convert <code>my_model_name</code> to a form that is acceptable to <code>run_mlm_wwm.py</code> but I could not figure out how.</p>
<p>Can anyone suggest how you do this? I would greatly appreciate your help. Thank you!</p>

ptrblck: <p>Based on <a href="https://stackoverflow.com/a/38534992">this post</a> it seems you are trying to encode/decode an invalid character. What seems to be  bit weird is that apparently a JSON file is expected (<code>_dict_from_json_file</code> is used), so my guess would be that the decoder tries to decode a wrong file format.</p>
Conceptual Confusion about designing an architecture

suegi: <p>I’m attempting to do a sequence CNN for a simple regression. I am wondering about the process of doing so.</p>
<p>I isolated a single datapoint, and changed it’s dimensions to accomodate the future model ( to also work with batches). e.g. my sequences are 35 length so i pass in a [1,1,35] TO DEBUG my network as I add layers I add the layer to my model class’s <strong>init</strong> then include it in the forward() call.</p>
<p>This works for the following code…</p>
<p>‘’’<br>
class myCNN(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(myCNN, self).<strong>init</strong>()</p>
<pre><code>    self.conv1d = torch.nn.Conv1d(in_channels=1, out_channels=8,kernel_size=2)
    self.norm = torch.nn.BatchNorm1d(8)
    self.relu = torch.nn.ReLU()
    self.pool = torch.nn.MaxPool1d(kernel_size=2)
    
    self.fc1 = torch.nn.Linear(136,1) #136 is correct for a SINGLE data point
    
def forward(self,x):
    #reshape x
    x = x.unsqueeze(1) #adds necessary dimension for Conv1D
    x = self.conv1d(x)
    x = self.norm(x)
    x = self.relu(x)
    x = self.pool(x)
    x = x.view(-1) # this flattens the layer
    x = self.fc1(x)
    
    return x
</code></pre>
<p>‘’’</p>
<p>However, as you can see I hardcoded the exact size in the self.fc1() layer. This is size for a single data point. When I change my batch size, it increases accordingly.</p>
<p>My questions are</p>
<ol>
<li>
<p>Do I need to pass batch_size in the init() call.</p>
</li>
<li>
<p>shouldn’t there be a way to make it flexible to batch size, whether it is 1 or 128… this was possible with my linear models.</p>
</li>
<li>
<p>how can I put in the correct size to the input_dimension for the linear layer?</p>
</li>
</ol>
<p>Second question; I’m trying to replicate models that do stacks of Convolutions/Pooling.</p>
<p>I’m still looking for an example that demonstrates how this is done. It seems like I need to know the number of channels after each conv/pooling block to pass into the following block. How can I do this without hardcoding these values?</p>
<p>I hope this makes sense!</p>
<p>thanks</p>

ptrblck: <p>You could treat the use case in the same way variable input shapes are treated in vision models, i.e. by adding an adaptive pooling layer before passing the activation to the linear layer.<br>
This will make sure that the expected number of features are constant and you will be able to pass different sequence length.<br>
Also, flatten the activation via <code>x = x.view(x.size(0), -1)</code>.</p>
<p>You could try <code>torch.nn.LazyConv2d</code> to avoid the need to define the number of input channels.</p>
Torchvision does not support mobilenet_v3?

Arohan_Ajit: <p>I was basically running a model on kaggle and while<br>
<code>model = torchvision.model.resnet18(pretrained=true)</code> works well, this<br>
<code>model = torchvision.models.mobilenet_v3_large(pretrained=True)</code> generates an error:<br>
<code>AttributeError: module 'torchvision.models' has no attribute 'mobilenet_v3'</code> This error does not occur with mobilenet_v2, but all the versions of mobilenet_v3. Am I doing something wrong here?</p>

ptrblck: <p>I cannot reproduce the issue using a recent source build and get:</p>
<pre><code class="lang-python">model = models.mobilenet_v3_large(pretrained=True)
print(model)

MobileNetV3(
  (features): Sequential(
    (0): ConvBNActivation(
      (0): Conv2d(3, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
      (2): Hardswish()
    )
    (1): InvertedResidual(
      (block): Sequential(
        (0): ConvBNActivation(
          (0): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
          (1): BatchNorm2d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
          (2): ReLU(inplace=True)
    ...
</code></pre>
<p>Which <code>torchvision</code> version are you using?</p>

Arohan_Ajit: <p>I am using the default torchvision version on kaggle i.e. 0.8.1</p>
<p>Edit: Apparently models like mobilenet_v3 weren’t supported in 0.8.1 because I tried on 0.10.1 on colab and it worked perfectly, Moreover kaggle doesn’t allow to manually upgrade the torchvision either. Gotta use colab for this one, thanks!</p>
Memory leak with autograd create_graph=True

111556: <p>I read several posts online about memory leak, could not find a solution. thanks in advance for any help. I also posted here and make a link so chances are higher for other people facing the same problem can find the solution. <a href="https://github.com/pytorch/pytorch/issues/7343" class="inline-onebox" rel="noopener nofollow ugc">[memory leak] [PyTorch] .backward(create_graph=True) · Issue #7343 · pytorch/pytorch · GitHub</a></p>
<p>below is the distilled code that give memory leak with autogrid create_graph=True</p>
<pre><code class="lang-auto">torch.manual_seed(2332)
dim = 10
winit = torch.randn([dim])
w1 = torch.randn([dim+1],requires_grad=True)
w2 = w1[dim:]
torch.autograd.set_detect_anomaly(True)

for e in range(1000):
    param0 = winit.clone().detach().requires_grad_()

    p = torch.sum((param0-w2)*(param0-w2))
    param0grad = torch.autograd.grad(p,param0,create_graph=True)[0] #mem leak
    param1 = param0 - 0.1*param0grad

    del param1 # del to try to free computational tree
    del param0grad
    del p
    del param0
    gc.collect()

    if (e%100==0): 
        p = psutil.Process()
        mem_info = p.memory_info()
        print('epoch ',e,' mem info ',mem_info)

========
run time output:
epoch 0 mem info pmem(rss=95322112, vms=4828065792, pfaults=24924, pageins=135)
epoch 100 mem info pmem(rss=95776768, vms=4828065792, pfaults=25040, pageins=135)
epoch 200 mem info pmem(rss=96342016, vms=4830425088, pfaults=25178, pageins=135)
epoch 300 mem info pmem(rss=96907264, vms=4830687232, pfaults=25316, pageins=135)
epoch 400 mem info pmem(rss=97456128, vms=4831997952, pfaults=25450, pageins=135)
epoch 500 mem info pmem(rss=97988608, vms=4832260096, pfaults=25580, pageins=135)

thank you.</code></pre>

ptrblck: <p>I’ve answered in the linked issue.</p>

111556: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> you are right, removed this line and no more memory leak<br>
<span class="hashtag">#torch</span>.autograd.set_detect_anomaly(True)</p>
<p>copy your suggestions here: “…you are using anomaly detection via <code>torch.autograd.set_detect_anomaly(True)</code> , which is a debug tool and was causing an increasing memory usage in the past. Could you disable it, once you’ve debugged your code (this mode would also slow down your code so should only be used while debugging)?..”</p>
<p>thank you.</p>
What are the benefits of using class when defining a dataloader?

Bled_Clement: <p>I have the following dataloader below:</p>
<pre><code class="lang-auto">def load_dataset(size_batch, size):
    data_path = "/home/bledc/dataset/test_set/crops_BSD"
    transformations = transforms.Compose([
        transforms.Grayscale(num_output_channels=1),
        transforms.ToTensor()
        ])

    train_dataset = datasets.ImageFolder(
        root=data_path,
        transform=transformations
    )
    train_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=size_batch,
        shuffle=True,
        num_workers=0,
        drop_last=True
    )
    return train_loader
</code></pre>
<p>I iterate through it in my training loop with the following:</p>
<pre><code class="lang-auto">data_loader = load_dataset(batch_size, width)
for data in data_loader:
        model.zero_grad()
        optimizer.zero_grad()
        img, _ = data
        img = img.to(device)
</code></pre>
<p>Can someone explain to me what the benefit would be of writing the load_dataset() function into a class? The reason for this is that I have been using the above template for dataloading from code I found online but it seems that most codebases use <code>class LoaderName(Dataset)</code> followed by definiting initial conditions and super().</p>
<p>Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="Bled_Clement" data-post="1" data-topic="129765">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bled_clement/48/21076_2.png" class="avatar"> Bled_Clement:</div>
<blockquote>
<p>Can someone explain to me what the benefit would be of writing the load_dataset() function into a class?</p>
</blockquote>
</aside>
<p><code>load_dataset</code> would be a function, which returns the <code>DataLoader</code> not a <code>class</code>.<br>
I would claim it depends on your coding style and wrapping the <code>DataLoader</code> creation into a function might allow you to move it into another file more easily.</p>
<aside class="quote no-group" data-username="Bled_Clement" data-post="1" data-topic="129765">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bled_clement/48/21076_2.png" class="avatar"> Bled_Clement:</div>
<blockquote>
<p>but it seems that most codebases use <code>class LoaderName(Dataset)</code></p>
</blockquote>
</aside>
<p>This <code>class</code> definition would be used to create a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>. In your example you are using <code>ImageFolder</code> so wouldn’t need the custom <code>Dataset</code> class.</p>

Bled_Clement: <p>Thanks for clearing this up <a class="mention" href="/u/ptrblck">@ptrblck</a> . As always your help on the forum is greatly appreciated.</p>
Libtorch C++ meson build

s0H57: <p>Hello,<br>
i want to use Libtorch with a C++ project using meson build.<br>
Currently i am running on Ubuntu 21.04 and downloaded <a href="https://packages.ubuntu.com/hirsute/libtorch-dev" rel="noopener nofollow ugc">libtorch-dev</a> via apt-get. Apparently the build now still has problems locating the torch headers when including them in my project file.</p>
<pre><code class="lang-auto">fatal error: torch/torch.h: No such file or directory
    2 | #include &lt;torch/torch.h&gt;
</code></pre>
<p>I also tried following the <a href="https://pytorch.org/cppdocs/installing.html" rel="noopener nofollow ugc">tutorial</a> but i can’t make use of the specific CMAKE commands in meson as far as i know. So do i have to manually link the source files from the unzipped libtorch somehow into my build? Or is the package from apt-get sufficient?<br>
Thanks.</p>
Conditional statement for a tensor in pytorch

Mari: <p>Hi,<br>
Is there any suggestion about how I can implement below conditions for kc which is a tensor in pytorch?<br>
following conditions give this error:<br>
“Boolean value of Tensor with more than one value is ambiguous.”<br>
net_in = torch.cat((x,y),1)<br>
kc = net2_kc(net_in)<br>
kc = kc.view(len(kc),-1)</p>
<p>if kc&gt;0.5:<br>
f = (kc-0.5)^2<br>
elif 0 &lt;= kc &lt;= 0.5:<br>
f = 0<br>
else:<br>
f = kc^2<br>
and then I need to add f function to my loss function.</p>

ptrblck: <p>The <code>if</code> condition checks for a single return value, so applying it to a tensor won’t work out of the box and you could index the tensor instead.<br>
I.e. using the posted conditions, you could apply the changes to parts of the tensor.<br>
Something like this might work:</p>
<pre><code class="lang-python">kc = torch.randn(10, 2)
f = torch.zeros_like(kc)

idx1 = kc &gt; 0.5
f[idx1] = (kc[idx1] - 0.5)**2

idx2 = (0 &lt;= kc) &amp; (kc &lt;= 0.5)
f[idx2] = 0 # wouldn't be necessary, as f is already initialized as zeros

used_idx = torch.zeros_like(idx1).bool()
used_idx |= idx1
used_idx |= idx2
f[~used_idx] = kc[~used_idx]**2
</code></pre>

Mari: <p>Thank you for the reply. I think it worked.</p>

Mari: <p>ptrblck,<br>
I have another question for you. The reason why I defined function f it is to restrict outputs of kc to be positive and a value between 0 and 0.5. But it did not work for me. I tried different activation functions like sigmoid and relu to force the output to be positive but these activation functions make the gradient very small and the output of kc would be very very small value close to zero. Only activation function that gives a none zero value is swish but the output is negative. Do you have any suggestions how I can force the output to be a positive and in range (0,0.5)?</p>
Prepare DataFrame for the dataset and dataloder

hamid_20: <p>hello every body.<br>
my data is pd.DataFrame :</p>
<pre><code>                   time                                 open      high         low       close 
</code></pre>
<p>|—|—|—|—|—|—|—|—|—|<br>
|0|         2021-08-05 12:03:00|            109.587|109.594|109.585|109.593|<br>
|1|         2021-08-05 12:04:00|            109.593|109.595|109.580|109.582|<br>
|2|         2021-08-05 12:05:00|            109.581|109.591|109.572|109.585|<br>
|3|         2021-08-05 12:06:00|            109.585|109.590|109.585|109.588|<br>
|4|         2021-08-05 12:07:00|            109.589|109.607|109.589|109.600|<br>
|…|…|…|…|…|…|…|…|…|<br>
|9995|  2021-08-16 10:55:00|             109.369|109.379|109.364|109.378|<br>
|9996|  2021-08-16 10:56:00|             109.377|109.392|109.377|109.386|<br>
|9997|  2021-08-16 10:57:00|             109.385|109.398|109.384|109.398|<br>
|9998|  2021-08-16 10:58:00|             109.397|109.422|109.397|109.416|<br>
|9999|  2021-08-16 10:59:00|             109.415|109.419|109.408|109.416|</p>
<p><strong>rates_frame[‘close’] for dataset and dataloder</strong></p>
<p><span class="hashtag">#DATA</span><br>
data_ld = torch.FloatTensor(rates_frame[‘close’].values[0:80000])<br>
data_va = torch.FloatTensor(rates_frame[‘close’].values[80000:])</p>
<h1>
<a name="mnist-dataset-1" class="anchor" href="#mnist-dataset-1"></a>MNIST dataset</h1>
<p>train_tensor_dl = data_utils.TensorDataset(data_ld)<br>
train_tensor_vl = data_utils.TensorDataset(data_va)</p>
<h1>
<a name="data-loader-2" class="anchor" href="#data-loader-2"></a>Data loader</h1>
<p>train_loader = torch.utils.data.DataLoader(dataset=train_tensor_dl,<br>
batch_size=batch_size,<br>
shuffle=True,<br>
num_workers=0)</p>
<p>test_loader = torch.utils.data.DataLoader(dataset=train_tensor_vl,<br>
batch_size=batch_size,<br>
shuffle=False,<br>
num_workers=0)</p>
<p>Is this true?<br>
How can I prepare rates_frame[‘close’]  for the dataset and dataloder?</p>

ptrblck: <p>Assuming <code>rates_frame['close']</code> returns a numpy array, you could use:</p>
<pre><code class="lang-python">data = torch.from_numpy(rates_frame['close'])
</code></pre>
<p>and <code>.clone()</code> it if needed.<br>
Besides that I would assume your approach should work.</p>
<p>PS: you can post code snippets by wrapping them into three backticks, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

ptrblck: <p><a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html">This tutorial</a> might be a good starter.</p>

hamid_20: <p>Unfortunately, I could not find a good training for Deep Learning in the financial markets (PyTorch), but Keras has a lot of good training and moved to Keras</p>
Using Pytorch's AMP with multiple scaler backwards per epoch

Bahman_Rouhani: <p>I’m trying to implement <a href="https://arxiv.org/abs/1704.00028" rel="noopener nofollow ugc">Wasserstein-GP</a> with Pytorchs’s Automatic Mixed Precision. In Wasserstein GAN the critic networks gets updated more frequently than generator (e.g. 5 times per each iteration).<br>
My code goes something like:</p>
<pre><code class="lang-auto">for i in range(5):
   with torch.cuda.amp.autocast():
     discriminator_optimizer.zero_grad()
         # do some stuff
      loss_discriminator = compute_discriminator_loss()
      loss_gradient = compute_gradient_penalty_loss()
      ld = loss_gradient +  loss_discriminator
   scaler.scale(ld).backward()
   scaler.step(discriminator_optimizer)
with torch.cuda.amp.autocast():
   generator_optimizer.zero_grad()
   # do some stuff
   loss_generator = compute_generator_loss()
scaler.scale(loss_generator ).backward()
scaler.step(generator_optimizer) 
scaler.update()
</code></pre>
<p>but I get the error</p>
<pre><code class="lang-auto">RuntimeError: step() has already been called since the last update().
</code></pre>
<p>When I call the <code>backward()</code> on discriminator loss for the second time.<br>
The error does not appear if I update the scaler after each <code>backward()</code> call, but the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#working-with-multiple-models-losses-and-optimizers" rel="noopener nofollow ugc">Pytorchs examples</a> states that " <a href="https://pytorch.org/docs/stable/amp.html#torch.cuda.amp.GradScaler.update" rel="noopener nofollow ugc"><code>scaler.update</code></a> should only be called once".<br>
So what’s the correct way to update the scaler here?</p>
<p>(I’m using pytorch 1.9.0)</p>

sbelharbi: <p>hi,<br>
not an expert on amp, but it seems you cant repeat step() with the same scale factor. it has to be updated. i think it is by design because it makes sens that way. the scale factor is designed to follow the dynamics of gradients which change through step(). so, every time you step(), you need to update(). you cant step several times without update, because you will leave the scale factor behind.</p>
<p>i think you should use 2 gradient scalers because your both losses are progressing at different rates. so, each one needs its own scale factor that is updated using update. this will allow you to update the scale factor of the discriminator as often as you like. it does not seem that you retaining any gradient to share between disc and gen, so, you can scale their gradient separately.</p>
<p>i think the pytorch example works because both losses advance with the same rate.</p>

Bahman_Rouhani: <p>That makes sense. I just wasn’t sure if it’s ok to use two different Scalers. Thanx.</p>
Access SSD Layers for Grad Cam Visualization

anujd9: <p>Hi. I am trying to visualize the Class Activation Maps for my trained Object Detection model using Grad Cam. The model is a SSD model provided on torch hub <a href="https://pytorch.org/hub/nvidia_deeplearningexamples_ssd/" rel="noopener nofollow ugc">here</a> which I further finetune on my custom dataset.<br>
I need help with two questions:</p>
<ol>
<li>What’s the correct way or layer to use here to hook onto to get the gradients and visualize the activation maps? I am currently looking at the last ReLU activation layer in the <code>feature_extractor</code> layer.</li>
</ol>
<pre><code class="lang-auto">Bottleneck(
  (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
)
</code></pre>
<ol start="2">
<li>I am trying to understand how to access that layer. I have tried to access the children of the model using something like this:</li>
</ol>
<pre><code class="lang-auto">modules = list(model.feature_extractor.children())
modules[0][6][5]
</code></pre>
<p>but I guess this is not a correct way to grab this layer outputs.<br>
Can someone please guide me here as to how to proceed?<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

yoshitomo-matsubara: <p>Hi <a class="mention" href="/u/anujd9">@anujd9</a></p>
<ol>
<li>
<p>Assuming your ssd model is instantiated as <code>model</code>, you should be able to access to the last ReLU layer of its feature_extractor by <code>model.feature_extractor[-1][-1].relu</code><br>
By using backward hook function (i.e., register it by <code>model.feature_extractor[-1][-1].relu</code>.register_backward_hook(your_backward_hook_func)`), you can get gradients at the layer.</p>
</li>
<li>
<p>To get the output from the last ReLU layer of its feature_extractor, you can use <code>ForwardHookManager</code> in <a href="https://github.com/yoshitomo-matsubara/torchdistill" rel="noopener nofollow ugc">torchdistill</a>. Here is <a href="https://github.com/yoshitomo-matsubara/torchdistill/blob/main/demo/extract_intermediate_representations.ipynb" rel="noopener nofollow ugc">an example notebook</a> to demonstrate the feature.<br>
In Section 3, you can use <code>feature_extractor.-1.-1.relu</code> as the module path for torchdistill like <code>forward_hook_manager.add_hook(model, 'feature_extractor.-1.-1.relu', requires_input=True, requires_output=True)</code></p>
</li>
</ol>
Data augmentation changed the mask in semantic segmentation

Sarmad_GTU: <p>Hi all<br>
I am working on a semantic segmentation problem using satellite image<br>
I want to ask about data augmentation (offline or on the fly ) when I apply it like adding noise or change the contrast so we apply it to both image and mask but in this case, the values of the mask pixels (pixel-wise class value ) changed (ex: value 1 change to 1.5 or something like that ) so, is this a problem? if so, how can I solve it?&gt;</p>

Sarmad_GTU: <p>any answer guys !!!</p>

ptrblck: <p>I guess you are resizing the mask with an interpolation method other than <code>NEAREST</code>, which would interpolate the output values and would thus create new “target” values.<br>
So yes, I think this might be a problem assuming you are working on a multi-class segmentation use case and you should make sure that a nearest neighbors interpolation method is used.</p>
<p>PS: I would recommend to have a bit more patience, as I wouldn’t expect many users to be active on a Sunday and bumping after 3h of creating the topic might not have the desired effect.</p>

Sarmad_GTU: <p>Thank you so much<br>
I am sorry for that</p>
<p>so, besides interpolation what is the best augment method that can I apply to image and mask without changing the mask values?</p>
<p>i used this methods:</p>
<p>train_transform = A.Compose(<br>
[   A.PadIfNeeded(min_height=512, min_width=512, border_mode=4),<br>
A.Resize(512, 512),<br>
A.RandomBrightnessContrast(brightness_limit=0.3,<br>
contrast_limit=0.3, p=0.5),<br>
A.HorizontalFlip(p=0.5),<br>
A.VerticalFlip(p=0.5),<br>
A.MedianBlur(blur_limit=3, always_apply=False, p=0.1),<br>
])</p>
<p>are these methods ok ?</p>

ptrblck: <p>No, I don’t think <code>RandomBrightnessContrast</code> and <code>MedianBlur</code> would make sense to be applied on a mask containing class indices as both transformations would change the “pixel values”, which represent class indices (not color values as in e.g. RGB input images).</p>

Sarmad_GTU: <p>so, which transform operations can I apply to my train dataset?</p>

ptrblck: <p>Spatial transformations applied on the input image should also be applied on the mask tensor to make sure that the input pixel location still correspond to the mask (e.g. rotaions, resizing etc.).<br>
Besides that you should treat the mask as a classification target and should not change its values.</p>

Sarmad_GTU: <p>yes, thank you very much<br>
so if i apply the add noise or bullring or something like that  operation just to image without mask , is this right?</p>

Sarmad_GTU: <p>I found this answer in Albumentations website:<br>
Many non-spatial transformations like, [RandomBrightness], [RandomContrast] [RandomGamma] can be also added. They will be applied only to the image and not the mask.</p>
Speed up training with lazy loading a lot of data

rku1999: <p>Hi everyone,</p>
<p>Here is my question:</p>
<p>I have roughly 400,000 training data and each one is stored as a csv (~35 GB in total). I have a custom dataset object that reads these csv files in <code>__getitem__</code>. Currently, each epoch takes roughly 70 minutes with a batch size of 512.</p>
<p>So, I was wondering if there’s anyway to speed up the training without adding additional resources?</p>
<p>Thanks!</p>

arman-yekkehkhani: <p>You should consider using <code>torch.utils.data.DataLoader</code>, and specify number of workers. These workers retrieve data from the dataset and will significantly improve the read spead. Here is a little snippet:</p>
<pre><code class="lang-auto">dataloader = torch.utils.data.DataLoader(dataset,
 batch_size=512,
 shuffle=True,
num_workers=10)
</code></pre>

rku1999: <p>Thanks for the suggestion. I tried it out but whenever I set <code>num_workers</code> to something greater than 1 the VM just freezes (running on GCP instance with 1 GPU). Is this perhaps a problem with memory?</p>

arman-yekkehkhani: <p>Well, it seems there is an open issue for it: <a href="https://github.com/pytorch/pytorch/issues/13246" rel="noopener nofollow ugc"> CPU memory gradually leaks when num_workers &gt; 0 in the DataLoader</a>. You can find a diverse set of possible solutions in the aforementioned link.</p>

googlebot: <p>converting data to binary file format should help (e.g. read csv with pandas, write parsed data with numpy.save, numpy.memmap or torch.save). make sure to write 4-byte floats, unless you train with float64.</p>
'aten::native_batch_norm' with arguments from the 'SparseCUDA' backend

pytorching: <p>Does <code>torch.nn.batch_norm</code> have support for sparse tensor?<br>
In my test:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/914c04226d400f58949824d5a12db5be2e59e6f7.png" data-download-href="https://discuss.pytorch.org/uploads/default/914c04226d400f58949824d5a12db5be2e59e6f7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914c04226d400f58949824d5a12db5be2e59e6f7_2_690x115.png" alt="image" data-base62-sha1="kJm617yjTD4487MvFzgUvL6sbqv" width="690" height="115" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914c04226d400f58949824d5a12db5be2e59e6f7_2_690x115.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914c04226d400f58949824d5a12db5be2e59e6f7_2_1035x172.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914c04226d400f58949824d5a12db5be2e59e6f7_2_1380x230.png 2x" data-dominant-color="30343A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1602×268 38.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Both ‘SparseCPU’ 和 ‘SparseCUDA’  backend doesn’t support.</p>
How to get all combinations of two data loaders of different sizes

oasjd7: <p>I want to get all combintations of two dataLoaders which have different dataset size.<br>
I can’t contain the whole data because of the GPU memory. (batch size == 32)<br>
What I want to do is:</p>
<pre><code class="lang-auto">loader_a : [a1,a2,a3] # Acutal size: 2,500
loader_b: [b1,b2,b3,b4] # Actual size: 5,430
Get Possible Combinations:
[a1-b1]
[a1-b2]
[a1-b3]
[a1-b4]
[a2-b1]
[a2-b2]
...
[a3-b4]
</code></pre>
<pre><code class="lang-auto">for step, (data_a, data_b) in enumerate(zip(loader_a, loader_b)):
... ?

</code></pre>

ptrblck: <p>You could try to use <code>itertools.product</code>:</p>
<pre><code class="lang-python">from itertools import product

from torch.utils.data import TensorDataset, DataLoader

loaderA = DataLoader(
    TensorDataset(torch.arange(6)), batch_size=2)
loaderB = DataLoader(
    TensorDataset(torch.arange(1, 7) * 10), batch_size=2)

for a, b in product(loaderA, loaderB):
    print(a, b)

&gt; [tensor([0, 1])] [tensor([10, 20])]
  [tensor([0, 1])] [tensor([30, 40])]
  [tensor([0, 1])] [tensor([50, 60])]
  [tensor([2, 3])] [tensor([10, 20])]
  [tensor([2, 3])] [tensor([30, 40])]
  [tensor([2, 3])] [tensor([50, 60])]
  [tensor([4, 5])] [tensor([10, 20])]
  [tensor([4, 5])] [tensor([30, 40])]
  [tensor([4, 5])] [tensor([50, 60])]
</code></pre>
Setting values of a tensor based on given indices of corresponding rows

Runqiu_Hu: <p>I’ve got a tensor <code>A</code> with shape <code>(M, N)</code>, and have another tensor <code>B</code> with shape <code>(M, P)</code> and with values of given indices in corresponding rows of <code>A</code>. Now I would like to set the values of <code>A</code> with corresponding indices in <code>B</code> to <code>0</code>.</p>
<p>For example:</p>
<pre><code class="lang-python">In[1]: import torch
       A = torch.tensor([range(1,11), range(1,11), range(1,11)])
       A
Out[1]: 
tensor([[ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10],
        [ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10],
        [ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10]])
</code></pre>
<pre><code class="lang-python">In[2]: B = torch.tensor([[1,2], [2,3], [3,5]])
       B
Out[2]: 
tensor([[1, 2],
        [2, 3],
        [3, 5]])
</code></pre>
<p>The objective is to set the value of the element with index <code>1,2</code> in the first row, <code>2,3</code> in the second row, and <code>3,5</code> in the third row of <code>A</code> to <code>0</code>, i.e., setting <code>A</code> to</p>
<pre><code class="lang-python">tensor([[ 1,  0,  0,  4,  5,  6,  7,  8,  9, 10],
        [ 1,  2,  0,  0,  5,  6,  7,  8,  9, 10],
        [ 1,  2,  3,  0,  5,  0,  7,  8,  9, 10]])
</code></pre>
<p>I have applied row by row for loop, and also tried <code>scatter</code>:</p>
<pre><code class="lang-python">zeros = torch.zeros(A.shape, dtype=torch.float).to("cuda")
A = A.scatter_(1, B, zeros)
</code></pre>
<p>The two methods work fine, but all give quite poor performance. Actually, I infer that some efficient approach should exist based on an error before. I initially used <code>A[:, B] = 0</code>. This would set all the indices of appeared in <code>B</code> to <code>0</code>, regardless of the row. However, the training speed improved drastically when doing <code>A[:, B] = 0</code>.</p>
<p>Is there any way to implement this more efficiently?</p>

ptrblck: <p>You could try this indexing method:</p>
<pre><code class="lang-python">A = torch.tensor([range(1,11), range(1,11), range(1,11)])
B = torch.tensor([[1,2], [2,3], [3,5]])
A[torch.arange(A.size(0)).unsqueeze(1), B] = 0.
print(A)
&gt; tensor([[ 1,  0,  0,  4,  5,  6,  7,  8,  9, 10],
          [ 1,  2,  0,  0,  5,  6,  7,  8,  9, 10],
          [ 1,  2,  3,  0,  5,  0,  7,  8,  9, 10]])
</code></pre>
<p>but I’m unsure if you would see a speedup compared to <code>scatter_</code>.</p>
How does Frequency Pooling make sense in CNNs that operates fully in the Frequency Domain?

eduardo4jesus: <p>I am reading this <a href="https://ieeexplore.ieee.org/abstract/document/8060431/authors#authors" rel="noopener nofollow ugc">paper</a>, image below,</p>
<blockquote>
<p><a href="https://i.stack.imgur.com/53fb0.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/53fb0.png" alt="enter image description here" width="473" height="500"></a></p>
</blockquote>
<p>which uses Frequency Pooling, from <a href="https://dl.acm.org/doi/abs/10.5555/2969442.2969513" rel="noopener nofollow ugc">this paper</a>, image below.</p>
<blockquote>
<p><a href="https://i.stack.imgur.com/OVBfZ.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/OVBfZ.png" alt="enter image description here" width="690" height="380"></a></p>
</blockquote>
<hr>
<p><strong>Question</strong></p>
<p>For simplicity, let me assume there is no <code>sinc</code> interpolation, and the convolution kernels are stored in the frequency domain having the same size of the input (<code>N</code> x <code>N</code>).<br>
My understanding of the frequency pooling in the proposed method is the following:</p>
<ol>
<li>On the forward-pass, we have that the lower frequencies components of the convolution output (N x N) are shifted to the center of the representation.</li>
<li>Next, the high frequency components outside of a central area (say N/2 x N/2) are discarded.</li>
<li>Then, the remaining frequencies as passed to the next layers ( non-linear activation; convolution; frequency pooling; non-linear activation; and so on)</li>
</ol>
<p>Here is an illustration, notice low frequencies are assumed to be in the center.</p>
<blockquote>
<p><a href="https://i.stack.imgur.com/R6xiD.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/R6xiD.png" alt="enter image description here" width="690" height="284"></a></p>
</blockquote>
<p>Given that, it seems that the discarded frequencies in the frequency pooling process do not contribute at all to the output. If that is the case, then all the frequencies that are discarded in all pooling layers, up to the last frequency pooling, would not be used for the computation of the output.</p>
<p>Wouldn’t this be equivalent to discarding all those frequencies in the input layer?<br>
I seem to be missing something very important on this.</p>
<hr>
<p>PS: Is there a specific channel to discuss research with this community?</p>

ptrblck: <p>I haven’t checked the paper in detail, but wouldn’t the steps between the frequency pooling (activation, conv etc.) be able to create activation maps with a wider frequency output, which then could be pooled again?</p>
How to modify internal ImageFolder object after creation?

JimW: <p>I am trying to port some dataloader codes from codebase 1 to codebase 2.<br>
For codebase 2,  the dataloader was based on ImageFolder, while the dataloader in codebase 1 was something where.  What I did was to replace the <strong>getitem</strong> of codebase2 with the one from codebase1, and it worked well.</p>
<p>However,  the other part of codebase 2 (which I cannot modify due to some internal policy)  will read the total number of samples by doing len(train_loader), where “train_loader” is the ImageFolder object.</p>
<p>So I am thinking if I can hack it a little bit to modify the attributes of this ImageFolder object “train_loader”, so that len(train_loader) will return the desired value.  I need to do that after this object was created, and I tried “self.samples = []” to see whether len(train_loader) will become 0, but nothing happened.</p>
<p>Could someone provide some guidance on this?  Thanks.</p>

ptrblck: <p>Removing all internal <code>self.samples</code> should work, as the <code>DatasetFolder</code> class uses this attribute to get the length as seen <a href="https://github.com/pytorch/vision/blob/7947fc8fb38b1d3a2aca03f22a2e6a3caa63f2a0/torchvision/datasets/folder.py#L240-L241">here</a>.<br>
Alternatively, you could also try to monkey-patch the <code>__len__(self)</code> method and assign the desired value to it, assuming you want to reduce the number of samples and remove the last samples.<br>
With that being said, I don’t know what your use case is and would generally recommend to create either a new <code>Dataset</code> with the desired attributes or create a custom one.</p>
Circular repeat of one dimension

m.hassanin: <p>Hi guys,</p>
<p>I have a tensor A = (10,1,4), I need to convert it to B = (10, 5, 4) provided that the second dimension is filled out from the first one in a circular way. That is, B[:,0,:] has values of A[1:5,:] and B[:,1,:] has values of A[2:6,:].</p>
<p>I tried repeat but all the items of B will be exactly as same as A</p>

ptrblck: <p>I’m unsure how this assignment would be possible, as the desired shapes would be different:</p>
<aside class="quote no-group" data-username="m.hassanin" data-post="1" data-topic="129942">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/m.hassanin/48/31935_2.png" class="avatar"> m.hassanin:</div>
<blockquote>
<p>That is, B[:,0,:] has values of A[1:5,:]</p>
</blockquote>
</aside>
<p>Example:</p>
<pre data-code-wrap="python"><code class="lang-python">A = torch.randn(10, 1, 4)
print(A[1:5].shape)
&gt; torch.Size([4, 1, 4])

B = torch.randn(10, 5, 4)
B[:, 1:5, :].shape
&gt; torch.Size([10, 4, 4])
</code></pre>
<p>As you can see, the slice in <code>A</code> contains 16 elements while the slice in <code>B</code> contains 160 elements.<br>
Could you post an example how <code>B</code> should be filled?<br>
I think you might be able to use <code>A.unfold</code>, but of course the shape mismatch should be solved first.</p>
Pythonic implementation of F.linear

ian_malcolm: <p>Hi, I try to learn the actual function by reading the code. For example, I want to understand what is happening in a linear layer (in forward and backward pass). I track from <a href="https://github.com/pytorch/pytorch/blob/0f2c60f0e37fe1738d30d3db3fd48e88b8087af7/torch/nn/modules/linear.py#L99" rel="noopener nofollow ugc">here</a> to <a href="https://github.com/pytorch/pytorch/blob/99e28baeba4f1ffb2623e64694b2aac13df5e0fb/torch/nn/functional.py#L1848" rel="noopener nofollow ugc">here</a> and get lost.</p>
<p>My question is: Is it possible for me to find a pythonic (or jit or C) implementation, of the actual operation, e.g., x*w+b, in the pyTorch official repo? I am also looking for that of a Conv2d layer.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/where-is-the-nn-linear-cuda-implementation/128310/2">This post</a> shows where the underling methods for a linear layer are defined. Convolutions dispatch to different implementations from <a href="https://github.com/pytorch/pytorch/blob/d6133b2fe6b863dd49fb21641bd04e24e19ac794/aten/src/ATen/native/Convolution.cpp">here</a> (i.e. the CPU kernel, native CUDA kernel, cuDNN, etc.).</p>