akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Using variable number of modules as children when subclassing torch.nn.Module

username: JamesDickens
cooked: <p>I have a subclass of torch.nn.Module for which I have multiple output heads, differing by one parameter that I pass into them. I want to have all of them accessible in the .children() function call, but with using a loop. In the code below, it seems only testClass3 will show any children at all (testClass1 does not). Is there a way to achieve this with loops?</p>
<pre><code class="lang-auto">import torch


class testClass2(torch.nn.Module):
    def __init__(self, i):
        super(testClass2, self).__init__()
        self.i = i


class testClass1(torch.nn.Module):
    def __init__(self):
        super(testClass1, self).__init__()
        self.test_classes = []
        for i in range(5):
            self.test_classes.append(testClass2(i))
        self.checkParams()

    def checkParams(self):
        for children in self.children():
            print(children)

class testClass3(torch.nn.Module):
    def __init__(self):
        super(testClass3, self).__init__()
        self.test_classes = []
        self.t1 = testClass2(1)
        self.t2 = testClass2(2)
        self.checkParams()

    def checkParams(self):
        for children in self.children():
            print(children)

t = testClass1()
t = testClass3()

</code></pre>
<p>I’m not sure how torch.nn.Module determines its “children”, I’m guessing it just checks all of its parameters is they are an instance of torch.nn.Module or something like that.<br>
Any insights appreciated.</p>

username: ptrblck
cooked: <p>PyTorch will register the submodules as children, if you use an <code>nn.ModuleList</code> instead of a Python <code>list</code>.<br>
Use <code>self.test_classes = nn.ModuleList()</code> and it should work.</p>
Sparse Matrix Forcing 0 Weights Crashing

username: pklvls
cooked: <p>Setup:<br>
I have a matrix of weights, a lot of which are originally 0. I do not want these weights to change.</p>
<p>As I understand, if I use these weights as follows:</p>
<pre><code class="lang-auto">output = torch.sum(torch.addmm(biases, weights, inputs))
output.backward()
optim.step()
</code></pre>
<p>Then the weights which were initially 0 can become non-zero.</p>
<p>However, I do not want this happening. Can I use a sparse matrix to represent the weights and force it to ignore gradients for zero-valued elements so that they never become non-zero?</p>
<p>Additionally:</p>
<pre><code class="lang-auto">RuntimeError: set_indices_and_values_unsafe is not allowed on a Tensor created from .data or .detach().
If your intent is to change the metadata of a Tensor (such as sizes / strides / storage / storage_offset)
without autograd tracking the change, remove the .data / .detach() call and wrap the change in a `with torch.no_grad():` block.
</code></pre>
<p>seems to be coming up whenever I try to update the weights in my sparse matrix for the second time. Does this mean we’re not allowed to backprop through and update sparse matrices?</p>

username: ptrblck
cooked: <p>I’m not sure if the sparse approach would work, but you could use <code>register_hook</code> for the dense weight tensor and zero out the gradients instead.</p>
How to find the 3D coordinates from the 2D coordinates of two different views video feeds?

username: Mona_Jalal
cooked: <p>I have used deep learning to predict the 2D keypoints of a deformable object in two different views of the object video feeds.</p>
<p>How can I retrieve the 3D keypoints if I have 2D keypoints for each corresponding frame? Is there a framework that already does that? Looking for some expert advice here.</p>

username: ptrblck
cooked: <p>This sounds like a general <a href="https://en.wikipedia.org/wiki/Triangulation_(computer_vision)">Triangulation</a> use case. If I’m not mistaken, OpenCV should have some algorithms for it, but unsure what their requirement is (besides calibrated cameras).</p>
Processing batch 

username: vjayd
cooked: <p>for i, data in enumerate(train_dataloader, 0):</p>
<p>File “/home/vijay/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 637, in <strong>next</strong><br>
return self._process_next_batch(batch)</p>
<p>File “/home/vijay/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py”, line 657, in _process_next_batch<br>
if isinstance(batch, ExceptionWrapper):            raise batch.exc_type(batch.exc_msg)</p>
<p>error</p>

username: ptrblck
cooked: <p>Could you describe your error, what you are trying to do, and where you are stuck?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
WHAT IS TORCH.NN REALLY?

username: airget0919
cooked: <p>In the PyTorch tutorial “WHAT IS TORCH.NN REALLY?”, I cannot access the link “<a href="http://deeplearning.net/data/mnist/" rel="noopener nofollow ugc">http://deeplearning.net/data/mnist/</a>” to get the MNIST data.</p>
<p>Is this a temporary fix?</p>

username: vaisakh_m
cooked: <p>It seems like a temporary issue.<br>
<em><a href="http://deeplearning.net/" rel="noopener nofollow ugc">http://deeplearning.net/</a></em> is down it seems.</p>
Why use DxHxW for 3d input data instead of HxWxD?

username: NightRain
cooked: <p>When inputting a 3d array, Conv3d expects as input NxCxDxHxW. Why not use NxCxHxWxD?</p>
<p>This only transposes the input anyway when the dimension is equal for all 3 axes.</p>

username: ptrblck
cooked: <p>What would be the advantage of NCHWD?</p>

username: NightRain
cooked: <p>It’s probably more intuitive since volumes are usually in the format HWD. In TF they simply specify (dim_1, dim_2, dim_3), so I can’t see the reason for using DHW…</p>

username: ptrblck
cooked: <p>Could you explain more about the “usual” format? I.e. does a specific library return volumes as HWD or is there any other convention? I’m currently seeing it more as CHW (PIL) vs. HWC (OpenCV), which might be annoying sometimes but both memory layouts have certain advantages. I’m not sure what the advantage of HWD vs. DHW would be. E.g. is there any expected performance gain using your layout?</p>
LSTM model giving repeated ending token as output when using BERT embedding

username: rectified_sigmoid
cooked: <p>I am trying to train a sequence to sequence LSTM model. I am using pre-trained BERT model for getting word embeddings. BERT needs [CLS] and [SEP] tokens at start and end, so that the tokenized sequence becomes ["[CLS]", …,"[SEP]"]. My model has encoder, that takes in this sequence as input and gives an intermediary output, and the Decoder takes this output and gives the original sequence as output. But problem is, after training, the model is giving “[CLS]” followed by [SEP] for every input as output. It is biasing to [SEP]. So, if input is [CLS] , a, b, c, [SEP] the output I get is [CLS] [SEP] [SEP] [SEP] [SEP].How do I solve this issue? here is my encoder code:</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
  def __init__(self, hidden_size, output_size=3):
    super(Encoder, self).__init__()

    self.bert = BertModel.from_pretrained("bert-base-uncased")
    self.bert.eval()
    #self.embed = nn.Embedding(input_size, embed_size)

    self.lstm1 = nn.LSTM(768, hidden_size)  

    
    self.linear1 = nn.Linear(hidden_size, 128)
    self.bn1 = nn.BatchNorm1d(128)
    self.linear2 = nn.Linear(128, 64)
    self.linear3 = nn.Linear(64, 32)
    self.bn2 = nn.BatchNorm1d(32)
    self.linear4 = nn.Linear(32, output_size)

    

  def forward(self, input, mask):
    input = input.long()
    with torch.no_grad():
      out,_ = self.bert(input, attention_mask = mask)
    out = out.permute(1,0,2)
    out,_ = self.lstm1(out)
    out = self.bn1(self.linear1(out[-1]))
    out = self.linear2(out)
    out = self.linear3(out)
    out = self.bn2(out)
    out = self.linear4(out)

    out = torch.sigmoid(out)

    return out
</code></pre>
<p>And here is my decoder code:</p>
<pre><code class="lang-auto">class Decoder(nn.Module):
  def __init__(self, input_size, output_size, hidden_size=128):
    super(Decoder, self).__init__()

    self.hidden_size = hidden_size

    self.output_size = output_size

    self.dropout = nn.Dropout(0.2)

    self.linear5 = nn.Linear(input_size,32)
    self.bn3 = nn.BatchNorm1d(32)
    self.linear6 = nn.Linear(32, 64)
    self.linear7 = nn.Linear(64, hidden_size)
    self.lstm2 = nn.LSTM(hidden_size,hidden_size)
    self.linear8 = nn.Linear(hidden_size,output_size)


  def forward(self, x, seq_len):
    out = self.bn3(self.linear5(x))
    #x = self.dropout(self.linear6(x))
    out = self.linear6(out)
    out = self.linear7(out)
    outt = out.unsqueeze(0)
    hidden = (torch.zeros(1, x.size(0),self.hidden_size, device=device), torch.zeros(1,x.size(0),self.hidden_size, device=device))
    out = []
    for t in range(seq_len):
      outt, hidden = self.lstm2(outt, hidden)
      out.append(outt.squeeze(0))
    out = torch.stack(out, dim=0).to(device)
    out = self.linear8(out)
    
    out = torch.softmax(out, dim=2)

    return out

</code></pre>
<p>I am making use of the pre-trained BERT model from the transformers library</p>
Get the last hidden state of sequence on a batch with variable sequence length

username: Luvata
cooked: <p>I want to get the last hidden state in a batch (with different length) after feeding through unidirection <code>nn.LSTM</code> (not the padded state). My current approach is:</p>
<p><code>List[Tensor] -&gt; Padded Tensor -&gt; PackPaddedSequence -&gt; LSTM -&gt; PadPackedSequence -&gt; Select hidden state of last step using length</code></p>
<pre><code class="lang-python">a = torch.ones(25, 300)
b = torch.ones(22, 300)
c = torch.ones(15, 300)
padded_seq = pad_sequence([a, b, c]) # torch.Size([25, 3, 300])
lengths = torch.Tensor([25, 22, 15]).int()
inp_seq = pack_padded_sequence(padded_seq, lengths=lengths)

lstm = torch.nn.LSTM(input_size=300, hidden_size=150, num_layers=2)
out, _ = lstm(inp_seq)
out_tensor, inp_length = pad_packed_sequence(out)
b_size = list(inp_length.size())[0]
last_hidden = out_tensor[inp_length - 1, range(b_size)].contiguous()
</code></pre>
<p>My question is:</p>
<ul>
<li>Am I doing a correct way to get that hidden state ? (I felt it a little clumsy)</li>
<li>How can I use it with <code>bidirectional=True</code> ?</li>
</ul>
Some basic tensor operations with powers

username: Bixqu
cooked: <p>I have a matrix A and a tensor b of size (1,3) - so a vector of size 3.</p>
<p>I want to compute</p>
<p>C = b1 * A + b2 * A^2 + b3 * A^3   where ^n is the n-th power of A.</p>
<p>At the end, C should have the same shape as A. How can I do this efficiently?</p>

username: vaisakh_m
cooked: <p><code>C = b[:,0]* A + b[:,1]* A**2 + b[:,2]* A**3</code></p>
How to share torch models across python processes

username: Tuyen_Vo_Quang
cooked: <p>I currently want to initialize a torch model in the main process and want to pass it to child thread (due to model size is large and thus initialization is long), but the IPC overhead is too large.<br>
I try to use model.share_memory() but it seems that there is no effect.<br>
My question is that is there any way to overcome this. The final objective is to use the model in child thread without bearing from long model initialization</p>
A question about backward propagation speed in cpp extension in tutorial

username: WMF1997
cooked: <p>hello, everyone. I have a question about backward propagation speed in writing cpp extension.</p>
<p>I download the source code of “Custom C++ and CUDA extensions” in PyTorch Tutorial, which is at <a href="https://github.com/pytorch/extension-cpp" rel="noopener nofollow ugc">https://github.com/pytorch/extension-cpp</a> .</p>
<p>when I ran <code>benchmark.py</code>, I found that the backward propagation time in <code>cpp</code> is much longer than that in <code>python</code>. I really do not know why this happen.</p>
<p>The result is listed as below. And I use the original code in <code>extension-cpp</code> repo.</p>
<pre><code class="lang-bash">(base) wmf997@wmf997-E743-Q7C08:~/extension-cpp-master$ python benchmark.py py
Forward: 246.525/250.680 us | Backward 365.973/376.601 us
(base) wmf997@wmf997-E743-Q7C08:~/extension-cpp-master$ python benchmark.py cpp
Forward: 178.099/180.986 us | Backward 536.442/549.297 us
</code></pre>
<p>I ran 10000 times, and the result is listed as below. (the result above ran 100 times)</p>
<pre><code class="lang-bash">(base) wmf997@wmf997-E743-Q7C08:~/extension-cpp-master$ python benchmark.py --runs 10000 py
Forward: 245.571/334.904 us | Backward 363.827/486.081 us
(base) wmf997@wmf997-E743-Q7C08:~/extension-cpp-master$ python benchmark.py --runs 10000 cpp 
Forward: 177.860/290.786 us | Backward 537.395/834.295 us
</code></pre>
<p>I run my code on Kubuntu 20.04, with Python 3.8.5(Anaconda), and PyTorch 1.7.0+cpu is installed via <code>pip</code>. The CPU is intel core i7-3540m.</p>