akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Does model serialization have any restriction?

username: simoloca
cooked: <p>Hi to everybody,<br>
I’m new with this mobile framework provided by Pytorch.<br>
I’m trying to customize the HelloWorld app using a model trained by me, but as I launch the app on Android Studio, the app on the virtualized device crashes.<br>
I put my model into the asset folder and change the value of the variable that contains the proper name of the model.<br>
So I cannot understand why it doesn’t work, could anybody help me with this?<br>
Thanks a lot.</p>
Torchtext custom dataset without applying a field to a column

username: greedsin
cooked: <p>In torch text it is possible to create custom datasets by extend <code>torchtext.data.Dataset</code> which requires calling the super constructor with a list of examples and the corresponding fields.</p>
<p>However for my data structure I need to apply numericalization and padding only to certain columns</p>
<pre><code class="lang-python">  fields=[("transcript", self.SOURCE),("label",None)] #For the "label" field apply nothing, just keep the original data structure
</code></pre>
<p>the field label is an array of strings</p>
<pre><code class="lang-auto">label=[[value1,value2],[value1, value3]]
</code></pre>
<p>For now I need to apply a nested field for this structure and at runtime I need to compile the indices back to words which makes it significally harder to process</p>
Different inference results when running on arm64 and x86_64

username: Klaus_Voltmer
cooked: <p>Hi,</p>
<p>I get different inference results when running my c++ code on arm64 and x86_64. I build pytorch from source as described here <a href="https://pytorch.org/mobile/ios/#build-pytorch-ios-libraries-from-source" rel="nofollow noopener">https://pytorch.org/mobile/ios/#build-pytorch-ios-libraries-from-source</a></p>
<p>The results on x86_64 are actually correct and the results on arm64 are <strong>incorrect</strong>. I double checked in the python world.</p>
<p>I am stepping in the dark, so any hint is most appreciated!</p>

username: LeviViana
cooked: <p>can you provide the code ?</p>

username: Klaus_Voltmer
cooked: <p>sure:</p>
<ul>
<li>
<p>load the model:<br>
module = torch::jit::load(fileName);</p>
</li>
<li>
<p>process:<br>
torch::Tensor result;<br>
std::vector<a>torch::jit::IValue</a> inputs;<br>
at::Tensor tensor = torch::from_blob(data.data(), {1, 1, 11, 144});<br>
inputs.push_back(tensor);<br>
torch::autograd::AutoGradMode guard( <strong>false</strong> );<br>
result = module.forward(inputs).toTensor();</p>
</li>
</ul>
<p>i have written unit tests and integrations tests that show that the results are equal to python. but once i run this on the iphone the results are different.</p>

username: ptrblck
cooked: <p>How are you creating the <code>data</code> and how large is the difference?</p>

username: Klaus_Voltmer
cooked: <p>I am creating the data from a std::vector&lt; <strong>float</strong> &gt; . It contains 11 spectrums with the size of 144.</p>
<p>The difference is quite big, by something like factor of 2.</p>

username: tom
cooked: <p>So arm64 and amd64 will use different backends. It is quite possible that you found a bug in the arm64 one, in particular if you use less-common modules. (e.g. I had that with transposed convs a year ago on arm32, where a network would run fine on amd64 but the output was messed up on my phone.)<br>
I know it is a lot of work, but if you want, the ideal reproducing case would be to narrow down the network to a single module where things go wrong and then provide Module+Parameters and inputs. This would also limit how much you need to tell us about it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Best regards</p>
<p>Thomas</p>
Layer IDs in the optimizer's state_dict

username: sigma_x
cooked: <p>I started looking into the state_dict of Adam optimizer, found this:</p>
<p><code>d['optimizerG']['state'].keys()</code></p>
<p>returns</p>
<p><code>dict_keys([140266712220800, 140266712220224, 140266712159432, 140266712219792, 140266712162168, 140266712220008, 140266712158784, 140266712219864, 140266712162096, 140267508645536, 140266712160656, 140266712220584, 140266712161376, 140267508643376, 140267508644456, 140266712161664, 140266712159648, 140266712220728, 140266712220152, 140266712159936, 140266712220368, 140266712161592])</code></p>
<p>After printing them out, I found out that  keyvalues refer to the weights and biases in the model, and allow access to the stored <code>exp_avg</code> and <code>exp_avg_sq</code> for each weight/bias. What I don’t quite understand is where the values above come from. Is this a time of creating the reference to the tensor?</p>

username: ptrblck
cooked: <p>These should be the <code>id</code>s of the parameters.<br>
This small code snippet demonstrates this behavior:</p>
<pre><code class="lang-python">x = torch.randn(10, requires_grad=True)
x_id = id(x)

y = torch.randn(10, requires_grad=True)

optimizer = torch.optim.Adam([x, y], lr=1.)

param_id = optimizer.state_dict()['param_groups'][0]['params'][0]
print(param_id == x_id)
&gt; True
</code></pre>

username: sigma_x
cooked: <p>OK then</p>
<p>I didn’t know tensors have ids</p>
Fine-tune pre-trained optical flow

username: akashs
cooked: <p>Does anyone have the experience or resources of fine-tunning a pretrained optical flownet as a sub-learning module in a PyTorch learning model? I found a couple of implementation of the Pytorch Flownet pretrained model. However, as a novice, I want to know the best-practice or the convenient way to incorporate the optical flow learning model as a sub-model in a PyTorch learning model.<br>
Any type of help is really appreciated. Thanks in advance.</p>
Log-likelihood for normal distribution

username: Isaac_Kargar
cooked: <p>Hi,<br>
I have a VAE that generates mean and var as output. So the size of the output heads is [B, 1, 28, 28]. I want to calculate the loglike loss. How can I do that? In pytorch vae tutorial, I see that they used bce loss but it doesn’t work here. Does anyone have any idea how to implement it?</p>
Unpooling or Transposed convolution for Deconvolution network

username: davidlee
cooked: <p>Hi guys,</p>
<p>I am wondering the influences of <strong>both the unpooling operator and Transposed convolution</strong><br>
<em>for upsampling the feature maps in the Deconvolution networks in output image prediction tasks.</em></p>
<p><strong>Are there any pros and cons between two operators?</strong><br>
It would be great if you introduce some papers regarding this issue.</p>
<p>Thanks in advance!!</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>The main difference is that transposed convolutions are similar to ordinary convolutions which means they can learn weights based on your tasks but for sure you need to also train them (time and power). But unpooling operator is just a math operator and does not do any learning.</p>
<p>Sometimes in depend on your task, for instance, you can do something similar to pooling using stride in convolutions but still you can see many SOTA papers that use pooling layers. So, it is hard to say one is better than the other.</p>
<p>If you just google “unpooling vs deconvolution” you will find so many discussions about it.</p>
<p>Here are some:</p>
<ol>
<li><a href="https://arxiv.org/pdf/1311.2901v3.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1311.2901v3.pdf</a></li>
<li><a href="https://stats.stackexchange.com/questions/252810/in-cnn-are-upsampling-and-transpose-convolution-the-same" rel="nofollow noopener">https://stats.stackexchange.com/questions/252810/in-cnn-are-upsampling-and-transpose-convolution-the-same</a></li>
<li><a href="https://github.com/facebookresearch/SparseConvNet/issues/75" rel="nofollow noopener">https://github.com/facebookresearch/SparseConvNet/issues/75</a></li>
</ol>
<p>bests</p>
How Pytorch compute gradient for division operator

username: Giang_Hoang
cooked: <p>I’m trying to compute the gradient of 1/x without using Pytorch’s autograd. I use the formula grad(1/x, x) = -1/x**2 but when I compare my result with the gradient given by Pytorch’s autograd, they’re different.</p>
<p>Here is my code</p>
<pre><code class="lang-auto">a = torch.tensor(np.random.randn(), dtype=dtype, requires_grad=True)
loss = 1/a
loss.backward()
print(a.grad - (-1/(a**2)))
</code></pre>
<p>The output is</p>
<pre><code class="lang-auto">tensor(5.9605e-08, grad_fn=&lt;ThAddBackward&gt;)
</code></pre>
<p>Can anyone explain why ?</p>

username: liangbright
cooked: <pre><code class="lang-auto">import torch
a = torch.tensor(-0.0061, dtype=torch.float32, requires_grad=True)
loss = 1.0/a
loss.backward()

print(-1/(a**2))
print(a.grad - (-1/(a**2)))
print((-1/a)/a-(-1/(a**2)))
</code></pre>

username: Giang_Hoang
cooked: <p>Can you tell me where to see such implementations of backward() of Pytorch builtin operators ?</p>

username: liangbright
cooked: <p>It is just my guess… wait for Pytorch guys</p>

username: HumberMe
cooked: <aside class="quote no-group" data-username="Giang_Hoang" data-post="3" data-topic="29448">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/giang_hoang/48/7575_2.png" class="avatar"> Giang_Hoang:</div>
<blockquote>
<p>implementations</p>
</blockquote>
</aside>
<p>they are wrote in cpp codes, If you can build pytorch from source, it will be easy to find the implementations.</p>
Accessing TorchScript module methods implemented in Python in C++

username: paulus
cooked: <p>Hi,</p>
<p>I’ve written a model in Python, translated it to TorchScript with torch.jit.script(model), and serialized with torch.jit.save(). In a C++ program, I can load the model with torch::jit::load() and use the inference with model.forward().</p>
<p>Is there a way of accessing other (than forward) methods from the model in the C++ domain?</p>
<p>A direct model.my_method() call would require that the C++ compiler knows how to access this function. If the model is loaded during the execution, it may have functions of unknown name and the compiler (i.e., I) doesn’t know how to use them. It there a way to accomplish this, e.g., with something like model.call(‘my_method’, method_args)?</p>

username: zdevito
cooked: <p>You can use <code>mymodule.get_method("name_of_your_method")(method_args);</code></p>

username: paulus
cooked: <p>Excellent! My guess wasn’t so far, but somehow I didn’t manage to find the correct function.</p>
<p>Many thanks!</p>
Multi CPU Conv2D

username: Daw
cooked: <p>Hi all,<br>
When I run the model that is based on nn.linear, the server shows ~2000% cpu usage, I do not mind that, however, when I used  nn.Conv2D I noticed the cpu is 100% maximum, and the iterations/epochs are extremely slow.<br>
I am wondering why is that, is there is a default parallel option for nn.linear, and is it possible to enable such thing for  nn.Conv2D()?</p>
<p>I appreciate your help.</p>
Is it necessay to keep the gradients when computes the perceptual loss using VGG net?

username: acoder_acoder
cooked: <p>Hi, there!</p>
<p>I argued with my friends whether it was necessay to keep the gradents of VGG net to obtain the perceptual loss.</p>
<p>loss_total = loss_net + loss_vgg.</p>
<p>I thought there was no use to update the parameters of the VGG net, so there was no reason to keep the gradients.</p>
<p>Any suggestions? Thank you!</p>

username: ptrblck
cooked: <p>Could you link to a reference implementation, which shows the desired behavior?<br>
If I’m not mistaken, the perceptual loss is calculated by using a norm between the intermediate activations between two input images, and summing it to a final loss.<br>
If my understanding is correct, I don’t see where the gradients can be used at this point, as they aren’t calculated yet.</p>

username: acoder_acoder
cooked: <p>Yes, exactly. Vgg acts as a ruler, we should not change a ruler.</p>
<p>Thanks!</p>
Indices for different columns of a tensor

username: Abueidda
cooked: <p>Hi,</p>
<p>I am experiencing weird behavior while dealing with the following scenario. tne is the batch size, while N is the total number of possible locations for each example. Each example has exactly 24 scalar outputs, where their locations are stored in dof tensor (size of dof is (tne X 24)). Fint_e has a size of tneX24 (i.e., the 24 outputs for each example). I am trying to construct a large tensor, which has a size of tne X N. When I do the following, it fills in the wrong manner. Any advice?</p>
<pre><code class="lang-auto">Fint_MAT        = torch.zeros((tne,N))
Fint_MAT[:,dof[:,:24]]  = Fint_e[:,:24]
</code></pre>
<p>The dof tensor, which has the size of batch size X 24, has different indices for each example, but each example has in total 24 indices.</p>
<p>For instance,<br>
dof[0,:] = 0, 1, 6, 9, … (24 in total)<br>
dof[1,:] = 1,100, 151, 300,… (24 in total)</p>
<p>Any hint would be appreciated.</p>

username: iamgroot42
cooked: <p>Array-based indexing beyond the first dimension does not work in Pytorch like it does in Numpy. Use <code>torch.gather</code>. Example <a href="https://discuss.pytorch.org/t/select-specific-columns-of-each-row-in-a-torch-tensor/497">here</a></p>

username: Abueidda
cooked: <p>Thanks! It seems that it is along the lines of what I am trying to do, but still I am not able to make it work for my case.</p>

username: Abueidda
cooked: <p>It seems that gather cannot make what I need, as the dof tensor, which has the size of batch size X 24, has different indices for each example, but each example has in total 24 indices.</p>
<p>For instance,<br>
dof[0,:] = 0, 1, 6, 9, … (24 in total)<br>
dof[1,:] = 1,100, 151, 300,… (24 in total)</p>
<p>Any hint would be appreciated.</p>
Trying to reuse some of Resnet Layers but getting duplicate layers

username: jc711
cooked: <p>I am trying to reuse some of the resnet layers for a custom architecture and ran into a issue I can’t figure out. Here is a simplified example; when I run:</p>
<pre><code class="lang-auto">import torch
from torchvision import models
from torchsummary import summary

def convrelu(in_channels, out_channels, kernel, padding):
    return nn.Sequential(
        nn.Conv2d(in_channels, out_channels, kernel, padding=padding),
        nn.ReLU(inplace=True),
    )


class ResNetUNet(nn.Module):
    def __init__(self):
        super().__init__()

        self.base_model = models.resnet18(pretrained=False)
        self.base_layers = list(self.base_model.children())


        self.layer0 = nn.Sequential(*self.base_layers[:3])
    

    def forward(self, x):
        print(x.shape)

        output = self.layer0(x)

        return output
    
base_model = ResNetUNet().cuda()
summary(base_model,(3,224,224))
</code></pre>
<p>Is giving me:</p>
<pre><code class="lang-auto">----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 64, 112, 112]           9,408
            Conv2d-2         [-1, 64, 112, 112]           9,408
       BatchNorm2d-3         [-1, 64, 112, 112]             128
       BatchNorm2d-4         [-1, 64, 112, 112]             128
              ReLU-5         [-1, 64, 112, 112]               0
              ReLU-6         [-1, 64, 112, 112]               0
================================================================
Total params: 19,072
Trainable params: 19,072
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 36.75
Params size (MB): 0.07
Estimated Total Size (MB): 37.40
----------------------------------------------------------------
</code></pre>
<p>This is duplicating each layer (there are 2 convs, 2 batchnorms, 2 relu’s) as opposed to giving one layer each. If I print out <code>self.base_layers[:3]</code> I get:</p>
<p><code>[Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False), BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True), ReLU(inplace=True)]</code></p>
<p>which shows just three layers without duplicates. Why is it duplicating my layers?</p>
<p>I am using pytorch version 1.4.0</p>

username: ptrblck
cooked: <p>I assume <code>summary</code> loops over all registered modules inside the model and prints them out.<br>
Since you’ve registered <code>resnet18</code> as <code>self.base_model</code> and again its first three layers as <code>self.layer0</code> in an <code>nn.Sequential</code> container, these modules will be printed as duplicates.</p>
Resume training with LR scheduler

username: numpee
cooked: <p>Hi all,<br>
I’m trying to resume training on a model with a stepLR scheduler, and I keep getting the warning from the following link:<br>
<a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate</a></p>
<pre><code class="lang-auto">UserWarning: Detected call of 'lr_scheduler.step()' before 'optimizer.step()'
</code></pre>
<p>Basically, the issue is that I’ve saved the last recorded step (epoch) along with my model and optimizer’s state dicts. Thus, I if i’ve already trained for <code>x</code> amount of epochs, I would call <code>lr_scheduler.step()</code> <code>x</code> amount of times before passing everything to the trainer.</p>
<p>So is this warning something I should worry about in my case? The warning itself isn’t informative enough for me to understand what the real issue is (maybe this is only problematic during training?). I do see that, by comparing with a previous run, my training isn’t being reproduced 100%, but it’s not off by too much.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I think you can ignore the warning, as you are calling this method before the training to get to the same epoch value.<br>
The warning should be considered, if you are seeing it inside your training loop.</p>
Mixup data augmentation

username: IamSparky
cooked: <p>Can anybody please help me in implementing <strong>mixup</strong> data augmentation process in pytorch <a class="mention-group" href="/groups/admins">@admins</a>?<br>
Please refer if any documentations are available .</p>

username: ptrblck
cooked: <p><a href="https://github.com/facebookresearch/mixup-cifar10">This code</a> is a little bit older by now, but might be a good starter.</p>
<p>PS: Please don’t tag certain people or groups, as this will discourage others to answer. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
TensorFlow, rnn.static_rnn in pytorch

username: 11182
cooked: <p>Hi, I am curious about the PyTorch function.</p>
<p>Is there a function that functions like TensorFlow rnn.static_rnn?</p>
<p>Thank you for letting me know.</p>

username: ptrblck
cooked: <p>It seems the difference between static and dynamic rnn was that the static implementation used a strict maximal sequence length, while the dynamic implementation used a while-loop.</p>
<p>If you want to limit your RNN implementation to a static sequence length, you could use a condition and return the current activation/output.</p>
Expected object of scalar type Float but got scalar type Double

username: Stephen_O_Brien
cooked: <p>I’m writing a regression program in pytorch and when it runs, the model gives the following error:</p>
<p><em>Expected object of scalar type Float but got scalar type Double for argument <span class="hashtag">#4</span></em></p>
<p>The error occurs at this location:</p>
<p>class LinearRegression(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(LinearRegression, self).<strong>init</strong>()<br>
self.linear = torch.nn.Linear(1, 1)<br>
def forward(self, x):<br>
y_pred = self.linear(x)  &lt;---- ERROR OCCURS HERE<br>
return y_pred</p>
<p>The full program and data can be downloaded here:  <a href="https://drive.google.com/open?id=1Ct9hsYoLNeBhua6-Ch4e5xS4Mds1oyWR" rel="nofollow noopener">https://drive.google.com/open?id=1Ct9hsYoLNeBhua6-Ch4e5xS4Mds1oyWR</a></p>
<p>Any help would be greatly appreciated.</p>

username: ptrblck
cooked: <p>This error is raised, if the module parameters (<code>weight</code> and <code>bias</code>) do not have the same type as the input data.<br>
I guess your linear layer has <code>float()</code> parameters, while you are passing <code>double()</code> tensors into it.<br>
Try to call <code>x = x.float()</code> before feeding the data into the model.</p>
<p>This error is often raised, if you created the tensors from numpy arrays, as numpy uses double as the default array type, while PyTorch uses float.</p>
Which NLP model to use to handle long context?

username: billconan
cooked: <p>I’m trying to process product data for an e-commerce platform. The goal is to understand products’ size.</p>
<p>Just to show you some examples on how messy product dimension description is:</p>
<pre><code class="lang-auto">Overall Dimensions: 66 in W x 41 in D x 36 in H
Overall: 59 in W x 28.75 in D x 30.75 in H
92w 37d 32h",
86.6 in W x 33.9 in D x 24 in H
W: 95.75\" D: 36.5\" H: 28.75\"",
W: 96\" D: 39.25\" H: 32\"",
"118\"W x 35\"D x 33\"T.",
"28 L x 95 W x 41 H"
"95\" W x 26.5\" H x 34.75\" D"
"98\"W x 39\"D x 29\"H"
"28\" High x 80\" Wide x 32\" Deep"

</code></pre>
<p>Now assume that the product dimension description is short &lt; 60 characters, I trained a two layer bidirectional LSTM, which can handle this task perfectly.</p>
<p>But the problem is, the above dimension description is usually embedded in a long context (as a part of the product description). How can I extract the useful information from the long context and understand it? My LSTM can only accept context size of 60 (the general idea is the same as <a href="https://towardsdatascience.com/addressnet-how-to-build-a-robust-street-address-parser-using-a-recurrent-neural-network-518d97b9aebd" rel="nofollow noopener">https://towardsdatascience.com/addressnet-how-to-build-a-robust-street-address-parser-using-a-recurrent-neural-network-518d97b9aebd</a>)</p>
<p>What language model is more suitable for this?</p>

username: vdw
cooked: <p>Just a suggestion: I don’t think your data looks that messy. Writing one or more RegEx to extract all dimensions correctly might be a challenge, but it shouldn’t be too difficult to come up with a RegEx that finds a relevant substring of size &lt;60 in a longer text, which you then can give to your existing LSTM.</p>
<p>The RegEx doesn’t even have to be that specific since it’s just a preprocessing step. For example: All substrings containing 3 numbers. Then you include the 3 word before and after that substring and check if it’s below &lt;60 tokens. That should do it.</p>

username: billconan
cooked: <p>I only showed the tip of the ice berg. trust me, it is very very messy.</p>
<p>Just to give you an example where the 3 numbers idea will fail:</p>
<p>Seat height: 32, arm height: 12, back height: 50, seat length: 72, depth: 89</p>
<p>This could be the dimensions of a sofa.</p>
<p>plus, for learning purpose, I still want to know how to handle long context with a nlp model.</p>
Resizing the torch tensor

username: Kahn
cooked: <p>I have a torch tensor of size  <code>torch.Size([1, 128, 56, 128])</code></p>
<p>1 is channel, 128 is the width, and height. 56 is the number of images.</p>
<p>How can I resize it to  <code>torch.Size([1, 56, 128, 128])</code>  ?</p>
<p>thankyou</p>

username: russellizadi
cooked: <p>You need to <code>permute</code> or <code>transpose</code> the tensor:</p>
<pre><code class="lang-auto">import torch
a = torch.randn(1, 128, 56, 128)
print(a.permute(0, 2, 1, 3).shape)
print(a.transpose(2, 1).shape)
</code></pre>
<pre><code class="lang-auto">torch.Size([1, 56, 128, 128])
torch.Size([1, 56, 128, 128])
</code></pre>

username: Kahn
cooked: <p>thankyou, it worked! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Avoiding for loop in the forward method with multi head FC layers

username: amirhf
cooked: <p>I have multiple heads of FC layers defined with a <code>nn.ModuleList</code> as:</p>
<pre><code class="lang-auto">self.multihead_fc_layers = nn.ModuleList([nn.Sequential(
    nn.Linear(64, 2)
) for _ in range(512)])
</code></pre>
<p>The input to this layer is a tensor of size (bs, 64) where bs is the batch-size. The goal for the sub-network is to yield 512 tensors of 2 neurons so a tensor of size (bs, 512, 2). Currently, I’m managing to do this as follows:</p>
<pre><code class="lang-auto">A = []
for h in range(512):
    A.append(softmax(self.multihead_fc_layers[h](E), dim=1)[:, 1].unsqueeze(1))
A = torch.cat(A, dim=1)
</code></pre>
<p>where E is the input tensor of size (bs, 64). Is there a way to code this more efficiently and without the for loop?</p>

username: SimonW
cooked: <pre><code class="lang-auto">def __init__(...):
    self.head = nn.Linear(64, 2 * 512)


def forward(....):
    A = self.head(E).reshape(-1, 512, 2).softmax(dim=-1)[:, :, 1]
</code></pre>
<p>But if you are doing a 2-element softmax, you might as well just use a 1-element output + sigmoid instead. I.e.,</p>
<pre><code class="lang-auto">def __init__(...):
    self.head = nn.Linear(64, 512)


def forward(....):
    A = self.head(E).sigmoid()
</code></pre>

username: amirhf
cooked: <p>Thank you <a class="mention" href="/u/simonw">@SimonW</a> for your answer. Are all of these heads trained separately?</p>

username: SimonW
cooked: <p>Yeah, just the weight matrices are concatenated together as one FC. Applying multiple FCs to the same input and concatenating the results is the same as concatenating hte FCs and applying once to the input.</p>

username: amirhf
cooked: <p>Great! Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Sample a batch from multiple text files

username: omer_as
cooked: <p>Hey<br>
I am training a language model with several text files (“books”) as input.<br>
In each training iteration, I want the batch to be composed of equally-sized “minibatches” from all books, and I wonder what is the correct way to implement it. If I understand correctly, there’s no way to access multiple datasets while maintaining the ‘identity’ of each dataset and using the same dataloader. Is multiple datasets the the way to go? Should I implement new <code>Sampler</code> and <code>BatchSampler</code>?</p>
<p>An important constraint is that the entire training set doesn’t fit in my computer’s memory so I have to read it as chunks in for each batch.</p>
<p>Thanks for the help!</p>
Custom backward with ctx

username: maralm
cooked: <p>Hi,</p>
<p>I have two questions regarding writing custom backward.<br>
1- Can I add a list of tensors to ctx.save_for_backward? I need to have a list of  24 activation outputs in my backward function.</p>
<p>2- Can I output something in the custom backward function? Let’s say I have a second copy of the model and would like to copy gradients in backward to that. How can I get gradients in the second model when calling loss.backward()</p>

username: albanD
cooked: <p>Hi,</p>
<ol>
<li>
<p>yes, call it as <code>ctx.save_for_bacward(*your_tensor_list)</code>. And get them back as <code>your_tensor_list = list(ctx.saved_tensors)</code> in the backward (if you’re fine with a tuple, the list is not necessary).</p>
</li>
<li>
<p>The custom backward must output the gradients wrt to the inputs of the forward function. Not sure to understand what you want here. But you have access to global variables and anything you saved in the ctx here.</p>
</li>
</ol>

username: maralm
cooked: <p>Thak you!</p>
<p>Regarding <span class="hashtag">#2</span>, my actual model is on gpu and I have an extra copy of the model (which resides on cpu) and when I calculate gradient for a layer, I copy the gradients to the cpu model too. So can I save the cpu model with ctx like ctx.cpu_model = cpu_model in forward pass, receive it in the backward pass, copy gradients from gpu model to cpu model during backward and after calling loss.backward(), get the gradients from the cpu model?</p>

username: albanD
cooked: <p>Yes that’s the way to do it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to select 5% of total values from a tensor randomly?

username: Umair_Javaid
cooked: <p><code>a = torch.rand(2,5,10)</code><br>
I want to select at most 5% of values from tensor <code>a</code> randomly and then multiply those values with <code>-1</code>? How to do that? kindly, give a generic solution as the shape of the tensor is not fixed</p>
user_suspended: yes

username: Umair_Javaid
cooked: <p><a class="mention" href="/u/alband">@albanD</a> any suggestions?</p>
user_suspended: yes

username: albanD
cooked: <p>Hi,</p>
<p>I guess you can do <code>out = (torch.rand_like(a) - 0.05).sign().type_as(a) * a</code> assuming a is a floating point number. This should select around 5% (the bigger the Tensor, the more precise it will be).</p>
ValueError: max() arg is an empty sequence

username: Michele_Guerra
cooked: <p>Hi everyone,</p>
<p>I was trying to do the tutorial on reinforcement learning but this error shows up when trying to plot the initial configuration of the PoleCart. After a bit research I think that the problem is with the backend of matplotlib. The tutorial says to set up matplotlib in this way:</p>
<h1>set up matplotlib</h1>
<p>is_ipython = ‘inline’ in matplotlib.get_backend()<br>
if is_ipython:<br>
from IPython import display</p>
<p>but the string that comes out of matplotlib.get_backend() for me is:</p>
<p>module://backend_interagg</p>
<p>Am I doing something wrong?</p>

username: russellizadi
cooked: <p>Are you using IPython (Jupyter notebooks or Google Colab)? If yes, one solution is setting <code>%matplotlib inline</code> which is a magic function. If you can share a link to the tutorial, I can check that out myself.</p>

username: Michele_Guerra
cooked: <p>I’ve been using PyCharm, in a normal setting, so I’m guessing no IPython, also because I’ve tried to write the magic function and it gives me a syntax error.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html</a></p>
<p>This is the link to the tutorial <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: russellizadi
cooked: <p>Have you tried this?</p>
<p><a href="https://stackoverflow.com/questions/49048520/how-to-prevent-pycharm-from-overriding-default-backend-as-set-in-matplotlib" rel="nofollow noopener">How to prevent PyCharm from overriding default backend as set in matplotlib?</a></p>

username: Michele_Guerra
cooked: <p>Oh I forgot to mention that I was connecting remotely to another computer and that was giving me problems! Now everything has worked out, since it’s just a tutorial I will run it locally. But now I will also try to run it remotely, even though it seems trivial: <a href="https://stackoverflow.com/questions/41892039/how-to-enable-x11-forwarding-in-pycharm-ssh-session" rel="nofollow noopener">https://stackoverflow.com/questions/41892039/how-to-enable-x11-forwarding-in-pycharm-ssh-session</a></p>
<p>Thanks a lot for your help!</p>
Using pos_weight in BCEWithLogitsLoss to optimize for F2 score

username: Lakshay_Sharma
cooked: <p>I have a multi-class multi-label classification problem, and so I’ve been using the <a href="https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss" rel="nofollow noopener">BCEWithLogitsLoss</a>. I’d like to optimize my model for a higher F2 score, and so want to bias it to have greater recall (with decent precision too of course). Are there any guidelines for setting the values in <code>pos_weight</code> to be able to do this?</p>
Multiple convolution filters with different sizes in parallel

username: minzwon
cooked: <p>Hi,</p>
<p>I want to use multiple convolution filters in parallel with initial weights (I want the filter values to be fixed).</p>
<p>Below is an example of the desired code. Filter lengths are different but the output dimension will be the same due to the padding. In this example, I only used three filters but I would like to use more than a hundred filters. In this case,</p>
<ol>
<li>What is the best practice to initialize multiple (&gt;100) convolution filters?</li>
<li>And how can I concatenate them all?<br>
(Let’s assume I already have the filter_weights array)</li>
</ol>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__():
        super(Model, self).__init__()

        self.filter_weights = [torch.tensor([[[0.1, 0.2, 0.3]]]),
                               torch.tensor([[[0.1, 0.2, 0.3, 0.4, 0.5]]]),
                               torch.tensor([[[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7]]])]

    def forward(x):
        out1 = F.conv1d(x, self.filter_weights[0], padding=1)
        out2 = F.conv1d(x, self.filter_weights[1], padding=2)
        out3 = F.conv1d(x, self.filter_weights[2], padding=3)
        out = torch.cat([out1, out2, out3], dim=1)
        return out
</code></pre>

username: ptrblck
cooked: <p>If you can come up with the padding logic inside a loop (or use some kind of lookup table), this should work:</p>
<pre><code class="lang-python">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.filter_weights = [torch.tensor([[[0.1, 0.2, 0.3]]]),
                               torch.tensor([[[0.1, 0.2, 0.3, 0.4, 0.5]]]),
                               torch.tensor([[[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7]]])]

    def forward(self, x):
        out = []
        for idx, filt in enumerate(self.filter_weights):
            padding = idx+1 # try to come up with logic here
            out.append(F.conv1d(x, filt, padding=padding))
        out = torch.cat(out, dim=1)
        return out
</code></pre>

username: DavidWRomero
cooked: <p>Hi, thanks for your answer. What about efficiency? would they be performed in parallel? I was trying to do something similar with cuda.Streams but have not been able to make it work. I would love to hear your thoughts about it. Thanks in advance.</p>
<p>David</p>

username: ptrblck
cooked: <p>Kernels will be put into a queue for execution on the GPU.<br>
Depending on the utilization of the GPU, you might try to use streams, but if a single conv operation is already fully utilizing the device, you most likely won’t see much difference.</p>