akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How do I use Cyclical Learning Rates built in pytorch?

username: vauthority
cooked: <p>Hi everyone! Currently, I’m doing image segmentation task with U-net architecture, but the result not that promising, so I came across cyclical learning rates which gives high performance and more efficient. I found that CLR is built-in pytorch but I’m not sure that I used it in a correct way? Here is my code:</p>
<p>optimizer= torch.optim.SGD(filter(lambda p: p.requires_grad ,model.parameters()),lr=1e-1,momentum =0.9)<br>
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer,base_lr =1e-3,max_lr = 0.1)<br>
criterion= nn.CrossEntropyLoss()</p>
<p>Training loop:<br>
epochs = 0<br>
for i in range(30):<br>
scheduler.step()<br>
for img,mask in loader:<br>
model.train()<br>
optimizer.zero_grad()<br>
img = img.to(device,dtype=torch.float)<br>
mask = mask.to(device,dtype=torch.float)<br>
preds = model(img)<br>
loss =criterion(preds,mask.long())<br>
loss.backward()<br>
optimizer.step()</p>
Weird Linear Layer Behaviour when using ModuleList

username: Kale-ab_Tessera
cooked: <p>So I have a list of layers and I apply a mask to some of the weights in the layers, by doing the following:</p>
<pre><code class="lang-auto"> def forward(self, input, mask=None):
        if mask is not None:
            self.weight.data.t().mul_(mask)
        f = F.linear(input, self.weight, self.bias)
        return f
</code></pre>
<p>This works when I store the list of layers like <code>self.hidden_layers = []</code>, but when I change this to <code> self.hidden_layers = nn.ModuleList()</code> the mask code above doesn’t work.</p>
<p>This is how I append to the list of layers:</p>
<pre><code class="lang-auto">        for i in np.arange(self.num_hidden_layers - 1):
            self.hidden_layers.append(LinearAutoMl(hidden_size, hidden_size).to(device))
</code></pre>
<p>Any reason why using ModuleList would do this?</p>

username: Kale-ab_Tessera
cooked: <p>Fixed it by changing line of code that applies the mask to<br>
<code>  self.weight = nn.Parameter(self.weight * mask.t())</code> .</p>

username: albanD
cooked: <p>Does the following works?</p>
<pre><code class="lang-auto">with torch.no_grad():
    self.weight.mul_(mask.t())
</code></pre>

username: Kale-ab_Tessera
cooked: <p>So that worked. Any reason why the above 2 methods don’t work?</p>

username: albanD
cooked: <p><code>.data</code> should not be used anymore as it has weird semantic.<br>
And rewrapping into an nn.Parameter has the side effect of calling <code>.detach()</code>, meaning that you would get a brand new set of parameters that are not necessarily the same as the ones you passed the optimizer.</p>

username: SimonW
cooked: <p>also you are changing weight in place at every forward, which is likely not what you want to do.</p>
Regarding custom layer

username: vijaytida
cooked: <p>Hi,<br>
Can anyone help me how to define addition of two vectors (adding  element one by one) with user defined forward and backward propagation using custom layer.<br>
Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p><a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd" rel="nofollow noopener">This section</a> of the doc should contain all the informations you need <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: vijaytida
cooked: <p>Actually this kind of example is confusing. Is there any simplified examples like addition of two vectors ?<br>
Thanks</p>

username: albanD
cooked: <p>There is an example in that tutorial that presents multiplication by a constant (called <code>MulConstant</code>). Isn’t that close enough to adding two vectors?</p>

username: vijaytida
cooked: <p>I tried to run the custom c++ extension code as mentioned in the post (<a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>) but the time taken for python and cpp versions are entirely different for backward propagation especially.<br>
(test1) gputest@gputest-OptiPlex-745:~/PycharmProjects/test1/extension-cpp-master$ python benchmark.py py<br>
Forward: 524.282/560.827 us | Backward 844.955/890.770 us<br>
(test1) gputest@gputest-OptiPlex-745:~/PycharmProjects/test1/extension-cpp-master$ python benchmark.py cpp<br>
Forward: 473.022/497.065 us | Backward 1243.830/1299.400 us</p>
<p>Because he mentioned that backward propagation time should be similar for both. But here I got different values will that would be an issue?<br>
Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>A difference of less than 0.4ms for a single run is not really significant I think. This can be caused by any other program running at the same time and competing for the same ressource.</p>
Named tensors: why don't contracting operations check the dimension names?

username: agarriga
cooked: <p>I discovered this <a href="https://pytorch.org/docs/stable/name_inference.html#contracts-away-dims" rel="nofollow noopener">documented behaviour</a> in the REPL: when matrix-multiplying two tensors with incompatible names (but compatible dimensions), it just works.</p>
<p>I thought the point of named tensors was to check for correct behaviour at runtime. Why aren’t the names checked here?</p>
Using Caffe2 C++ framework in 2019

username: Postermaestro
cooked: <p>Does anyone care to explain how to use the Caffe2 C++ framework today? Does it come with pytorch now since they are merged? I am trying to build a simple project in Visual Studio 2019 using Caffe2 in Windows 10.</p>

username: tom
cooked: <p>Quite likely, you are on your own there:</p>
<aside class="quote" data-post="4" data-topic="50526">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/caffe2-vs-libtorch-realistically/50526/4">Caffe2 vs libtorch, realistically</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    the caffe2 bits have no public support, and will be changed and deprecated at will. 
libtorch will be maintained, supported and improved, but at this point it wont have feature parity with caffe2 as a goal.
  </blockquote>
</aside>

<p>Libtorch is rapidly approaching feature parity with PyTorch for the “beyond Tensor and Autograd bits”, too.</p>
<p>Best regards</p>
<p>Thomas</p>

username: rilesdg3
cooked: <aside class="quote no-group" data-username="Postermaestro" data-post="1" data-topic="56538">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/postermaestro/48/16133_2.png" class="avatar"> Postermaestro:</div>
<blockquote>
<p>e merged? I am trying to build a simple project in Visual S</p>
</blockquote>
</aside>
<p>It comes with PyTorch so if you download and install pytorch  you will also be installing caffe2 c++ api. From there you just include the header files for caffe2. I am currently using Caffe2 c++ api on linux.</p>
<p>Check this link out <a href="https://github.com/rilesdg3/caffe2_cpp_tutorial" class="inline-onebox" rel="noopener nofollow ugc">GitHub - rilesdg3/caffe2_cpp_tutorial: C++ transcripts of the Caffe2 Python tutorials and other C++ example code</a></p>
PyTorch model with Nvidia Inference Server

username: kaan
cooked: <p>I want to run my PyTorch model over Nvidia Inference Server with high performance. Instead of convert PyTorch model to Torch Script, is it possible to convert pytorch -&gt; onnx -&gt; tensorrt -&gt; nvidia inference server? Which one is better for performance?</p>

username: ptrblck
cooked: <p>If I’m not mistaken, ONNX uses TorchScript under the hood, so you won’t be able to directly export to ONNX.<br>
Please correct me, if I’m wrong.</p>
Keep Repeating the Same Index in Data Loader N times

username: Rafael_R
cooked: <p>Hi,</p>
<p>I have a case when for a given index in the dataset, i want my dataloader to return permutations of data[‘index’] n times.</p>
<p>How can i do so?</p>
<p>i.e. can i manually control the updating of index and yet keep returning the data smples?</p>
<p>So, artifically if my dataset has k samples, the new approach will return k * n samples in one epoch?</p>

username: ptrblck
cooked: <p>I’m not sure how your actual logic is, but I think a custom <code>sampler</code> might do what you are looking for.</p>
Dataloader for importing 1D signal (EGG)

username: pedram1
cooked: <p>HI everyone</p>
<p>I have a data set size of [65536,1,94] , it is a tensor that 65555 is number of my data , 1 is channel number and 94 is length of the signal. My data is a not a pic, its like a signal.</p>
<p>I would like to use dataloader like this due to device my data to 32 epochs and 2048 batch size.<br>
is this the right way of doing that:<br>
///////////////////////////////////////////////////<br>
num_epochs = 32<br>
batch_size = 2048<br>
dataloader = DataLoader(data, batch_size=batch_size)</p>
<p>//////////////////////////////////<br>
thank you</p>

username: ptrblck
cooked: <p>Your code should work.<br>
Generally I would recommend to create a custom <code>Dataset</code> (where you can use transformations etc.) or pass your tensors to a <code>TensorDataset</code>.<br>
However, in your simple example, it should work nevertheless to pass the data tensor directly to the <code>DataLoader</code>.</p>