akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Preprocess image without F.interpolate

username: manthan3C273
cooked: <p>I’m doing image prepressing during training as below:</p>
<pre><code class="lang-auto">im_transform = transforms.Compose(
  [
      transforms.ToTensor(),
      transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
  ]
)

im = Image.open('/content/image.jpg')

# unify image channel to 3 here

im = im_transform(im)

# add mini-batch dim here

im = F.interpolate(im, size=(im_rh, im_rw), mode='area')
</code></pre>
<p>Then I’m passing it to model. I checked how it looks like before passing to the model.</p>
<pre><code class="lang-auto">im_np = im.numpy()
im_np = im_np.reshape((im_rh, im_rw, 3))
im_np = (im_np * 255).astype(np.uint8)
check_im = Image.fromarray(im_np)
check_im
</code></pre>
<p>The input and processed image is looking like this,<br>
<a href="https://drive.google.com/file/d/1YqMf0PvNjQG_jiumuKce6Kktv6gl4tyJ/view?usp=sharing" rel="noopener nofollow ugc">Input Image</a><br>
<a href="https://drive.google.com/file/d/1skpk_xR_HhG269M6pNqA3IUtWZ1PBArP/view?usp=sharing" rel="noopener nofollow ugc">Processed Image</a></p>
<p><strong>So my question is, how can I get the same result like this without using F.interpolate for inference.</strong> I’m reading image using opencv. And also normalize it to bound values between -1 to 1.</p>

username: ptrblck
cooked: <p>The interleaving in the processed image is most likely created by a <code>view</code> or <code>reshape</code> operation using a wrong memory layout (channels-last vs. channels-first).<br>
I assume this operation might cause the trouble:</p>
<pre><code class="lang-python">im_np = im_np.reshape((im_rh, im_rw, 3))
</code></pre>
<p>Could you check the shape of <code>im_np</code> before applying the <code>reshape</code> operation? If the channels are in <code>dim0</code>, either change the reshaping opertation or permute the dimensions first.</p>

username: manthan3C273
cooked: <p>Exactly, the problem is not with <code>F.interpolation</code> or <code>cv2.resize</code> . Actually, <code>im_np.reshape</code> is not the correct choice here. I did it by this way and it works very well.</p>
<pre><code class="lang-auto">im = cv2.imread('/content/image.jpg')
im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)

# normalize values to scale it between -1 to 1
im = (im - 127.5) / 127.5   

# find scale factor - fx &amp; fy

# resize image
im = cv2.resize(im, None, fx = x, fy = y, interpolation = cv2.INTER_AREA)

# prepare input shape
im = np.transpose(im)
im = np.swapaxes(im, 1, 2)
im = np.expand_dims(im, axis = 0).astype('float32')
</code></pre>
<p>Thank you for help.!</p>
How to handle Out-of-vocabulary token in inference using torchtext Field?

username: aryopg
cooked: <p>Hi guys, I am facing a problem using the torchtext package. So, in the data building phase, I created a text field using the data.Field and I build the vocabulary using training data:</p>
<pre><code class="lang-auto">shared_text_field = data.Field(sequential=True, tokenize=self.tokenizer.tokenize,
                                            init_token=self.sos_token, eos_token=self.eos_token,
                                            pad_token=self.pad_token, unk_token=self.unk_token)
shared_text_field.build_vocab(train)
</code></pre>
<p>Unfortunately, when I tried to use it with my test data, I got a KeyError problem (I’m sorry for the truncated error message):</p>
<pre><code class="lang-auto">File "/Users/aryopg/.local/share/virtualenvs/learning-Y_vf_ZaD/lib/python3.7/site-packages/torchtext/data/field.py", line 336, in &lt;listcomp&gt;
arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
KeyError: 'hardship'
</code></pre>
<p>Did I do something wrong or is this at the moment not supported? (A lil bit bizarre if it’s not supported yet) I’ll be very happy to provide more details. Thanks</p>

username: dunefox
cooked: <p>There is a similar issue here but with a whitespace being looked up: <a href="https://discuss.pytorch.org/t/out-of-vocabulary-keyerror-on-vocab-stoi-in-torchtext/65910" class="inline-onebox">Out-of-vocabulary KeyError on vocab.stoi in TorchText</a><br>
Even if the comment there is correct – that a whitespace shouldn’t really be part of your vocabulary – shouldn’t it be mapped to the unknown token by default? Unfortunately, they’re not really answering the question about oov words. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>Issue without much discussion: <a href="https://github.com/pytorch/text/issues/337" rel="nofollow noopener">https://github.com/pytorch/text/issues/337</a><br>
… Maybe you can revive the issue or create a new one.</p>

username: aryopg
cooked: <p>Thanks for the response and for pointing out those links! I am quite aware of the unresponsiveness, unfortunately <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:"><br>
This may not be the place to ask, but is there any other library (other than torchtext) that is more robust and well-maintained? Ps: Hopefully this will ring a little bell for the developers lol</p>

username: abishekchiff
cooked: <p>There are bugs in torchtext, try what you did after installing it straight from the Github repository like this <strong>pip install --upgrade git+https://github.com/pytorch/text</strong></p>

username: cramraj8
cooked: <p>Are there any answers to the OOV issue so far ?</p>
Round of floating point value of tensor

username: cbd
cooked: <p>I have tensor of value ([0.1234, 0.5678]). I want to round of it to 3 decimal point like  ([0.123, 0.567]).</p>

username: Dwight_Foster
cooked: <p>You could do something like this</p>
<pre><code class="lang-auto">n_digits = 3
rounded = (x * 10**n_digits).round() / (10**n_digits)
</code></pre>
<p>from <a href="https://discuss.pytorch.org/t/round-tensor-to-x-decimal-places/25832/3">this</a> forum post but it will keep the zeros at the end.</p>
Image Denoising using Unet

username: Hugh_T
cooked: <p>I’m new to computer vision and deep learning. I’m trying to train this Unet model <a href="https://github.com/kevinlu1211/pytorch-unet-resnet-50-encoder" class="inline-onebox" rel="noopener nofollow ugc">GitHub - rawmarshmellows/pytorch-unet-resnet-50-encoder</a> with Resnet50 as encoder. I want to implement it in a way that I pass two rgb images which are first processed by resnet50 and then the layers are concated before being passed to the decoder. I tried doing it and changed n_classes in the code to 3 to output a 3 channel rgb image just like the inputs but it gives me a distorted image <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/576ab14f51693d3359540d3bb3eb58e3f70ad3d1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/576ab14f51693d3359540d3bb3eb58e3f70ad3d1" title="20210211_223158"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/576ab14f51693d3359540d3bb3eb58e3f70ad3d1_2_690x419.jpeg" alt="20210211_223158" data-base62-sha1="ctk7VhHc5q6hiomeRjGb1cygrLz" width="690" height="419" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/576ab14f51693d3359540d3bb3eb58e3f70ad3d1_2_690x419.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/576ab14f51693d3359540d3bb3eb58e3f70ad3d1_2_1035x628.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/576ab14f51693d3359540d3bb3eb58e3f70ad3d1.jpeg 2x" data-dominant-color="878878"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20210211_223158</span><span class="informations">1323×805 745 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>  which I don’t understand why. Please help me with this</p>
Why `torch.is_floating_point` and `torch.is_complex` but `is_integer`?

username: dtoniolo
cooked: <p>PyTorch currently provides <code>torch.is_floating_point</code> and <code>torch.is_complex</code> to check whether a tensor contains data of a certain numeric types. Why hasn’t an equivalent method for integers been implemented yet?</p>
<p>My guess is that it isn’t as widely requested as the other two, but maybe there’s a more profound reason. I’m considering opening an issue on GitHub for creating a feature request</p>

username: ptrblck
cooked: <p>Creating a feature request sounds good.<br>
Maybe you could check, if your current tensor is not a floating point tensor to get an idea, if it’s an integer tensor?</p>

username: dtoniolo
cooked: <p>Thanks for you answer. I’ve created a <a href="https://github.com/pytorch/pytorch/issues/52161" rel="noopener nofollow ugc">feature request</a>, let’s see what comes out of it.</p>
[D] - Self attention in computer vision

username: black0017
cooked: <p>Hey all !</p>
<p>This is Nikolas from AI Summer.</p>
<p>I am pretty interested in self-attention and transformers in computer vision. I have started an open source project to collect my process of re-implementing different modules in self attention and transformers architectures in computer vision.</p>
<p>If there is anybody that is interested in the same stuff please do let me know.</p>
<p>Here is a list of articles that I wrote:</p>
<ul>
<li><a href="https://theaisummer.com/attention/" rel="noopener nofollow ugc">How Attention works in Deep Learning: understanding the attention mechanism in sequence models</a></li>
<li><a href="https://theaisummer.com/transformer/" rel="noopener nofollow ugc">How Transformers work in deep learning and NLP: an intuitive introduction</a></li>
<li><a href="https://theaisummer.com/vision-transformer/" rel="noopener nofollow ugc">How the Vision Transformer (ViT) works</a></li>
<li><a href="https://theaisummer.com/einsum-attention/" rel="noopener nofollow ugc"> Understanding einsum for Deep learning: implement a transformer with multi-head self-attention from scratch</a></li>
</ul>
<p>Based on all my research from zero to hero for self attention I also created this repo:</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/The-AI-Summer/self-attention-cv" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://repository-images.githubusercontent.com/334760721/5b618d00-6b44-11eb-999f-8347bda3562e" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/The-AI-Summer/self-attention-cv" target="_blank" rel="noopener nofollow ugc">The-AI-Summer/self-attention-cv</a></h3>


  <p><span class="label1">Implementation of self-attention mechanisms for general purpose. Focused on computer vision modules. Ongoing repository. pip package coming soon...</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Would love to hear that more people are interested to re-implement papers in this direction.</p>
<p>Have a fab. day!</p>
torch.FloatTensor and torch.tensor returns different shape of tensor

username: nobodykid
cooked: <p>Hi,</p>
<p>Recently, I’ve been facing the same issue as in <a href="https://discuss.pytorch.org/t/how-to-fix-mismatch-in-shape-when-using-backward-function/58865" class="inline-onebox">How to fix Mismatch in shape when using .backward() function</a></p>
<p>Originally, I use <code>torch(.cuda).FloatTensor</code> to create the tensor. Comparing it to the solution on the post, this is what I got</p>
<pre><code class="lang-auto"># using FloatTensor
a_one = torch.FloatTensor(1)
a_one
&gt;&gt;&gt; tensor([0.])
a_one.dtype
&gt;&gt;&gt; torch.float32

# using torch.tensor
b_one = torch.tensor
b_one
&gt;&gt;&gt; tensor(1.)
b_one.dtype
&gt;&gt;&gt; torch.float32
</code></pre>
<p>Can someone explain how both of this tensor initialization returns different shape of tensor?</p>

username: Dwight_Foster
cooked: <p>In torch.FloatTensor the parameter you put into it is the shape not the values in the tensor while in torch.tensor you pass in the values of the tensor not the shape.</p>

username: nobodykid
cooked: <p>But if you put this in, it return the same dimension as <code>torch.FloatTensor(1)</code> except now it has value?</p>
<pre><code class="lang-auto">a = torch.FloatTensor([1])
a
&gt;&gt;&gt; tensor([1.])
</code></pre>

username: Dwight_Foster
cooked: <p>If you put it as a list than FloatTensor uses it as an input instead of a shape.</p>

username: ptrblck
cooked: <p>The short answer is: don’t use <code>torch.FloatTensor</code> to create tensors, but stick to the factory methods, such as <code>torch.randn</code>, <code>torch.ones</code>, <code>torch.empty</code>, <code>torch.tensor</code>, etc.<br>
The former approach might have unexpected behavior (as see in this topic), could yield an uninitialized tensor (as explained by <a class="mention" href="/u/dwight_foster">@Dwight_Foster</a>) etc.</p>
Why are the gradients not disabled when decorating the function

username: Carsten_Ditzel
cooked: <p>I have a function in which prompting the require_grad attribute for a tensor that has <strong>requires_grad = True</strong> set, correctly yields <strong>False</strong> because the function itself is decorated with <span class="mention">@torch.no_grad</span>(). Turning of the decoration results in <strong>True</strong>.</p>
<p>Trying to reproduce this behaviour with the following snippet</p>
<pre><code class="lang-auto">x = nn.Parameter(torch.tensor(1.))
print(x.requires_grad)

@torch.no_grad()
def foo(t):
    print(t.requires_grad)

foo(x)
</code></pre>
<p>gives <strong>True</strong> both times when I expected the second print statement to output <strong>False</strong>.</p>
<p>Am I missing something here?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The decorator won’t change the <code>requires_grad</code> property on existing Tensor. The only thing it does is disable tracking of gradient computations within that block.<br>
In particular, that means that if you do any op in your function: <code>t2 = t + 1</code>, then t2.requires_grad=False because that op was not tracked and so the output does not require gradients.</p>

username: Carsten_Ditzel
cooked: <p>thank you, yes that is clear. However, I experienced the case where requires_grad would actually change from printing True to False if output within the function decorated like above which seems strange to me. As soon as I put torch.enable_grad() as a decorator, requires_grad output True as it was meant to. Thats why I figured that the decorater itseld causes this output to be different oO</p>
BCEWithLogitsLoss

username: Tommaso_Iaquinta
cooked: <p>I have multilabel onehot encoded targets. As definition each of my examples can fall in more classes but it can fall also in None of them. Should I create a new class corresponding to the examples that fall in no class or is it ok to have example with all 0 vector target.</p>
<p>Some example:<br>
‘text’–&gt;8 element target</p>
<p>‘i do not like this movie’ → [0,0,0,0,0,1,0,0]<br>
‘i do not like this movie because i hate thriller’ → [1,0,0,0,0,1,0,0]<br>
‘hi I’m tommy’ → [0,0,0,0,0,0,0,0]</p>
<p>My question referring to the example is if it’s, in general, better to change everything to:<br>
‘text’–&gt;9 element target</p>
<p>‘i do not like this movie’ → [0,0,0,0,0,1,0,0,0]<br>
‘i do not like this movie because i hate thriller’ → [1,0,0,0,0,1,0,0,0]<br>
‘hi I’m tommy’ → [0,0,0,0,0,0,0,0,1]</p>

username: jeremyfix
cooked: <p>My understanding is that your additional “other” logit would not bring in any additional information in the sense that saying it is the 9th element is equivalent to say it is none of the others.</p>

username: Tommaso_Iaquinta
cooked: <p>I agree with you that it brings no additional information, in my training set the NoClass examples are around 70% of the dataset and I wonder which of the two explained situation are better in order to add the “pos_weight” parameter of BCEWithLogitsLoss.  Also the other classes are unbalanced, which is in the 30% of the dataset the classes are not equally distributed.</p>

username: cpeters
cooked: <p>The pos_weight parameter is specified per-class. e.g. if the 0th class only has one positive occurrence to a thousand negative occurrences then you would set it to 1000/1, 1000, regardless of the relative occurrences of other classes. It only concerns the relative appearance of a positive value <em>for that class</em> in relation to the number of times it’s 0.</p>
<p>All that adding a class for the no-class case would do for pos_weight is add another column where the weight is approximately 30/70 (i.e. weighting the negative cases more strongly to offset how many positive values there are).</p>
<p>All this is assuming that you wish to weight the classes as if they’re all equally likely to occur within the dataset.</p>
Export RNN (GRU) as ONNX, how to set the batch size

username: Johannes_L
cooked: <p>Hello to all!<br>
I’m trying to export a trained GRU as ONNX. As I want to test it online (take each timestep in a loop and hand over hidden state), I would like to have a fixed input length of 1. With batch_first=True I use an input tensor of size (batch_sz=1, seq_len=1, feat=10). When I try to export the ONNX I get the Warning:</p>
<p>"/Users//miniforge3/envs/arm_native/lib/python3.9/site-packages/torch/onnx/symbolic_opset9.py:1909: UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with GRU can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model.<br>
warnings.warn("Exporting a model to ONNX with a batch_size other than 1, " + "</p>
<p>What am I doing wrong? I also tried to set dynamic axis for the input-length, with no effect.<br>
I can’t find any examples for exporting recurrent nets, only old workarounds for when torch didn’t support this feature.</p>
<pre><code class="lang-auto">import torch

class Test(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.net = torch.nn.GRU(input_size=10, hidden_size=5, num_layers=2, batch_first=True)
        self.net.eval()

    def forward(self, x, h):
        return self.net(x, h)


m = Test()
_x = torch.rand((1, 1, 10))
_h = torch.rand((2, 1, 5))
_out, _h_n = m(_x, _h)

torch.onnx.export(m,
                  (_x, _h),
                  'def.onnx',
                  export_params=True)
</code></pre>
Customize Embedding layer in Pytorch Transformer

username: rubenszmm
cooked: <p>Hi guys, I followed the Harvard Annotated Transformer at <a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="noopener nofollow ugc">Annotated Transformer</a> and everything runs ok with text and integers. However, as the Transformer is an autoregressive model, I’d like to bypass the Embedding layer, given that it only accepts .long() data type (integers) and I have float data for Time Series forecasting. I searched for nn.Embedding source code in PyTorch but I guess it is written in C. How can I customize the nn.Embedding in a way it keeps the original input data as output  or accepts floats as inputs without integer requirements/transformations?</p>
<p>Here is the class being used.</p>
<pre><code class="lang-auto">class Embeddings(nn.Module):
    def __init__(self, d_model, vocab):
        super(Embeddings, self).__init__()
        self.lut = nn.Embedding(vocab, d_model)         #here
        self.d_model = d_model

    def forward(self, x):
        return self.lut(x) * math.sqrt(self.d_model)
</code></pre>
<p>Ahead of the code, we have:</p>
<pre><code class="lang-auto">def make_model(src_vocab, tgt_vocab, N=6, 
               d_model=512, d_ff=2048, h=8, dropout=0.1):
    "Helper: Construct a model from hyperparameters."
    c = copy.deepcopy
    attn = MultiHeadedAttention(h, d_model)
    ff = PositionwiseFeedForward(d_model, d_ff, dropout)
    position = PositionalEncoding(d_model, dropout)
    model = EncoderDecoder(
        Encoder(EncoderLayer(d_model, c(attn), c(ff), dropout), N),
        Decoder(DecoderLayer(d_model, c(attn), c(attn), 
                             c(ff), dropout), N),
        nn.Sequential(Embeddings(d_model, src_vocab), c(position)),
        nn.Sequential(Embeddings(d_model, tgt_vocab), c(position)),
        Generator(d_model, tgt_vocab))
    
    for p in model.parameters():
        if p.dim() &gt; 1:
            nn.init.xavier_uniform(p)
    return model```

I tried to use `torch.cat` without success.

Thanks in advance</code></pre>
Opposite of grid_sample

username: RevoGen
cooked: <p><a href="https://pytorch.org/docs/master/nn.functional.html#torch.nn.functional.grid_sample" rel="nofollow noopener">grid_sample</a> samples values from the coordinates of the input. I was wondering if there was a way to do the reverse: assigning values to particular coordinates of the output, with the coordinates being within [-1,1].<br>
Is there a way to do this?</p>
<p>Any help is appreciated.</p>
<p>Solved. I inverted the affine transformation I used to generate the grid, and used grid_sample as normal.</p>

username: mr-mojo
cooked: <p>I’m actually looking for the same thing! ideally with support for reversing the bilinear interpolation that was used when extracting with grid_sample. <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/simonw">@SimonW</a>  any insights on this? Thanks!</p>
Low level operations comparative performance

username: InfoMax
cooked: <p>In PyTorch there are often multiple ways to do the same simple operations while having sometimes very different computational costs.<br>
A few examples:<br>
-indexing is faster than index_select<br>
-indexing on a later dimension rather than the first dimension causes subsequent operations such as torch.bucketize to be slower because the resulting tensor is no longer contiguous<br>
-torch.prod is faster than multiplying indexed tensors<br>
-torch.expand is faster than torch.repeat because there is no copy</p>
<p>I was wondering if there is a comprehensive list of such low level common performance “tricks”? Also is it possible for an implementation to be faster than the alternative on the forward pass but slower than the alternative on the backward pass? I am mainly interested in the comparative performance of low level basic operations like the ones above.</p>
A difference between np.clamp and torch.clamp, any possible workaround?

username: hrushi
cooked: <p>In numpy, while using np.clamp(x, min, max) we can pass an array of min/max values but pytorch only accepts an integer.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = np.array([0.4, 10, 0.1])
&gt;&gt;&gt; np.clip(a, [0.3, 0.4,0.5], [0.9, 0.99, 0.999])
array([0.4 , 0.99, 0.5 ])

&gt;&gt;&gt; a = torch.tensor([0.4, 10, 0.1])
&gt;&gt;&gt; torch.clamp(a, [0.3, 0.4,0.5], [0.9, 0.99, 0.999])
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
TypeError: clamp(): argument 'min' (position 2) must be Number, not list

</code></pre>
<p>Is there any way I can use torch directly to clamp the values using an array instead of converting the torch.tensor to numpy array and then use np.clip to clip the values and then reconverting them back to torch.tensor?</p>
<p>Or is there any method that would clip the elements of an array to a percentage value? For example, if I pass the value of 10, an element of array with value 100 will be clipped minimum of 90, maximum of  110.</p>

username: ptrblck
cooked: <p>I’m not aware of this functionality in PyTorch and <code>torch.clamp</code> as well as <code>torch.clip</code> (alias for <code>clamp</code>) expect single numbers as the min and max arguments.</p>
<p>CC <a class="mention" href="/u/mruberry">@mruberry</a> for numpy compatibility.</p>

username: mruberry
cooked: <p>Sorry, <a class="mention" href="/u/hrushi">@hrushi</a>, unfortunately I don’t think there’s a great workaround available today (at least not one better than you’re using). This is a popular request, and I’ve updated the Github issue tracking it (see <a href="https://github.com/pytorch/pytorch/issues/2793" class="inline-onebox" rel="noopener nofollow ugc">Support `clamp()` with tensor min and max · Issue #2793 · pytorch/pytorch · GitHub</a>) to be “high priority” so this functionality is provided quickly.</p>