akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why is the U-net not translationally invariant?

username: Sorench
cooked: <p>Hi,<br>
I have a general question that I was hoping for some pointers to information on.<br>
Briefly, I thought a U-net was invariant to translation, so I made a simple simulation placing squares at random locations, but only labelling the squares if in the left side of the image. I put 10 pixel squares in a 128 pixel image. Used 1 max pool step only, so lowest feature map matrix size is 64x64.<br>
I was expecting the Unet NOT to be able to fit this problem well, but turns out it did. Even with a single max-pool level it can successfully neglect squares on the right side of the image.<br>
How is this possible?  It’s conv + max pool + bilinear upsampling only. All are local operations and translation invariant.<br>
Inspection of the activation layers, seems to indicate that the encoding arm is translation invariant (feature maps simply move when input moves, intensity is constant). Output in the decoding arm vary in intensity depending on translation and seems to be responsible for the good fit. I just don’t understand how convolutions with 3x3 filters are able to produce said dependence on location.<br>
What am I missing?</p>
<p>Thanks!<br>
Soren</p>

username: Sorench
cooked: <p>Turns out I used align_corners=True in the upsampling step. This adds a location dependent phase shift to the upsampled image relative to decoder arm. Enough for the network to create filters to exploit that apparently…  align_corners=False leaves it invariant to translations. Nice fig here:</p><aside class="quote quote-modified" data-post="9" data-topic="22663">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/ebca7d/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-we-should-use-align-corners-false/22663/9">What we should use align_corners = False</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/6a242715685b8192f07c93a57a1d053b8add97bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/6a242715685b8192f07c93a57a1d053b8add97bf" title="align_corners.png">[align_corners]</a> 
Here is a simple illustration I made showing how a 4x4 image is upsampled to 8x8. 
When align_corners=True, pixels are regarded as a grid of points. Points at the corners are aligned. 
When align_corners=False, pixels are regarded as 1x1 areas. Area boundaries, rather than their centers, are aligned.
  </blockquote>
</aside>

<p>So I guess it may in general be a bad idea to use align_corners=True if you want translational equivariance!</p>
Interpolate the value from a discreted image with grid_sample function

username: GtxMiracle
cooked: <p>I want to interpolate the value from a discreted image. Speciafically, I have a grid, a mask and an input. Only the points whose mask is 1 of the input will contribute the interpolation. Is there any way to achieve this with grid_sample function?</p>

username: GtxMiracle
cooked: <p>Any one can help me? Thanks a lot!</p>

username: ptrblck
cooked: <p>How would the output values be calculated if all neighbors of the output pixel locations have a 0 in their mask?</p>

username: GtxMiracle
cooked: <p>Hi, for the interpolation, the nearest points whose mask is one will be taken into account. The point with mask 0 will be ignored.</p>

username: GtxMiracle
cooked: <p>Hi, for the interpolation, the nearest points whose mask is one will be taken into account. The point with mask 0 will be ignored. Is there any way to achieve this by grid sample?</p>

username: ptrblck
cooked: <p>I don’t understand the use case completely.<br>
An interpolation would use neighboring values to calculate the value at the new output location using a defined method, such as linear interpolation etc.<br>
E.g. for a simple linear interpolation of a 1D singal, the output location at coordinate <code>[0.2]</code>, would get the new value as <code>input[0] * 0.8 + input[1] * 0.2</code>. How would this value be calculated, if the mask has a zero value for this coordinate?</p>

username: GtxMiracle
cooked: <p>Thank you for the reply. For my case, suggest the mask is [1, 0, 1, 0], for the output location at [1.2], the output value will be input[0]<em>(1.8/2)+input[1]</em>(0.2/2). Which means, I will find the nearest points whose masks are not zero for the linear interpolation.</p>

username: ptrblck
cooked: <p>This would mean that <code>input[1]</code> would be multiplied by zero, as its the mask value there?<br>
In that case you could multiply the input tensor with the mask and perform a “standard” interpolation.</p>

username: GtxMiracle
cooked: <p>I’m afraid not. Take the input [2,10,3,4] as an example. The output in my case should be 2<em>0.9+3</em>0.1=2.1. But if I multiply 10 with zero and perform a “standard” interpolation, the output will be 2<em>0.8+0</em>0.2=1.6.</p>

username: GtxMiracle
cooked: <p>sorry the formulas are <code>2*0.9+3*0.1=2.1</code> and<code>2*0.8+0*0.2=1.6</code></p>

username: ptrblck
cooked: <p>Ah OK, so in the previous fomula <code>input[2]</code> would be used, no?<br>
I don’t know, how this would work out of the box and would need to run some code first.</p>

username: GtxMiracle
cooked: <p>No. The source code of the grid_sample will choose the nearest point of the target position as <code>(x,y)</code> and use <code>(x,y), (x+1, y), (x, y+1) and (x+1, y+1)</code> for the interpolation process.</p>
The difference between i, batch and step

username: ky_Pa
cooked: <p>What do i, batch, and step represent? what differences are there?</p>
<pre><code class="lang-auto">for i, batch in enumerate(dataloader):
        real_A = Variable(batch["A"].type(Tensor))
        real_B = Variable(batch["B"].type(Tensor))
</code></pre>
<pre><code class="lang-auto">for step, (x, y) in enumerate(data_loader):
            images = make_variable(x)
            labels = make_variable(y.squeeze_())</code></pre>

username: ptrblck
cooked: <p><code>i</code> and <code>step</code> represent the same variable returned by <code>enumerate</code> (the counter of the current loop).<br>
The return value of the <code>dataloader</code> seems to be a <code>dict</code> in the first example, so <code>batch</code> will contain the keys <code>'A'</code> and <code>'B'</code>, while <code>data_loader</code> will return a <code>tuple</code> which is unpacked to <code>x</code> and <code>y</code>.</p>

username: ky_Pa
cooked: <p>In Example 1, A and B respectively represent two image data sets. So does batch here refer to batches of images?Is batch and batchsize related here?<br>
In Example 2, ImageFolder is used. Do x and y refer to the images in the folder and the folder name under the path?</p>

username: ptrblck
cooked: <p>Your code snippets just show the usage of the return value of the <code>DataLoader</code>s.<br>
I assume that no custom collate functions etc. were used, but the “standard” approach of creating a <code>Dataset</code> and wrapping it into a <code>DataLoader</code>.</p>
<p><code>batch</code> is the name of the returned variable from the <code>DataLoader</code>. Based on the following lines of code, it seems that it’s a <code>dict</code> containing batches of <code>'A'</code> and <code>'B'</code>, which both seem to be tensors.</p>
<p>A batch contains <code>batch_size</code> number of samples.</p>
<p><code>ImageFolder</code> will return the loaded image and the corresponding target, not the folder name.<br>
The subfolders will be sorted and <code>ImageFolder</code> will assign a class index to each folder.</p>

username: ky_Pa
cooked: <p>Thank you very much for your answers!</p>
Finetuning embeddings - nn.Embedding vs. nn.Embedding.from_pretrained

username: Jeremy_Rutman
cooked: <p>I have been working with pretrained embeddings (Glove) and would like to allow these to be finetuned. I currently use embeddings like this:</p>
<pre><code>    word_embeddingsA = nn.Embedding(vocab_size, embedding_length)
    word_embeddingsA.weight = nn.Parameter(TEXT.vocab.vectors, requires_grad=False)
</code></pre>
<p>Should I simply set requires_grad=True to allow the embeddings to be trained? Or should I do something like this</p>
<pre><code>    word_embeddingsA = nn.Embedding.from_pretrained(TEXT.vocab.vectors, freeze=False)
</code></pre>
<p>Are these equivalent, and do I have a way to check that the embeddings are getting trained?</p>

username: ptrblck
cooked: <p>The approaches should yield the same result (if you use <code>requires_grad=True</code> in the first approach).<br>
To make sure this layer is trained, you could check the gradients after the <code>backward</code> call via:</p>
<pre><code class="lang-python">print(model.word_embeddings.weight.grad)
</code></pre>
<p>and you should see valid gradients.<br>
If you are seeing <code>None</code> as the return value, the computation graph might have been detached at some point.</p>

username: Jeremy_Rutman
cooked: <p>Ok thanks.  If I now want to infer using the trained weights can I still do</p>
<pre><code>test_sen1 = TEXT.preprocess(test_sen1)
test_sen1 = [[TEXT.vocab.stoi[x] for x in test_sen1]]
test_sen1 = np.asarray(test_sen1)
test_sen1 = torch.LongTensor(test_sen1)
test_tensor1 = Variable(test_sen1, volatile=True)
output = model(test_tensor1,1)
</code></pre>
<p>I suppose this may still be ok as the TEXT object is just supplying the indices?</p>

username: ptrblck
cooked: <p>I’m not familiar enough with <code>torchtext</code> unfortunately.<br>
Could you test this code snippet using some training examples and run a sanity check to see, if the predictions are expected?</p>
<p>PS: <code>Variables</code> are deprecated since PyTorch <code>0.4</code>, so you can use tensors now. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
To save memory, wrap the inference code in a <code>with torch.no_grad()</code> block.</p>
Checking embedding coverage

username: Jeremy_Rutman
cooked: <p>I was wanting to check how much of my corpus a particular embedding covered and looked at</p>
<p><code>TEXT.vocab.freqs['&lt;unk&gt;']</code></p>
<p>and this showed 0 , but it also shows 0 for any nonsense word I know is not covered, so I;m not sure there are 0 unknowns - I guess I can sum the frequencies and compare this to the number of words in my corpus?</p>
DataParallel Reshape Problems

username: WR01
cooked: <p>I have a model that I would like to parallelize so that inference would be much faster. The input vector that I have to this network is at one point 195xHxW. My network then reshapes it -1x3xHxW, which should normally work (65x3xHxW). But because DataParallel wraps the network it divides the 195xHxW tensor into n pieces, where n is the amount of gpus. However, when dividing the tensor it does it in such a way that the last tensor can no longer be reshaped. Is there a way to get DataParallel to work with the model so that the tensor can still be properly reshaped?</p>

username: rvarm1
cooked: <p>I’m assuming that since your tensor of shape 195<em>H</em>W is a single training example, you don’t want it to be split by N GPUs, since 195 is not the batch size in this case? If you pass in your training examples in the form such as batch_size * C * H * W for example then DataParallel/DDP should divide along the batch size dim.</p>
<p>Could you potentially paste a reproduction of this issue and the associated error message that you get?<br>
Btw, if you are using utilities provided by PyTorch such as the <a href="https://pytorch.org/docs/stable/data.html" rel="nofollow noopener">DataLoader</a>  you can configure the <code>drop_last</code> argument to ensure that batch sizes are even.</p>
PyTorch filename print to csv file not defined

username: alameer
cooked: <p>I am trying to print the file name into a csv file but getting an error message file is not defined:</p>
<pre><code class="lang-auto">        output = model(images) 
        preds = output.sum(dim=[1,2,3]) 
        print(preds)
        with open('predsLilas1.csv','a') as fd:
            fd.write( ','.join(map(str, preds.detach().tolist())) + '\n')
            fd.write( ','.join(map(str, targets.detach().tolist())) + '\n')
            fd.write( ','.join(map(str, filename.detach().tolist())) + '\n')</code></pre>

username: ptrblck
cooked: <p>Based on the error message I guess <code>predsLilas1.csv</code> is not created, while you are trying to append to the file. You could try to pass <code>a+</code> as the mode, which should create the file, if it doesn’t exist or make sure that the file is in the right folder via <code>os.path.exists('predsLilas1.csv')</code>.</p>
Multi worker data loader in multi processing context

username: rui_zhang_331
cooked: <p>Hi, I am using pytorch multi processing to speed up some data processing in CPU side(in order to feed to GPU faster ), <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-best-practices" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-best-practices</a></p>
<p>In each processing func, I also use multi-worker data loader to speed up the data loading processing time.<a href="https://pytorch.org/docs/stable/data.html" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html</a></p>
<p>suppose I have <code>N</code> processing, each has <code>M</code> dataloader worker, so total <code>NxM</code> underneath threading there.</p>
<p>If in my dataloader, I want to get all data in a sequential way, which means <code>__get_item__(self, idx)</code> in data loader will will have a index as para and different processing and dataloader worker processing different index, how can I ensure they do not process duplicate or miss process some?</p>
__getitem__ in custom dataset is called twice

username: antran96
cooked: <p>I am building a custom Dataset where I read the audio from a path (given in a pandas dataframe). When I tried to print the path in <code>__getitem__</code>, it prints the same path twice, which means it processes the same thing twice. Not sure if everything works properly.</p>
<p>Here is my custom Dataset:</p>
<pre><code class="lang-auto">class BirdDatasetTrain:
    def __init__(self, folds,freq_mask=False, crop = 512):
        df = pd.read_csv("./input/train_folds.csv")

        df = df[["filename","ebird_code", "ebird_lbl", "duration", "kfold"]]
        df = df[df.kfold.isin(folds)].reset_index(drop=True)
        print(df.head())
        self.filenames = df.filename.values
        self.ebird_lbls = df.ebird_lbl.values
        self.ebird_codes = df.ebird_code.values
        self.freq_mask = freq_mask
        self.crop = crop 
    
    def __len__(self):
        return (len(self.filenames))

    
    def __getitem__ (self,item):
        fp = BASE_DIR + self.ebird_codes[item] + "/" +self.filenames[item]
        print(fp)  # This is printed twice with same value
        mel_spec = build_spectogram(fp)
        print(mel_spec) # This is printed twice with same value
        mel_spec = do_random_crop(mel_spec, self.crop)

        mel_spec = (mel_spec - mel_spec.mean()) / (mel_spec.std()+1e-7)
        
        if self.freq_mask:
            mel_spec = freq_mask(mel_spec)
        
        mel_spec = mel_spec.reshape([1,mel_spec.shape[0],mel_spec.shape[1]])
        return {
            "audio": torch.tensor(mel_spec,dtype=torch.float),
            "ebird_lbl":torch.tensor(self.ebird_lbls[item], dtype=torch.long)
        }
</code></pre>

username: BTNug
cooked: <p>Hi <a class="mention" href="/u/antran96">@antran96</a>, could you double check:</p>
<ul>
<li>
<strong>BirdDatasetTrain.<strong>len</strong>()</strong> output is &gt; 1?</li>
<li>Please make sure your filenames, ebird_lbls, and ebird_codes outputs*<br>
-. Do you use batch size 2?</li>
<li>In the image generation, you could also double check whether your I/O is correct by printing inside your code. Because you doubt that you are using the same dataset, you could try to print “item” value in your <strong><strong>getitem</strong></strong> function.</li>
</ul>
<p>*you could try to instantiate this class, then check all of those parameters first</p>
Tensorflow to PyTorch Conversion

username: isaacg
cooked: <p>I have been converting a large amount of tensorlfow 2.0 code to PyTorch and I must say, the pytorch documentation and tutorials have been very helpful.  This transition has been going very smooth and I am writing to say that I appreciate what you are doing to help your user base; documentation seems very clear and consistent.  Thank you and looking forward to more pytorch in the future!</p>

username: ptrblck
cooked: <p>Thanks for the kind words!<br>
Let us know, if you get stuck somewhere. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Torch Mobile android app is huge (90mb apk, but 30mb download size)

username: aditya-hari
cooked: <p>I’ve used a torchvision for a mobile image classification app but it is absolutely massive for what it does.<br>
The apk itself is 90mb, but the download size is 30mb. What’s causing this massive disparity in size? Is there any way for me to reduce this?</p>

username: IvanKobzarev
cooked: <p>For pytorch-android 1.5.0 size of the aar is 25mb,<br>
<a href="https://bintray.com/pytorch/maven/org.pytorch%3Apytorch_android/1.5.0#files/org%2Fpytorch%2Fpytorch_android%2F1.5.0" class="onebox" target="_blank" rel="nofollow noopener">https://bintray.com/pytorch/maven/org.pytorch%3Apytorch_android/1.5.0#files/org%2Fpytorch%2Fpytorch_android%2F1.5.0</a></p>
<p>Nightly builds are a bit bigger.</p>
<p>What is the size of your vision model that you pack in application?</p>
<p>(As an idea to have smaller model you may try to use quantized model or models with smaller size like mobileNetV2 or download model on the device at runtime)</p>
NAS training dynamic parameters update?

username: Scott_Hoang
cooked: <p>I am training a NAS network containing a controller and a model. For each training epoch, the controller dynamically configures the model network to optimize model’s reward. Since this is NAS training, the search space is relatively large, and the model itself is very large. However, 99% of the time, many parameters will not be activated. My question is, is it possible for me to change my optimizers (an AdamOpt) on the fly to only update parameters that were activated at the epoch? My hope is that it decreases overall training time.</p>
torch.utils.cpp_extension.BuildExtension

username: ilhem
cooked: <p>hi , i like using the method back-translation (data augmentation)  but appears me an error and i don’t know how solves . this is my code :</p>
<blockquote>
<p>import torch<br>
en2de = torch.hub.load(‘pytorch/fairseq’, ‘transformer.wmt19.en-de.single_model’, tokenizer=‘moses’, bpe=‘fastbpe’)<br>
de2en = torch.hub.load(‘pytorch/fairseq’, ‘transformer.wmt19.de-en.single_model’, tokenizer=‘moses’, bpe=‘fastbpe’)<br>
and this is error:</p>
</blockquote>
<blockquote>
<p>DistutilsClassError: command class &lt;class ‘torch.utils.cpp_extension.BuildExtension’&gt; must subclass Command</p>
</blockquote>
Why my "manual" GRU is slower than nn.GRU?

username: David_Refaeli
cooked: <p>I’m trying to build a GRU network using simpler blocks of nn. While using nn.GRU works really fast, my version is painfully slow, even though it also should run on GPU. I’m trying to understand why that is.</p>
<p>Here’s the nn.GRU version:</p>
<pre><code class="lang-auto">def __init__(self, in_dim, h_dim, out_dim, n_layers, dropout=0):
        self.rnn = nn.GRU(in_dim, h_dim, n_layers)
        self.decoder = nn.Linear(h_dim, out_dim)
</code></pre>
<pre><code class="lang-auto">def forward_(self, input: Tensor, hidden_state: Tensor=None):
        output, hidden = self.rnn(input, hidden_state)
        layer_output = self.decoder(output)
        return layer_output, hidden
</code></pre>
<p>And here’s my “manual” version:</p>
<pre><code class="lang-auto">def __init__(self, in_dim, h_dim, out_dim, n_layers, dropout=0):
        for i in range(n_layers):
            # params = dict()
            if i != 0:
                in_dim = h_dim
            params[f'L{i}W1'] = nn.Linear(in_dim, h_dim)
            params[f'L{i}W2'] = nn.Linear(h_dim, h_dim, bias=False)
            params[f'L{i}W3'] = nn.Linear(in_dim, h_dim)
            params[f'L{i}W4'] = nn.Linear(h_dim, h_dim, bias=False)
            params[f'L{i}W5'] = nn.Linear(in_dim, h_dim)
            params[f'L{i}W6'] = nn.Linear(h_dim, h_dim, bias=False)
        params['W7'] = nn.Linear(h_dim, out_dim)
        for param in params:
            self.add_module(param, params[param])
        self.params = params
</code></pre>
<pre><code class="lang-auto">def forward(self, input: Tensor, hidden_state: Tensor=None):
    if hidden_state is None:
            layer_states.append(torch.zeros(batch_size, self.h_dim, device=input.device))

    for i in range(self.n_layers):
        layer_input = layer_middle
        layer_middle = torch.zeros(batch_size, seq_len, self.h_dim, device=input.device)
        params = self.params
        for j in range(seq_len):
            x = layer_input[:, j, :]
            z = F.sigmoid(params[f'L{i}W1'](x) + params[f'L{i}W2'](layer_states[i]))
            r = F.sigmoid(params[f'L{i}W3'](x) + params[f'L{i}W4'](layer_states[i]))
            g = F.tanh(params[f'L{i}W5'](x) + r*params[f'L{i}W6'](layer_states[i]))
            layer_states[i] = z*layer_states[i] + (1-z)*g
            layer_middle[:, j, :] = layer_states[i]
        layer_middle = nn.Dropout(self.dropout)(layer_middle)
    layer_output = torch.zeros((batch_size, seq_len, self.out_dim), device=input.device)
    for j in range(seq_len):
        x = layer_middle[:, j, :]
        layer_output[:, j, :] = params['W7'](x)
</code></pre>
[JIT] [Mobile] Op may not exist in TorchScript

username: zetyquickly
cooked: <p>Hello everyone!</p>
<p>Once again I’ve run into something I don’t understand, need your help.</p>
<p>I’ve compiled <code>speed_benchmark_torch</code> for mobile to test my TorchScript RCNN model.</p>
<p>The model is from detectron2. I’ve slightly varied caffe2 conversion of RCNN to use <code>torch.jit.trace</code> instead <code>torch.onnx.export</code> <a href="https://github.com/facebookresearch/detectron2/blob/master/tools/deploy/caffe2_converter.py" rel="nofollow noopener">https://github.com/facebookresearch/detectron2/blob/master/tools/deploy/caffe2_converter.py</a><br>
That gives a valid model to run on PC but on Mobile there is an issue.</p>
<pre><code class="lang-auto">Unknown builtin op: _caffe2::GenerateProposals.
Could not find any similar ops to _caffe2::GenerateProposals. This op may not exist or may not be currently supported in TorchScript.
...
Serialized   File "code/__torch__/detectron2/export/c10.py", line 22
    scores = torch.detach(_6)
    bbox_deltas = torch.detach(_7)
    _16, _17 = ops._caffe2.GenerateProposals(scores, bbox_deltas, im_info, _4, 0.25, 1000, 100, 0.69999999999999996, 0., True, -180, 180, 1., False, None)
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    scores0 = torch.detach(_8)
    bbox_deltas0 = torch.detach(_9)
</code></pre>
<p>The question is possible to un TorchScript model with caffe2 ops or we should use old <code>speed_benchmark</code> for caffe2 models?<br>
Why there are caffe2 related options in <code>./speed_benchmark_torch --help</code>?</p>

username: Kyle_Armstrong
cooked: <p>Hi <a class="mention" href="/u/zetyquickly">@zetyquickly</a>, I was curios if you every found a solution to your problem. I am currently running into basically the same error trying to deploy a model to ios.  Any tips would be greatly appreciated thanks!</p>

username: zetyquickly
cooked: <p>Hello, <a class="mention" href="/u/kyle_armstrong">@Kyle_Armstrong</a></p>
<p>I’m afraid that you won’t be able to use PyTorch backend to run Caffe2 operations on mobile. If you are interested in deploy of Caffe2 models you should build a Caffe2 .pb model files and then run it with Caffe2 backend on mobile, e.g. using <code>speed_benchmark</code> (not <code>speed_benchmark_torch</code>)<br>
To run models using only PyTorch backend your TorchScript model must contain only PyTorch operators and only supported on with mobile PyTorch (e.g. you’re unable to run <code>torchvision::nms</code> op on mobile, at least that was the case recently)</p>
Pytorch 3d augmentation

username: bioinfo-dirty-jobs
cooked: <p>Dear  all, I would like to do some augmentation on my 3d cube from mri.<br>
I want to use TORCHVISION.TRANSFORMS but I start using numpy array.<br>
I try to use</p>
<pre><code class="lang-auto">transformations = transform_train = transforms.Compose([
                                    
                                      transforms.ToPILImage(),
                                      transforms.RandomHorizontalFlip(),
                                      transforms.ToTensor(),
                                      
                               ])
</code></pre>
<p>Any suggestion? thanks so much for any help</p>

username: ptrblck
cooked: <p><code>torchvision.transforms.ToPILImage</code> might not work for MRI data directly, but <a class="mention" href="/u/christianperone">@christianperone</a> has written some transformation for medical data in the <a href="https://github.com/perone/medicaltorch">medical torch repo</a>, which might be useful.</p>

username: fepegar
cooked: <p>Hey <a class="mention" href="/u/bioinfo-dirty-jobs">@bioinfo-dirty-jobs</a>,</p>
<p>You can use the transforms in <a href="https://torchio.readthedocs.io/transforms/augmentation.html#randomflip" rel="nofollow noopener">TorchIO</a>:</p>
<pre><code class="lang-python">import torchio.transforms as transforms

transformations = transform_train = transforms.Compose([
    transforms.ToCanonical(),
    transforms.RandomFlip(),
])</code></pre>
Data Augmentor for 3D images

username: bluesky314
cooked: <p>I have MRI images of brain scans of shape [bs, modalities, h,w,d] and would like to know if there is a library for 3D image augmentations. Also which augmentations would be good for this kind of data?</p>

username: ptrblck
cooked: <p>Have a look at <a class="mention" href="/u/christianperone">@christianperone</a>’s <a href="https://github.com/perone/medicaltorch">…MedicalTorch</a> repo.<br>
It might be a good starter for your project.</p>
<p>Regarding the data augmentations, you could try to apply the augmentation on each slide of your scans.<br>
In this case I would use the functional API of <code>torchvision.transforms</code> to make sure that each “random” transformation is applied in with the same parameters on each slide.<br>
<a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7?u=ptrblck">Here</a> is a small example on using the same transform parameters on the data and target. You would need to adapt this code for all image slides.</p>
<p>Alternatively you could write your own (affine) transformations. <a href="https://discuss.pytorch.org/t/how-to-rotate-90-and-270-degrees-of-5d-tensor/22476/13?u=ptrblck">Here</a> is a small example I’ve written for a rotation using rotation matrices.</p>

username: bluesky314
cooked: <p>Thanks, I will look into these</p>

username: fepegar
cooked: <p>Hey <a class="mention" href="/u/bluesky314">@bluesky314</a>,</p>
<p>This might come a bit late, but you could take a look at the <a href="https://torchio.readthedocs.io/transforms/transforms.html" rel="nofollow noopener">TorchIO transforms</a>.</p>
Multiple workers dataloader with big input files

username: TheoEst
cooked: <p>Hello evereybody,</p>
<p>I am working on a autoencoder for 3D medical imaging. My data input are big (around 150<em>150</em>150 pixels). I have 517 different images (around 150 Go) in a hard drive. I have 4 Tesla K 40 GPU. I am working inside a docker and I put the options --ipc=host. I design my own DataSet classes and I use the DataLoader of Pytorch.</p>
<p>I am trying to improve the loading of the data because currently it is taking 80% of the calculation time.</p>
<p>When the numbers of workers is equal to 0, the loading time of my data is between 1 and 2 seconds (I tested with numpy.load or torch.load, it doesn’t make any differences).</p>
<p>But when I put num workers &gt; 0, the loading time of the data is between 20 and 40 seconds. My hypothesis is that because of the size the data, pickling the array between the workers slow down the loading function.</p>
<p>Is there any way to load huge data with multiple workers and not having such a slown down ?</p>
<p>Thanks,</p>

username: fepegar
cooked: <p>Hi <a class="mention" href="/u/theoest">@TheoEst</a>, you might want to take a look at the <a href="https://torchio.readthedocs.io/data/dataset.html" rel="nofollow noopener"><code>ImagesDataset</code> in TorchIO</a>.</p>
When will unfold and fold support 5D tensors?

username: liangbright
cooked: <p>Currently, only 4-D input tensors (batched image-like tensors) are supported by unfold and fold</p>
<p>I assume the size is (N, C, H, W)</p>
<p>a set of 3D medical images are 5D tensors:  (N, C, D, H, W)</p>
<p>When will unfold and fold support 5D tensors?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think there is any plan to support generalised versions of fold and unfold.<br>
You have Conv3D if you need to do convolutions though.</p>
<p><a class="mention" href="/u/fmassa">@fmassa</a> migh be more familiar about how easy/hard it would be to generalise fold and unfold?</p>

username: liangbright
cooked: <p>It is about custom convolution operation like this one</p><aside class="quote" data-post="1" data-topic="14992">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lucamocerino/48/3908_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/custom-convolution-dot-product/14992">Custom Convolution Dot Product</a> 
  </div>
  <blockquote>
    Hi, 
my question is quite trivial. Is it possible to change the conv2d Functional ? 
I need a very deep but small change in the dot product in convolution is it possible ? 
Thanks every one.
  </blockquote>
</aside>


username: John1231983
cooked: <p><a class="mention" href="/u/liangbright">@liangbright</a>: What does it mean fold and unfold? Pytorch supports almost 3D operators likes conv, pool, upsampling…</p>

username: fepegar
cooked: <p><a class="mention" href="/u/liangbright">@liangbright</a>, if you want to use a 3D sliding window you can use the grid sampler in <a href="https://torchio.readthedocs.io/data/patch_based.html" rel="nofollow noopener">TorchIO</a>, as I explained in this topic: <a href="https://discuss.pytorch.org/t/how-to-extract-2d-patches-from-3d-medical-image/76646/2" class="inline-onebox">How to extract 2D patches from 3D medical image</a>.</p>
Medical images mean and std

username: David_Jitca
cooked: <p>I have a dataset with medical images int16. I have tried to calculate mean and std to use Normalize.<br>
Like this<br>
mean_tr = (train_x.float().mean()/10209)<br>
std_tr = (train_x.float().std()/10209)<br>
mean_te = (test_x.float().mean()/11108)<br>
std_te = (test_x.float().std()/11108)<br>
Where 10209 is the value of the biggest pixel in train, and 11108 in test.<br>
I have something like this :<br>
mean_tr = tensor(0.0439) std_tr = tensor(0.0616)<br>
mean_te = tensor(0.0425) std_te = tensor(0.0586)<br>
I then used this values to Normalize data, with tranform, and I print some values to see what happened. I saw values like 60000, after this step, very big values, and smalest was -0.7. Why? Can you help me with some ideas how to deal with this images?<br>
Thanks</p>

username: Intel_Novel
cooked: <p>Do you use 3D medical images?</p>

username: MariosOreo
cooked: <p>Hi,</p>
<p>Did you perform  the transformation <code>transform.functional.to_tensor()</code> firstly? Because it will scale your data in range[0, 1] and then perform normalization on it, you could have a try.</p>

username: David_Jitca
cooked: <p>This images are 2D slices from 3D medical images, from what I understand T1.</p>

username: David_Jitca
cooked: <p>In documentation, it says that will converts PIL Images, or numpy array that are in range [0, 255].</p>

username: David_Jitca
cooked: <p>No one have an answer?</p>

username: fepegar
cooked: <p>Hi <a class="mention" href="/u/david_jitca">@David_Jitca</a>,</p>
<p>Maybe you can use the <a href="https://torchio.readthedocs.io/transforms/preprocessing.html#preprocessing" rel="nofollow noopener">preprocessing transforms in TorchIO</a>. Also, sharing a <a href="https://en.wikipedia.org/wiki/Minimal_working_example" rel="nofollow noopener">minimal working example</a> would help us help you.</p>
How to train 3D Volume Medical image ? using Deep Learning PyTorch?

username: selvakarna
cooked: <p><strong>How to train 3D Volume image ?</strong> using DL -PyTorch? for 3D Objects recognition?</p>
<p>DL -Deep Learning</p>

username: fepegar
cooked: <p>Hi <a class="mention" href="/u/selvakarna">@selvakarna</a>,</p>
<p>You can try using <a href="https://torchio.readthedocs.io/index.html" rel="nofollow noopener">TorchIO</a>, which was specifically designed for 3D medical images.</p>
Free parameter self convolutional block attention module

username: developer0hye
cooked: <p>I designed a free parameter self-attention convolutional block module using pytorch.</p>
<p>My work is inspired from <a href="https://arxiv.org/pdf/1807.06521.pdf" rel="nofollow noopener">CBAM</a>.</p>
<p>I tested my work on CIFAR-100 dataset and residual18 network.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Model</th>
<th>Param.</th>
<th>Acc1.</th>
<th>Acc2.</th>
<th>Acc3.</th>
<th>Acc4.</th>
<th>Acc5.</th>
<th>Best Acc.</th>
<th>Avg Acc.</th>
</tr>
</thead>
<tbody>
<tr>
<td>resnet18</td>
<td>11.22M</td>
<td>76.36%</td>
<td>75.94%</td>
<td>76.38%</td>
<td>76.03%</td>
<td>76.37%</td>
<td>76.38%</td>
<td>76.22%</td>
</tr>
<tr>
<td>with CBAM</td>
<td>11.39M</td>
<td>76.20%</td>
<td>76.55%</td>
<td>76.23%</td>
<td>76.26%</td>
<td>76.16%</td>
<td>76.55%</td>
<td>76.28%</td>
</tr>
<tr>
<td>with ZCBAM(Max)</td>
<td>11.22M</td>
<td>75.64%</td>
<td>75.97%</td>
<td>76.20%</td>
<td>75.99%</td>
<td>75.87%</td>
<td>76.20%</td>
<td>75.93%</td>
</tr>
<tr>
<td>with ZCBAM(Avg)</td>
<td>11.22M</td>
<td>76.89%</td>
<td>76.77%</td>
<td>76.51%</td>
<td>76.45%</td>
<td>76.68%</td>
<td>76.89%</td>
<td><strong>76.66%</strong></td>
</tr>
<tr>
<td>with ZCBAM(Avg&amp;Max)</td>
<td><strong>11.22M</strong></td>
<td>76.46%</td>
<td>76.95%</td>
<td>76.62%</td>
<td>76.34%</td>
<td>76.12%</td>
<td><strong>76.95%</strong></td>
<td>76.50%</td>
</tr>
</tbody>
</table>
</div><p>If you are interested in my work, please refer to<br>
<a href="https://github.com/developer0hye/ZCBAM" rel="nofollow noopener">ZCBAM</a>.</p>

username: ptrblck
cooked: <p>Interesting work!<br>
Thanks for sharing! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: developer0hye
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thanks!</p>
<p>Would you mind if I upload my works like this post here later?</p>
<p>I wonder if my post is not proper to be posted in this community.</p>

username: ptrblck
cooked: <p>I think it would be great to see your work here and discuss it.<br>
I’m not familiar with your current approach, but we have a lot of experienced user in our community, who could discuss your work here. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Manually calculated gradient not matching autograd

username: Kai_Eiji
cooked: <p>Hi</p>
<p>I’m trying to calculate a gradient of a function that is  <code>A*A*X*W0*W1 w.r.t A</code>  in PyTorch. I have calculated the gradient manually but I’m not getting the same answer as the pytorch’s auto grad function. Can anyone help me with what I did wrong?</p>
<p>The manually calculated function is</p>
<pre><code class="lang-auto">def torch_grad(A, W0, W1, X):
  # Forward
  W0W1 = torch.mm(W0, W1)
  AX = torch.mm(A, X)
  AXW0W1 = torch.mm(AX, W0W1)
  AAXW0W1 = torch.mm(A, AXW0W1)

  # Backward / Gradients
  rows, cols = AAXW0W1.size()
  grad_AX = torch.mm(torch.eye(rows, cols), W0W1.t())
  grad_AA = torch.mm(grad_AX, X.t())
  grad_A = torch.mm(grad_AA, (2*A).t())
  return grad_A
</code></pre>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>The problem most likely lies with the <code>torch.eye(rows, cols)</code> that you use.<br>
Keep in mind that the backward pass computes a vector jacobian product. So you need to make sure that the vector you backprop matches this identity matrix here.</p>

username: Kai_Eiji
cooked: <p>I’m getting the row and col for the identity matrix by <code>rows, cols = AAXW0W1.size()</code> so I thought it’s matching the shape of vector I’m backpropagating. Am I wrong?</p>
<p>Also I was comparing the result with this function</p>
<pre><code class="lang-auto">def torch_autograd(A, W0, W1, X):
    # Forward
    W0W1 = torch.mm(W0, W1)
    AX = torch.mm(A, X)
    AXW0W1 = torch.mm(AX, W0W1)
    AAXW0W1 = torch.mm(A, AXW0W1)
    XW0W1 = torch.mm(X, W0W1)
    
    # Backward / Gradients
    rows, cols = AAXW0W1.size()
    AAXW0W1.backward(torch.eye(rows, cols))
    return A.grad
</code></pre>

username: albanD
cooked: <p>I don’t think these are doing the same computation.<br>
You can see this by changing the shape of you matrices so that they’re not all the same (but still match the mm) and you’ll see that your function does not return something on the shape of A.</p>
How can I implement shape on Custom Dataset?

username: bfeeny
cooked: <p>I am trying to get “shape” to work on my Custom Dataset:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, data, window):
        self.data = data
        self.window = window

    def __getitem__(self, index):
        x = self.data[index:index+self.window]
        return x

    def __len__(self):
        return len(self.data) -  self.window + 1
</code></pre>
<pre><code class="lang-auto"># my_dataset=MyDataset(train_data_normalized, 14)
my_array = np.arange(99)
my_array.shape
</code></pre>
<p>(99,)</p>
<pre><code class="lang-auto">my_dataset=MyDataset(my_array,10)
my_dataset.shape
</code></pre>
<pre><code class="lang-auto">---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-42-72becebb3571&gt; in &lt;module&gt;
      1 my_dataset=MyDataset(my_array,10)
----&gt; 2 my_dataset.shape

AttributeError: 'MyDataset' object has no attribute 'shape'

</code></pre>

username: Nikronic
cooked: <p>Hi,</p>
<p><code>shape</code> actually is an attribute for tensors or numpy arrays, if you are interested in adding any attribute to your classes, just define <code>self.shape</code>. Here is an example:</p>
<pre><code class="lang-auto">
class MyDataset(data.Dataset):
    def __init__(self, data, window):
        self.data = data
        self.window = window
        self.shape = self.__getshape__()

    def __getitem__(self, index):
        x = self.data[index:index+self.window]
        return x

    def __len__(self):
        return len(self.data) -  self.window + 1

    def __getshape__(self):
        return (self.__len__(), *self.__getitem__(0)[0].shape)

x_train = np.random.randn(6000, 1, 32, 32)

my_data = MyDataset(x_train, 1)
</code></pre>
<p>Bests</p>
Why torch.tensor(source_tensor) is not preferred?

username: I_H_Yoo
cooked: <p>As the title says, why is <code>torch.tensor(source_tensor)</code> not preferred and why <code>tensor.clone().detach()</code> is more preferred when a tensor is copied?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Mostly for clarity. <code>tensor.clone().detach()</code> makes it very clear what happens to the Tensor: You first allocate new memory for it then detach it from the autograd graph from the original one.</p>
<p><code>torch.tensor(source_tensor)</code> does the same thing but you can easily forget it and have hard-to-debug issues when using it.</p>

username: I_H_Yoo
cooked: <p>Thank you very much.</p>
How to use torch.flip to flip an image?

username: Scott_Hoang
cooked: <p>let input be of the shape of (BxCxHxW), and I want to flip along the Width.<br>
in numpy I would have my input = input[:, :, :, ::-1]. How can I recreate the same results using torch.flip? I can’t seem to be able to get it right.</p>

username: Scott_Hoang
cooked: <p>I am stupid…<br>
it is just torch.flip(input, (3,))<br>
doc said it should be a tuple of ints so I was confuse…</p>
AttributeError: 'BucketIterator' object has no attribute 'text'

username: mayurat22
cooked: <p>I created an iterator using Bucketiterator on train_data, which has two columns ‘text’ and its label, I also developed the vocab using LABEL.build_vocab(train_data,).</p>
<p>The train_data is a text which is convert into ids</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65fa58f32b98c07adb3989360649feef8b90543c.png" data-download-href="https://discuss.pytorch.org/uploads/default/65fa58f32b98c07adb3989360649feef8b90543c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65fa58f32b98c07adb3989360649feef8b90543c_2_690x399.png" alt="image" data-base62-sha1="ey8AFDekuHvwO94tdUVdZOeNsVC" width="690" height="399" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65fa58f32b98c07adb3989360649feef8b90543c_2_690x399.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65fa58f32b98c07adb3989360649feef8b90543c_2_1035x598.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/5/65fa58f32b98c07adb3989360649feef8b90543c.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1105×639 40.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Creating iterator</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/a/cac423226451b79d95eedb9c929b2729e2b8deec.png" alt="image" data-base62-sha1="sVKFBQIHljIzGwvAP7t52prh3MU" width="597" height="125"></p>
<p>Error:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cab3596a80539b7a59fe5213767ea6e3efdf257.png" data-download-href="https://discuss.pytorch.org/uploads/default/5cab3596a80539b7a59fe5213767ea6e3efdf257" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cab3596a80539b7a59fe5213767ea6e3efdf257_2_690x220.png" alt="image" data-base62-sha1="ddMJYD6I1hNfB5pXr5edwcwgIh9" width="690" height="220" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cab3596a80539b7a59fe5213767ea6e3efdf257_2_690x220.png, https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cab3596a80539b7a59fe5213767ea6e3efdf257.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cab3596a80539b7a59fe5213767ea6e3efdf257.png 2x" data-dominant-color="F7F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">966×308 32.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
PyTorch Nighly concrete version in environmen.yml

username: WaterKnight
cooked: <p>Is there any way of saving a concrete PyTorch nightly version on a environment.yml of Anaconda:</p>
<p>I tried as follows:</p>
<pre><code class="lang-auto">name: Test
channels:
  - pytorch-nightly
  - defaults
  - conda-forge
dependencies:
  - python=3.7
  - pip  
  - ipywidgets
  - jupyterlab
  - cudatoolkit=10.2
  - pytorch=1.6.0.dev20200610=py3.7_cuda10.2.89_cudnn7.6.5_0
  - torchvision=0.7.0.dev20200610=py37_cu102
</code></pre>
<p>But it don’t works</p>

username: ptrblck
cooked: <p>I haven’t tried it, but the current nightly binary should be shipped in the <code>pytorch</code> channel, not <code>pytorch-nightly</code>. Could you change it and check it again?</p>
<p>EDIT: I’m wrong, the package name was changed (I think), not the channel name.</p>

username: WaterKnight
cooked: <p>pytorch-nightly from pytorch channel hasn’t been updated for a year.</p>
<p>Looking into versions of pytorch nighly channel I have seen that after several versions they get removed</p>
<p><a href="https://anaconda.org/pytorch-nightly/pytorch/files" class="onebox" target="_blank" rel="nofollow noopener">https://anaconda.org/pytorch-nightly/pytorch/files</a></p>
Custom Dataset is changing shape of data

username: bfeeny
cooked: <p>I am trying to implement a custom Dataset for sliding window over my data to be fed into an LSTM.</p>
<p>As you can see, my data is 1913,30490…I wish to grab 14 at a time (batch_size=14)…which should give me an object that is 14,30490</p>
<pre><code class="lang-auto">train_data_normalized.shape
(1913, 30490)
</code></pre>
<p>My custom Dataset below:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, data, window):
        self.data = data
        self.window = window

    def __getitem__(self, index):
        x = self.data[index:index+self.window]
        return x

    def __len__(self):
        return len(self.data) - self.window
</code></pre>
<p><code>my_dataset=MyDataset(train_data_normalized, 1)</code><br>
<code>my_loader = torch.utils.data.DataLoader(my_dataset, batch_size=14, shuffle = False)</code></p>
<pre><code class="lang-auto">d = next(iter(my_loader))
print(d.shape)
torch.Size([14, 1, 30490])
</code></pre>
<p>I am trying to understand why it is producing 14, 1, 30490 instead of 14, 30490</p>
<p>The end result I want to be able to get sequences like 1-14, then 2-15, 3-16, etc so they are of size 14 and advance forward one index at a time.</p>

username: bfeeny
cooked: <p>I see what I am doing wrong. I needed to request a window of 14 and a batch size in may loader of 1.  It returns a 3d object, but actually that is what I want to feed the LSTM.</p>
<p>Another issue I have though is how can I make sure that it doesn’t return incomplete windows?  Say for example I have a dataset that is 99x10. If I use a window size of 10, and a batch size on my loader of 20 it would return:<br>
Batch 0.   20x10<br>
Batch 1.   20x10<br>
Batch 2.   20x10<br>
Batch 3.   20x10<br>
Batch 4.    9x10</p>

username: bfeeny
cooked: <p>Looks like I just needed to add drop_last=True to my loader.</p>