akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
This is not overfitting but something else, right?

username: Carsten_Ditzel
cooked: <p>I trained a VAE on a dataset containing 1k images. The VAE itself is convolutional, downsamples 256x256 rgb images four times before reconstruction and uses both relu and BatchNorm Layers as well as ResNet-like Skip Connection prior and after the bottleneck. Input is normalized to unit scale and no data augmentation takes place. The modal consists in total roughly of about 5e6 parameters and the visual inspection of the reconstructed images look decent enough. This is represented for both training and validation runs by the orange curve. No overfitting occurs as far as I can tell. However, if I use the entire dataset, i.e. ~ 35k training images and 5k validation images, the same model performs like shown by the green curve. Taking into account only the lower validation plot, I had said, this is overfitting, although it would have struck me as odd, that the same model that performed decently on the 1k images dataset now overfits on a much larger dataset. But also the training loss shows a clear inflection point around epoch 15.</p>
<p>Can somebody tell me a likely source of error here? Is the models capacity not large enough to sufficiently capture the complexity of the data? I dont think so, since I am using a <a href="https://github.com/openai/DALL-E/blob/master/dall_e/encoder.py" rel="noopener nofollow ugc">sota convolutional VAE used by OpenAI for a dataset comprising millions of images</a></p>
<p>Any help would be appreciated. Thank you in advance<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14785eea2d85ce4e697d7cfde7c9d6d72d7b4600.png" data-download-href="https://discuss.pytorch.org/uploads/default/14785eea2d85ce4e697d7cfde7c9d6d72d7b4600" title="curves"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14785eea2d85ce4e697d7cfde7c9d6d72d7b4600_2_436x499.png" alt="curves" data-base62-sha1="2V5rrfU6X62Ut1CDi1IhXFT4EsE" width="436" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14785eea2d85ce4e697d7cfde7c9d6d72d7b4600_2_436x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14785eea2d85ce4e697d7cfde7c9d6d72d7b4600_2_654x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/4/14785eea2d85ce4e697d7cfde7c9d6d72d7b4600.png 2x" data-dominant-color="FDFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">curves</span><span class="informations">846×968 31.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Problems with PyTorch Installation

username: anonymous_mar
cooked: <p>Hello there! I’ve been trying to install PyTorch, but so far all I got is error messages. I tried the command line</p>
<p>“pip install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio===0.8.1 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a>”,</p>
<p>that was given on the tutorial, and I get the error:</p>
<p>“ERROR: Could not find a version that satisfies the requirement torch==1.8.1+cpu<br>
ERROR: No matching distribution found for torch==1.8.1+cpu”</p>
<p>Later I tried to simplify the command to just “pip install torch”, which gives me:</p>
<h2>ERROR: Command errored out with exit status 1:<br>
…<br>
ModuleNotFoundError: No module named ‘tools.nnwrap’</h2>
<p>ERROR: Failed building wheel for torch<br>
Running setup.py clean for torch<br>
ERROR: Command errored out with exit status 1:<br>
…<br>
ERROR: Failed cleaning build dir for torch</p>
<p>I’d be glad if you could help me, thanks!</p>
Correct way to calculate the final test accuracy

username: randr100
cooked: <p>I found a <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#convnet-as-fixed-feature-extractor" rel="noopener nofollow ugc">resnet example</a> where the best eval accuracy is given. I want to know if that is the final test/eval accuracy?<br>
It is in the function below</p>
<pre><code class="lang-auto">def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    start_time = time.time()
    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0
for epoch in range(num_epochs):
print('Epoch {}/{}'.format(epoch, num_epochs - 1))
print('-' * 10)
# Each epoch has a training and validation phase
for phase in ['train', 'val']:
if phase == 'train':
                model.train() # Set model to training mode
else:
                model.eval() # Set model to evaluate mode
            running_loss = 0.0
            running_corrects = 0
# Iterate over data.
for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)
# forward
# track history if only in train
with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)
# backward + optimize only if in training phase
if phase == 'train':
                        optimizer.zero_grad()
                        loss.backward()
                        optimizer.step()
# statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
if phase == 'train':
                scheduler.step()
            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))
# deep copy the model
if phase == 'val' and epoch_acc &gt; best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

    time_elapsed = time.time() - start_time
print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
print('Validation Accuracy: {:4f}'.format(best_acc))
# load best model weights
    model.load_state_dict(best_model_wts)
return model
</code></pre>
Parameter is not getting iniatlized

username: ADONAI_TZEVAOT
cooked: <pre><code class="lang-auto">class A(nn.Module):
    def __init__(self):
        super(A,self).__init__()
        self.p = nn.Parameter(torch.randn(1))
</code></pre>
<p>In def forward when I call <code>print(self.p.shape, self.p)</code>, it is returning :</p>
<pre><code class="lang-auto">torch.Size([0]) Parameter containing:
tensor([], requires_grad=True)
</code></pre>
<p>So, self.p is not initialized. How to initialize this ? Please help.</p>

username: sio277
cooked: <p>It works for me.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class A(nn.Module):
    def __init__(self):
        super(A,self).__init__()
        self.p = nn.Parameter(torch.randn(1))
        
    def forward(self):
        print(self.p.shape, self.p)
        
a = A()
a()
</code></pre>
<p>which prints out:</p>
<p>torch.Size([1]) Parameter containing:<br>
tensor([-0.3897], requires_grad=True).</p>

username: ADONAI_TZEVAOT
cooked: <p>I dunno what exactly the error was, it didn’t run in the beginning , now its running. I’m getting errors if I dont remove <code>A,self</code> inside <code>super()</code>. Any idea why.</p>

username: ptrblck
cooked: <p>What kind of errors are you seeing if you keep <code>super(A, self).__init__()</code> and which Python version are you using?<br>
The arguments were only necessary in Python2, if I’m not mistaken, but I haven’t seen it’s raising an error in Python3.</p>
RNN model built from scratch doesn't overfit a sample of data

username: AchrafASH
cooked: <p>Hi,</p>
<p>I’ve tried to implement a simple RNN cell from scratch for sentiment classification (positive/negative) and overfit a sample of data, but for some reason, the model doesn’t seem to learn anything (loss doesn’t decrease). Do you have any clue what could cause this?</p>
<p>Thanks for your help!</p>
<p>Here is the model:</p>
<pre><code class="lang-auto">class SentimentClassifier(nn.Module):
    def __init__(self, input_size, hidden_size, output_size=2):
        super(SentimentClassifier, self).__init__()
        self.hidden_size = hidden_size
        self.whh = nn.Linear(hidden_size, hidden_size)
        self.wxh = nn.Linear(input_size, hidden_size)
        self.fc = nn.Linear(hidden_size, output_size)
    
    def init_hidden_state(self, batch_size):
        return torch.zeros(batch_size, self.hidden_size)
    
    def forward(self, x, hidden_state): # x shape: (batch_size, input_size)
        hidden_state = torch.sigmoid(self.whh(hidden_state) + self.wxh(x))
        output = F.softmax(self.fc(hidden_state), dim=1)
        return output, hidden_state
    
    def fit(self, dataset, batch_size, epochs, lr=0.001):
        optimizer = torch.optim.SGD(self.parameters(), lr=lr)
        criterion = torch.nn.CrossEntropyLoss()
        
        for epoch in range(epochs):
            total_loss = 0
            for idx, (texts, sentiments) in enumerate(dataset):
                hidden_state = self.init_hidden_state(batch_size=batch_size)
                
                # forward
                for i in range(texts.shape[0]): # texts.shape[0] = sequence length
                    output, hidden_state = self.forward(texts[i], hidden_state)
                
                loss = criterion(output, sentiments)
                
                # backward
                optimizer.zero_grad()
                loss.backward()
                torch.nn.utils.clip_grad_norm_(self.parameters(), max_norm=1)
                
                # gradient descent or Adam step
                optimizer.step()
                
                total_loss += loss
</code></pre>

username: AbdulsalamBande
cooked: <p>Hello, I also can not find any bug , but check out this tutorial <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" rel="noopener nofollow ugc">RNN From Scratch</a>, it might help. You can also check <a href="https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html" rel="noopener nofollow ugc">RNN From Scratch</a>. Hope it helps</p>

username: AchrafASH
cooked: <p>Thanks for your reply. I’ve done them before and looked at them a couple of time but can’t see what I’m doing wrong.<br>
Maybe I’m missing something about the backpropagation as I loop through the sequence input and only do loss.backward() - the loss is computed with the target and the final output only.</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects raw logits as the model output, so you should remove the <code>F.softmax</code> operation to create the <code>ouput</code> tensor.</p>
Bounded spectral norm

username: Dylan_Yung
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3ce148ba07179f7cace604a2ecf8295ce624049.png" data-download-href="https://discuss.pytorch.org/uploads/default/b3ce148ba07179f7cace604a2ecf8295ce624049" title="image (1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3ce148ba07179f7cace604a2ecf8295ce624049_2_690x286.png" alt="image (1)" data-base62-sha1="pECZvIlIcxOC5at25jveo3tIl3H" width="690" height="286" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3ce148ba07179f7cace604a2ecf8295ce624049_2_690x286.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3ce148ba07179f7cace604a2ecf8295ce624049_2_1035x429.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3ce148ba07179f7cace604a2ecf8295ce624049.png 2x" data-dominant-color="EBEAE5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image (1)</span><span class="informations">1245×517 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So I’ve been reading the spectral norm gaussian process paper. They have this part where they normalize the spectral norm weights based on the denominator of the equation. I was wondering how this can be done in PyTorch? Currently the issue is the spec norm layer is applied to a module and I don’t think it returns the weights denominator and numerator and if it did I’d have to go layer by layer and check if sigma(Wl) is above the threshold then multiply that, but all this isn’t possible because you instantiate SN over a module.</p>
How to resolve runtime error

username: octavian21
cooked: <p>Hello mates,</p>
<p>I am struggling to solve below error while I trigger model training with pytorch1.4 framework.</p>
<p>I’ve already added “libcaffe2_nvrtc.so” file to PATH and could see the so file through “echo $PATH”.</p>
<p>Do you have any other idea to solve below issue? Thank you ahead.</p>
<h1>RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so: cannot open shared object file: No such file or directory</h1>
Averaging upper and lower triangle of a matrix

username: Usman_Mahmood
cooked: <p>Hello, I have a tensor of shape N * X * Y where N is the number of subjects. I want to average the upper and lower triangle for each matrix i in N. For example if  subject 1 has the matrix:</p>
<p>tensor([[1, 2, 3],<br>
[4, 5, 6],<br>
[7, 8, 9]])<br>
Then the output should be:</p>
<p>tensor([[1, 3, 5],<br>
[3, 5, 7],<br>
[5, 7, 9]])<br>
So, adding upper and lower triangle and dividing each value of two, the main diagonal remains same as you add each number with itself and divide by 2. I know I can loop through each subject, get the upper and lower triangle using <code>torch.</code> <code>triu</code> and <code>torch.</code> <code>tril</code>, add them together and take average and put them back in the array.<br>
But I am looking for a faster method , specially without the loop.<br>
Thank you</p>

username: Usman_Mahmood
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any help on this?</p>

username: KFrank
cooked: <p>Hi Usman!</p>
<aside class="quote no-group" data-username="Usman_Mahmood" data-post="1" data-topic="104257" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/usman_mahmood/48/22826_2.png" class="avatar"> Usman_Mahmood:</div>
<blockquote>
<p>I want to average the upper and lower triangle for each matrix i in N.</p>
</blockquote>
</aside>
<p>Probably the most direct approach is to average the matrix with its<br>
transpose.  Here is an illustration for the 3-dimensional-tensor use<br>
case you mention:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; t = torch.tensor ([[[1, 2, 3], [4, 5, 6], [7, 8, 9]], [[10, 20, 30], [40, 50, 60], [70, 80, 90]]])
&gt;&gt;&gt; (t + t.transpose (1, 2)) / 2
tensor([[[ 1.,  3.,  5.],
         [ 3.,  5.,  7.],
         [ 5.,  7.,  9.]],

        [[10., 30., 50.],
         [30., 50., 70.],
         [50., 70., 90.]]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Saving final tensor output as tif images

username: Fahd_Jerbi
cooked: <p>Hi, I made a CNN model for edge detection task. The model did good and I can see the results through the feature maps (I’m not using the fc part of the model). I’m using the model to accept an input size as (1, 3, 224, 224) but the output tensor is (1, 4, 13, 13).<br>
I want to save tensor output as (224, 224, 3) and to sum feature maps of each Conv layer as one image. If anyone could help it would appreciated.</p>
Make cross validation parallelized

username: udo
cooked: <p>I have a cuda9-docker with tensorflow and pytorch installed, I am doing cross validation on an image dataset. Currently I am using a for loop to do the cross validation. Something like</p>
<pre><code>for data_train, data_test in sklearn.kfold(5, all_data):
  train(data_train)
  test(data_test)
</code></pre>
<p>But the for loop takes too long, will the following code work to parallelize the for loop? Maybe there is already a solution. But this is not Data Parallelization.</p>
<pre><code>from multiprocessing import Pool

def f(trainset, testset):
    train_result = train(trainset)
    test_result = test(testset)
    save_train_result()
    save_test_result()

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, sklearn.cvfold(5, all_data)))
</code></pre>
<p>I am not sure if the multiprocessing will only paralize the cpu or both cpu and gpu? This might be easiler than doing parallel in side  a model i guess like <a href="https://discuss.pytorch.org/t/parallelize-simple-for-loop-for-single-gpu/33701" class="inline-onebox">Parallelize simple for-loop for single GPU</a><br>
since in my case, there is no need to communicate across each process?</p>

username: deeperlearner
cooked: <p>I have the same issue. Anyone has solution?</p>
<p>I currently found <a href="https://discuss.pytorch.org/t/split-single-gpu/18651/8">this</a> may help, but haven’t check the performance.</p>

username: wayi
cooked: <p><code>I am not sure if the multiprocessing will only paralize the cpu or both cpu and gpu?</code></p>
<p>According to the <a href="https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool" rel="noopener nofollow ugc">official documentation</a> of Process Pools:</p>
<p><em>processes</em> is the number of worker processes to use. If <em>processes</em> is <code>None</code> then the number returned by <a href="https://docs.python.org/3/library/os.html#os.cpu_count" rel="noopener nofollow ugc"><code>os.cpu_count()</code></a> is used.</p>
<p>Therefore, I believe it should only do the parallelization on CPU.</p>
<blockquote>
<p>There is no need to communicate across each process?</p>
</blockquote>
<p>I think even if you parallelize the loops here, you will still have to sync gradients/weights across processes, because there is actually an implicit data dependency of the model weights.</p>
<p>If you parallelize the cross validation for loop w/o communication, you will evolve a separate models on each process, and this will slower down the convergence.</p>
Updating custom DataLoader class fields with multiple workers

username: wmpauli
cooked: <p>I wrote a pretty involved custom DataLoader class. It works fine if I only use “0” workers, but as soon as I try to have 1+ workers, I run into issues.</p>
<p>In my <code>__getitem__(self, index)</code> method, in addition to returning the data and labels, I also want to cache intermediate results, so that they can be reused in the future. For this purpose I have a dictionary <code>self.cache[index]</code> where I store these results.</p>
<p>The puzzling thing is not that two workers might try to read/write the same dictionary entry simultaneously. Rather, <code>self.cache</code> doesn’t seem to be shared by workers at all. For example, if I tried to access the field from the training script (i.e. <code>dataset.cache[index]</code>) there is nothing there.</p>
<p>Is there a detailed guide/documentation on how to create a workaround for this? I’m obviously new to this …</p>

username: ptrblck
cooked: <p>You could try to use a shared array as described <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py">here</a>.</p>

username: wmpauli
cooked: <p>Thanks. That did the trick. Also glad to see that the Manager object supports many other types types (list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value and Array).</p>
Serialiazation of state_dict changes after loading into new model instance

username: jnsplt
cooked: <p>Hi all, does someone know why the bytes obtained from serializing state_dicts change after loading the state dict into a new instance of the same model architecture?</p>
<p>Have a look:</p>
<pre><code class="lang-auto">import binascii
import torch.nn as nn
import pickle

lin1 = nn.Linear(1, 1, bias=False)
lin1s = pickle.dumps(lin1.state_dict())
print("--- original model ---")
print(f"hash of state dict: {hex(binascii.crc32(lin1s))}")
print(f"weight: {lin1.state_dict()['weight'].item()}")

lin2 = nn.Linear(1, 1, bias=False)
lin2.load_state_dict(pickle.loads(lin1s))
lin2s = pickle.dumps(lin2.state_dict())
print("\n--- model from deserialized state dict ---")
print(f"hash of state dict: {hex(binascii.crc32(lin2s))}")
print(f"weight: {lin2.state_dict()['weight'].item()}")
</code></pre>
<p>prints</p>
<pre><code class="lang-auto">--- original model ---
hash of state dict: 0x4806e6b6
weight: -0.30337071418762207

--- model from deserialized state dict ---
hash of state dict: 0xe2881422
weight: -0.30337071418762207
</code></pre>
<p>As you can see, the hashes of the (pickles of the) state_dicts are different whereas the weight is copied over correctly. I would assume that a state_dict from the new model equals the old one in every aspect. Seemingly, it does not, hence the different hashes.</p>
Data_source parameter in initializer of Sampler base class

username: gncs
cooked: <p>What is the purpose of the <code>data_source</code> argument in the initializer of the <code>Sampler</code> base class?<br>
See <a href="https://github.com/pytorch/pytorch/blob/f9ca0d87a7cd2afb3ae5866acf7a77af54c5ab91/torch/utils/data/sampler.py#L20" rel="noopener nofollow ugc">here</a>. It seems unnecessary. What am I missing? Thank you!</p>

username: ptrblck
cooked: <p>It’s used e.g. <a href="https://github.com/pytorch/pytorch/blob/f9ca0d87a7cd2afb3ae5866acf7a77af54c5ab91/torch/utils/data/sampler.py#L54-L69">here</a> to create the <code>iterator</code> and to define the length of the sampler.</p>

username: gncs
cooked: <p>Thank you for the quick reply. I understand that it’s necessary for the derived classes, but the parameter doesn’t seem to have any effect in the <em>base class</em>. A class deriving from this base class will have to fill this parameter for no reason.</p>

username: ptrblck
cooked: <p>I don’t think that’s true, as e.g. <a href="https://github.com/pytorch/pytorch/blob/f9ca0d87a7cd2afb3ae5866acf7a77af54c5ab91/torch/utils/data/sampler.py#L130-L147"><code>SubsetRandomSampler</code></a> is not using this argument.<br>
Note that the constructor of the parent class is not called, which would expect this argument.<br>
I’m not deeply familiar with the typing system in Python, but my best guess is that the optional <code>data_source</code> argument in the parent class was added to make typing compatible (somehow).</p>

username: gncs
cooked: <p>Exactly like in the <code>SubsetRandomSampler</code>, my class also derives from <code>Sampler</code> and does not have a <code>data_source</code>.  However, I don’t think it’s common practice not to call the base classes’ initializer to avoid having to specify an argument that does nothing in the base class.</p>
<p>Unfortunately, I don’t understand how this is related to type inspection. Could you clarify? Thank you.</p>
Image Pre-Process Comparision for Segmentation Models

username: AlperEkmekci
cooked: <p>Hello,</p>
<p>I have 2 copies of same dataset which is processed and non-processed to segmentation purpose. I want to compare them to analyze impact of pre-process step(s) without do a labeling-training-testing.</p>
<p>Here is my question, Is it possible to compare pre-process step(s) each other avoid labeling and how can I compare these methods properly ?</p>
<p>Best.<br>
Alper</p>
When I use mini batch gradient descent, what optimizer should I use?

username: 111492
cooked: <p>When I use mini batch gradient descent, what optimizer should I use?<br>
I see that some people use optim.SGD(), but Stochastic gradient descent is not mini batch gradient descent.There is some direct difference between them. Why can I use optim.SGD() when I use mini batch gradient descent?</p>
<p>i saw <a href="https://discuss.pytorch.org/u/chenyuntc">Yun Chen</a>  say that “SGD optimizer in PyTorch actually is Mini-batch Gradient Descent with momentum” Can someone please tell me the rationale for this?</p><aside class="quote" data-post="2" data-topic="8060">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/48/299_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-sgd-works-in-pytorch/8060/2">How SGD works in pytorch</a> 
  </div>
  <blockquote>
    You are right. SGD optimizer in PyTorch actually is  Mini-batch Gradient Descent with momentum.
  </blockquote>
</aside>

<p>Thank you for reading my query.<br>
I look forward to hearing from you all.</p>
<p>My English is not very good, so I took the help of DEEPL translation. There may be some grammatical errors or improper use of words！Please forgive me</p>
Loading Tif Files with ImageLoader

username: CDWatson
cooked: <p>When loading Tif files using ImageLoader, the values of the files change. This does not happen when using the exact same process on JPG files.</p>
<p>Numpy array before loading with ImageLoader:<br>
[[231.8403  231.82645 231.8113  … 230.36873 230.36305 230.35547]<br>
[231.85472 231.84184 231.82722 … 230.3746  230.36903 230.36107]<br>
[231.86945 231.8577  231.84369 … 230.37767 230.37189 230.36334]<br>
…<br>
[235.34044 235.3341  235.3279  … 232.78148 232.77174 232.76143]<br>
[235.3523  235.34482 235.33734 … 232.79488 232.78423 232.77327]<br>
[235.36395 235.35553 235.34727 … 232.8095  232.79814 232.78664]]</p>
<p>Numpy array after using ImageLoader:<br>
[[[231 231 231]<br>
[231 231 231]<br>
[231 231 231]<br>
…<br>
[230 230 230]<br>
[230 230 230]<br>
[230 230 230]]</p>
<p>[[231 231 231]<br>
[231 231 231]<br>
[231 231 231]<br>
…<br>
[230 230 230]<br>
[230 230 230]<br>
[230 230 230]]</p>
<p>[[231 231 231]<br>
[231 231 231]<br>
[231 231 231]<br>
…<br>
[230 230 230]<br>
[230 230 230]<br>
[230 230 230]]</p>
<p>…</p>
<p>[[235 235 235]<br>
[235 235 235]<br>
[235 235 235]<br>
…<br>
[232 232 232]<br>
[232 232 232]<br>
[232 232 232]]</p>
<p>[[235 235 235]<br>
[235 235 235]<br>
[235 235 235]<br>
…<br>
[232 232 232]<br>
[232 232 232]<br>
[232 232 232]]</p>
<p>[[235 235 235]<br>
[235 235 235]<br>
[235 235 235]<br>
…<br>
[232 232 232]<br>
[232 232 232]<br>
[232 232 232]]]</p>
<p>As can be seen here the float values are either being cut off or rounded to the nearest int, how can I fix this?</p>
Replace matrix with new value

username: haelee_bae
cooked: <p>Hello, I want to replace the matrix element with the new value.</p>
<p>For example, I have (512,45,1000) matrix.</p>
<p>I also have two tensors (512,1) , (512,1).</p>
<p>In two tensors there are values which are smaller than 45 and 1000 each.</p>
<p>I want to pad the matrix to 0 for the row and column which 0 padding starts with values of each tensor.</p>
<p>now I am using for statement,</p>
<p>for i in range(similarity_score.shape[0]):<br>
similarity_score[i][first_len[i]:]=num<br>
similarity_score[i][:,second_len[i]:] = num</p>
<p>In my thought, it takes much time because of for statement.<br>
Is there any built-in function or more efficient way to reduce the time?</p>
How to pad zeros on Batch

username: Jingles
cooked: <p>Is there a better way to do this? How to pad tensor with zeros, without creating new tensor object? I need inputs to be of the same <code>batchsize</code> all the time, so I want to pad inputs that are smaller than <code>batchsize</code> with zeros. Like padding zeros in NLP when sequence length is shorter, but this is padding for batch.</p>
<p>Currently, I created a new tensor, but because of that, my GPU will go out of memory. I don’t want to reduce batchsize by half to handle this operation.</p>
<pre><code class="lang-python">import torch
from torch import nn

class MyModel(nn.Module):
    def __init__(self, batchsize=16):
        super().__init__()
        self.batchsize = batchsize
    
    def forward(self, x):
        b, d = x.shape
        
        print(x.shape) # torch.Size([7, 32])

        if b != self.batchsize: # 2. I need batches to be of size 16, if batch isn't 16, I want to pad the rest to zero
            new_x = torch.zeros(self.batchsize,d) # 3. so I create a new tensor, but this is bad as it increase the GPU memory required greatly
            new_x[0:b,:] = x
            x = new_x
            b = self.batchsize
        
        print(x.shape) # torch.Size([16, 32])

        return x

model = MyModel()
x = torch.randn((7, 32)) # 1. shape's batch is 7, because this is last batch, and I dont want to "drop_last"
y = model(x)
print(y.shape)
</code></pre>

username: Eta_C
cooked: <p>If you use <code>DataLoader</code>, <code>drop_last=True</code> may work.</p>

username: Jingles
cooked: <aside class="quote no-group" data-username="Jingles" data-post="1" data-topic="116290">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/46a35a/48.png" class="avatar"> Jingles:</div>
<blockquote>
<pre><code class="lang-auto">x = torch.randn((7, 32)) # 1. shape's batch is 7, because this is last batch, and I dont want to "drop_last"
</code></pre>
</blockquote>
</aside>
<p>Intentionally want to avoid <code>drop_last=True</code>.</p>
<p>My current solution is:</p>
<pre><code class="lang-auto">x = F.pad(x, (0,0,0,(self.batchsize-b)))
</code></pre>
Even though the seed is fixed, I still get different results

username: MarcusNerva
cooked: <p>This is my main function:</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    args = get_settings()
    set_random_seed(args.seed)
    train(args=args)
    test(args=args)
    print('===================All is finished====================')
</code></pre>
<p>Every time I run this code with the same hyper-parameters, I got different performance on test dataset.<br>
And this is my set_random_seed() function:</p>
<pre><code class="lang-auto">def set_random_seed(seed):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
</code></pre>
<p>I even add torch.manual_seed(seed) and torch.cuda.manual_seed_all(seed) in <strong>init</strong>() of every model or submodel. The trained models still got different performance on test dataset.</p>
<p>By the way, the problem won’t lie in test code. Because I have test the same trained model several times, and every time got the same results.</p>
<p>I am so confused <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:">. Can anyone give me a hand?</p>

username: ptrblck
cooked: <p>Have a look at the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> and in particular set <code>torch.use_deterministic_algorithms(True)</code>, which would yield an error, if no deterministic algorithms can be found for your workload.</p>

username: MarcusNerva
cooked: <p>Thanks. Very helpful! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
Converting Fully Connected to Convolutional Layer

username: hachiro
cooked: <p>Hello</p>
<p>I’m trying to convert a fully - connected layer to a convolutional one. My framework pipeline consists of two modules , a featurizer and a classifier. The architecture of the classifier is a simple network as described above<br>
self.clf = nn.Sequential(<br>
nn.Linear(M, 32),<br>
nn.LeakyReLU(0.2),<br>
nn.Dropout(0.2),<br>
nn.Linear(32, 16),<br>
nn.LeakyReLU(0.2),<br>
nn.Dropout(0.2),<br>
nn.Linear(16, 2),<br>
)<br>
I did some research but I am a bit confused how to do the transofrmation. I want to use Conv1d instead of the FC layers.</p>
<p>Thank’s in Advance .</p>

username: Eta_C
cooked: <p>Convolution with <code>in_channels=M</code>, <code>out_channels=32</code>, <code>kernel_size=1</code>, <code>stride=1</code></p>

username: hachiro
cooked: <p>Everything works fine by adding         :<br>
self.clf = nn.Sequential(<br>
nn.Conv1d(in_channels=M, out_channels=32, kernel_size=1, stride = 1),<br>
nn.LeakyReLU(0.2),<br>
nn.Conv1d(in_channels=32, out_channels=16, kernel_size=1, stride=1),<br>
nn.LeakyReLU(0.2),<br>
nn.Conv1d(in_channels=16, out_channels=2, kernel_size=1, stride=1),<br>
)<br>
The only thing that i had to modify was  at the forward function :</p>
<pre><code>    z = z.unsqueeze(-1)
    logit = self.clf(z)
    logit = logit.squeeze(-1)
</code></pre>
<p>Where z was the output of an embedding + attention pooling stage<br>
Thank’s for the help</p>
Torch tensor of objects

username: pingu
cooked: <p>If I have a class</p>
<pre><code class="lang-auto">class Operation():
    
    def __init__(self):
                
        self.layer = nn.Linear(16*16,8,bias=True)

        
    def eval(self,input):
        
        return = torch.relu(self.layer(input))
</code></pre>
<p>How can I create a 2D torch tensor of this type ? For example, the equivalent of the follwing in numpy;</p>
<pre><code class="lang-auto">matrix = np.empty((x_dim,y_dim),dtype=object)
 for x in range(x_dim):
    for y in range(y_dim):
        matrix[x,y] = Operation()
</code></pre>
<p>I have tried in pytorch but seems like object is not a valid data type…</p>
<p>If it is really not possible is there another way it can be done ? Using ModuleList or ModuleDict while being able to refer to objects within as coordinates ?</p>
What are the differences between a pytorch and tensorflow tensor?

username: knoriy
cooked: <p>In the example below TF and PT operate on different channels, what is the reasoning behind this? is the order of the data different in TF than that of PT?</p>
<pre><code class="lang-auto">pt_tensor = torch.rand((5,80,256))
tf_tensor = tf.random.uniform((5,80, 256))

tf_conv = tf.keras.layers.Conv1D(128,3)
pt_conv = torch.nn.Conv1d(80, 128,3)

tf_conv(tf_tensor).shape # shape = TensorShape([5, 78, 128]) 

pt_conv(pt_tensor).shape # shape = torch.Size([5, 128, 254])

</code></pre>
<p>How are tensors stored? Are PT tensor stored as <code>batch_size + [in_hight,  in_width]</code> and TF tensors stores as <code>batch_size + [in_width, in_height]</code>?</p>

username: Dwight_Foster
cooked: <p>The difference is not in the way tf and pytorch store tensors it is the fact that their convolutional layers output different shapes. In tensorflow the conv1d layers have an output of (batch size, new steps, filters) while in pytorch the output of conv1d is shaped (batch size, filters, new steps). This is what makes the difference not the tensors themselves.</p>
How to retain gradient after converting tensor->numpy->tensor

username: saikumar_Joru
cooked: <p>Hello,<br>
I am working on Graph Convolutional neural networks using PyTorch. The input vectors are fed into series of GCN layers where it accumulates its neighbor information and generates an embedding vector for each node. Until here I am using PyTorch tensors for tracking gradients. But now for the prediction, I should feed these embedding vectors individually into a downstream model(XGBoost in my case). But as tensors don’t work on XGBoost I need to convert them to NumPy, make prediction, compute loss, and backpropagate through the model until the beginning of GCN layers. But as I have converted the tensor to numpy I will lose the gradients after that. Is there a way I could retain my gradients including the operations on numpy array.</p>

username: albanD
cooked: <p>Hi,</p>
<p>We cannot automatically track gradients for numpy Tensors I’m afraid. You will have to either stick with pytorch Tensor or write the backward computation explicitly as described here: <a href="https://pytorch.org/docs/stable/notes/extending.html" class="inline-onebox">Extending PyTorch — PyTorch 1.8.0 documentation</a></p>
Please help me，Error installing torch by Jetson nano

username: AoLie
cooked: <p>I want to install torch vision &gt; = 0.8.1 on Jetson nano , The version downloaded from GitHub is 0.8.1, but it is 0.8.0a0 after installation<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb545d6bb16083a640b58a7fc9055c0da75400c1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fb545d6bb16083a640b58a7fc9055c0da75400c1" title="IMG_20210329_194018"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb545d6bb16083a640b58a7fc9055c0da75400c1_2_665x499.jpeg" alt="IMG_20210329_194018" data-base62-sha1="zRmAXphUOzXxs6fkaj5S6N6Jpx7" width="665" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb545d6bb16083a640b58a7fc9055c0da75400c1_2_665x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb545d6bb16083a640b58a7fc9055c0da75400c1_2_997x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb545d6bb16083a640b58a7fc9055c0da75400c1.jpeg 2x" data-dominant-color="693D62"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20210329_194018</span><span class="informations">1280×961 180 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Padding and stride for transfer learning

username: yuri
cooked: <p>I want to transfer learning from vgg16 pretrained layer to a layer in a new model<br>
is it compulsory to use the same stride and padding values like in the original (pretrained) layer?</p>
<p>say I wan to transfer learning from the second conv layer in vgg16 that looks as below:<br>
A)<br>
nn.Conv2D(64, 64, kernel_size=(3,3),padding=(1,1), stride=1)<br>
and my new layer:<br>
B)<br>
nn.Conv2D(64, 64, kernel_size=(3,3),<strong>padding=0</strong>, stride=1)<br>
can I still transfer learning from A to B?<br>
thx</p>

username: Dwight_Foster
cooked: <p>Yes you should still use the same padding and stride because that will change the output size if you mess around with them. The pretrained weights are trained on a specific size so other sizes might mess up the training.</p>

username: yuri
cooked: <p>thanks for your ansewr.</p>
<p>say I want to have an output layer that has the half size of the original (pretrained) one. does not this allow me to tweak the stride value to 2 instead of the oraginally value 1 ?<br>
you also mentioned that pretrained layers are trained to deal with specific spatial size. does this mean if vgg16 for example was trained on 224 by 224 images then we can input only this image size?</p>

username: Dwight_Foster
cooked: <p>Yes you can tweak the stride. No you do not have to input a 224 by 224 image. You can input any  image size but the performance might not be the same as you expect because the weights won’t match to the image size.</p>
Is PyTorch supposed to be thread-safe?

username: bdaskalov
cooked: <p>Is invoking PyTorch functions thread-safe wrt Python threads?</p>
<p>I am trying to run model inference in a multi-threaded server and I get segmentation faults from MKLDNN convolution kernel. I expected that PyTorch calls will drop the GIL and be thread-sage similar to numpy, opencv and other similar libraries but was not able to find this clearly documented anywhere.</p>

username: ezyang
cooked: <p>PyTorch underlying C++ library is expected to be thread safe (although the Tensor object is not thread-safe for multiple writers; you need to synchronize that yourself). So if you are getting multithreading related segfaults, if you have a repro, please report us an issue.</p>

username: bdaskalov
cooked: <p>Here is the issue with the script to reproduce it:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/16828" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/bdaskalov" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/2833552?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/16828" target="_blank" rel="nofollow noopener">Issue: Process crashes with segmentation fault when calling conv2d from multiple threads.</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/bdaskalov" target="_blank" rel="nofollow noopener">bdaskalov</a>
	on <a href="https://github.com/pytorch/pytorch/issues/16828" target="_blank" rel="nofollow noopener">2019-02-06</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
Process crashes with segmentation fault when calling conv2d from multiple threads.
To Reproduce
Steps to reproduce the behavior:
Run the attached script. (might...</pre>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Konstantin_Burlachen
cooked: <p>Why do you think so? Can you provide proof?<br>
Are any official documentation that states which parts are are thread-safe and what parts are not?</p>
<p>I faced with a problem that if DataLoaders has several worker_threads and several threads use each one DataLoaders but on the same Dataset - then it sometimes brings to disaster.</p>
<p>Should I fill a ticket?</p>
Parallel operations on samples of a batch

username: Ajinkya_Ambatwar
cooked: <p>I have a batch sample of size (32, 1024, 3) where 32 is the batch size each of which contains (1024) points with 3 features each.<br>
For each of the sample(among 32) [size (1024,3)] , I am performing few operations which <strong>does not</strong>  include <code>torch.nn</code> operations(like <code>conv</code> or <code>linear</code> etc) but rather the numpy like operations.<br>
For eg. the operations are like this</p>
<pre><code class="lang-auto">curr = 0
batch = torch.randint(0,55,[32,1024,3])
for i in batch:
     for j in i:
          id_ = torch.unique(j)
          id_ = id_[id_!=curr]
         &lt;some other operations&gt;
</code></pre>
<p>I wanted to know, can I perform these operations parallely(eliminating the <code> for i in batch</code> loop)?<br>
The <code>torch.nn</code> APIs automatically perform these tasks on an entire batch without having to explicitly mention it in the code. I wonder if such a similar method can be applied here.</p>
<p>If so, how can I do that?<br>
Tagging <a class="mention" href="/u/ptrblck">@ptrblck</a> for reference!<br>
Thank you!</p>
Metric Learning For Classification - How good is it?

username: ElisonSherton
cooked: <p>Hello guys!</p>
<p>I am trying to figure out how and if I could use metric learning for the task of image classification. The approach I am trying out is to cluster objects of same class together while maximizing the distance between clusters of different class.</p>
<p>To give a concrete example, I am trying to classify if a dress is a summer dress or a winter dress; the former is more free, flowy and loose-fitting while the latter is a body-hugging, thick fabric whole body covering kind of dress; the classes are kinda self-explanatory that way…</p>
<p>For the model to do metric learning, I have used a resnet50 head followed by average pooling and a linear layer to reduce the embedding dimension from 2048 to 512 respectively.</p>
<p>In the dataset, I have sampled 2 classes with a batch size of 32 i.e. 16 images per class for comparison.</p>
<p>I have used Multi-Similarity Loss, with hyper-parameters alpha = 2 (positive pairs weightage), beta = 50 (negative pairs weightage) and base = 0.5 as mentioned in the <a href="https://arxiv.org/abs/1904.06627" rel="noopener nofollow ugc">MS Loss paper.</a> Any suggestions on tuning these are also welcome.</p>
<p>After training for around 20 epochs with using a differential learning rate strategy and tuning lr after almost every 5 epochs, the model that I trained gives the following embedding plot (512 feature vector embeddings)</p>
<p><img src="https://forums.fast.ai/uploads/default/optimized/3X/f/7/f7fa5bb5164a76a561d0dd3d523fbbd74e35dec8_2_690x414.png" alt="" width="690" height="414"></p>
<p>However, when I do the same task with a traditional resnet50, I get the following embedding plot</p>
<p><img src="https://forums.fast.ai/uploads/default/optimized/3X/f/6/f6dc8a7319788507c56297a4f1f6553073d1d623_2_690x407.png" alt="" width="690" height="407"></p>
<p>What I don’t understand is that the MSLoss, which is specifically designed to make sure the embeddings of similar class should get closer and that across classes the embeddings should be far apart, fails to cluster the images well whereas a simple classification cross-entropy loss performs better.</p>
<p>Could someone from the community shed light on this topic; any insights are appreciated.</p>
<p>Thanks &amp; Regards,<br>
Vinayak.</p>
Difference between Adam and AdamW implementation

username: NightRain
cooked: <p>What is the difference between the implementation of Adam(weight_decay=…) and AdamW(weight_decay=…)?<br>
They look the same to me, except that AdamW has a default value for the weight decay.</p>

username: fadetoblack
cooked: <p>Please check the paper behind AdamW:<br>
</p><aside class="onebox pdf">
  <header class="source">
      <a href="https://arxiv.org/pdf/1711.05101.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>
  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1711.05101.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://arxiv.org/pdf/1711.05101.pdf" target="_blank" rel="noopener nofollow ugc">1711.05101.pdf</a></h3>

<p class="filesize">5.78 MB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: AOZMH
cooked: <p>I consulted the official documentation of Adam &amp; AdamW and noticed that the implementation of weight-decay in Adam also followed the <a href="https://arxiv.org/abs/1711.05101" rel="noopener nofollow ugc">Decoupled Weight Decay Regularization</a> (<a href="https://pytorch.org/docs/stable/optim.html#torch.optim.Adam" rel="noopener nofollow ugc">torch.optim — PyTorch 1.7.0 documentation</a>) which is the same for Adam. Does that mean that currently, Adam &amp; AdamW are the same w.r.t. weight decay?</p>

username: hoangle_tttm
cooked: <p>I have the same question!</p>

username: AOZMH
cooked: <p>I guess the issue is addressed in <a href="https://discuss.pytorch.org/t/difference-between-adam-and-adamw-in-pytorch/109173">this thread</a> which led to a (pending) update on the official document.<br>
A quick conclusion is that the actual implementation of weight decay in Adam still follows the original L2-regularization, despite the documentation, thus, maybe using AdamW is still a better choice.</p>

username: hoangle_tttm
cooked: <p>Thank you!<br>
One more question… Have you used AdamW with 1CycleLearningRate policy in Pytorch? How to use them correctly?</p>

username: AOZMH
cooked: <p>Haven’t tried it yet… Maybe you can find some <a href="https://pytorch.org/tutorials/" rel="noopener nofollow ugc">tutorials</a> on your topic/case?</p>