akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Storing and loading a model whose modules have extra attributes

Branikas: <p>Hello everyone.</p>
<p>I am trying to create an auto-encoder architecture for image segmentation using a framework for rotation invariance. The modules of this framework inherit from the torch.nn classes but have some extra attributes as well.<br>
For the Conv2d module, for example, they don’t have only the weight and bias attributes but some extra as well. When I train my network and save it normally (with with torch.save(model.state_dict(), PATH)) i notice that when I load it with model.load_state_dict(torch.load(PATH)) and try to predict an image I get errors referring to the keys of these extra attributes. More specifically, I get:  RuntimeError: Error(s) in loading state_dict :<br>
Missing key(s) in state_dict: …<br>
for every convolutional block there is no value assigned for the extra attributes in the dictionary.<br>
I suppose when the dictionary is created only the standard attributes of the model’s components (pooling, conv. etc) are considered for adding to the dictionary.<br>
Can I somehow change that for custom classes that inherit from torch.nn modules?</p>
<p>Thank you!</p>

JuanFMontesinos: <p>I would say that it should work always you add nn.parameters or buffers which are the ones tracked by the state_dict.<br>
If you try to save other type of variables it will fail. I would recommend to pass them in the init function such that the constructor can recover them at the time of instantiating the class.</p>
<p>Another possibility is you are kind of hardcoding them not using the tools provided which properly register parameters.</p>

Branikas: <p>Thank you very much, I actually didn’t think of passing them in the init, struggling to integrate 2 models. I will try your suggestions.</p>
Right way to apply self attention on lstm outputs?

hadaev8: <p>For now, I have it like this<br>
Still, it’s not clear how to apply attention to lstm outputs</p>
<pre><code>class Encoder(nn.Module):
	def __init__(self, hparams):
		super(Encoder, self).__init__()

		self.conv = ResidualBlock1d(in_channels=hparams.encoder_embedding_dim,
						out_channels=hparams.encoder_embedding_dim,
						kernel_size=hparams.encoder_kernel_size,
						activation=hparams.activation, normtype=hparams.normtype,
						num_layers=hparams.encoder_n_convolutions - 1)

		self.lstm = nn.LSTM(hparams.encoder_embedding_dim,
					int(hparams.encoder_embedding_dim / 2), 1,
					batch_first=False, bidirectional=True)

		self.attn = nn.MultiheadAttention(hparams.encoder_embedding_dim, 8)

	def attn_pad_mask(self, lengths):
		max_len = torch.max(lengths).item()
		mask = torch.arange(max_len, out=torch.cuda.LongTensor(max_len))[
			None, :] &gt;= lengths[:, None]
		return mask

	def forward(self, x, input_lengths):
		x = x.transpose(1, 2)

		x = self.conv(x)

		x = x.transpose(1, 2).transpose(0, 1)

		x = nn.utils.rnn.pack_padded_sequence(
			x, input_lengths, batch_first=False)

		self.lstm.flatten_parameters()
		outputs, _ = self.lstm(x)
		outputs, _ = nn.utils.rnn.pad_packed_sequence(
			outputs, batch_first=False)

		attn_mask = self.attn_pad_mask(input_lengths)
		outputs = self.attn(
			outputs, outputs, outputs, key_padding_mask=attn_mask, need_weights=False)[0]
		outputs = outputs.transpose(0, 1)

		return outputs</code></pre>
Advance indexing of tensor to get rid of nested for-loops

Siddharth_Vashishtha: <p>I have a situation for which I am using nested for-loops, but I want to know if there’s a faster way of doing this using some advanced indexing in Pytorch.</p>
<p>I have a tensor named <code>t</code>:</p>
<pre><code class="lang-auto">t = torch.randn(3,8)
print(t)
tensor([[-1.1258, -1.1524, -0.2506, -0.4339,  0.8487,  0.6920, -0.3160, -2.1152],
        [ 0.4681, -0.1577,  1.4437,  0.2660,  0.1665,  0.8744, -0.1435, -0.1116],
        [ 0.9318,  1.2590,  2.0050,  0.0537,  0.6181, -0.4128, -0.8411, -2.3160]])
</code></pre>
<p>I want to create a new tensor which indexes values from <code>t</code>.<br>
Let’s say these indexes are stored in variable <code>indexes</code></p>
<pre><code class="lang-auto">indexes = [[(0, 1, 4, 5), (0, 1, 6, 7), (4, 5, 6, 7)],
           [(2, 3, 4, 5)],
           [(4, 5, 6, 7), (2, 3, 6, 7)]]

</code></pre>
<p>Each inner tuple in <code>indexes</code> represents four indexes that are to be taken from a row in t.</p>
<p>As an example, based on these indexes my output would be a 6x4 dimension tensor (6 is the total number of tuples in <code>indexes</code>, and 4 corresponds to one value in a tuple)</p>
<p>For instance, this is what I want to do:</p>
<pre><code class="lang-auto">#counting the number of tuples in indexes
count_instances = sum([1 for lst in indexes for tupl in lst])

#creating a zero output matrix 
final_tensor = torch.zeros(count_instances,4)

final_tensor[0] = t[0,indexes[0][0]]
final_tensor[1] = t[0,indexes[0][1]]
final_tensor[2] = t[0,indexes[0][2]]
final_tensor[3] = t[1,indexes[1][0]]
final_tensor[4] = t[2,indexes[2][0]]
final_tensor[5] = t[2,indexes[2][1]]
</code></pre>
<p>The final output looks like this:<br>
print(final_tensor)</p>
<pre><code class="lang-auto">tensor([[-1.1258, -1.1524,  0.8487,  0.6920],
        [-1.1258, -1.1524, -0.3160, -2.1152],
        [ 0.8487,  0.6920, -0.3160, -2.1152],
        [ 1.4437,  0.2660,  0.1665,  0.8744],
        [ 0.6181, -0.4128, -0.8411, -2.3160],
        [ 2.0050,  0.0537, -0.8411, -2.3160]])

</code></pre>
<p>I created a function <code>build_tensor</code> (shown below) to achieve this with nested for-loops, but I want to know if there’s a faster way of doing it with simple indexing in Pytorch. I want a faster way of doing it because I’m doing this operation hundreds of times with bigger index and t sizes.</p>
<p>Any help?</p>
<pre><code class="lang-auto">def build_tensor(indexes, t):
    #count tuples
    count_instances = sum([1 for lst in indexes for tupl in lst])
    #create a zero tensor
    final_tensor = torch.zeros(count_instances,4)
    final_tensor_idx = 0

    for curr_idx, lst in enumerate(indexes):
        for tupl in lst:
            final_tensor[final_tensor_idx] = t[curr_idx,tupl]
            final_tensor_idx+=1
    return final_tensor
</code></pre>

mmisiur: <p>Have you tried <a href="https://pytorch.org/docs/stable/torch.html#torch.index_select" rel="nofollow noopener">index_select</a> ? It’s an easy way to select parts of tensor along one dimension by the indexes, so one for loop can be replaced.</p>
Measuring time of backpropagation

Raphikowski: <p>Hi all,</p>
<p>I wanted to ask if it is possible to not just measure the time of the backward pass as a whole, but also to measure the time of specific parts of the backwards pass. I.e., I have different models within my algorithm and I would want to find out if there is any particular model of these which is requiring particularly much time for backpropagation.<br>
Or would you just assume that the time of for backpropagation is proportional to the forward pass in each separate model?</p>
Dataset doesn't work well for distributed training

marcelgwerder: <p>I have been working on implementing distributed training for NER. In the process I implemented a version using Horovod and one using DistributedDataParallel, because I initially thought my issues were related to my implementation. Both work as expected with a public dataset. I can scale the learning rate by the number of processes (or the batch size) and I get results that are very close to the non-distributed training, yet faster. With my private dataset, which served me for testing along the way, the behavior is different. The distributed training on e.g. 4 processes performs almost exactly like when I train on a single process on 1/4 of the data but using the scaled learning rate. Debugging showed that the different processes have different losses and that the gradients are correctly syncronized in the backward pass.</p>
<p>The only two explanations I have for this: 1) There is still something wrong in my code. 2) The gradients computed are so similar for each process that there is not much or no gain in averaging them and the result is similar to working with 1/4 of the data and a scaled learning rate.</p>
<p>This is my first experience with distributed training so I can’t tell if 2) is reasonable and I’d be keen to know more about your experience with this.</p>
Setting value of parameter manually through another variable

marco_pytorch: <p>The following works:<br>
<code>foo = nn.Parameter( torch.randn(4) )</code><br>
<code>model.hidden.bias = foo</code></p>
<p>However when I change it to this:<br>
<code>bar = model.hidden.bias</code><br>
<code>bar = nn.Parameter( torch.randn(4) )</code><br>
it doesn’t seem like the bias parameter within the model actually updates, but only the local variable <code>bar</code> changes.</p>
<p>How can I create a sort of “pointer” to the bias parameter so that I can set it?</p>
Matmul function

111227: <p>We create an approximate multiplier. So, we want to change matmul function into our multiplier. Can we modify the function?</p>

tom: <p>You can’t without modifying the main PyTorch code, change the backend.<br>
What you could do if you use the JIT is to create a custom optimization pass that would replace all calls to matmul with calls to your matmul function.</p>
<p>Best regards</p>
<p>Thomas</p>

111227: <p><a class="mention" href="/u/tom">@tom</a><br>
Thanks for your reply. but I don’t understand your meaning. (I am new to PyTorch, and I appreciate you can explain my questions.)<br>
1.How to change backend<br>
2.How to create a custom optimization pass use JIT</p>
<p>Simplify my question. I used PyTorch function <strong>matmul</strong> now</p>
<blockquote>
<p>xx = torch.matmul(windows[channel], self.weights[i_convNumber][channel])</p>
</blockquote>
<p>So far, I want to customize function(use our multiplier) and call this function but isn’t works and very complicated. I would like to know is there have another way or some advice to make it easier?</p>
Displaying the depth map

Carolmeir: <p>Hello,</p>
<p>I am new to programming with python. How can I display the depth map images using <a href="https://github.com/mileyan/AnyNet/blob/master/finetune.py" rel="nofollow noopener">this</a> code?</p>

ptrblck: <p>Which variable would correspond to the depth map?<br>
Generally you can use <code>matplotlib.pyplot</code> to display image-like arrays.<br>
If you are dealing with tensors, you can transform it to a numpy array via <code>tensor.numpy()</code>.</p>
GPU memory usage

kkudus: <p>HI all,</p>
<p>I have a question about Pytorch GPU memory usage. I am playing with a couple of different architectures for my fully connected network, each of which have n hidden units in each hidden layer, though the number of hidden layers varies. I was expecting the memory footprint for each of the models to be roughly the same during inference (no backprop). The model itself is quite small and does not take up much GPU memory, but, my deeper networks seem to use much more VRAM. Given that they have the same number of hidden units the matrix multiplication operations are done on matrices of the same size, I was expecting all the models to use roughly the same amount of VRAM. What am I missing here? Why are the deeper networks need more memory to train?</p>
<p>Thanks in advance!</p>

ptrblck: <p>Could you give an example of a model, which uses little memory and another one of a model which uses more memory?<br>
I’m not sure to understand the question completely, but since each layer will use its own parameters, the more layers you are creating the more memory will be used.</p>
Blended Precision

Nick_8229: <p>I am trying blended precision for tensor operation, and find the runtime is not improved, and actually worse. Anything i missed?</p>
<p><code>torch.cuda.get_device_name()</code></p>
<p>‘Tesla T4’</p>
<pre><code class="lang-auto">import torch
import time

a = torch.zeros(60000, 200, dtype=torch.float16, device='cuda')
b = torch.tensor([0.01, 0.03, 0.05, 0.07], dtype=torch.float32, device='cuda')

for i in range(5):
    start = time.time()
    for j in range(1000):
        a.normal_()
        d = a * b.view(-1, 1, 1)
    print(time.time() - start)
</code></pre>
<p>0.9754595756530762<br>
1.9229979515075684<br>
1.913642168045044<br>
1.9154479503631592<br>
1.9257619380950928</p>
<pre><code class="lang-auto">import torch
import time

a = torch.zeros(60000, 200, dtype=torch.float32, device='cuda')
b = torch.tensor([0.01, 0.03, 0.05, 0.07], dtype=torch.float32, device='cuda')



for i in range(5):
    start = time.time()
    for j in range(1000):
        a.normal_()
        d = a * b.view(-1, 1, 1)
    print(time.time() - start)
</code></pre>
<p>0.910468339920044<br>
1.8464961051940918<br>
1.8373432159423828<br>
1.841585397720337<br>
1.8456008434295654</p>

ptrblck: <p>Since CUDA operations are asynchronous, you would have to synchronize the code via <code>torch.cuda.synchronize()</code> before starting and stopping the timer.<br>
Also, do you want to use FP16 and FP32 in your first example or should both be FP16?</p>
Loading InceptionV3 taking a lot of time

ASMIftekhar: <p>I am trying to load inceptionV3 from torchvision models using the following command:</p>
<p><code> import torchvision.models as models</code><br>
<code>inception=models.inception_v3(pretrained=True,progress=True)</code><br>
But it is taking close to three minutes to load. I am running it on a local server so I dont think computation power is the issue.</p>
<p>Is there anyway to load the model quickly?</p>

ptrblck: <p>Are you seeing this long duration only when initializing the model the first time (the checkpoint might be downloaded in the background) or also when you just rerun the same code?</p>

ASMIftekhar: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="68067" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Are you seeing this long duration only when initializing the model the first time (the checkpoint might be downloaded in the background) or also when you just rerun the same code</p>
</blockquote>
</aside>
<p>Hello, Every time I am running the code it is happening. The model is already downloaded.</p>

ptrblck: <p>Is your home folder on a local disk or mounted as a network drive?<br>
In the latter case, you could try to set <code>TORCH_HOME</code> to a local folder.</p>

ASMIftekhar: <p>I did that. It doesn’t help. One more thing the other models (e.g. Googlenet, Alexnet) are loading fine.</p>

ptrblck: <p>Which PyTorch and <code>torchvision</code> versions are you using?</p>

ASMIftekhar: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="68067">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>using</p>
</blockquote>
</aside>
<p>torch            1.1.0<br>
torchvision      0.3.0<br>
CUDA 10.0</p>

Hanxiang_Hao: <p>Any update on this issue? I also met the same issue every time when I load inception v3. My setting is torch: 1.4.0 and torch vision 0.5.0.</p>

ptrblck: <p>The slowdown might be related to <a href="https://discuss.pytorch.org/t/torchvisions-inception-v3-takes-much-longer-to-load-than-other-models/68756/6">this issue</a>. Lets continue the discussion in the other thread please.</p>
Getting conflict between ipaddress and torchvision when trying to install on Anaconda

Chriexpe: <p>First of all, I don’t know if this problem is supposed to be reported here or only on Conda’s Github, but there isn’t active as here so anyway…<br>
I’m trying to use Super-SloMo, wich require PyTorch to be installed on (in my case, Anaconda2-2018.12-Windows-x86_64), but when I try to install with the following command:</p>
<p><code>conda install pytorch torchvision cudatoolkit=10.1 -c pytorch</code></p>
<p>It start “Solving environment:” and after a while it return the following error:</p>
<pre><code class="lang-auto">Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - ipaddress
  - torchvision
Use "conda info &lt;package&gt;" to see the dependencies for each package.

</code></pre>
<p>I don’t know what to do or where I did something wrong…<br>
And yes, it’s an older version of Anaconda because newer ones for some reason don’t get installed correctly (i mean, the installation is abnormally fast and don’t give any error, but there is a lot of files missing, thus Anaconda don’t even open, and yes, I don’t have any antivirus beside Windows Defender, wich is supposed to be deactivated)</p>

ptrblck: <p>Could you try to uninstall the <code>ipaddress</code> package (if you don’t need it) or alternatively create a new conda environment and try to reinstall pytorch and torchvision?</p>

Chriexpe: <p>I couldn’t unistall ipaddress, but creating another environment worked like a charm, thank you!</p>
Converting a List to a Tensor

rjmiller927: <p>I am using the Wave-U-Net neural network for audio source separation but there is a bug in the code I have. When performing separation, the outputs are stored in a list and each item in the list is a Tensor. However, the tensors are on a GPU and need to be converted to CPU. My issue is that I cannot convert the list to CPU because numpy lists don’t have a .cpu() attribute. How can I convert all the tensors to CPU?</p>
<p>Here is the code I’m working with. The offending list is called “listofpred0”, and the error occurs when it is passed to mu_law_decode.</p>
<p>def test(epoch):  # testing data<br>
model.eval()<br>
start_time = time.time()<br>
with torch.no_grad():<br>
for iloader, xtrain in loadtest:<br>
iloader=iloader.item()<br>
print(iloader)<br>
listofpred0 = []<br>
for ind in range(0, xtrain.shape[-1] - sampleSize, sampleSize):<br>
output = model(xtrain[:, :,ind:ind + sampleSize].to(device))<br>
listofpred0.append(output.reshape(-1))<br>
ans0 = mu_law_decode(np.concatenate(listofpred0))<br>
if not os.path.exists(‘vsCorpus/’): os.makedirs(‘vsCorpus/’)<br>
sf.write(savemusic.format(iloader), ans0, sample_rate)<br>
print(‘test stored done’, np.round(time.time() - start_time))</p>

rjmiller927: <p>Found a solution…</p>
<p>Prior to the line <code>ans0 = mu_law_decode(np.concatenate(listofpred0))</code>, I added a for loop that went through all elements in <code>listofpred0</code> and assigned to CPU one by one. Here is the updated code:</p>
<pre><code class="lang-auto">def test(epoch):  # testing data
    model.eval()
    start_time = time.time()
    with torch.no_grad():
        for iloader, xtrain in loadtest:
            iloader=iloader.item()
            print(iloader)
            listofpred0 = []
            for ind in range(0, xtrain.shape[-1] - sampleSize, sampleSize):
                output = model(xtrain[:, :,ind:ind + sampleSize].to(device))
                listofpred0.append(output.reshape(-1))
            # New For Loop
            for i in range(0,len(listofpred0)):
                listofpred0[i] = listofpred0[i].cpu()
            ans0 = mu_law_decode(np.concatenate(listofpred0))
            if not os.path.exists('vsCorpus/'): os.makedirs('vsCorpus/')
            sf.write(savemusic.format(iloader), ans0, sample_rate)
            print('test stored done', np.round(time.time() - start_time))
</code></pre>
Has torch.autograd.Variable been replaced?

Abdelpakey: <p>Hi, I’m just wondering is torch.autograd.Variable  replaced by torch.tensor (…, Requires_grad= True)?</p>
<p>and why when I use this code</p>
<p>t = torch.from_numpy(matrix_np).clone()<br>
matrix_torch =torch.tensor(t, requires_grad=True)</p>
<p>it throws this error<br>
matrix_torch =torch.tensor(t.clone(), requires_grad=True)<br>
<strong>main</strong>:1: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).</p>
<p>what is the best way to make the Numpy array a tensor to use it later to backprop. ?</p>

albanD: <p>Hi,</p>
<p>The idea is that every Tensor is a Variable now.<br>
In your case, you can do: <code>matrix_torch = torch.from_numpy(matrix_np).clone().requires_grad_()</code>.</p>

Abdelpakey: <p>Thanks <a class="mention" href="/u/alband">@albanD</a><br>
Do I really need .clone() here if so why?</p>

albanD: <p>No you don’t really need the clone.<br>
Note that when you do <code>t = torch.from_numpy(matrix_np)</code> then <code>t</code> and <code>matrix_np</code> actually share memory. So modifying <code>matrix_np</code> inplace would modify <code>t</code> as well. So you might want to actually clone to avoid any issue.</p>
What to do when all errors are due to one class?

alx: <p>Hi all, I recently realized that 80% of my errors are due to the fact that my model predicts my class 1 as a class 0. I have 5 classes in total.</p>
<p>I had the idea to pretrain the model using binary classification for class 0 and 1 to make sure the model has initial weights that can make the difference between the two. However I’ve never seen a densenet used for binary classification so I’m not sure how it could be done.</p>
<p>In general, what are some techniques to use when most of the errors are due to a wrong class prediction? augmentation? transformations?</p>
<p>Thanks</p>

ptrblck: <p>Loss weighting might be another approach.<br>
If you are dealing with an imbalanced dataset, then <code>WeightedRandomSampler</code> might counter this effect.</p>

alx: <p>Thanks patrick, I tried that but it decreased the accuracy by around 20%, is there anything to adjust when using a balanced dataloader? LR maybe?</p>

ptrblck: <p>If you are overfitting on the majority class, you could use weighted sampling to trade the sensitivity for the specificity (in a binary use case).<br>
Note that the accuracy for an imbalanced dataset might be misleading as described in the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">accuracy paradox</a>.</p>
How to determine if cudnn.benchmark is finished?

maaft: <p>I need to measure the GPU memory consumption of my model and therefore have to wait until CUDNN’s benchmark has finished.</p>
<p>Is there any way to detect this easily (like cudnn.benchmark_finished == True or something similar) ?</p>
<p>I plotted the GPU Memory consumption over time with benchmarking enabled (orange) and disabled (blue).</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/3/6356dadf1b15e58b013930a74872a9abd25b5894.png" alt="Figure_1" data-base62-sha1="eaNm2ttcDC3jf1N0XlQXDnQVzda" width="640" height="480"></p>
<p>Of course I could use this profile to “detect” when the benchmark has finished but it feels awfully hacky.</p>

albanD: <p>Hi,</p>
<p>The benchmark will happen every time you call a convolution with inputs of different size.<br>
So it never really “finishes” as if you try to do a convolution with new shape, it will benchmark that new shape to find the best algorithm.<br>
Note that for a given shape, the benchmark will happen during the first call and when the call to conv returns (and you sync the device with torch.cuda.synchronize()), there won’t be any benchmark for that shape anymore.</p>

maaft: <p>Alright, that’s what I’ve thought.</p>
<p>So no way around measuring “convergence” manually, I guess!</p>
<p>Thanks!</p>
Batching training DAE with LSTM encoder, decoder properly

Dylan_Yung: <pre><code class="lang-auto">def denoise_train(x: DataLoader):
    loss = 0
    x_padded = list(map(lambda s: pad_string(s), x))
    x_idx_tensor = strings_to_index_tensor(x_padded)
    noisy_x = list(map(lambda s: noise_name(s), x))
    noisy_x_padded = list(map(lambda s: pad_string(s), noisy_x))
    noisy_x_idx_tensor = strings_to_index_tensor(noisy_x_padded)
    noisy_x_rnn_tensor = to_rnn_tensor(noisy_x_idx_tensor)
    batch_sz = len(x)
    encoder_hidden = encoder.init_hidden(batch_size=batch_sz)

    for i in range(noisy_x_rnn_tensor.shape[0]):
       _, encoder_hidden = encoder(noisy_x_rnn_tensor[i].unsqueeze(0), encoder_hidden)
    
    decoder_input = strings_to_tensor([SOS] * batch_sz)
    decoder_hidden = encoder_hidden
    names = [''] * batch_sz

    for i in range(x_idx_tensor.shape[0]):
        decoder_probs, decoder_hidden = decoder(decoder_input, decoder_hidden)
        nonzero_indexes = x_idx_tensor[i]
        best_indexes = torch.squeeze(torch.argmax(decoder_probs, dim=2), dim=0)
        decoder_probs = torch.squeeze(decoder_probs, dim=0)
        best_chars = list(map(lambda idx: index_to_char(int(idx)), best_indexes))
        loss += criterion(decoder_probs, nonzero_indexes.type(torch.LongTensor))

        for i, char in enumerate(best_chars):
            names[i] += char

        decoder_input = strings_to_tensor(best_chars)

    loss.backward()
    return names, noisy_x, loss.item()
</code></pre>
<p>I have this code for a denoising autoencoder on first names. It takes in a first name then pre-pads it with PAD char which I denote. I set it up like this so I could do batch training, but after training it for a couple days, I ran a name through it and it just printed out pads, which makes sense because it backprops on every iteration of a char so it’s being rewarded for just generating pads. How do I set it up not to learn this? I need the padding for batch training, because the names aren’t all consistent length so the padding allows them to be a consistent length. Currently when I put a noised name through it after training it just puts out all pads</p>
Crossed Element-wise multiplication

benoriol: <p>I have two tensors. t1 of shape NxD and t2 of shape MxD.</p>
<p>I want to get tensor t3 of shape NxMXD in which t3[n,m,:] = t1[n,:] * t2[m,:], being * the element-wise multiplication.</p>
<p>Is there an efficient way to do this? I can do it with fors but I need a better way to do it.</p>
<p>Thank you very much.</p>
<p>Ben</p>

JuanFMontesinos: <pre><code class="lang-auto">t3=torch.einsum('nd,md-&gt;nmd',t1,t2)
</code></pre>

benoriol: <p>This seems to work!</p>
<p>I wonder if it is implemented efficiently.</p>
<p>Thank you very much!</p>
<p>Ben</p>

JuanFMontesinos: <p>I have no idea. I really hope so. For sure it is cuds-paralellized.<br>
<a class="mention" href="/u/alband">@albanD</a> Can you give us any insight about einsum implementation? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>That might depend on size and device.<br>
I’m afraid it might not always be optimal as it is much more general than other ops.<br>
In this case, you can use broadcasting to get the same result:</p>
<pre><code class="lang-python">In [1]: import torch

In [2]: N, M, D = 10, 20, 30

In [3]: t1 = torch.rand(N,D)

In [4]: t2 = torch.rand(M,D)

In [5]: %timeit t3=torch.einsum('nd,md-&gt;nmd',t1,t2)
41.2 µs ± 263 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [6]: %timeit t3=torch.einsum('nd,md-&gt;nmd',t1,t2)
41.3 µs ± 325 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [7]: %timeit t3=t1.unsqueeze(1) * t2.unsqueeze(0)
18.4 µs ± 130 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [8]: %timeit t3=t1.unsqueeze(1) * t2.unsqueeze(0)
23.1 µs ± 3.72 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [9]: (t1.unsqueeze(1) * t2.unsqueeze(0) - torch.einsum('nd,md-&gt;nmd',t1,t2)).abs().max()
Out[9]: tensor(0.)
</code></pre>
<p>Note that the timing may vary wildly for different sizes or if you use a GPU.<br>
Also for other op, the einsum might be faster.</p>

albanD: <p>Also <a href="https://optimized-einsum.readthedocs.io/en/stable/">opt_einsum</a> might improve the einsum performances, but I haven’t tried for this particular example.</p>
Sending a PR to pytorch

kl_divergence: <p>I’ve previously contributed to torchvision, pytorch docs but I’ve not contributed to pytorch. My request to send a PR has been approved. I can’t simply clone the forked repo because it has not been compiled (import torch didn’t work) .</p>
<p>Do I need to build Pytorch from source to see the updated change that I do with python code ?</p>
<p>Building it from source hangs my laptop after sometime and it’s very slow. I also want to check first if my change is working or not. What should I do ?</p>
<p>Can you list the minimal steps required?</p>

kl_divergence: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, could you please suggest?</p>

ptrblck: <p><a class="mention" href="/u/tom">@tom</a> created a great tutorial on how to fix your first bug in PyTorch (including a video) <a href="https://lernapparat.de/fixing-your-first-pytorch-bug/">here</a>.</p>

tom: <p>Three quick comments (though we’d really need more context to see where you want input):</p>
<ul>
<li>Building and testing PyTorch your changes before submission is essential to a successful PR.</li>
<li>You may want to skip it for developing the patch, in particular when you are working on Python parts. I sometimes pick the “very quick and very dirty” way of just editing the installed .py files and then incorporating the changes to the source.</li>
<li>Building PyTorch is quite intense in terms of resources, but the <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md" rel="nofollow noopener">CONTRIBUTING.md</a> has hints how to speed up at least rebuilds.</li>
<li>My personal way of doing things (I think it is in the video) is to use <code>python3 setup.py build</code> (even though PyTorch always helpfully reminds me that I’m doing it wrong) and then using the packages in <code>./build/lib.SOMETHING/</code> (either with <code>(cd build/lib.*/ ; ipython3)</code>  or with <code>PYTHONPATH=./build/lib.SOMETHING python3 test/sometest.py</code>. In this way of working, you <em>cannot</em> run python from the checkout path because it’ll find the <code>torch</code> module there but that won’t find the compiled C extension modules.<br>
The fancy people have venvs and <code>setup.py develop</code> but I never had the ambition for that type of sophistication.</li>
</ul>
<p>Just give us a shout when there is something specific you want input on.</p>
<p>Best regards</p>
<p>Thomas</p>

kl_divergence: <p>Thanks a lot for your support.</p>
How to append n dimension list

Hwarang_Kim: <p>To append the elements to list of 1 dimension,<br>
list.append(somthing) may be okay.<br>
but how can i make the list which has the shape of (3,11,100,100) to (3,16,100,100)<br>
I want to concatenate two list, (3,11,100,100) + (3,5,100,100)</p>
<p>Is there any functions for doing this kind of work? or do i have to make my own code?</p>

peter: <p>You can use <strong>torch.cat</strong>.</p>
<p>So in your example torch.cat([a,b], dim=1) should fo the trick.</p>
Image segmentation on autoencoder output

Topsoil: <p>I have an autoencoder NN:</p>
<pre><code class="lang-auto">class Autoencoder(nn.Module):
  def __init__(self):
    super(Autoencoder, self).__init__()

    self.Encoder = nn.Sequential(
        nn.Conv2d(3,64,kernel_size = (8,8),padding = 2,padding_mode = 'replicate',bias = True),
        nn.LeakyReLU(0.1),
        nn.MaxPool2d(kernel_size = (2,2)),
        ###Now 128*128*64
        nn.Conv2d(64,64,kernel_size = (5,5),padding = 1,padding_mode = 'replicate',bias = True),
        nn.LeakyReLU(0.1),
        nn.MaxPool2d(kernel_size = (2,2)),
        ###Now 64*64*32
        nn.Conv2d(64,32,kernel_size = (3,3),padding = 2,padding_mode = 'replicate',bias = True),
        nn.LeakyReLU(0.1),
        nn.MaxPool2d(kernel_size = (2,2)),
        ###Now 32*32*16
        #nn.Conv2d(16,16,kernel_size = (10,10),padding_mode = 'same',bias = True),
        #nn.LeakyReLU(0.1),
        #nn.MaxPool2d(kernel_size = (2,2)))
    )
        ###Now 16*16*16
    
    self.Decoder = nn.Sequential(
        #nn.ConvTranspose2d(16,16,kernel_size = (3,3)),
        #nn.LeakyReLU(0.1),
        nn.ConvTranspose2d(32,64,kernel_size = (3,3),stride = 2, padding = 1),
        nn.LeakyReLU(0.1),
        nn.ConvTranspose2d(64,64,kernel_size = (6,6),stride = 2,padding = 1),
        nn.LeakyReLU(0.1),
        nn.ConvTranspose2d(64,3,kernel_size = (8,8), stride = 2, padding = 3),
        nn.Tanh())
    
  def forward(self,x,corruption):
      x = self.Encoder(x)
      #print('Encoder output: ',x.shape)
      if x.shape == corruption.shape:
        x = x * corruption
      else:
        corruption = corruption[0:len(x)]
        x = x * corruption
      x = self.Decoder(x)
      #print('Decoder output: ',x.shape)
      return x
</code></pre>
<p>But i have a problem, where output images of 256<em>256</em>3 are segmented into 3x3 grid even after some learning (it also appeared on simplier models)<br>
looking at results after first epoch, i can make a theory on what happens, but i still cant understand how to fix it:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/5933edfc13f64054dec69d38fc1970f7407eac9d.jpeg" alt="epoch 1" data-base62-sha1="cJ7KIMbO9pDyLgpUbeXGiySpXIN" width="598" height="288"></p>
<p>It looks like something splits images into top-middle-bottom fashion and then replicates it three times, but what is doing it, i am not sure</p>
<p>Full code available here:<br>
<a href="https://github.com/Grubzerusernameisavailable/autoencoder/blob/master/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/Grubzerusernameisavailable/autoencoder/blob/master/</a></p>

JuanFMontesinos: <p>is it possible you are applying some kind of reshaping? that can create that effect. Besides corruption is a bit suspicious too. Is it related to the original image?</p>
<p>It’s not related to the model but some sort of bug in the data nor reshaping.</p>
Issue tensor dimension

Levis: <p>Hi,</p>
<p>I have an annoying issue that I can’t handle.</p>
<p>Here is a reproducible code in colab :</p>
<p><a href="https://colab.research.google.com/drive/1LixjriTogEgCs_zPRnNgoXb-jxEBOuYw" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1LixjriTogEgCs_zPRnNgoXb-jxEBOuYw</a></p>
<p>It appears in the Running train-loop session,</p>
<p>First of all I have this line :</p>
<p><strong>1/5 * Epoch (train): 0% 0/1</strong>  , which is quite weird, something like 0/1.</p>
<p>Then I have the classical issue :  <strong>The size of tensor a must match the size of tensor b at non-singleton dimension 4</strong> , that I can’t understand as in pytorch everything is moreor less hidden, I don’t know where it could come from.</p>
<p>I’m quite new in Pytorch, may you have an idea ?</p>
<p>Thanks</p>

kl_divergence: <p>I didn’t have a look at the notebook. But this is a common error. You should check the input dimensions of the data and the dimension of the input your network expects. A good way to debug this is to print the dimensions of the tensor as training progresses, if the dimension change in the way, you’ll know what to do.</p>

Levis: <p>Yes, I think this is a dimension tensor problem, but as I’m very new I have no idea how to do it, and then if there is a change, how to reshape, but very kind of you, thanks a lot</p>

JuanFMontesinos: <p>Could you just paste a minimal snipped with tensors generated by torch.random(…) ? So that we can easily run it</p>

kl_divergence: <p>Just post a snippet regarding your problem here. That’d be helpful.</p>

Levis: <p>Everything is in colab, I think this is a reproducible exemple/error, I can’t see what are you expecting, sorry.</p>

JuanFMontesinos: <p>Indeed, but it’s not the same debugging in colab than debugging in a local machine with a proper IDE.<br>
It’s also a waste of time to keep reading code which is not related to the problem, to wait for colab to initialize and download packages and a long list of etceteras.</p>
<p>If there’s a problem in a network, the simplest case is to provide something like:</p>
<pre><code class="lang-auto">class Mynet(nn.Module):
  init...
  forward(x):
  ...

model= Mynet(...)
output =model(torch.rand(A,B,C))
</code></pre>
<p>So that copying those few lines of code we can reproduce it not paying attention to a bigger bunch of code.</p>
How to vectorize rotation of 3D points?

lui.bonati: <p>Dear all,<br>
I have a matrix (B, 3N), where the rows are the elements in the batch and the columns correspond to N points in a 3D space, like (x1,y1,z1,x2,y2,z2,…,xN,yN,zN). I can reshape it to be (B,N,3).</p>
<p>Now I want to apply a rotation to the dataset, projecting each 3D point on three versors e_x, e_y, e_z. Each of them is a matrix of shape (B,3) since the reference frame is different for each element in the batch.</p>
<p>I have implemented this with a for-loop over the batches and each atom, in the following way:</p>
<pre><code class="lang-auto"># x.shape =&gt; (B,N,3)
# e_x.shape =&gt; (B,3) {also e_y and e_x}

x_new = torch.zeros((B,N,3))

for b in range(B): # loop over batch
   for j in range(N): # loop over atoms
      x_new[b,j,0]=torch.dot(x[b,j,:], e_x[b,:]) # x component of atom j in the new reference frame 
      x_new[b,j,1]=torch.dot(x[b,j,:], e_y[b,:]) # y component
      x_new[b,j,2]=torch.dot(x[b,j,:], e_z[b,:]) # z component
</code></pre>
<p>but of course it is really slow. Do you have any suggestion about how to vectorize this calculation avoiding the for loops? Should I concatenate e_x, e_y, e_z in a batch of matrices and then apply it to the data?</p>
<p>Thank you very much,<br>
Luigi</p>

JuanFMontesinos: <p>Soz, version 2 just to avoid doing a for loop over each dimension:</p>
<pre><code class="lang-auto">import torch
# x.shape =&gt; (B,N,3)
# e_x.shape =&gt; (B,3) {also e_y and e_x}
B=2
N=4
x=torch.rand(B,N,3)
x_custom=x.unsqueeze(3).expand(B,N,3,3)
e_x=torch.rand(B,3)
e_y=torch.rand(B,3)
e_z=torch.rand(B,3)

x_new = torch.zeros((B,N,3))

for b in range(B): # loop over batch
   for j in range(N): # loop over atoms
      x_new[b,j,0]=torch.dot(x[b,j,:], e_x[b,:]) # x component of atom j in the new reference frame
      x_new[b,j,1]=torch.dot(x[b,j,:], e_y[b,:]) # y component
      x_new[b,j,2]=torch.dot(x[b,j,:], e_z[b,:]) # z component


einsum = torch.einsum('bijk,bjk-&gt;bik',x_custom,torch.stack([e_x,e_y,e_z],dim=2))

print(torch.norm((x_new-einsum)))
</code></pre>
Multi-processing training, GPU0 has more memory usage

zeal: <p>I use the torch.distributed.launch module to multi-processing my training program. Everything seems fine but I don’t know why some process in 1-N gpu will has another memory usage in GPU 0.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b8ccb1511cb8e9b83c89d38a2502250b52b6ccc6.png" alt="image" data-base62-sha1="qmOpcHoK1QrAQrXYGKJC0ga0fsy" width="179" height="295"><br>
As depicted in the picture, the process in gpu4,6 have something in gpu0, this two usage are about 700+M memory. And sometimes other processes will also have similar behavior, but not all the other process will have memory usage in gpu0.</p>
<p>I don’t know why this thing happen? Since the memory unbalances, the training sometimes will be close due to 'out of memory error.</p>

pietern: <p>I agree this can be annoying. As you see not all processes initialize this context. Is there perhaps some path in your code that conditionally initialized some memory, or sets the CUDA device manually? We don’t have any facilities that I know of to point to the culprit here, aside from simply looking at code.</p>

11153: <p>Maybe, it’s because that you doesn’t set your device to load when use distributeddataparallel.</p>
<pre><code class="lang-auto">loc = 'cuda:{}'.format(args.gpu)
checkpoint = torch.load(SAVE_PATH, map_location=loc)
</code></pre>
<p>adding map_location option in your main_worker will solve your problem.</p>
PyTorch Build from Source - Non-zero exit status 1

yamand16: <p>Hello everyone,</p>
<p>I am trying to use Tesla K20M. Since it has 3.x compute capability, I have to install PyTorch from source. I am following instructions on [1]. I installed Cuda 10.0 and Cudnn 7.6 with Anaconda. When I tried to run “python setup.py install” command, I got an error.</p>
<p>– Configuring incomplete, errors occurred!<br>
See also “…/pytorch/build/CMakeFiles/CMakeOutput.log”.<br>
See also “…/pytorch/build/CMakeFiles/CMakeError.log”.<br>
Traceback (most recent call last):<br>
File “setup.py”, line 737, in <br>
build_deps()<br>
File “setup.py”, line 316, in build_deps<br>
cmake=cmake)<br>
File “…/pytorch/tools/build_pytorch_libs.py”, line 59, in build_caffe2<br>
rerun_cmake)<br>
File “…/pytorch/tools/setup_helpers/cmake.py”, line 325, in generate<br>
self.run(args, env=my_env)<br>
File “…/pytorch/tools/setup_helpers/cmake.py”, line 141, in run<br>
check_call(command, cwd=self.build_dir, env=env)<br>
File “…/miniconda3/envs/conda_pytorch/lib/python3.7/subprocess.py”, line 363, in check_call<br>
raise CalledProcessError(retcode, cmd)<br>
subprocess.CalledProcessError: Command ‘[‘cmake’, ‘-GNinja’, ‘-DBUILD_PYTHON=True’, ‘-DBUILD_TEST=True’, ‘-DCMAKE_BUILD_TYPE=Release’, ‘-DCMAKE_INSTALL_PREFIX=…/pytorch/torch’, ‘-DCMAKE_PREFIX_PATH=…/miniconda3/envs/conda_pytorch’, ‘-DNUMPY_INCLUDE_DIR=…/miniconda3/envs/conda_pytorch/lib/python3.7/site-packages/numpy/core/include’, ‘-DPYTHON_EXECUTABLE=…/miniconda3/envs/conda_pytorch/bin/python’, ‘-DPYTHON_INCLUDE_DIR=…/miniconda3/envs/conda_pytorch/include/python3.7m’, ‘-DPYTHON_LIBRARY=…/miniconda3/envs/conda_pytorch/lib/libpython3.7m.so.1.0’, ‘-DTORCH_BUILD_VERSION=1.5.0a0+3655975’, ‘-DUSE_NUMPY=True’, ‘…/pytorch’]’ returned non-zero exit status 1.</p>
<p>Since I am using the server in my company, I can only install something in my virtual environment. Besides, the GCC version of the system is 4.8.4.</p>
<p>Could you please give me any suggestions? Thank you.</p>
<p>Best.</p>
<p>[1] <a href="https://github.com/pytorch/pytorch#from-source" rel="nofollow noopener">https://github.com/pytorch/pytorch#from-source</a></p>

ptrblck: <aside class="quote no-group" data-username="yamand16" data-post="1" data-topic="69396">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yamand16/48/23423_2.png" class="avatar"> yamand16:</div>
<blockquote>
<p>I installed Cuda 10.0 and Cudnn 7.6 with Anaconda</p>
</blockquote>
</aside>
<p>To build PyTorch from source with CUDA support, you would need a local CUDA and cudnn installation.<br>
The runtime from conda shouldn’t be sufficient to build.</p>

yamand16: <p>Thank you for the response. Unfortunately I cannot install CUDA and cudnn to out of the virtual environment. Because it is not my personal computer. So, is there any other way to install PyTorch with CUDA support on this system?</p>

ptrblck: <p>You could use docker and derive the base container from one with CUDA.</p>
C++ 3rd party library for optimizer

pytorch_user13: <p>Hi all,</p>
<p>I am using C++ version of pytorch.<br>
I would like to use 3rd party optimizer for training.<br>
Is this possible to achieve? Did anybody attempt that? Is there any example?</p>

Lornatang: <p>you can try NVIDIA apex tool</p>

ptrblck: <p>What kind of 3rd party implementation are you looking for?<br>
<code>apex</code> works using the Python frontend and is used for mixed precision training.</p>

pytorch_user13: <p>I have several methods for constrained minimization implemented in pure c++ and I would like to use them for training DNNs.<br>
I was hoping that there is easy way to access raw pytorch data (trainable parameters, gradients), so I could just hook up solver.<br>
Can something like this be done?</p>
Will torch.jit.script change the output of model?

BobChen: <p>I have converted a model to scipt module using <code>torch.jit.scipt</code>. However, I find there is some difference between the output of the original model and the output of the scripted model like:</p>
<pre><code class="lang-auto">model.eval()
scripted_model = torch.jit.script(model)

# there is some difference between output and scripted_output
output = model(inputs)
scripted_output = scripted_model(inputs)
</code></pre>
<p>Each element has an average difference of 1e-6</p>
<p>Is this phenomenon normal?</p>

ptrblck: <p>Yes, this small difference is expected due to the limited precision for float32.</p>

BobChen: <p>Thans for your fast reply:)</p>
USPS normalization

tux: <p>Hello everyone,</p>
<p>Any lead on how to normalize the USPS dataset ? I am using <code>transforms.Normalize((0.5,), (0.5,))</code> but I am not fully convinced it is the right normalization.</p>

ptrblck: <p>What is your idea and why are you not convinced? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
What exactly is a tensor?

BroccoliFever: <p>I am new to machine learning, and I am taking an online class to develop an AI. I would like to understand how it all works, rather than just copying the code so I can implement my own AIs in the future, but my first question is what exactly is a tensor? From what I can find online, it is basically a multi-dimensional array.</p>
<p>For instance, a tensor with size [2, 2, 2] means that there is there is a 3-d array with size 2x2x2. If I were to extend that to [2, 2, 2, 2], each dimension of the 3-d object would have an additional dimension. Is that correct? So a [2, 2, 2, 2] array would hold 16 objects (2x2x2x2 = 16)?</p>
<p>If that is the case, what is the benefit of using a tensor object versus simply a numpy array? Can a tensor hold additional data or are there things that can be done on a tensor that cannot be done in a numpy array?</p>
<p>Thanks to all who aid me in the beginning of my adventure into machine learning!</p>

richard: <p>A tensor of size (2, 2, 2, 2) is four-dimensional. Yes, it can hold 2 * 2 * 2 * 2 = 16 elements.</p>
<p>A tensor is a generic n-dimensional array, very similar to numpy arrays. The nice thing about using pytorch tensors in general is that you can easily put them on the gpu (as opposed to numpy arrays, which only exist on the cpu).</p>

BroccoliFever: <p>Awesome, thank you so much for the quick response! A question related to size then:</p>
<p>torch.unsqueeze(tensor, 0) adds a 1 dimensional object to the beginning of the tensor?</p>
<p>So tensor_test = (2, 2, 2)<br>
tensor_test.unsqueeze(0) then makes tensor_test = (1, 2, 2, 2)?</p>

richard: <p>Yes, torch.unsqueeze(tensor, 0) <a href="http://pytorch.org/docs/master/torch.html" rel="nofollow noopener">adds a dimension</a> to the beginning of the tensor.</p>

junaidtariq_tariq: <p>Dear Richard sir,<br>
can we use tensors for extracting visual features as i have training and testing data in tensors now i want to extract the visual features from them so please is it possible and how can i do it?</p>

junaidtariq_tariq: <p>Am working on movielens dataset of 10M and i have its data testing training in tenors form (numpy ) file can we extract visual features from the tensors.</p>

ptrblck: <p>Could you give an example of a visual feature, please?</p>
How to choose number of epochs

Ibrahim_Banat: <p>How Can i know the the best number of epochs for my model and how could i increase accuracy?<br>
(73)|690x387](upload://mzCTyVnjMycDSnlwYec8Ecld3kC.png)</p>

ptrblck: <p>You could observe the validation loss and use e.g. early stopping.<br>
Also, you could use a threshold to stop the training if the loss decreases less than this threshold.</p>