akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Manual Streaming?

Jordan_Howell: <p>Hello,</p>
<p>I’m having a hard time with a tight deadline trying to get a custom data set to load correctly.  I have thus created a label, numerical column, categorical column and image tensors separably.  However, when I load them into the GPU, I get a memory error.</p>
<p>Is there a way a manually stream batches out of the tensors into the gpu, 10 at a time?</p>

ptrblck: <p>What kind of “memory error” are you getting?<br>
Could you post the error message with the stack trace, please?</p>

Jordan_Howell: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>.  I already  ran a process where I pulled all of the data prep out of the custom data set to include transformation to tensors.</p>
<p>Now my custom data set looks like this:</p>
<pre><code class="lang-auto">class tensor_data(Dataset):
    def __init__(self, num, cats, labels, images):
        self.num = num_tensor
        self.cats = cats
        self.labels = labels
        self.images = images
    
    def __len__(self):
        return len(self.num)
    
    def __getitem__(self, idx):
        return self.num[idx], self.cats[idx], self.labels[idx], self.images[idx]
</code></pre>
<p>That said, it’s also solved the single image per batch issue and it seems to be running faster.  It’s just as easy to define a function to do the prework so I’m happy.  Thank you, as always, for the great support. The community is lucky to have you.</p>
How to initialize zero loss tensor

rajcscw: <p>Hi,<br>
I would like to initialize a zero loss tensor which will be used to accumulate losses dynamically based on some condition. Sometimes, there will not be any loss accumulation and in this case, when I backpropagate, it throws the following run time error</p>
<p><code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code></p>
<p>Sample code looks like this:</p>
<pre><code class="lang-python">import torch

# some inputs
x = torch.rand(10, 3)

# some targets
y = torch.rand(10, 1)

# some parameter
W = torch.nn.Parameter(torch.rand(3, 1))

# total loss
total_loss = torch.tensor([0.0])

# some condition
if torch.rand(1) &gt; 0.5:
    loss = torch.sum((torch.matmul(x, W) - y) ** 2)
    total_loss += loss
total_loss.backward()
</code></pre>
<p>So, the question is how to initialize this zero loss tensor for handling this?</p>

albanD: <p>Hi,</p>
<p>I guess you have two options:<br>
either you do the backward only when it happened:</p>
<pre><code class="lang-auto">total_loss = 0.
# You code
if torch.is_tensor(total_loss):
    total_loss.backward()
</code></pre>
<p>Or create it with requires grad all the time.</p>
<pre><code class="lang-auto">total_loss = torch.tensor([0.0], requires_grad=True)
# You code
total_loss.backward() # This will always work now.
</code></pre>

rajcscw: <p>I tried your second solution, it throws this run time error.</p>
<pre><code class="lang-python">total_loss += loss
RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
</code></pre>

albanD: <p>Ho right you can’t use it inplace… The first one is definitely best.<br>
Or use you original code and check that <code>total_loss.requires_grad</code> before calling backward on it.</p>

rajcscw: <p>Thanks, I will go with this solution then.</p>
Newbie Question: dcgan_faces_tutorial variable adaptation

Quadwalter: <p>Hello,<br>
sorry for the newbie question but: I am trying to learn pytorch by example with: <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a></p>
<p>Having successfully executed dcgan_faces_tutorial.ipynb with the provided 64x64 celeba dataset, i now try to adapt it to the 250x250 lfw dataset for more interesting results, adapting image_size, ngf and ndf to 250.</p>
<p>Dataset loading, Generator and Discriminator construction work out fine. However the training loop exits on:</p>
<p><em>ValueError                                Traceback (most recent call last)</em><br>
<em> in </em></p>
<ul>
<li>
<pre><code>27         output = netD(real_cpu).view(-1)*
</code></pre>
</li>
<li>
<pre><code>28         # Calculate loss on all-real batch*
</code></pre>
</li>
</ul>
<p><em>—&gt; 29         errD_real = criterion(output, label)</em></p>
<ul>
<li>
<pre><code>30         # Calculate gradients for D in backward pass*
</code></pre>
</li>
<li>
<pre><code>31         errD_real.backward()*
</code></pre>
</li>
</ul>
<p><em>…</em><br>
<em>ValueError: Target and input must have the same number of elements. target nelement (128) != input nelement (18432)</em></p>
<p>Aparently, some other variables need adapting (batch size? latent vector?). Is there a formula for variable adaptation?</p>
<p>Any help would be greatly appreciated!!</p>
<p>Walter</p>

ptrblck: <p>Based on the error message it seems that <code>netD</code> is not returning a probability for each sample in the batch (which would have the shape <code>[batch_size=128]</code>), but instead an output of <code>[batch_size=128, 1, 12, 12]</code> due to the increased spatial size. After the <code>view(-1)</code> operation this would create a tensor of <code>128*12*12=18432</code>.<br>
You would have to make sure the output is a single pixel, e.g. by adding more layers (pooling, strided convs etc.).</p>
Converting NN outputs to Long in custom Loss function

K_B: <p>Hi,<br>
I wanted to ask that is it possible to convert an output of a NN to a long dtype in the custom loss function and still get a non-zero gradient?<br>
As per my understanding, any of the operations like floor/ceil will give me a zero gradient.</p>

ptrblck: <p>This won’t work, as Autograd cannot differentiate through <code>LongTensors</code> and you would therefore detach the computation graph as seen here:</p>
<pre><code class="lang-python">x = torch.randn(1, requires_grad=True)
y = x * 1
print(y.grad_fn)
&gt; &lt;MulBackward0 object at 0x000001D991AB65E0&gt;

z = y.long()
print(z.grad_fn)
&gt; None
</code></pre>
<p>You could use <code>round</code>, <code>ceil</code> or <code>floor</code> instead.</p>
CRNN prediction accuracy

Zrufy: <p>I’m working with the crnn and a doubt came to me.<br>
How can I have the value of accuracy when predicting an image never seen.<br>
Let me explain better: the network gives me the right text but how do I know how accurately the text is right?</p>

ptrblck: <p>You won’t be able to calculate the accuracy automatically without targets.<br>
Of course you (personally) could have a look at the output and determine, how well your model predicts, but usually you would run a final test of the model using an unseen testset with targets to get an estimation of the “real world performance”.</p>

Zrufy: <p>sorry for late i solve with my softmax function to retrieve the exact value!</p>
How can I print the image of each layer in CNN?

Yolkandwhite: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/8/78d73a367ece71d1adcd82258c7c41ed3bba41db.png" alt="image" data-base62-sha1="hf0pYHWswSp4qCN87JBgfmGq73R" width="580" height="326"><br>
I want to print the output in image of each layer just like picture above<br>
how can I do it??</p>
<pre><code>class CNN(nn.Module):

    def __init__(self):
        super(CNN, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size = 3, stride = 1, padding = 1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size = 2, stride = 2))
    
        self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size = 3, stride = 1, padding =1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size = 2, stride = 2))
    
        self.fc = nn.Linear(7 * 7 * 64, 10, bias = True) #0~9 10개
        torch.nn.init.xavier_uniform(self.fc.weight)
    
    def forward(self, x):
        out = self.layer1(x)
        imgprt = out.numpy()
        imgplot = plt.imshow(imgprt)
        out = torch.from_numpy(out)
        out = self.layer2(out)
        out = out.view(out.size(0), -1)
        out = self.fc(out)
        return out


RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.
</code></pre>
<p>i tried to change the data type in numpy and use plt.imshow() in forward but didn’t work.</p>

ptrblck: <p>Was the suggested workaround also not working? (<code>Use var.detach().numpy() instead.</code>)</p>
<p>For a more general approach, I would recommend to store the wanted activation in a <code>list</code> or <code>dict</code> by using forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>

111346: <p>The image is really awesome, would you please share where you get the original picture?<br>
thank you very much !</p>

Yolkandwhite: <p>I just copied image from somewhere.</p>
<p>I forgot where I copied it from. It was quite a long time.</p>
<p>but you can try right click on that image and search image in google.<br>
(If you are using google chrome browser)</p>
Training loss graph

Rahul_pillai: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/9394c52752b5b2e9e4f5e2940eaf59cd390b95fc.png" alt="download" data-base62-sha1="l3yVscI8XLxmxvtIVjFL8NKubmI" width="595" height="411"></p>
<p>Every time i train ,my training loss just spikes during the 2nd epoch and drops hard,<br>
i want to know why is that, is it a bad thing?</p>

ptrblck: <p>Could have various reasons, such as a high learning rate, which could cause some divergence at the beginning of the epoch, and later on let your model converge again (as your validation loss is low).<br>
You could check the predictions during training and check the loss for each batch to isolate it further e.g. to a “bad” sample.</p>
Making a list of layers

887574002: <p>Hi, I want to make a list of layers for a given number n in a given loop on j iterating over n and apply them on input x as a whole. Something that I am looking for is like:</p>
<pre><code class="lang-auto">layers = ......

for j in range(n):

   layers = ...
  output = layers(x) 
</code></pre>
<p>Precisely, for j =1 I want to have only one FC layer and apply it on x, for j=2 I want to have two FC and apply them on x and etc. How can I do that?</p>
<p>I looked at <code>MyModule</code> class, but it doesn’t help me since the size of out_put would change. I would be thankful if you guide me.</p>
Question on next(iter(dataloader))

ooodragon: <p>Hello peop!<br>
So I know the following code<br>
<code>next(iter(dataloader))</code><br>
makes dataloader object to be a iterable and “next” helps it to iterate over it.<br>
I thought this code will give me a size of (batch_num, channel_num, image_size1, image_size2)<br>
but this isn’t true, instead next(iter(dataloader))[0] (or fixing it with any index) gives me what I thought it should…</p>
<p>is there any answer for this?</p>

ptrblck: <p>The <code>dataloader</code> object might return multiple values as it’s calling into <code>Dataset.__getitem__</code> to create the batch.<br>
For a “standard” classification use case, your would return a data and target batch, so the first output would correspond to your desired shape.</p>

ooodragon: <p>thanks for the answer!<br>
hm… so it’s getting multiple batches??<br>
(In my case I think I specified 8 samples per batch and <strong>getitem</strong> gave me len() of 3 batches)<br>
I’m still a little confused<br>
you mind clarifying a bit?</p>
<p>thank you so much</p>

ptrblck: <aside class="quote no-group" data-username="ooodragon" data-post="3" data-topic="86639">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ooodragon/48/24206_2.png" class="avatar"> ooodragon:</div>
<blockquote>
<p>hm… so it’s getting multiple batches??</p>
</blockquote>
</aside>
<p>If your <code>Dataset.__getitem__</code> returns multiple tensors, <code>next(iter(loader))</code> will return a batch for each output.<br>
For the “standard” use case you would thus get a batch of inputs and a batch of targets.</p>
<aside class="quote no-group" data-username="ooodragon" data-post="3" data-topic="86639">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ooodragon/48/24206_2.png" class="avatar"> ooodragon:</div>
<blockquote>
<p>In my case I think I specified 8 samples per batch and <strong>getitem</strong> gave me len() of 3 batches</p>
</blockquote>
</aside>
<p>Are you calling <code>__getitem__</code> manually or how did you use the <code>len()</code> operation on it?<br>
If <code>next(iter(loader))</code> returned a batch of shape 3 even though you specified <code>batch_size=5</code>, then your <code>Dataset</code> might not have enough samples.<br>
In that case you should check how the data is loaded and make sure you are able to load more than 3 samples.</p>
Implementing Squeeze and Excitation block with pretrained resnet

sgaur: <p>Hi<br>
I am trying to implement SE block with any pretrained resnet architecture.</p>
<pre><code class="lang-auto">self.encoder = torchvision.models.resnet34(pretrained=pretrained)

self.conv1 = nn.Sequential(
    self.encoder.conv1,
    self.encoder.bn1,
    self.encoder.relu
    SeBlock(64,2),
    nn.MaxPool2d(2, 2) )

self.conv2 = nn.Sequential(
    self.encoder.layer1,
    SeBlock(64,2),
    )

self.conv3 = nn.Sequential(
    self.encoder.layer2,
    SeBlock(128,2),
    )

self.conv4 = nn.Sequential(
    self.encoder.layer3,
    SeBlock(256,2),
    )

self.conv5 = nn.Sequential(
    self.encoder.layer4,
    SeBlock(512,2),
    )
</code></pre>
<p>Above is my code but It is not working out. Looking into githubs they build resnet from scratch and induce SE block and then load model.state_dict() weights for the layers of resnet and train the remaining model.<br>
I just need to know what is the correct procedure to use SE block with pretrained resnet.?</p>
<p>Thanks.!!</p>

chetan_patil: <p>Can you paste the trace ?</p>
Values of outputs keep growing

fsh: <p>Hi all,</p>
<p>In a simple network with a customized loss function, the outputs of the network keep growing during training. The loss function decreases in a few epochs, then stops in a constant value.</p>
<p>The data are normalized and I have used batch normalization and gradient clipping to try to keep the values limited. I also tried with smaller learning rate with SGD and Adam optimizers. But again the same problem.</p>
<p>Any idea what could be the problem?</p>

ptrblck: <p>What kind of loss function are you using?<br>
It seems that the loss value might not reflect the “difference” between the model output and the target, since the loss is constant, while the output is growing?</p>
<p>Based on this, I guess the loss function implementation might be buggy or “unexpected” behavior.</p>

fsh: <p>The loss function is a little complicated, and you were right, the implementation was incorrect.</p>
Slicing tensor using boolean list

dpernes: <p>I have a boolean Python list that I’d like to use as a “mask” for a tensor (of the same size as the list), returning the entries of the tensor where the list is true.</p>
<p>For instance, given the list <code>mask = [True, False, True]</code> and the tensor <code>x = Tensor([1, 2, 3])</code>, I would like to get the tensor <code>y = Tensor([1, 3])</code>. In numpy, this would be simply <code>y = x[mask]</code>, but in PyTorch indexing tensors with lists is not (yet?) supported.</p>
<p>Moreover, I need an efficient implementation for this slicing, since this would be performed in every forward pass of my model. What do you suggest?</p>
<p>Thank you in advance.</p>

dpernes: <p>Well, currently my solution is:</p>
<pre><code>idx = np.argwhere(np.asarray(mask))
y = x[idx]
</code></pre>
<p>Is there any better option?</p>

killeent: <p>Hi Diogo -</p>
<p>We currently have a subset of support for numpy indexing but what I believe you are referring to is Boolean array indexing as described here: <a href="https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#boolean-array-indexing" rel="nofollow noopener">https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#boolean-array-indexing</a>.</p>
<p>The closest PyTorch equivalent is masked select, as described here: <a href="http://pytorch.org/docs/master/torch.html?highlight=masked_select#torch.masked_select" rel="nofollow noopener">http://pytorch.org/docs/master/torch.html?highlight=masked_select#torch.masked_select</a>. If you could instead generate your mask in the ByteTensor format described there, you could achieve what you want.</p>

dpernes: <p>Thanks, <a class="mention" href="/u/killeent">@killeent</a>, this perfectly does the job!</p>

goldengrape: <pre><code class="lang-python">y=torch.arange(0,3)
x=torch.Tensor([True,False,True])==True
print(y[x])
</code></pre>
<p>this works</p>

tengerye: <p>This solution is better. Assume you have a two dimensional tensor <code>y=torch.range(1, 8).reshape(2, 4)</code>, and <code>x=torch.Tensor([True,False])==True</code>. In this case, you can’t use <code>torch.masked_select</code>, but you can easily execute <code>y[x, :]</code>.</p>
[Transformer] how to set the src_mask and memory_mask In generation task?

jkwang93: <p>I try to build a transformer to generate text, my src_input is sentence, tgt_input is +sentence, tgt_output is sentence+, masks is the default setting.  But when I training, the loss is very low (about 0.0001, I have trained same dataset using RNN, the loss is 0.1),  this results in the generated text being almost unique (no matter how I adjust the temperature sampling value).<br>
Is there something wrong with my masks, should I set my src_mask and memory_mask?</p>
<p>thanks！</p>
Using pytorch from Cython with nogil for multithreading

Joseph_Gardi: <p><strong>Goal</strong>: I am interested in doing multithreading with Pytorch and would prefer to use Cython rather than C++ if possible. I would like to take advantage of nogil (no global interpreter lock) in Cython which allows for multithreading in parallel rather than just concurrently.</p>
<p><strong>Problem</strong>: The Cython compiler does not allow calls to the Pytorch Python frontend when the gil is released. I think the solution to this would be to call the C++ frontend for Pytorch directly from Cython because C++ code can be called when the gil is released.<br>
<strong>Question</strong>: Is using Pytorch with nogil in Cython feasible and reasonable? If so could someone point me to an example?<br>
<strong>How I got here</strong>: I tried using torch.multiprocessing but I discovered that does not work for my use case because I got this error:<br>
RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries. If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).</p>
<p>This is because I wanted to evaluate my neural network on the main process and then compute the loss function (which is computationally expensive in my case) on several different batches.</p>
Quantized model consists of ReLU6

thancaocuong: <p>Hi, Can I quantize model which consists of ReLU6 activation function?</p>

pshashk: <p>Yes you can. ReLU6 was added to DEFAULT_MODULE_MAPPING. See <a href="https://discuss.pytorch.org/t/quantized-hard-sigmoid/59013" class="inline-onebox">Quantized hard sigmoid</a></p>

thancaocuong: <p>hi <a class="mention" href="/u/pshashk">@pshashk</a>, I did tried to fuse model with ReLU6 activation function. It throws an error. I see in pytorch source code that fuse modules currently support 4 types sequence of modules:</p>
<pre><code class="lang-auto">  Fuses only the following sequence of modules:
    conv, bn
    conv, bn, relu
    conv, relu
    linear, relu
</code></pre>
<p><a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/fuse_modules.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/quantization/fuse_modules.py</a></p>

raghuramank100: <p>That is correct, we will work on adding support for fusing relu6 soon. For now, if you are doing post training quantization, you could replace relu6 with relu and proceed as a work around.<br>
Thanks,</p>

thancaocuong: <p>thank you. Hope you release it soon</p>

Raghav_Gurbaxani: <p><a class="mention" href="/u/raghuramank100">@raghuramank100</a> can you provide an example of how to replace relu6 with relu ? I am trying to quantize a network with relu6 activations.</p>
Create 2D axial slices from 3D vol of medical image

CS.Enthu: <p>Hello everyone.<br>
I have a 3D volume of medical data storred as numpy array of size [284,143,143]. Now I want to slice the images axially and save them in tiff format. How can I acheive that. Can someone please give an example.</p>
<p>Thanks</p>

ptrblck: <p>You could iterate the tensor, index each sample, transform it to <code> PIL.Image</code> and store it.<br>
Here is a small code snippet:</p>
<pre><code class="lang-python">import torchvision.transforms.functional as TF

x = torch.randn(5, 24, 24)
for idx in range(x.size(0)):
    tmp = x[idx]
    img = TF.to_pil_image(tmp)
    img.save('img{}.tiff'.format(idx))
</code></pre>

CS.Enthu: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Is multiprocess Dataloader for IterableDataset keeping the order of original interable?

Emerald01: <p>Hi,</p>
<p>I have a question here.</p>
<p>I derive the IterableDataset so it can yield my data. In particular, my input data is a list that I care for the order because later I need to make sure that my output still follows the same order as input. Although more complicated than this, essentially it can be thought as  [1,2,3,4…]</p>
<p>Then I use Dataloader with multiple workers like the following</p>
<pre><code class="lang-auto">loader = DataLoader(iterable_dataset, batch_size=256, num_workers=2, worker_init_fn=worker_init_fn)
for batch in loader:
    print(batch)
</code></pre>
<p>However, I find the batch is strictly following the original order of the iterable [1,2,3,4]. Even I on purpose delay the first worker which is outputting 1 and 2, and the second worker for 3 and 4, this for loop is still yielding data in the original order, i.e., 1, 2, 3, 4. That makes me believe that the async DataLoader here, although they are processing data in parallel (I timing the worker process right before each yield, and I am pretty sure both workers start to work separately at time 0), they will communicate to always keep the original order of the dataset. For example, if 2,3,4 are all ready at the earlier time but they will be blocked and wait for 1 to finish and finally only allow the yield order to be 1,2,3,4</p>
<p>Is this an expected behavior? It looks true to me and I feel it is sub-optimal as it is not a real queue. If this is the case, I am okay but I am curious which line of source code takes care of that, if it is not, could you please hint if I have some misunderstanding or code error so it looks like this?</p>
<p>Thank you so much!</p>

mrshenli: <p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> for dataloader questions</p>
PyTorch Transforms on images/videos with more than 3 channels

HaziqRazali: <p>Is there an easy way to use PyTorch’s transforms on images/videos with more than 3 channels? Right now I have a video with the following shape <code>(60, 170, 128, 40)</code> i.e. there are 60 frames in the video and each frame has 40 channels to it. It seems like PIL is unable to work with anything that has more than 3 channels.</p>
<p>I currently only need the center crop function and so can directly crop the numpy array but was just wondering if there is a way to use the torchvision transform functions on images that contain more than 3 channels or if I have to script everything.</p>

Nikronic: <p>Hi,</p>
<p>Have you tried <a href="https://kornia.readthedocs.io/" rel="nofollow noopener">Kornia</a> which supports many different augmentations on both images and tensors.</p>
<p>For instance, your case be achieved this way:</p>
<pre><code class="lang-auto">import kornia as K

x = torch.randint(0, 1, (60, 40, 170, 128)).float()
K.center_crop(x, (90, 90)).shape
</code></pre>
<p>Bests</p>

HaziqRazali: <p>Thanks for introducing me to Kornia!</p>
Cross compiling Pytorch C++ extension

linimus: <p>I am using Pytorch C++ extension, and I want to cross-compile (target is arm64) the C++ file instead of <code>python setup.py install</code> in the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">tutorial</a>. Is there any way to do this?</p>
Error while trying to build module using pyBind

chinmay5: <p>I am trying to build a module which uses C++ code for non max suppression and using pyBind in order to use it with PyTorch. However, while trying to import the module, I get the following error.</p>
<pre><code class="lang-auto">ImportError: /home/chinmay/Desktop/setup/temp/lib/python3.6/site-packages/
special_layers-0.1-py3.6-linux-x86_64.egg/nms_bridge_sample.cpython-36m-x86_64-linux-gnu.so:
undefined symbol:
ZN6caffe28TypeMeta21_typeMetaDataInstanceIN3c107complexIfEEEEPKNS_6
detail12TypeMetaDataEv
</code></pre>
<p>I have already created a new virtual environment and installed pyTorch but the error still remains. I tried it on pyTorch 1.5.1 and Ubuntu 18.04</p>
Randomly initialized embeddings for torchtext

Jeremy_Rutman: <p>I’d like to randomly initialize word embeddings - can  I do the following:</p>
<pre><code class="lang-auto">
    TEXT.build_vocab(train_data) 
    vocab_size = len(TEXT.vocab)
    embedding_vectors = torch.FloatTensor(np.random.rand(vocab_size,embedding_length))
    word_embeddings = nn.Embedding(vocab_size, embedding_length)
    word_embeddings.weight = nn.Parameter(embedding_vectors, requires_grad=True)

</code></pre>
<p>to do so ?<br>
I have heard tales of a parameter for build_vocab that allows for this out of the gate but have yet to sight it myself.</p>

ptrblck: <p>This code snippet would assign <code>embedding</code> vectors to the <code>nn.Embedding</code> layer.<br>
Note that <code>nn.Embedding</code> will already randomly initialize the <code>weight</code> parameter, but you can of course reassign it.</p>
<p>You could use <code>torch.from_numpy(np.random.rand(...)).float()</code> to avoid a copy, but your code should also work.</p>
When using torch.nn.functional.conv3d are the filters not repeats of themselves along the second dimension of the tensor?

BossyMicrobe: <p>I am currently trying to utilise the torch.nn.functional.conv3d function as I am trying to implement my own filters. The function calls for the filters to be of shape (out_channels, in_channels, kT, kH, kW).</p>
<p>Say I have a network thats first convolutional layer has 1 input and 8 filters of size 5,5,5 the filter would be a tensor of [8,1,5,5,5]. Then if the second layer has 16 filters it is [16,8,5,5,5]. But from what I understand along the second dimension wouldn’t the filters all be the same.</p>
<p>as to say you were to run the second layer of filters (SecondLayerFilters)</p>
<p>for filters in SecondLayerFilters[0]:<br>
print(filters)</p>
<p>shouldn’t all the filters be the same? Am I miss understanding something</p>
<p>EDIT: I realise I didn’t making my problem that clear. I understand that the out_channels is basically the number of filters in that layer, but I don’t understand what the need for the in_channels. It’s my understand that the process will use the same filters for each in_channel? so are these filters just duplicates or are they all different?</p>

ptrblck: <aside class="quote no-group" data-username="BossyMicrobe" data-post="1" data-topic="87114">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/a5b964/48.png" class="avatar"> BossyMicrobe:</div>
<blockquote>
<p>but I don’t understand what the need for the in_channels. It’s my understand that the process will use the same filters for each in_channel? so are these filters just duplicates or are they all different?</p>
</blockquote>
</aside>
<p>Each filter will use different weights for the input channels, i.e. the <code>weight</code> of <code>[16, 8, 5, 5, 5]</code> will contain 16 filters each with a weight matrix of <code>[8, 5, 5, 5]</code> (with different values). The weight matrix will be multiplied with the input (of the same shape) for each window.</p>
<p><a href="https://cs231n.github.io/convolutional-networks/#conv">CS231n - Convolution</a> explains this quite well for <code>nn.Conv2d</code>.</p>
Num_workers effect

pooya_khandel: <p>I know that setting <code>num_workers &gt; 0</code> in <code>Dataloader</code> may speed training and data access. According to <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">documentation</a> <code>num_workers</code> is about</p>
<blockquote>
<p>how many subprocesses to use for data loading.  <code>0</code>  means that the data will be loaded in the main process.</p>
</blockquote>
<p>Does the loading refer to loading data into the RAM? For instance, suppose that there is enough memory to read the dataset; hence, our data is read from a text file when creating an instance of <code>Dataset</code> class. Now if we use <code>num_workers &gt; 0</code> when defining <code>Dataloader</code>, do we gain any advantage of these workers?</p>

ptrblck: <p>If your data is already loaded, you might get an advantage of using multiple workers for the preprocessing of each sample. If you don’t need to process the samples, but would simply slice the data, you won’t see a huge performance benefit. Just in case, you could nevertheless profile the code and verify it.</p>
Proper way to initialize h_0 c_0

bfeeny: <p>I realize that the default in nn.LSTM is it will automatically initialize h_0 and c_0 states if it has to with zeros.  I am using zeros below as an example.  I often see people initialize like so:</p>
<pre><code class="lang-auto">    def init_hidden(self):
        self.h_0 = torch.zeros(self.num_layers, self.batch_size, self.hidden_size).to(device)
        self.c_0 = torch.zeros(self.num_layers, self.batch_size, self.hidden_size).to(device)
</code></pre>
<p>then call <code>lstm.init_hidden()</code> before they start training.</p>
<p>The issue I have seen with this, is that if you then goto do a prediction (eval) and pass it something less than batch_size, say a single batch, it errors.  Example, lstm was trained with <code>batch_size = 20</code> and now try to pass it a single prediction:</p>
<p><code>RuntimeError: Expected hidden[0] size (3, 1, 128), got (3, 20, 128)</code></p>
<p>I have also seen people initialize these in the <code>forward</code> function like so:</p>
<pre><code class="lang-auto">    def forward(self, x):
         self.h_0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
         self.c_0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
</code></pre>
<p>So this is being called on every forward step, and instead of<code> batch_size</code> it’s using <code>x.size(0)</code>.  But which way is correct?  How do you make sure that if you train with say a <code>batch_size=20</code> that you can pass a single data to get a prediction?</p>
Half vs Full Precision with CUDA

jakemdaly: <p>I am comparing a half precision workload (tensors are torch.HalfTensor) to a full precision workload (tensors are torch.FloatTensor) by using a command line profiling tool from NVIDIA which reports FLOPS.</p>
<p>I noticed that even when running the model with half precision tensors, the tool reports that only single precision operations happened, albeit many less FLOPS than occured with the full precision model.</p>
<p>My suspicion is that there are some bitpacking optimizations made occuring somewhere in the pytorch/cuda library bindings.</p>
<p>I’d like to confirm this, or if thats not right, hear what the explanation is. Thanks!</p>

Scott_Hoang: <p>Are you using gpus that support half precision like the 20xx series nvidia? If not then, the number of glop between half and full should be the same since you still have to use the same number of registers in the gpus.</p>

jakemdaly: <p>Yes it’s a Jetson Tegra X1, so I believe it supports half<br>
Interestingly, I am seeing a similar number of flops when comparing half precision to full precision (actually a few more for half precision than full precision) but ~1/2 the amount of bytes are being fetched from memory</p>

Scott_Hoang: <p>I’m sorry. Gflop is the count the number of operations. half cuda is about the actual size of the data. The number of ops should be the same</p>

jakemdaly: <p>The number of flops I am seeing is a little bit higher on half precision than full precision, and the bytes fetched from memory is half (which makes sense).<br>
My leading theory for the discrepancy (which I am looking for more information about), is that the half precision data are being bit packed onto 32-bit data to be sent on chip. Then there are operations which unpack the data so they can be operated on as 16-bit once they are on chip?</p>

Scott_Hoang: <p>you are seeing higher gflop on half precision because of conversion ops where 32 bit data are converted into 16.</p>

ptrblck: <p>Also answered <a href="https://discuss.pytorch.org/t/tegra-x1-half-precision-data-with-pytorch/87057/2">here</a>.</p>