akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch installs the CPU version no matter what

joshmcorreia: <p>I’ve been trying to install PyTorch with CUDA support enabled, but have been unsuccessful no matter what I try.</p>
<p>GPU: GTX 1080 Ti<br>
OS: Windows 10<br>
Environment: Anaconda<br>
Python: 3.9.8</p>
<p>I installed Anaconda and ran the following command in the Anaconda Prompt:<br>
<code>conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch</code>.<br>
As far as I can tell, this is how I download the version of PyTorch with GPU support.</p>
<p>This is the output of <code>nvidia-smi</code>:<br>
<code>NVIDIA-SMI 496.49       Driver Version: 496.49       CUDA Version: 11.5</code></p>
<p>This is what happens when I try to access PyTorch from within the Python3 interpreter:</p>
<pre><code class="lang-auto">import torch

torch.cuda.is_available()
&gt; False

torch.__version__
&gt; '1.10.0+cpu'

torch.cuda.device_count()
&gt; 0

torch.cuda.init()
&gt; AssertionError: Torch not compiled with CUDA enabled
</code></pre>
<p>It appears that the CPU version is installed, even though I didn’t select the CPU version on the PyTorch website. What can I do to ensure I install the GPU version? Thanks for all your help!</p>

joshmcorreia: <p>I ended up solving my own issue through lots of trial and error. It turns out that <code>python3</code> on my machine is linked to a different python3 environment, not the one that was installed in my conda environment. Launching python with the command <code>python</code> instead of <code>python3</code> fixed the issue. I’ll leave this post up in case anyone else runs into this in the future <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to share weights of dilated convolutional kernels?

tslermen: <p>Hello all,</p>
<p>I would like to know how to share weights of dilated convolutional kernels. For example:</p>
<pre><code class="lang-auto">self.conv1 = nn.Conv2d(in_channels=2,out_channels=16,kernel_size=5, stride=1, padding=2, dilation=1)
self.conv2 = nn.Conv2d(in_channels=2,out_channels=16,kernel_size=5, stride=1, padding=2, dilation=2)
self.conv3 = nn.Conv2d(in_channels=2,out_channels=16,kernel_size=5, stride=1, padding=2, dilation=3)
self.conv4 = nn.Conv2d(in_channels=2,out_channels=16,kernel_size=5, stride=1, padding=2, dilation=4)
</code></pre>
<p>I would like to share weights between those convolutional layers.</p>
<p>Thanks!</p>

JuanFMontesinos: <p>Just define it once and call it as many times as you need.</p>

ayalaa2: <p>Interesting problem!</p>
<p>While I do see some murmurs here and there about the ability to share weights between modules, it seems pretty frown upon.</p>
<p>I believe the better idea would be to use trainable parameters and make a functional call to conv2d using those trainable parameters.</p>
<p>Something like this would probably work for you:</p>
<pre><code class="lang-auto">
import torch
import torch.nn as nn
import torch.nn.functional as F

class Conv(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size):
        super().__init__()

        # Register parameters that are trainable
        self.weight = nn.Parameter(torch.randn(out_channels, in_channels, kernel_size, kernel_size))
        self.bias = nn.Parameter(torch.randn(out_channels))

    def forward(self, x, stride, padding, dilation):
        # Do a functional call so we can use the same weights but different arguments
        return F.conv2d(
            x, self.weight, bias=self.bias, stride=stride,
            padding=padding, dilation=dilation
        )

# Example creation of module
conv = Conv(2, 16, 5)

# Example input
x = torch.randn((8, 2, 16, 16))

# Example usage with different dilation values
y1 = conv(x, 1, 2, 1)
y2 = conv(x, 1, 2, 2)
y3 = conv(x, 1, 2, 3)
print(y1.shape, y2.shape, y3.shape)
</code></pre>
<p>You probably want to consider initializing the weights differently however.</p>

JuanFMontesinos: <p>Hmmm I missed the part where he was using different dilations.<br>
So yep <a class="mention" href="/u/ayalaa2">@ayalaa2</a>  's code is the way to go.</p>

tslermen: <p>Hey! Thanks for the answer! It does make sense! Thank you very much! But I was wondering how the gradients are computed. Is it like a normal convolutional layer?</p>

ayalaa2: <p>Yes, it should be as if it was a typical normal convolution layer. In fact, if you peek at the source code, you’ll notice that the pytorch convolution modules end up just calling their functional counterpart.</p>
Will parameters not for forwarding be updated during backward propagation?

LiuWJ: <p>I have created the following network, where linear is for forwarding and ‘linears’ in modlst are for calculating loss function. The question is, will the parameters in modlst been updated after ‘loss.backward()’ and ‘optimizer.step()’? Thank you for your assistance!</p>
<p>class Net(torch.nn.Module):<br>
def <strong>init</strong>(self,lag_list,in_size,out_size):<br>
super(Net,self).<strong>init</strong>()<br>
self.modlst=torch.nn.ModuleList([])<br>
self.lag_list=lag_list<br>
self.linear=torch.nn.Linear(in_size,out_size,bias=None)<br>
self.l=len(lag_list)<br>
for i in range(self.l):<br>
self.modlst.append(torch.nn.Linear(in_size,out_size,bias=None))<br>
def forward(self,x):<br>
return self.linear(x)</p>

ptrblck: <p>In your current code snippet the <code>forward</code> method only uses <code>self.linear</code>. Since none of the linear layers in <code>self.modlst</code> is used, they won’t receive and gradients and thus won’t be updated.<br>
In case you are using these layers at a later stage, they should be updated as long as you don’t detach the computation graph etc.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

LiuWJ: <p>Thank you for your assistance！My current goal is to train a model whose output satisfies the property of autoregression (time lag might not be continuous), so my loss function hopes to include the loss function between the model output and target, as well as the property of autoregression of the model output itself. So how can I set and train the autoregressive parameter here? I’m quite confused with it…</p>

LiuWJ: <p>Here is the main part of my code</p>
<pre><code class="lang-auto">    def __init__(self, T: int, dim_size: int,lag_list):
        super(MF, self).__init__()
        self.t_embeddings = nn.Embedding(T, dim_size）
        
        self.modlst=torch.nn.ModuleList([])
        self.lag_list=lag_list
        self.l=len(lag_list)
        for i in range(self.l):
             self.modlst.append(torch.nn.Linear(dim_size,dim_size,bias=None))

    def forward(self, times):
         mt = self.t_embeddings(times)
         return(mt)

     def var(self, times):
        tmp = self.t_embeddings(times)
                
        for i in range(max(self.lag_list),tmp.shape[0]):
            for j in range(self.l):
                tmp[i]=tmp[i]+self.modlst[j](tmp[i-self.lag_list[j]])

        return(tmp)
            
def ARLoss(model,T,lag_list):
    times=torch.arange(T).to(device)
    ar_y=model.var(times)
    y=model(times,lag_list)
    loss=torch.linalg.norm(ar_y-y)
    return loss

model=MF(t,lag_list)
loss_func=torch.nn.MSEloss()
loss=loss_func(target,model(times,lag_list))+ARLoss(model,T,lag_list)

......


</code></pre>
Bias grad in linear regression remains small compared to weight grad, and intercept is not properly learnt

Wouter_Durnez: <p>I have thrown together a dummy model to showcase linear regression in pytorch, but I find that my model is not properly learning. It’s doing well when it comes to learning the slope, but the intercept is not really budging. Printing out the grads at every epoch tells me that, indeed, the grad is a lot smaller for the bias. <strong>Why is that? How can I remedy it, so the intercept is properly learnt?</strong></p>
<p>This is what happens (a set to 0 to illustrate):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6dd36a605382e76600866a7e2e7abb8298ef8bdf.gif" alt="linear_regression_SGD" data-base62-sha1="fFyZZSmdh23EjnMAoVRcWUF2jDx" width="640" height="480" class="animated"></p>
<p>This is the code:</p>
<pre><code class="lang-auto"># Create some dummy data: we establish a linear relationship between x and y
a = np.random.rand()
b = np.random.rand()

a=0

x = np.linspace(start=0, stop=100, num=100)
y = a * x + b

# Now let's create some noisy measurements
noise = np.random.normal(size=100)
y_noisy = a * x + b + noise

# What's the overall error?
mse_actual = np.sum(np.power(y-y_noisy,2))/len(y)

# Visualize
plt.scatter(x,y_noisy, label='Measurements', alpha=.7)
plt.plot(x,y,'r', label='Underlying')
plt.legend()
plt.show()

# Let's learn something!
inputs = torch.from_numpy(x).type(torch.FloatTensor).unsqueeze(1)
targets = torch.from_numpy(y_noisy).type(torch.FloatTensor).unsqueeze(1)


# This is our model (one hidden node + bias)
model = torch.nn.Linear(1,1)
optimizer = torch.optim.SGD(model.parameters(),lr=1e-5)
loss_function = torch.nn.MSELoss()

# What does it predict right now?
shuffled_inputs, preds = [], []
for input, target in zip(inputs,targets):

    pred = model(input)
    shuffled_inputs.append(input.detach().numpy()[0])
    preds.append(pred.detach().numpy()[0])

# Visualize
plt.scatter(x,y_noisy, color='blue', label='Measurements', alpha=.7)
plt.plot(shuffled_inputs, preds, color='orange', label='Predictions', alpha=.7)
plt.plot(x,y,'r', label='Underlying')
plt.legend()
plt.show()

# Let's train!
epochs = 100
a_s, b_s = [], []

for epoch in range(epochs):

    # Reset optimizer values
    optimizer.zero_grad()

    # Predict values using current model
    preds = model(inputs)

    # How far off are we?
    loss = loss_function(targets,preds)

    # Calculate the gradient
    loss.backward()

    # Update model
    optimizer.step()

    # Quick check
    for p in model.parameters():
        print('Grads:', p.grad)

    # New parameters!
    a_s.append(list(model.parameters())[0].item())
    b_s.append(list(model.parameters())[1].item())

    print(f"Epoch {epoch+1} -- loss = {loss}")
</code></pre>

googlebot: <p>slope’s gradient is modulated by input, while bias only receives direct loss gradients. and it receives conflicting direction signals, so per sample gradients cancel each other. and global minimum is not zero, because you don’t use mini-batches or re-samples, the optimal line is not strictly horizontal.</p>
Compare two layers of the same model

Sarah: <p>Hello,<br>
I’m writing a code for an AutoEncoder and testing the relevance of each layer using a function.<br>
After calculating the relevance of each layer, how can I compare them and prune the one with the lower relevance??</p>
<p>Thank you</p>

eqy: <p>Are you attempting to prune the weights of each layer or each layer as a whole?<br>
IIRC “The Lottery Ticket Hypothesis” paper provides a good overview of common methods for weight pruning;<br>
<a href="https://arxiv.org/pdf/1803.03635v1.pdf" rel="noopener nofollow ugc">1803.03635v1.pdf (arxiv.org)</a><br>
Though I’m not sure how well these would generalize to pruning an entire layer all at once (something iterative might still be needed).</p>

Sarah: <p>yes I am looking for pruning the whole layer</p>
<p>But do you know how I can compare two layers starting from the last layer in the model, that will be helpful.</p>
<p>Thank you</p>

eqy: <p>In a feedforward model, it might be difficult to describe the relative importance of layers if they are in sequence because, the learned representations can be different (e.g., at different levels of abstraction such as simple patterns vs. high level concepts). It’s more common to compare the relevance of two layers in parallel because one can directly weigh the contribution of each to the output—but it sounds like you want to compare two layers in sequence.</p>
<p>In this case, a hack might be to use a ResNet-style model with identity or skip connections between certain layers, and to do an ablation study. For example, if you already know your desired metric (e.g., accuracy) you can simply ablate layers with this style of architecture and observe how the accuracy is affected (possibly with some finetuning after truncating the model).</p>

Sarah: <p>Thank you for your response<br>
I have done the ablation study on the model and got the results, but know I want to compare it with a dynamic algorithm that prune irrelevant layers according to some criteria</p>
<p>Thank you so much<br>
I’ll keep looking for it.</p>
How to split dataset into two considering fixed seed to ensure reproducibility in PyTorch?

GamarChoba: <p>I am working on one of my University assignments and there is one sub-task that says. Split the data in two (Train and Validation) while using a fixed seed to ensure reproducibility. I have written some code that is working fine but I want to know whether it is the correct way or not?</p>
<pre><code class="lang-auto">torch.manual_seed(0)
mnist_train, mnist_val = torch.utils.data.random_split(mnist_rest, [54000,6000])
</code></pre>
<p>I am working on the MNIST dataset.</p>

ayalaa2: <p>This works, but it’s not my preferred method.</p>
<p><code>torch.manual_seed(0)</code> sets the state of the global seed generator. So this will cause all subsequent calls that uses any kind of torch randomness to be reproducible. However, suppose you refactored your code later on:</p>
<pre><code class="lang-auto">torch.manual_seed(0)

# Adding another random call
x = torch.randn((3, 2))

mnist_train, mnist_val = torch.utils.data.random_split(mnist_rest, [54000,6000])
</code></pre>
<p>Then your train and validation set are no longer the same as before. I would recommend using <a href="https://pytorch.org/docs/stable/generated/torch.Generator.html" rel="noopener nofollow ugc">pytorch’s generator</a> for this.</p>
<pre><code class="lang-auto">gen = torch.Generator()
gen.manual_seed(0)
mnist_train, mnist_val = torch.utils.data.random_split(
    mnist_rest, [54000,6000],
    generator=gen
)
</code></pre>
<p>Basically we’re creating a generator for random numbers and we’re telling <code>random_split</code> to specifically use this generator and not the global generator.</p>
<p>That being said, I do want to clarify that your method <strong>does work</strong> and if you never broke those 2 lines apart you will always get the same train/val split. I just prefer the use of generators so make sure I’m precisely controlling the randomness in these calls.</p>
Wrong target format ImageFolder and DataLoader

Silviu-Alexandru_Din: <p>Hey guys,</p>
<p>I am new to working with Images and Pytorch and I have an issue with data format.<br>
I’m using ImageFolder utility to load my images and classes, and I have the following folder structure:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55a7eda994c9319f505fb303adab8518ff1523e5.png" alt="image" data-base62-sha1="cdKmVJMKPdjTdgcmpD1Rf80dUln" width="442" height="450"></p>
<p>I’m loading the images with:</p>
<pre><code class="lang-auto">training_data = datasets.ImageFolder(root=os.path.join(curr, r'fer2013/train/'),
                                     transform=transforms.ToTensor())
training_set = DataLoader(training_data, batch_size=10, shuffle=True)

</code></pre>
<p>and then I am looping over it like so:</p>
<pre><code class="lang-auto">for epoch in range(5):  # 3 full passes over the data
    for idx, data in enumerate(training_set):  # `data` is a batch of data
        X, y = data  # X is the batch of features, y is the batch of targets.
        net.zero_grad()
        output = net(X)
        loss = F.nll_loss(output, y.unsqueeze(1))
        loss.backward() 
        optimizer.step() 
    print(loss)
</code></pre>
<p>where net is an instance of my Net() class.</p>
<p>Below are the shapes of <strong>torch.argmax(output, dim=1), output, y, and y.unsqueeze(1)</strong> in this exact order:<br>
<code>torch.Size([10, 48, 7]) torch.Size([10, 3, 48, 7]) torch.Size([10]) torch.Size([10, 1])</code></p>
<p>and I am getting this error on the <strong>F.nll_loss</strong> function call:<br>
<em>RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 2</em></p>
<p>Is this something related to how the data is being loaded, or should I apply some operations to the target y?</p>
<p>Thank you for any help!</p>

ptrblck: <p>The output shape doesn’t match the target shape for the provided output.<br>
<code>nn.NLLLoss</code> expects a model output in the shape <code>[batch_size, nb_classes, *]</code> and a target in the shape <code>[batch_size, *]</code> containing class indices in the range <code>[0, nb_classes-1]</code>.<br>
Since you are using <code>ImageFolder</code> I would assume you are working on a multi-class classification use case.<br>
In that case the model output should have the shape <code>[batch_size, nb_classes]</code> and the target <code>[batch_size]</code> (note the missing additonal dimensions).<br>
Remove the <code>.unsqueeze</code> operation performed on <code>y</code> and make sure the model output matches the expected shape.</p>
Where to get a pytorch version with cuda 7.5

hitbuyi: <p>I  know pytorch support cuda 7.5,  but try to install online, it failed</p>

hitbuyi: <p>my platform is win10 +  anaconda3,not in linux</p>

ptrblck: <p>CUDA 7.5 was used when PyTorch 0.4.1 was released and if I’m not mistaken, no Windows binaries were built for this old release.<br>
Your best bet would thus be to check out this legacy release and build from source.<br>
I would generally recommend to stick to the latest release.</p>

hitbuyi: <p>My laptop’s  GPU  is  nVIDIA GeForceGT 650M,whic is very old, does the latest CUDA works for  this old GPU ?</p>

ptrblck: <p>CUDA 10 would support it, but I think that the lowest compute capability supported by PyTorch is sm_35, so you wouldn’t be able to build it for your device. In that case you could either try to use remove GPUs (e.g. via Colab) or indeed build an old release from source.</p>
Machine Learning

helpmeplease: <p>Hi. I’m a student and wanted to explore more about machine learning. I am trying to make a neural network model that classify child-adult. My process is by first  detecting their body then their head/face to get the ratio of the head over body. What I’m confused about is how many dataset should I make. Like do I have to make a separate dataset for head? body? or child? adult? I hope someone can help me with this problem. Thank you in advance and have a great day.</p>

JuanFMontesinos: <p>Hey!<br>
Well I would say that the ratio hea/body is not significant (correct me if I’m wrong) but I would say kids and adults are proportional. However the face is clearly related to the age.</p>
<p>The pipeline you can follow is:</p>
<ol>
<li>Download images from google by tagging child/adult  (use one folder for each)</li>
<li>Run a face detector over the images. Here you have two options:<br>
2a. <a href="https://www.pyimagesearch.com/2018/02/26/face-detection-with-opencv-and-deep-learning/" class="inline-onebox" rel="noopener nofollow ugc">Face detection with OpenCV and deep learning - PyImageSearch</a> A complete tutorial<br>
2b. If you feel confident about python  you can directly go for  <a href="http://dlib.net/" rel="noopener nofollow ugc">http://dlib.net/</a>  and use its face detector.<br>
To follow this option you should know or learn to open images and to work with numpy arrays.<br>
Basically the face detector will return you 4 points corresponding to a box around the face.</li>
<li>Crop the images within the box and saves them in two new different folders like <code>child_cropped</code>, <code>adult_cropped</code> or something similar.</li>
<li>Now you should have all the data you need! You just need to use a 2D convolutional neural network. This works with images basically. I recommend you to use an already-existing one. This way you ensure it’s well done.</li>
<li>Then you just need to slightly modify the network predicting a binary numer (0 child, 1 adult) or viceversa. Since you have them in different folders you already know the solution.</li>
<li>Train the network and that’s all!.</li>
</ol>
<p>Depending on your experience this may take you a while, let me know if i can further help u</p>
TPU: Resource exhausted, although there seems to be enough memory

gcascio: <p>I am trying to train a ViT on a cloud TPU using a pytorch implementation. For small datasets everything works fine. As soon as I use a larger dateset (Imagenet) I run in to an error I do not understand:</p>
<p><code>2021-11-10 17:01:24.065497: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at tpu_execute_op.cc:266 : Resource exhausted: Attempting to reserve 5.88G at the bottom of memory. That was not possible. There are 14.74G free, 0B reserved, and 4.62G reservable.</code></p>
<p>I am aware that the memory requirements increase with larger images and batch sizes but what I don’t understand is the error message. How can there be 14.74G free with 0B reserved but only 4.62G reservable and what direction could I look into to resolve this issue?</p>
<p>Some details to my setup:</p>
<p><strong>Model/Training</strong><br>
image_size: 224 x 224<br>
batch_size: 128<br>
model parameters: ~86M</p>
<p><strong>Hardware</strong><br>
TPU: v3-8</p>
<p><strong>Environment</strong><br>
Python 3.8.10<br>
Ubuntu 20.04.2 LTS</p>
<p><strong>Relevant Packages</strong><br>
tf-nightly==2.6.0<br>
torch==1.9.0<br>
torch-xla @ <a href="https://storage.googleapis.com/tpu-pytorch/wheels/tpuvm/torch_xla-1.9-cp38-cp38-linux_x86_64.whl" rel="noopener nofollow ugc">https://storage.googleapis.com/tpu-pytorch/wheels/tpuvm/torch_xla-1.9-cp38-cp38-linux_x86_64.whl</a><br>
torchvision==0.10.0</p>
Pack_padded_sequence with dynamic sequence lengths

serafej: <p>Pytorch offers a pack_padded_sequence function for RNNs which enables efficient batching of varying-length sequences when we know the length of the sequences in advance, saving computation on sequences that end earlier in the batch.<br>
In my problem case, the length of each sequence is not known in advance, but is decided while the batch of sequences is processed (e.g. based on the intermediate network output).</p>
<p>(1) Could pack_padded_sequence or a different existing PyTorch function be used to efficiently process such a batch of on-line-varying sequence lengths?<br>
(2) If not, would such a function be possible to implement in PyTorch?</p>
Clipped torch.cumsum

Peter_Featherstone: <p><code>torch.cumsum</code> is currently implemented as :<br>
y_{i} = x_{i} + y_{i-1}</p>
<p>How do i get a running clipped version like so:<br>
y_{i} = clip(x_{i} + y_{i-1}, min, max)</p>
<p>This isn’t the same as<br>
<code>y = torch.clip(torch.cumsum(x), min, max)</code><br>
The clipping needs to be evaluated every iteration.</p>
Averaging embedding in training + MeanBackwards Autograd

st-vincent1: <p>I’m implementing a neural model for a task where part of the input is supposed to be several categorical variables. At inference time, I wish to embed each attribute separately and then take the mean.</p>
<p>If I average the embeddings at train time similarly as during inference, will a model be able to learn each embedding correctly? I.e. will the model be able to figure out which embedding to update and how even if only a mean of them was fed as input? My intuition tells me that MeanBackwards autograd is supposed to help with this but I’m not sure I understand exactly how that is going to work!</p>

my3bikaht: <p>You want to pass multiple inputs in parallel and calculate mean of outputs or you want to calculate mean of inputs then pass it to model?</p>
Multiprocessing in IterableDataset for VideoLoading

abhiksark: <p>Hi, I am using IterableDataset for Loading my videos using the following code.</p>
<pre><code class="lang-auto">class RetinaFaceDatasetVideoNew(torch.utils.data.IterableDataset):
    def __init__(self,name:str):
        self.name = name
        self.cap = cv2.VideoCapture(str(name))
        if not self.cap.isOpened():
            raise IOError(f"Cannot open video {name}")
        self.scale = 0.5

    def __iter__(self):
        frame_idx = 0
        while True:
            try:
                ret, image = self.cap.read()
                if not ret:
                    # no frames has been grabbed
                    break
                img = cv2.resize(image,
                                 None,
                                 None,
                                 fx=self.scale,
                                 fy=self.scale,
                                 interpolation=cv2.INTER_LINEAR,
                                )
                yield image,self.name
            except StopIteration:
                return
</code></pre>
<p>Problem is that this process is slow.</p>
<p>To tackle this issue I tried multiprocessing over different videos as it is faster.<br>
For that I wrote</p>
<pre><code class="lang-auto">class RandomDatasetM(torch.utils.data.IterableDataset):
  def __init__(self,name: list):
    self.name = name
    self.cap = []
    self.cap.append(cv2.VideoCapture(name[0]))
    self.cap.append(cv2.VideoCapture(name[1]))

  def __iter__(self):
    frame_idx = 0
    worker_info = torch.utils.data.get_worker_info()
    if worker_info is None:  # single-process data loading, return the full iterator
      while True:
        try:
            ret, image = self.cap[0].read()
            if not ret:
                # no frames has been grabbed
                break
            yield image, self.name[worker_id]
        except StopIteration:
            return
    else:  # in a worker process
        # split workload
        worker_id = worker_info.id
        while True:
          try:
              ret, image = self.cap[worker_id].read()
              if not ret:
                  # no frames has been grabbed
                  break

              yield image,self.name[worker_id]
          except StopIteration:
              return
</code></pre>
<p>But this is not running and going in deadlock. Can someone suggest me approach to achieve multiprocessing here?</p>

JuanFMontesinos: <p>Save videos as numpy arrays if you have enough hardisk. Then just ignore multiprocessing and drop it on GPU and use pytorch. You don’t need to use multiprocessing, it is applied automatically when you call the dataloader.</p>
<p>Another option is to use Nvidia’s DALI library. Way faster than anything you can code. But you will spend some time to learn how to use it.</p>
Different weights with quantize_fx and quantize_per_tensor

Jonas_ar: <p>Hello everyone,</p>
<p>I am trying to quantize the MobileNetV3, which I trained on the MNIST handwritten dataset.</p>
<p>In the documentation for quantization here on the pytorch website, I stumbled upon the prototybe function of “FX GRAPH MODE POST TRAINING STATIC QUANTIZATION”. Which I used to quantize my model. It worked, since when all the layers and weights are quantized now.</p>
<p>But before moving forward I wanted to compare the weights of the fully quantized model to if I only quantize a specific tensor with the same scale and zero-point.</p>
<p>x = torch.quantize_per_tensor(model.state_dict()[‘features.0.0.weight’][0], scale = 0.0057970373891294 , zero_point = 0, dtype = torch.qint8)</p>
<p>y = model_quantized.state_dict()[‘features.0.0.weight’][0]</p>
<p>print(f’{x}\n{y}\n’)</p>
<p>Output:</p>
<p>tensor<br>
([[[-0.1333, -0.1971,  0.1623],<br>
[-0.0174, -0.2377,  0.0928],<br>
[ 0.1101, -0.2493,  0.1913]],</p>
<pre><code>    [[ 0.0754, -0.0406, -0.0232],
     [ 0.0232,  0.1797,  0.3536],
     [-0.0116,  0.0812,  0.1565]],

    [[-0.0985, -0.1043, -0.1681],
     [-0.0058, -0.1101, -0.0928],
     [-0.0174, -0.1101,  0.1217]]], size=(3, 3, 3), dtype=torch.qint8,
   quantization_scheme=torch.per_tensor_affine, scale=0.0057970373891294,
   zero_point=0)
</code></pre>
<p>tensor<br>
([[[-0.2261, -0.3304,  0.2667],<br>
[-0.0290, -0.4058,  0.1565],<br>
[ 0.1913, -0.4174,  0.3246]],</p>
<pre><code>    [[ 0.1275, -0.0638, -0.0348],
     [ 0.0406,  0.3014,  0.5971],
     [-0.0174,  0.1391,  0.2609]],

    [[-0.1681, -0.1797, -0.2783],
     [-0.0116, -0.1913, -0.1565],
     [-0.0290, -0.1855,  0.2087]]], size=(3, 3, 3), dtype=torch.qint8,
   quantization_scheme=torch.per_tensor_affine, scale=0.0057970373891294,
   zero_point=0)
</code></pre>
<p>The original weights:</p>
<p>model.state_dict()[‘features.0.0.weight’][0]</p>
<p>tensor([[[-0.1360, -0.1972,  0.1600],<br>
[-0.0184, -0.2405,  0.0924],<br>
[ 0.1127, -0.2482,  0.1923]],</p>
<pre><code>    [[ 0.0749, -0.0395, -0.0211],
     [ 0.0253,  0.1814,  0.3552],
     [-0.0111,  0.0820,  0.1559]],

    [[-0.1000, -0.1057, -0.1673],
     [-0.0062, -0.1127, -0.0930],
     [-0.0168, -0.1123,  0.1228]]])
</code></pre>
<p>As you can see the weights of the quantized model are quite different compared to the original ones, even in the qint8 dype.</p>
<p>Long story short, my qustion is what reason could be behind the weight change? From what I understood so far it looks like there is some kind of factor multiplied during the quantization process, could some kind of nomalization help?</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Output of torchvision.models.detection.fasterrcnn_resnet50_fpn

neeravnagda: <p>Hey, so I am using torchvision.models.detection.fasterrcnn_resnet50_fpn, and part of the output is a list of labels as integers.<br>
How do I find the words corresponding to these labels? I am not re-training the model, I am just using the pretrained model and passing in an image.</p>

ptrblck: <p>The detection models used COCO 2017 in their pretraining and the <a href="https://pytorch.org/vision/stable/models.html">models docs</a> mention the class labels:</p>
<blockquote>
<p>For object detection and instance segmentation, the pre-trained models return the predictions of the following classes:</p>
</blockquote>
<pre data-code-wrap="python"><code class="lang-python">COCO_INSTANCE_CATEGORY_NAMES = [
    '__background__', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus',
    'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'N/A', 'stop sign',
    'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow',
    'elephant', 'bear', 'zebra', 'giraffe', 'N/A', 'backpack', 'umbrella', 'N/A', 'N/A',
    'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball',
    'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket',
    'bottle', 'N/A', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl',
    'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza',
    'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'N/A', 'dining table',
    'N/A', 'N/A', 'toilet', 'N/A', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone',
    'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'N/A', 'book',
    'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush'
]
</code></pre>
Augmentation error - TrivialAugmentWide

azer: <p>Hi</p>
<p>I am trying to utilize the PyTorch augmentation  TrivialAugmentWide() but I get the error message.</p>
<p>cannot import name ‘TrivialAugmentWide’ from ‘torchvision.transforms.autoaugment’</p>
<p>Any help is appreciated.</p>
<p>Az</p>

ptrblck: <p>You might need to update <code>torchvision</code> as this transformation was added ~2 months ago.</p>
Removing datapoints from dataset

Maria_Fernanda_De_La: <p>I would like to remove specific indices from dataset. I tried this but it doesn’t work <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<pre><code class="lang-auto">class MyDataset(Dataset):
   def __init__(self,remove_list):
        self.cifar10 = datasets.CIFAR10(root='./data',

                                        download=False,
                                        train=True,
                                        transform=transforms.ToTensor())
        self.data = self.cifar10.data
        self.targets = self.cifar10.targets
        self.final_data, self.final_targets = self.__remove__(remove_list)
      
    def __getitem__(self, index):
        data, target = self.final_data[index], self.final_targets[index]
        return data, target, index

    def __len__(self):
        return len(self.final_data)

    def __remove__(self, remove_list):
        data = np.delete(self.data, remove_list)
        targets = np.delete(self.targets, remove_list)
        return data, targets
</code></pre>

Maria_Fernanda_De_La: <p>I realized it was an issue with the way I deleted items, should be:</p>
<pre><code class="lang-auto">   data = np.delete(self.data, remove_list, axis=0)
    
        
        targets = np.delete(self.targets, remove_list, axis=0)
</code></pre>
<p>But is it doing the correct thing overall?:  Removing specific images based on the index or is the index different every time it is loaded?</p>

ptrblck: <p>The data should be loaded in the same order, but of course you could verify it by comparing some random data samples.</p>
<p><code>np.delete</code> should work fine on numpy arrays. Alternatively, you could also slice the arrays by creating a <code>mask</code> array and setting the values at <code>remove_list</code> to <code>False</code>:</p>
<pre><code class="lang-python">mask = np.ones(len(arr), dtype=bool)
mask[remove_list] = False
data = self.data[mask]
</code></pre>

Maria_Fernanda_De_La: <p>This is so helpful, thank you so much!</p>

Torcione: <p>Hi ptrblck,<br>
can you explain me in more details what do the last two lines of code do?</p>

ptrblck: <p>The code snippet initializes a <code>mask</code> with <code>True</code> values for all entries first.<br>
The second line of code then uses the <code>remove_list</code> indices to index <code>mask</code> and sets these values to <code>False</code>. In the last line of code <code>self.data</code> is indexed with <code>mask</code> and reassigned to <code>data</code> which will then contain all entries from <code>self.data</code> where <code>mask</code> was set to <code>True</code>.</p>

Torcione: <p>Thank you, very clear! So (correct me if I’m wrong) you also need a further line to do the same on the targets, right? Something like:</p>
<pre><code class="lang-auto">targets = self.targets[mask]
</code></pre>

ptrblck: <p>Yes, if you are working with a target tensor and want to remove the same indices you would have to add your line of code.</p>
Can't build libtorch lite for iOS in linux

MD_TARIQUL_ISLAM: <p>Hello good people,<br>
I’m on Linux, and I was trying to convert a pytorch model optimized for IOS following this link: <a href="https://pytorch.org/tutorials/prototype/lite_interpreter.html#how-to-use-lite-interpreter-custom-build" class="inline-onebox" rel="noopener nofollow ugc">(Prototype) Introduce lite interpreter workflow in Android and iOS — PyTorch Tutorials 1.9.0+cu102 documentation</a></p>
<p>But what I found in <strong>Build libtorch lite for iOS</strong> step that it requires <strong>iOS SDK</strong>. As I’m on a Linux system, it’s quite impossible for me to use this.</p>
<p>So, my concern is whether there is any way to convert my Pytorch model for use on iOS. Or I can follow another workflow which can convert a pytorch model. I’d be glad to know that. Thanks in advance.</p>

Linbin: <p>Once converted the model (save_for_lite_interpreter) and obtained operator yaml file in Linux, you can build libtorch lite on Mac using the generated yaml file.</p>

Matt_Davis: <p>So we need to do the build on a mac. Ok, will try that.</p>

cccclai: <p>The current release is using mobile interpreter. If you don’t need a custom build (only build with the operators needed for a target model), this tutorial is probably easier to follow <a href="https://pytorch.org/tutorials/recipes/mobile_interpreter.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Efficient mobile interpreter in Android and iOS — PyTorch Tutorials 1.10.0+cu102 documentation</a></p>

Matt_Davis: <p>I do need a custom build for la pack</p>
Sampler with multiple dataset simoultaneous iteration

Torcione: <p>I need to iterate simoultaneously on multiple dataset (let’s say 2 dataset) keeping the element of each of them isolated (each batch must contain only element of one dataset and for each step I want to work with one batch from each dataset). To do so I think that <code>torch.utils.data.TensorDataset</code> can be the right tool, for example:</p>
<pre><code class="lang-auto">dataset = torch.utils.data.TensorDataset(dataset1, dataset2)
dataloader = DataLoader(dataset, batch_size=128, shuffle=True)
for index, (xb1, xb2) in enumerate(dataloader):
    ....
</code></pre>
<p>where  <code>xb1</code> refers to the input data and target associated to one of the 2 dataset.<br>
My first question is: have I understood well the use of <code>torch.utils.data.TensorDataset</code> ? Does this approach solve my problem?</p>
<p>My second question is: how to put a sampler in the dataloader in this situation? can I, for example, define 2 indeces tensor <code>Idx1</code> and <code>Idx2</code> and put in <code>DataLoader</code> an option like <code>sampler = (Idx1, Idx2)</code></p>
<p><strong>EDIT</strong>:<br>
An alternative approach could be to create a dataloader for each dataset, each one with his own sampler and use zip() to iterate simoultaneously on the 2 dataset. Is there a more clean solution for that (also beacosu I read that (<a href="https://newbedev.com/how-to-iterate-over-two-dataloaders-simultaneously-using-pytorch" rel="noopener nofollow ugc">source</a>):</p>
<blockquote>
<p><code>cycle()</code> and <code>zip()</code> <strong>might create a memory leakage problem</strong> - especially when using image datasets!</p>
</blockquote>
<p>)?</p>

someshfengde: <p>using custom dataset might be best option here</p>
<p><a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files</a></p>

Torcione: <p>Hi, thank you for your comment!<br>
Then how can I prevent the mixing of element associated to different dataset inside a given batch?<br>
And how to perform operation on a pair of batches (one for each dataset) at each step?</p>

someshfengde: <p>If you are gonna use the dataset class from Pytorch you will have whole control how your data is gonna load and you can ensure that no memory leak happens</p>
<p>for making pairs of batches you can use dataloader API and create batches of the dataset according to your need slr was quite preoccupied with work</p>
Torch.index_select out of memory

alwynmathew: <p>Ran out of memory while using <a href="http://pytorch.org/docs/master/torch.html#torch.index_select" rel="nofollow noopener">torch.index_select</a></p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(input.size())
(1, 1, 256, 512)
&gt;&gt;&gt; print(idx.size())
131072
&gt;&gt;&gt; pix = torch.index_select(input, 0, idx)
RuntimeError: cuda runtime error(2): out of memory at /pytorch/torch/lib/THC/generic/THCStorage.cu:58
</code></pre>
<p>Any fix? Thank you.</p>

ptrblck: <p>What exactly would you like to index?<br>
In your current approach, you will select from dimension <code>0</code>, which has only one entry.<br>
So basically you are repeating your <code>Tensor</code>, resulting in a size of <code>[131072, 1, 256, 512]</code>.</p>

alwynmathew: <p>Gather all <code>input</code> pixel value at index <code>idx</code>.</p>
<pre><code class="lang-auto">input_flat = input.contiguous().view(-1)
pix = torch.gather(input_flat, 0, idx)
</code></pre>
<p>Does this work?</p>

ptrblck: <p>How did you define <code>idx</code>? Does it store the “flat” indices, i.e. <code>[0, 256*512]</code>?</p>
<p>Could you try this:</p>
<pre><code class="lang-auto">x = torch.randn(1, 1, 256, 512)
idx = torch.arange(256*512).long()
idx = idx[torch.randperm(256*512)]

y = x.view(-1)[idx]
</code></pre>
<p>Let me know, if this suits your use case.</p>

deJQK: <p>I also meet similar issue when using <code>torch.index_select</code>. What I have are two tensors say <code>lut</code> (64 x 7281) and <code>idx</code> (64 x 943), where values in the <code>idx</code> tensor are 0 to 7280 and I need to use some way like <code>res = torch.stack([torch.index_select(l_, 0, i_) for l_, i_ in zip(lut, idx)])</code>. I wonder if there is any more memory-efficient way to do this.</p>

deJQK: <p>It is interesting that <code>res = torch.stack([l_[i_] for l_, i_ in zip(lut, idx)])</code> works well and there is no OOM issue. I do not understand why.</p>

ptrblck: <p>Instead of the list comprehension you could also index <code>lut</code> as seen here:</p>
<pre><code class="lang-python">lut = torch.randn(64, 7281)
idx = torch.randint(0, 7281, (64, 943))

res = torch.stack([l_[i_] for l_, i_ in zip(lut, idx)])

out = lut[torch.arange(lut.size(0)).unsqueeze(1), idx]
print((out == res).all())
&gt; tensor(True)
</code></pre>
How to modify graph schedule?

JUNSUNGKIM99: <p>Hi I am new to glow and DNN.<br>
I received the onnx file and tried to compile it. The question here is how to map onnx files to high-level IR.<br>
Looking at it, I could find a way to map high-level IR to low-level IR, but I could not find how to map network files to high-level IR.</p>
<p>Instead of modifying the original neural network file, I want to create a modified graph in the process of reading onnx and making it high-level IR.</p>
<p>Thanks</p>
Bilinear layer : RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead

Navid_Aftabi: <p>Hi, I am new in Pytorch. My model is the following:</p>
<pre><code class="lang-auto">emb = nn.Embedding(100, 100)
fc_emb = nn.Linear(100, 512, bias=False)
dropout_fc = nn.Dropout(0.3)

conv = nn.Conv1d(in_channels=2, out_channels=512, kernel_size=1, stride=1)

dropout_c = nn.Dropout(0.3)  

cat = nn.Bilinear(512, 512, 512)
drop_cat = nn.Dropout(0.3)  
</code></pre>
<p>I have tried this example:</p>
<pre><code class="lang-auto">x_c = dropout_c(F.relu(conv(torch.randn(200,2,20))))
x_c = torch.transpose(x_c, 1, 2)
x_n = emb(torch.randint(0,100,(200,20)))
x_n = dropout_fc(F.relu(fc_emb(x_n)))

x = drop_cat(cat(x_c, x_n))
</code></pre>
<p>But I keep getting this error:</p>
<pre><code class="lang-auto">RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
</code></pre>
<p>when I run the following code the <code>nn.Bilinear</code> works well. But it gives me the error when I run the above codes.</p>
<pre><code class="lang-auto">cat(torch.randn(256, 20, 512),torch.randn(256, 20, 512)).shape
</code></pre>
<p>Could you please help me?</p>

ayalaa2: <p>Bilinear doesn’t like the fact that <code>x_c</code> was transposed. You can replace the call with this: <code>x = drop_cat(cat(x_c.contiguous(), x_n))</code>. This basically just reshapes the data (in memory) to match what the transpose did.</p>

Navid_Aftabi: <p>Thank you <a class="mention" href="/u/ayalaa2">@ayalaa2</a> <img src="https://discuss.pytorch.org/images/emoji/apple/pray/2.png?v=10" title=":pray:t2:" class="emoji" alt=":pray:t2:"> <img src="https://discuss.pytorch.org/images/emoji/apple/v/2.png?v=10" title=":v:t2:" class="emoji" alt=":v:t2:"></p>
Add Residual Connections To Output

matthewfernst: <p>Hi There! I’m currently trying to figure out how to add a residual connection to my out layer. Essentially, I want every neuron in the hidden layers to be connected to the output neuron.</p>
<p>I’m currently making a sequential layer for every hidden layer and at the end of my forward pass, I am adding all the outputs up and making that the final output. Does anyone have any recommendations on how to do this? Thank you! (Sorry in advance for all of the code)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c0e14228833c47175057dee4e4576b78c7b92905.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c0e14228833c47175057dee4e4576b78c7b92905" title="residual"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0e14228833c47175057dee4e4576b78c7b92905_2_171x500.jpeg" alt="residual" data-base62-sha1="rwihSb8uM8PSr5Kap6aJSsiLH4V" width="171" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0e14228833c47175057dee4e4576b78c7b92905_2_171x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0e14228833c47175057dee4e4576b78c7b92905_2_256x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c0e14228833c47175057dee4e4576b78c7b92905_2_342x1000.jpeg 2x" data-dominant-color="2E3031"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">residual</span><span class="informations">1920×5614 451 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to install pytorch on jetson tx2?

jinooooo: <p>ive been trying to install pytorch on jetson tx2 for my company’s work but im not able to do it. Can someone please guide me here if at all pytorch is compactible with jetson tx2?</p>

eqy: <p>It should be compatible, and it sounds like there should be prebuilt pip wheels available:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-10-now-available/72048">
  <header class="source">
      <img src="https://aws1.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-10-now-available/72048" target="_blank" rel="noopener nofollow ugc" title="12:51AM - 27 March 2019">NVIDIA Developer Forums – 27 Mar 19</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:237/42;"><img src="https://aws1.discourse-cdn.com/nvidia/original/2X/8/8f17cc8f1a724d6ecea8a197a267ec8a05ef1490.png" class="thumbnail" width="237" height="42"></div>

<h3><a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-10-now-available/72048" target="_blank" rel="noopener nofollow ugc">PyTorch for Jetson - version 1.10 now available</a></h3>

  <p>Below are pre-built PyTorch pip wheel installers for Python on Jetson Nano, Jetson TX1/TX2, and Jetson Xavier NX/AGX with JetPack 4.2 and newer.   Download one of the PyTorch binaries from below for your version of JetPack, and see the installation...</p>

  <p>
    <span class="label1">Reading time: 232 mins 🕑</span>
      <span class="label2">Likes: 123 ❤</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How to speed up a for loop of calls

Rodolphe_Lampe: <p>In the forward method of my nn.Module, I have these 2 lines :</p>
<p>views = [z] + [transformation(z) for transformation in self.model.transformations]<br>
representations = torch.stack([self.model.encoder(view) for view in views])</p>
<p>where self.model.transformations is a list of nn.Modules (Convolutions actually).<br>
By profiling my code, I seed that these formulation as a list comprehension might be problematic in term of speed.</p>
<p>Is there a pytorch function/module that would take a list of modules and, in a fast way, apply it to my entry z and then stack the resuls ?</p>
<p>Thanks</p>

googlebot: <aside class="quote no-group" data-username="Rodolphe_Lampe" data-post="1" data-topic="136404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rodolphe_lampe/48/16870_2.png" class="avatar"> Rodolphe_Lampe:</div>
<blockquote>
<p>By profiling my code, I seed that these formulation as a list comprehension might be problematic in term of speed.</p>
</blockquote>
</aside>
<p>Very unlikely, unless you have thousands of transformations on tiny arrays. In normal circumstances, your hotspots should be in native (c++ or cuda); this may be invisible to your profiler, or you’re doing profiling without cuda synchronization.</p>