akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Multiprocessing PicklingError

username: 111357
cooked: <p>I am sorry that I can not reproduce my error because my code is too complex.<br>
When I try to train a model, I got the following error.</p>
<pre><code class="lang-bash">Traceback (most recent call last):
  File "/home/xxx/.conda/envs/env_name/lib/python3.8//queues.py", line 239, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/xxx/.conda/envs/env_name/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle &lt;class 'numpy.core._exceptions.UFuncTypeError'&gt;: it's not the same object as numpy.core._exceptions.UFuncTypeError
</code></pre>
<p>I cannot locate the location of the error. Can anyone give some advice about what’s the type of this error?</p>

username: ptrblck
cooked: <p>This type of error is raised by Python’s multiprocessing library, when it’s unable to <code>pickle</code> certain classes and seems to be related to <a href="https://github.com/numpy/numpy/issues/16490">this issue</a>, which crashes if numpy tries to raise an exception. Based on the discussion in the issue it seems this was fixed on Dec 4, 2020 in numpy.</p>
How to randomly pick integer interval/range

username: 111429
cooked: <p>Hi,<br>
I have a list 0-9: [0,1,2,3,…,9]<br>
I want randomly output 2 vector with given span.<br>
If span=3<br>
output may be<br>
([2,3,4], [7,8,9])<br>
or<br>
([3,4,5], [6,7,8])</p>
<ul>
<li>two vectors can not overlap.<br>
Any advice?<br>
Thanks.</li>
</ul>
How to load state_dict for ParameterList?

username: Lei_Shi
cooked: <p>I try to load pretrained model by:</p>
<p>saved = torch.save(model.state_dict())<br>
model.load_state_dict(torch.load(saved))</p>
<p>but the ParameterList([Parameter(Tensor1), Parameter(Tensor2) …]) of the model is not correctly loaded.</p>
<p>So, what is the correct method to load pretrained weights for nn.ParameterList()?</p>
<p>pytorch 1.6.0</p>

username: ptrblck
cooked: <p>This small code snippet seems to work, so could you share a minimal code example to reproduce this issue?</p>
<pre><code class="lang-python"># create a simple model using a linear layer
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.params = nn.ParameterList([
            nn.Parameter(torch.randn(1)),
            nn.Parameter(torch.randn(1)),
            nn.Parameter(torch.randn(1))
        ])
        self.fc1 = nn.Linear(1, 1)
        
    def forward(self, x):
        x =self.fc1(x)
        return x

model = MyModel()
print(model.state_dict())

sd = model.state_dict()
model = MyModel()
model.load_state_dict(sd)
print(model.state_dict())
</code></pre>
All experiments were written as one (without provided tags)

username: Vitalik
cooked: <p>Hi, I wanted to compare several runs that I did in the loop creating new SummaryWriter instances like this:</p>
<pre><code class="lang-auto">for experiment_name in experiments:
    logger = SummaryWriter(self._log_path, comment=experiment_name)
    ...
    for epoch in range(5):
        ...
        logger.add_scalar("Epoch Loss", loss, epoch)
        ...
    logger.close()
</code></pre>
<p>In the log path I got several files like this:</p>
<pre><code class="lang-auto">events.out.tfevents.1609675249.nlp-vm.13735.0
events.out.tfevents.1609679736.nlp-vm.13735.1
events.out.tfevents.1609687200.nlp-vm.13735.2
events.out.tfevents.1609691662.nlp-vm.13735.3
events.out.tfevents.1609699158.nlp-vm.13735.4
events.out.tfevents.1609703743.nlp-vm.13735.5
events.out.tfevents.1609711308.nlp-vm.13735.6
events.out.tfevents.1609716054.nlp-vm.13735.7
</code></pre>
<p>But the Tensorboard displays all runs as one:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d30e6592ab54112925928c1015c77d6f3c298f07.png" alt="image" data-base62-sha1="u75AEzEKlTCd4youFR5Uzk2QLDp" width="336" height="231"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba52cc1348a958c459ff48544369db3410b9afa8.png" alt="image" data-base62-sha1="qAicv1OsoQQe2C6KE5taZr46CSc" width="296" height="157"></p>
<p>What I should do to fix it and can I do it without re-running all experiments?</p>

username: Vitalik
cooked: <p>Found: the events files that I have in the one folder should be in the separate and the folder name will be displayed as an experiment name.<br>
Also found the important note in the SummaryWriter docstring:</p>
<pre><code class="lang-auto">comment (string): Comment log_dir suffix appended to the default
``log_dir``. If ``log_dir`` is assigned, this argument has no effect.
</code></pre>
What is the fastest way to index the knn tensor?

username: Zhiyuan_Zhang
cooked: <h2>I use the following codes to get the neighbor points. However, the speed is low. I use for loop to obain the desired neighbors. Is there more efficient way? torch.gather? or torch.take? How to index?</h2>
<h2>pts=torch.randn(2,1024,3)  # [batch, point_number, dim]<br>
qrs=torch.randn(2,512,3)<br>
D=torch.cdist(qrs, pts)<br>
K=64  <span class="hashtag">#number</span> of neighbors<br>
dist, idx = D.topk(k=K, dim=-1, largest=False)<br>
nn_pts = torch.stack([pts[n][i,:] for n, i in enumerate(torch.unbind(idx, dim = 0))], dim = 0) # [batch, queries_number, K, dim]</h2>

username: ptrblck
cooked: <p>You could directly index <code>pts</code> via:</p>
<pre><code class="lang-python">pts=torch.randn(2,1024,3) # [batch, point_number, dim]
qrs=torch.randn(2,512,3)
D=torch.cdist(qrs, pts)
K=64 #number of neighbors
dist, idx = D.topk(k=K, dim=-1, largest=False)
nn_pts = torch.stack([pts[n][i,:] for n, i in enumerate(torch.unbind(idx, dim = 0))], dim = 0) # [batch, queries_number, K, dim]


ret = pts[torch.arange(pts.size(0)).unsqueeze(1).unsqueeze(2), idx]
print((ret == nn_pts).all())
&gt; tensor(True)
</code></pre>
Broadcast torch.gather (4-D array with 2-D index)

username: zeyuyun1
cooked: <p>Hi,</p>
<p>I think this question is very similar to this <a href="https://discuss.pytorch.org/t/access-3d-tensor-by-2d-index/54889">post</a> (which explains how to access an 4-D tensor through 3-D index), but I still can’t figure it out.</p>
<p>I want to access an 4-D tensor through 2-D index like following:</p>
<pre><code class="lang-auto">dim1 = 3
dim2 = 4
dim3 = 5
dim4 = 10
source = torch.randn((dim1,dim2,dim3,dim4))
index = torch.randint(dim3, (dim1, dim2))
ret =  torch.FloatTensor(dim1, dim2, dim3)

for i in range(dim1):
    for j in range(dim2):
        ret[i,j,:] = source[i, j, index[i,j], :]
</code></pre>

username: ptrblck
cooked: <p>I assume <code>ret</code> should be initialized with the shapes <code>[dim1, dim2, dim4]</code>, otherwise you’ll get a shape mismatch error in the posted loop.<br>
If that’s the case, this code should work:</p>
<pre><code class="lang-python">dim1 = 3
dim2 = 4
dim3 = 5
dim4 = 10
source = torch.randn((dim1,dim2,dim3,dim4))
index = torch.randint(dim3, (dim1, dim2))
ret =  torch.zeros(dim1, dim2, dim4)

for i in range(dim1):
    for j in range(dim2):
        ret[i,j,:] = source[i, j, index[i,j], :]


ret2 = torch.zeros(dim1, dim2, dim4)
ret2[torch.arange(dim1).unsqueeze(1), torch.arange(dim2)] = source[torch.arange(dim1).unsqueeze(1), torch.arange(dim2), index]

print((ret == ret2).all())
&gt; tensor(True)
</code></pre>
Independent optimizations on multiple GPUs

username: imesery
cooked: <p>Hey,</p>
<p>I have a classic use-case of white-box adversarial attacks generation, where for each input image from the test set (using a dataloader and a pre-trained model) I generate an adversarial example using a standard training process with Adam optimizer.</p>
<p>I have a machine with multiple GPUs, so I want to parallelize the optimization on different images, since they’re independent.<br>
Essentially, we can look at it as completely different processes, except they use the same trained model (could be different copies of it if necessary) and there has to be a “main” process that in every iteration loads &lt;num_gpus&gt; samples from the dataloader, and sends each one of them to a different GPU.</p>
<p>what’s the cleanest way to implement this parallelization?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>One approach would be to store the model copies from each GPU in e.g. a <code>list</code>, push the data chunks to the appropriate device, and execute the training using each model:</p>
<pre><code class="lang-python">models = [modelA.to('cuda:0'), modelB.to('cuda:1), ...]
dataA = dataA.to('cuda:0')
dataB = dataB.to('cuda:1')
...
data = [dataA, dataB, ...]

for model, input in zip(models, data):
    output = model(input)
    ...
</code></pre>
<p>As long as the workload is not tiny and the CPU can schedule the kernel launches and run ahead, you should see overlapping computation.</p>
RTX 3090 compatibility with old pytorch versions

username: normster
cooked: <p>Hi,</p>
<p>I’m trying to run some code which requires an older version of pytorch (v1.1.0) on an RTX 3090 card and sending anything to the gpu hang for a really long time. I know this is to be expected since the GPU and software compute capability differ in the major version (7.5 vs 8.6) but is there anything I can do about this or do I just have to patch the code?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>The “hang” you are seeing is most likely the JIT compilation for the (new) GPU architecture.<br>
In PyTorch 1.1.0 CUDA&gt;=11.0 wasn’t released yet, so you would need to cherry-pick all CUDA11 commits to this branch and recompile it.<br>
I think the easier way would be to update the PyTorch code base and use the latest release (also for new bug fixes and new features).</p>
BCEWithLogitsLoss() and sigmoid layer

username: spring
cooked: <p>BCEWithLogitsLoss() is used in multi-label classification. Since BCEWithLogitsLoss() combines one sigmoid layer, there is no need to set a sigmoid layer in the model when training. By the way, when evaluating, should we put the sigmoid layer in the model and then binarize activations based on 0.5 threshold?</p>
<p>Dataset<br>
input<br>
[0, 0, 1, 1, 0, 0, 0, 1]</p>
<p>target(label)<br>
[0, 1, 1, 1, 0]</p>

username: caonv
cooked: <p>No, using threshold 0 on your raw output is equivalent to the threshold of 0.5 after a sigmoid layer.</p>

username: spring
cooked: <p>You are right. I was confused a little bit. Thanks!</p>
Image as input for VAE?

username: OuisYasser
cooked: <p>Hello mate’s,<br>
I have a question on VAE.<br>
I’m working on a project for image to image translation (colourization) and I wanna use the VAE to benefit from the two losses (KLD and MSE).<br>
Can I use VAE for a problem as colourization? Because all example’s are using noise input.<br>
Thank you,</p>

username: caonv
cooked: <p>I suggest using GANs or plain CNNs. Due to the instinct of VEA, the latent vector must be sampled from a  Gaussian distribution <code>N(0,1)</code>, otherwise you would not be able to train your model.</p>
Model.train() and model.eval() when performing training and evaluation simultaneously

username: spring
cooked: <p>I recently learned that there are evaluation and training modes in the model. If I do training and evaluation at the same time to check the overtitting, where do I set model.eval() and model.train()?</p>
<p>Are the below codes correct?</p>
<pre><code># Train the model
oneEpochLossList_train = []
for i, batch in enumerate(train_loader):
    inputs, labels = batch

    # Move tensors to the configured device
    inputs = inputs.to(device)
    labels = labels.to(device)

    # Forward pass
    model.train()
    y_pred = model(inputs)
    loss = criterion(y_pred, labels)

    oneEpochLossList_train.append(loss.item())

    # Backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

# Test the model with current parameters
oneEpochLossList_test = []
for j, batch in enumerate(test_loader):
    inputs, labels = batch

    # Move tensors to the configured device
    inputs = inputs.to(device)
    labels = labels.to(device)

    # Forward pass
    model.eval()
    y_pred = model(inputs)
    loss = criterion(y_pred, labels)

    oneEpochLossList_test.append(loss.item())
</code></pre>
<p>And I want to know what the model.eval() and model.train() are exactly.<br>
Thanks.</p>

username: caonv
cooked: <p>Your code works properly, but you can just put <code>model.train()</code> and <code>model.eval()</code> right before the corresponding <code>for</code> loop for training and validating, not do it for every loop.</p>
<pre><code class="lang-auto"># Train the model
oneEpochLossList_train = []
model.train()
for i, batch in enumerate(train_loader):
   ...

# Test the model with current parameters
oneEpochLossList_test = []
model.eval()
for j, batch in enumerate(test_loader):
    ...

</code></pre>

username: spring
cooked: <p>Thanks!<br>
I edited it as you taught me.</p>
LSTM Input after Spatial CNN

username: DrDumbenstein
cooked: <p>I am working on a model that has a number of frames input (nB<em>C</em>H<em>W) and outputs a steering angle prediction. The input goes through slice-by-slice conv then through LSTM layer to smooth the prediction. The LSTM input is 3D (seq_lenght, batch, input_size) so before feding the 3D tensor to LSTM i managed to have (1, nB, input_size = C</em>H*W). I am kinda confused about the seq_lenght what should it mean in such architecture when i want to capture the temporal info between frames with LSTM?</p>

username: caonv
cooked: <p>the squence length here should be your <code>nB</code>, and the <code>batch</code> for LSTM should be 1, because you have only 1 sequence of length <code>nB</code>.</p>

username: DrDumbenstein
cooked: <p>Thank you so much for the insight! I Changed it and the loss stopped oscillating and start to decrease!</p>
How to ignore specific method in TorchScript?

username: marka17
cooked: <p>Hi!</p>
<p>Let I have the <strong>Model</strong> and some method <code>check_input_shape</code>. I always want to use this method in pytorch but I want to ignore it in TorchScript. So, is there anything (maybe some decorator) to do it? I found torch.jit.unused but this decorator raise exception when trying to call decorated function <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: googlebot
cooked: <p><code>@jit.ignore </code> `````````````</p>

username: marka17
cooked: <p>Nope, it doesn’t work. If you use the <code>torch.jit.ignore</code> and try to run TorchScript model you either use python interpreter (when pass argument drop=False) or raise exception (when pass argument drop=True).</p>

username: googlebot
cooked: <p>sorry, I assumed you wanted to ignore compilation.</p>
<p>you can use something like</p>
<pre><code class="lang-auto">if not jit.is_scripting(): check_input_shape(x)
</code></pre>
<p>to skip execution.</p>
Is there any way of accessing the output from intermediate layers?

username: hughperkins
cooked: <p>Let’s say I have a function that receives:</p>
<ul>
<li>a network module, containing multiple layers</li>
<li>the output of that network module, from passing in a batch</li>
<li>nothing else</li>
</ul>
<p>… is there any way of somehow accessing the intermediate results in the intervening layers, from passing the batch through those layers, to get the output?</p>
<p>I’m wondering if maybe autograd stores this information somewhere/somehow, so we can access it ourselves?</p>

username: albanD
cooked: <p>Hey!</p>
<p>To reduce memory usage we try very hard not to save all of these no. So you cannot guarantee that they are actually saved.<br>
You could use global nn.Module forward hooks to force saving some of the results during the forward pass so that you can access them later.</p>

username: hughperkins
cooked: <p>Ah, thats an interesting idea. I can assume that I do have access to the network before the batch was passed through it, so I could add hooks at that point. interesting.</p>

username: albanD
cooked: <p>Ho I just realized that our doc binding for global hooks is broken  <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
But you can use <code>Module.register_module_forward_hook()</code> to register a hook on <em>all</em> the modules that will run. So that you don’t have to hook every single Module in there.<br>
You can find the doc inline for now <a href="https://github.com/pytorch/pytorch/blob/df88cc3f7f5c13221a93d7d3d38e681a3d5a6b6a/torch/nn/modules/module.py#L89-L115">here</a>.</p>

username: hughperkins
cooked: <p>Note that, since I forget each time, to use this function:</p>
<pre><code>from torch import nn
nn.modules.module.register_module_forward_hook(hook)</code></pre>
Saving model and weights for torchscript separately

username: zsameem
cooked: <p>Hi,</p>
<p>As usual, I create my model and load the saved weights using</p>
<pre><code class="lang-auto">torch_model = MyModel() #Create my model
state_dict = torch.load(model_weights_path)['state_dict']
torch_model.load_state_dict(state_dict)
</code></pre>
<p>I can successfully create a torchscript model from this <code>torch.jit.script</code> which I then save, load and use in my C++ app. The model and weights are now stored together in one serialized file which we get from <code>torch.jit.script</code></p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 5, 512, 512)
traced_script_module = torch.jit.script(torch_model, x)
traced_script_module.save("torchscript_model.pt")
</code></pre>
<p>The workflow in C++ is different because the model and weights are together stored in <code>torchscript_model.pt</code>. Is there any way to save the model and weights of torchsript model separately and use similar steps as python i.e first create the model and then load the weights?</p>
Torchaudio datasets has no attribute CMUARCTIC

username: sriharsha0806
cooked: <p>I am trying to download some speech datasets from torchaudio datasets. When I am running the code, I am facing the following issues</p>
<pre><code class="lang-auto">import torchaudio.datasets as dsets
dsets.CMUARCTIC(root='./Datasets/', download=True)
</code></pre>
<p>AttributeError: module ‘torchaudio.datasets’ has no attribute ‘CMUARCTIC’</p>
<p>From some other attributes, like LIBRISPEECH, the error is  “FileNotFoundError: [Errno 2] No such file or directory: ‘./Audio_datasets/train-clean-100.tar.gz’”</p>
<p>For VCTK it is</p>
<pre><code class="lang-auto">File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/sri/anaconda3/lib/python3.8/site-packages/torchaudio/datasets/vctk.py", line 87, in __init__
    download_url(url, root)
  File "/home/sri/anaconda3/lib/python3.8/site-packages/torchaudio/datasets/utils.py", line 128, in download_url
    req_info = urllib.request.urlopen(req).info()
  File "/home/sri/anaconda3/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/home/sri/anaconda3/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/home/sri/anaconda3/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/home/sri/anaconda3/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/home/sri/anaconda3/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/home/sri/anaconda3/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
</code></pre>
<p>My torchaudio version is “‘0.4.0a0+ba6d67b’”</p>
<p>Should I download a lesser version of torchaudio?</p>

username: ptrblck
cooked: <p>You are getting a <code>404</code> error, which points to a request error. Could you try to download the dataset again, as the server might have been down or the URL moved? I was able to download the dataset just now using <code>torchaudio==0.8.0.dev20201116</code>.</p>
3D input size to LSTM

username: Anshumaan_Dash
cooked: <p>Hi Everyone,</p>
<p>I am new to using LSTMs. I have the following requirements:</p>
<p>Input to lstm: [30, 16, 2]<br>
Output from lstm: [256, 1]</p>
<p>Currently, as per the documentation, the input can be of a specific length, say n. That is one dimensional. How to have a 3d input?</p>
<p>E.g. ( <em>batch size</em> ,  <em>sequence length</em> ,  <em>input dimension</em> ) : I want the “Input dimension” as [30,16,2]</p>
<p>Also, in this case, what exactly is the “Sequence length”?</p>
<p>Thanks</p>

username: DrDumbenstein
cooked: <p>Try using x = x.view(1,  30, 2) LSTM input (seq_lenght, batch, input_size). Gd Luck</p>
How to mask an area (or NAN) in the input of a CNN model

username: UCDuan
cooked: <p>Hi all,</p>
<p>I wonder if we can mask an area in the input fields of a convolutional network. The reason is that I want to see if there is any predictability out of this masked area. I was thinking fill in zeros in the masked area. Is there any better method other than that?</p>
<p>Thanks!</p>

username: InnovArul
cooked: <p>In my understanding, if the input is normalized to be zero-mean (either by subtracting mean, std from train set or ImageNet), you could fill it with zeros (the mean-value).</p>

username: UCDuan
cooked: <p>Hi,</p>
<p>Thanks for your reply and I filled it with zeros. The problem is the model focus on the edge of zeros and non-zeros, and I think that’s because of the sharp gradient of this area. Do you have any other ideas?</p>
<p>Thanks.</p>
<p>By focus, I mean the visualization with integrated gradient shows high attribution near the edge.</p>
What software was used to create the animation in Udacity's course

username: oat
cooked: <p>May I ask if anybody knows which <strong>software</strong> was used to create the <strong>animated diagrams</strong> shown in <a href="https://www.youtube.com/watch?v=_Ok5xZwOtrk&amp;feature=emb_logo" rel="noopener nofollow ugc">Udacity’s PyTorch tutorial</a>?</p>
How to create a gpu tensor base on data_ptr?

username: scirocc19900305
cooked: <pre><code class="lang-auto">int main(){
    std::cout&lt;&lt;torch::cuda::is_available()&lt;&lt;std::endl;
    auto tensor=torch::randn({3,10},torch::kCUDA);
    auto p=tensor.contiguous().data_ptr&lt;float&gt;();
    torch::Tensor te=torch::from_blob(p,{3,10});
    std::cout&lt;&lt; te&lt;&lt;std::endl;
    return 0;
}
</code></pre>
<p>then I got “Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)”<br>
Any advice?</p>

username: ptrblck
cooked: <p>It should work, if you specify the right device in <code>from_blob</code>:</p>
<pre><code class="lang-auto">auto t = at::empty({1, 2, 3}, TensorOptions(kCUDA, 0));
auto t_ = from_blob(t.data_ptr(), {1, 2, 3}, kCUDA);
</code></pre>
How to free graph after detach?

username: clavichord93
cooked: <p>I want to use a network for extract the features, and then use the features as the inputs to another network. As the first network doesn’t need backward, so I use <code>detach</code> after the first network. However, I find that the GPU memory used by the first network is not freed. Is there any methods to free the GPU memory used by the first network after extracting the features?</p>
<p>The code is like:</p>
<pre><code class="lang-auto">features = model1(inputs).detach()
outputs = model2(features)
loss = loss_func(outputs)
optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>

username: ptrblck
cooked: <p>You could potentially <code>detach</code> and <code>clone</code> the original output, assign it to a new variable, and delete the original output later.<br>
However, wrapping the first model into a <code>with torch.no_grad()</code> block should be easier and would avoid storing these intermediate activations from the beginning.</p>
Access labels in ConcatDataset

username: nianen88
cooked: <p>Is there a way to access the labels of a dataset after using ConcatDataset in a way like how the MNIST labels are accessed with dataset.targets?</p>

username: ptrblck
cooked: <p><code>ConcatDataset</code> will assign the passed <code>datasets</code> to an internal attribute <code>self.datasets</code> and won’t change these datasets directly.<br>
If the passed datasets had the <code>.target</code> attribute, you can still assess it via:</p>
<pre><code class="lang-python">concat_dataset.datasets[0].target # assuming the first internal dataset has the target attribute
</code></pre>
How do I solve the type casting error?

username: HarshRangwala
cooked: <p>Here’s the code that performs the classification task.</p>
<pre><code class="lang-auto">def classify_face(image):
    device = torch.device("cpu")
    img = process_image(image)
    print('Image processed')
    # img = image.unsqueeze_(0)
    # img = image.float()
    
    pred = model(img)[0]
    
    # Apply NMS
    pred = non_max_suppression(pred, 0.4, 0.5, classes = [0, 1, 2], agnostic = None )
    if classify:
        pred = apply_classifier(pred, modelc, img, im0s)
    #print(pred)
    
    model.eval()
    model.cpu()
    
    print(pred)
        
    # output = non_max_suppression(output, 0.4, 0.5, classes = class_names, agnostic = False)
        
    #_, predicted = torch.max(output[0], 1)
    #print(predicted.data[0], "predicted")

    classification = torch.cat(pred)[:, -1]
    index = int(classification)
    print(names[index])
    return names[index]
</code></pre>
<p>During prediction, pred consists of x1, y1, x2, y2, conf, and class.</p>
<p>Eg: pred = [tensor([[176.64380, 193.86154, 273.84702, 306.30405,   0.83492,   2.00000]])]</p>
<p>If there are no predictions made by the model then pred is simply empty.</p>
<p>Eg: pred = [tensor([], size=(0, 6))]</p>
<p>Presently my program stops prediction if it receives an empty tensor and throws an error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "WEBCAM_DETECT.py", line 168, in &lt;module&gt;
    label = classify_face(frame)
  File "WEBCAM_DETECT.py", line 150, in classify_face
    index = int(classification)
ValueError: only one element tensors can be converted to Python scalars
</code></pre>
<p>How do I make my program sort of just ignore if there are no predictions made at a certain frame and continue onto the next frame?</p>
<p>My goal is to access the class from the tensor stored in pred, in integer.</p>
<p>Thank you for the help!</p>

username: ptrblck
cooked: <p>You could use <code>pred[0].numel()</code> to check, if the current tensor inside the <code>list</code> contains any elements and, if not, skip it in the following calculation.</p>
Difference between torch.tensor() and torch.Tensor()

username: User4
cooked: <p>Hi,</p>
<p>My question is the title, in Pytorch’s document, as I understand, the performance of <code>torch.tensor()</code> and <code>torch.Tensor()</code> is similar.</p>
<p>However, when I train my model, my data was read by <code>torch.tensor()</code> and the performance was very poor, but when I switch to <code>torch.Tensor()</code>, the model predicts correctly what I want.</p>

username: InnovArul
cooked: <p>In my undertanding, <code>torch.tensor()</code> is just a wrapper to <code>torch.Tensor</code> with configurable options.<br>
Your observation seems strange.</p>
<p>Can you check if there is any difference in data loaded when you use <code>torch.tensor()</code> vs. <code>torch.Tensor</code>?</p>

username: User4
cooked: <p><a class="mention" href="/u/innovarul">@InnovArul</a> Thank you for your reply,</p>
<p>I just change <code>torch.tensor()</code> to <code>torch.Tensor()</code>, I am training a regression model and no matter what architecture I used, after a short time of training, all the output will tend to be similar. (at first, I think it is the batch norm problem)</p>
<p>Then I decided to create a new project and code again, this time when I use <code>torch.Tensor(my_target)</code>, I got something like this</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.Tensor(10)
tensor([2.8958e+32, 3.4970e-38, 1.0141e+31, 1.1210e-44, 9.9920e-38, 1.9793e-18,
        4.5036e+16, 4.0556e-08, 2.5038e-12, 4.0058e-11])
</code></pre>
<p>Which is normal in case <code>torch.tensor()</code></p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.tensor(10)
tensor(10)
</code></pre>
<p>Therefore, I decided to come back to my old repository and change <code>tensor</code> to <code>Tensor</code> and the miracle happened.</p>
<p>For additional information, I am using pytorch 1.7.0</p>

username: ptrblck
cooked: <p><code>torch.Tensor(10)</code> will return an uninitialized <code>FloatTensor</code> with 10 values, while <code>torch.tensor(10)</code> will return a <code>LongTensor</code> containing a single value (<code>10</code>).<br>
I would recommend to use the second approach (lowercase t) or any other factory method instead of creating uninitialized tensors via <code>torch.Tensor</code> or <code>torch.FloatTensor</code>.<br>
To explicitly create a tensor with uninitialized memory, you could still use <code>torch.empty</code>.</p>
Error during training

username: Angelina_Robert
cooked: <p>Hi all,<br>
I am using the mean of features for training the network. It generates the same error, sometimes after 4 epochs, sometimes after 10 epochs or 19 epochs. I ve faced this type of probles first time. Any help in this regard will be appreciated.</p>
<p>Error after 19 epochs of training:</p>
<p>File “main.py”, line 510, in train<br>
feature_mean_stack[idx]= torch.stack([feats[idx]],dim=0).cuda(non_blocking=True)<br>
KeyError: 6</p>
<p>Here 6 is the index of class.</p>
<p>Cheers</p>

username: ptrblck
cooked: <p>Based on the error message it seems that <code>feats</code> does not always have the desired size as the indexing operation fails. I don’t know, how <code>feats</code> is initialized, but you would have to make sure it’s not depending on e.g. the model predictions.</p>
In a bidirectional LSTM which columns correspond to the forward versus backward direction

username: krishnab
cooked: <p>I am using the <code>nn.LSTM()</code> layer with bidirectional set to True. I have a 3 layer network, so I get a <code>hidden</code> tensor that has is [6, BatchSize, input_size]. Now, the first dimension of <code>hidden</code> contains both the forward and backward direction of the LSTM for each of the 3 layers, but I was not clear on how that tensor is laid out. So does <code>hidden[0, :, :]</code> correspond to the first or bottom layer in the forward direction, and then <code>hidden[1,:,:]</code> correspond to the bottom layer in the backward direction?</p>
<p>I actually need to concatenate the hidden nodes from the forward and backward pass on each layer, so I imagine that would be something like the code below, but can someone correct me if I am wrong.</p>
<pre><code class="lang-auto">output, (hidden, cell) = self.lstm1(x)
new_hidden = torch.cat((hidden[0::2, :. :], hidden[1::2, :, :], dim=2)
</code></pre>
<p>So new dimension should be of size [3, BatchSize, input_size].</p>

username: ptrblck
cooked: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html?highlight=lstm#torch.nn.LSTM"><code>LSTM</code> docs</a>:</p>
<blockquote>
<p><strong>h_n</strong> of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len.<br>
Like <em>output</em>, the layers can be separated using <code>h_n.view(num_layers, num_directions, batch, hidden_size)</code> and similarly for <em>c_n</em>.</p>
</blockquote>
<p>You could thus use the mentioned <code>view</code> operation to separate the directions and index this tensor.</p>
Model is performing well in Colab, while it does not converge at all in local laptop

username: RRCode
cooked: <p>Hello All,</p>
<p>I created a pipeline for classification problem in Colab Notebook with pytorch version 1.7.0+cu101 and the model is converging very well there. But if I use the same pipeline to execute in my local laptop, the accuracy never goes above 10% with pytorch version 1.7.1</p>
<p>I have no clue, what am I missing here. Please help.</p>

username: ptrblck
cooked: <p>If you are using different operating systems (Linux in Colab vs. Windows on your Laptop), make sure the data paths are equal, as e.g. the sorting is not equal on those systems.<br>
If that doesn’t help, try to update Colab to 1.7.1 and check, if you are still able to train the model or if there might be a potential issue in the latest PyTorch release.</p>
Implementing One Class SVM in PyTorch

username: ambareeshravi
cooked: <p>Is it possible to implement One Class SVM as a part of a Neural Network in PyTorch?</p>
<p>I am doing an experiment for my research on outlier detection and I need to implement One Class SVM as a layer and its objective function as apart of the main objective / loss function. I know I could use ScikitLearn and LibSVM for the same. But my experiment needs it to be a part of a neural network ensemble I am building. So the weights should be iteratively learnable with the other parameters of the NN.</p>
<p>Any help or pointers would be appreciated.</p>

username: ptrblck
cooked: <p>You could try to use e.g. <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiLabelMarginLoss.html#torch.nn.MultiLabelMarginLoss"><code>nn.MultiLabelMarginLoss</code></a>, which is the multi-classification hinge loss and create the desired targets for your “One Class SVM”.</p>
How to get last element from the prediction?

username: HarshRangwala
cooked: <p>I am new to PyTorch. I have a variable pred which has a list of a tensor.</p>
<pre><code class="lang-auto">print(pred)
output: [tensor([[176.64380, 193.86154, 273.84702, 306.30405,   0.83492,   2.00000]])]
</code></pre>
<p>So I wanted to access the last element which is the class. I did that by first converting the list into a tensor.</p>
<pre><code class="lang-auto">x = torch.stack(pred)
output: tensor([[[176.64380, 193.86154, 273.84702, 306.30405,   0.83492,   2.00000]]])
</code></pre>
<p>Now, how do I access the last element or is there any better/efficient way of doing this??</p>
<p>Thank you for the help!</p>

username: ptrblck
cooked: <p>You can access the last element (<code>2.0</code>) via <code>x[0, 0, -1]</code>.</p>
3D convolutions of depth 1

username: the-dharma-bum
cooked: <p>Hello !</p>
<p>This may seem like a silly question but here it is:<br>
I need to develop a generic pipeline to classify pretty long video (typically 1 mn, with 25 rgb fps that is a tensor of shape (3, 25*60, width, height)).<br>
I thought a good way to do it at a reasonable computational cost would be to divide a video into a sequence of smaller segments (10sc for instance). This way, a network would predict each sequence, and the predictions would be aggregated at the end.<br>
One should be able to define the depth (that is, the number of frames) of one element of this sequence, as well as the 3D CNN that will be used to classify each element.<br>
Now my question is, what if one choose a depth of 1 ?<br>
It will give to a 3D CNN a cube of depth 1. Is that an issue ? Is that equivalent to giving a 2D image to a 2D CNN ?<br>
Of course I could detect if the chosen depth is equal to 1 and instantiate a 2D CNN in that case instead of a 3D CNN, but is it really necessary ?</p>
<p>I hope i’m being clear, and thank you in advance !</p>

username: ptrblck
cooked: <p>Yes, a depth of 1 would be equal to a 2D image and you could indeed use <code>nn.Conv2d</code> instead, since the kernel size in this dimension would be 1.</p>
Outputshape for cnn

username: swarup
cooked: <p>how to get the shape of the output image after each layer of CNN for verification?</p>

username: ptrblck
cooked: <p>I’m not sure what kind of verification you would like to perform, but based on the input shape you could manually calculate the output shape using the formula from the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d">docs</a>.</p>
Why CUDA runs out of memory when calling torch.save?

username: Rongzhao_Zhan
cooked: <p>Hello!</p>
<p>I was training a model for 3D semantic segmentation, which impose a very heavy memory pressure. <code>torch.__version__</code> is <code>0.1.11+37d9568</code></p>
<p>After I fit data and model into GPU, everything went well until I tried to save a checkpoint using <code>torch.save</code>. And I got the following trace back:</p>
<p><code>THCudaCheck FAIL file=/home/zhang/src/pytorch/torch/csrc/generic/serialization.cpp line=38 error=2 : out of memory Traceback (most recent call last): ... File "/home/zhang/pytorch/packages/torchmed/utils/trainer.py", line 152, in _snapshot torch.save(state_dict, filename) File "/home/zhang/anaconda3/lib/python3.6/site-packages/torch/serialization.py", line 120, in save return _save(obj, f, pickle_module, pickle_protocol) File "/home/zhang/anaconda3/lib/python3.6/site-packages/torch/serialization.py", line 192, in _save serialized_storages[key]._write_file(f) RuntimeError: cuda runtime error (2) : out of memory at /home/zhang/src/pytorch/torch/csrc/generic/serialization.cpp:38</code></p>
<p>Does that mean I should reserve some memory for checkpoint saving? If so, how much should I reserve?</p>
<p>BTW, when I make a checkpoint, the training and testing processes should have finished, and the output <code>Variable</code>, i.e. <code>loss</code>, has been out of scope, which, I think, means the used GPU memory could be freed, so that there should be enough memory for the snapshoting. Am I wrong about the memory free mechanism?</p>
<p>Many many thanks for any suggestion!!</p>

username: jekbradbury
cooked: <p>I think this is worth filing as a bug if you haven’t resolved it yet – there should be no need, in principle, for GPU memory to be allocated during serialization, so if the serialization code does that it could probably be improved.</p>

username: Rongzhao_Zhan
cooked: <p>I resolved this problem by moving the model to cpu and then performing seialization.<br>
I also think this is a bug, but I have no idea how to fix it …</p>

username: bfortuner
cooked: <p>+1. I had the same problem and moving the model to cpu before saving also worked.</p>

username: lan2720
cooked: <p>+1. I also had the problem… Is there any other method to solve it?</p>

username: Himanshu_Upreti
cooked: <p>+1, I am also getting the same error, but with device =  ‘cuda:1’. The code is executing properly with ‘cuda:0’. I don’t know why?</p>

username: Mongo
cooked: <p>I solve this problem by your method, thanks!</p>
MultivariateNormal backward error with respect to covariance

username: Qinsheng_Zhang
cooked: <p>Here is a simple piece of code to learn covariance of 2D gaussian.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.optim import Adam

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.cov = nn.Parameter(torch.tensor([0.],requires_grad = True))
        self.cur_cov = torch.cat((torch.tensor([1.0]), self.cov,self.cov,torch.tensor([1.0])),axis=0).reshape(2,2)
        self.dist = torch.distributions.multivariate_normal.MultivariateNormal(\
                torch.tensor([0.,0.]),self.cur_cov)

    def forward(self,x):
        return self.dist.log_prob(x)

model = Net()
op = Adam(model.parameters(),lr=0.1)


data = torch.tensor([1,2,3,5]).reshape(-1,2)
for i in range(10):
    op.zero_grad()
    log_p = model(data)
    loss = -torch.sum(log_p)
    loss.backward()
    op.step()
</code></pre>
<p>The code raise error</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>However, as I added the flag</p>
<pre><code class="lang-auto">for i in range(10):
    op.zero_grad()
    log_p = model(data)
    loss = -torch.sum(log_p)
    loss.backward(retain_graph=True)
    op.step()
    print(loss.item())
</code></pre>
<p>The loss and <code>model.cov</code> did not change.</p>
Why autograd will accumate gradients?

username: oat
cooked: <p>May I ask why PyTorch’s autograd will <strong>accumulate gradients</strong>, if <code>optimizer.zero_grad()</code> is not called?</p>
<p>What is an example of the use case of cumulated gradients?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>You could simulate a larger batch size by accumulating the gradients of smaller batches and scaling them with the number of accumulations. This can be useful e.g. if the larger batch size would be beneficial for training but doesn’t fit onto your GPU.<br>
Accumulating the gradients gives you the ability to scale them manually afterwards without enforcing any assumptions on your use case.</p>

username: oat
cooked: <p>Thanks for the explanation, <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
A question about AllenNLP

username: fan_zhimin
cooked: <p>AllenNLP is based on torch.</p>
<p>Recently, I was learning a model based on Elmo. It used Allen NLP, but after I installed it, I directly threw the following error. Is this my wrong installation? I tried to uninstall the installation again, but it didn’t solve it. Has anyone encountered such a problem? I can’t start with this problem.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5cc8c931910a36725b89849c016b8ba877ae3f1.png" data-download-href="https://discuss.pytorch.org/uploads/default/c5cc8c931910a36725b89849c016b8ba877ae3f1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5cc8c931910a36725b89849c016b8ba877ae3f1_2_690x226.png" alt="image" data-base62-sha1="sdOiYP8NcVpIkCoe968ObSiDL9v" width="690" height="226" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5cc8c931910a36725b89849c016b8ba877ae3f1_2_690x226.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5cc8c931910a36725b89849c016b8ba877ae3f1_2_1035x339.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5cc8c931910a36725b89849c016b8ba877ae3f1.png 2x" data-dominant-color="313336"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1214×399 25.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25d03e9dc76727506f71492d72ea3e9277b2a4d8.png" data-download-href="https://discuss.pytorch.org/uploads/default/25d03e9dc76727506f71492d72ea3e9277b2a4d8" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25d03e9dc76727506f71492d72ea3e9277b2a4d8.png" alt="image" data-base62-sha1="5ovPmlqvANSDBh3Y8JzfH4HE1zG" width="690" height="68" data-dominant-color="2F2E31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">879×87 7.43 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>If I’m not mistaken, the <code>resource</code> module is Unix-specific and might not be available on Windows.<br>
However, you could still try to install it via <code>pip install python-resources</code> and check, if the package can be found. If not, you would have to use a Unix-like OS or remove the <code>resource</code> usage from the script.</p>
Constructing reference/baseline for LSTM - Layer Integrated Gradients

username: devpramod
cooked: <p>`# compute attributions and approximation delta using layer integrated gradients</p>
<p>attributions_ig, delta = lig.attribute((input_indices,h), (reference_indices,h),<br>
n_steps=500, return_convergence_delta=True)<br>
`<br>
When I run the above code, I get the following error.<br>
AssertionError: baseline input argument must be either a torch.Tensor or a number however &lt;class ‘tuple’&gt; detected</p>
<p>But the documentation says that a tuple can be passed as baseline argument.</p>
<p>Am I doing this right?</p>

username: anniezhi
cooked: <p>Similar problem here. Did you find a way out?</p>
The default value of dataloader multiprocessing_context is "spawn" in a spawned process?

username: Euruson
cooked: <p>The default value of dataloader <code>multiprocessing_context</code> seems to be “spawn” in a spawned process on Unix.  I will get OOM unless I set  <code>multiprocessing_context="fork"</code> explicitly.</p>
<p>It doesn’t behave as documentation says:</p>
<blockquote>
<p>On Unix, <code>fork()</code> is the default <a href="https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing" rel="noopener nofollow ugc"><code>multiprocessing</code></a> start method. Using <code>fork()</code> , child workers typically can access the <code>dataset</code> and Python argument functions directly through the cloned address space.</p>
</blockquote>

username: osalpekar
cooked: <p>Do you have any logs indicating spawn is being used? Also, by any chance are you on MacOS? (Asking because spawn is the default on MacOS).</p>

username: Euruson
cooked: <p>Running on Ubuntu.</p>
<p>I use <code>torch.multiprocessing.spawn(main, ...)</code> to run DDP training. The dataloaders are initialized in <code>main()</code>.  The memory usage without expilict setting <code>multiprocessing_context</code> is out of limit, as well as setting <code>multiprocessing_context="spawn"</code>. However, setting <code>multiprocessing_context="fork"</code> makes memory usage much smaller, like 25G/ 63G. The OOM happens at <code>enumerate(dataloader)</code>.</p>

username: osalpekar
cooked: <p>I tried running a simple script that mimics your described code, but I am not able to reproduce the issue - this might be due to having different environments, etc. Do you mind creating a GitHub issue in the PyTorch repo and filling out the detailed environment description there (PT version, CUDA, NCCL version, etc.)? That way, we can try to reproduce this issue in an environment similar to yours. Thanks!</p>
Lasso regularization problem

username: lifeblack
cooked: <p>Hi… I need to  solve this minimization problem in one part of my code…<br>
min_x {torch.norm(x_n - x,2)+lambda1*torch.norm(y_n - Ax,2)}<br>
I used cvxpy library for this problem,I coded this like bellow:</p>
<pre><code class="lang-auto">x_es = cp.Variable(64)
lambda1 = cp.Parameter(nonneg=True)
lambda2 = cp.Parameter(nonneg=True)
problem = cp.Problem(cp.Minimize(objective_fn(x,x_n,y_n,A,lambda1,lambda2)))
</code></pre>
<p>I got this error during Ax part:</p>
<pre><code class="lang-auto">matmul: Input operand 1 does not have enough dimensions (has 0, gufunc core with signature (n?,k),(k,m?)-&gt;(n?,m?) requires 1)
</code></pre>
<p>how could I can solve that minimization? I  want to solve  this part without training…</p>