akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How do I load multiple images using only their file name and path?

Cindy_Liu: <p>I’ve been given a list of file paths for a ton of images and need to load them, then have AlexNet return the lc7 feature vector of these images. They are almost all from different folders. I don’t know how to really approach this so any help would be great.</p>

Shisho_Sama: <p>the easies way is to create a new Dataset (inherit from torch.utils.data.Dataset), and implement <code>__getitem__(self, index)</code><br>
it makes your life much easier.</p>
Serialization - saving multiple models in a single file

cheggars: <p>I am trying to save several models in a single file. Strangely, I faced an issue that a single Linear layer cannot be saved and subsequently read, but a Linear inside Sequential can be saved and read.</p>
<p>The following code works ok:</p>
<pre><code class="lang-auto">	const std::string FN = "d:\\test.dat";
	struct mtype : torch::nn::Sequential 
	{
		mtype() : torch::nn::Sequential(torch::nn::Linear(5, 2)) {};
	};
	// torch::nn::Linear MLP(10,2);

	mtype MLP;
	torch::serialize::OutputArchive MLPArchive;
	
	for (int i = 0; i &lt; 10; i++)  
	{	
		torch::serialize::OutputArchive nestedarchive;
		MLP-&gt;save(nestedarchive);
		MLPArchive.write(std::to_string(i), nestedarchive);
	}
	MLPArchive.save_to(FN);
        torch::serialize::InputArchive IA;
	IA.load_from(FN);	
</code></pre>
<p>but if i change saved model from Sequential to Linear as follows, than it throws an exception on the last line load_from(file). What’s the problem?</p>
<p>The following code throws an exception on the last line.</p>
<pre><code class="lang-auto">	const std::string FN = "d:\\test.dat";
	torch::nn::Linear MLP(10,2);
	torch::serialize::OutputArchive MLPArchive;
	
	for (int i = 0; i &lt; 10; i++)  
	{	
		torch::serialize::OutputArchive nestedarchive;
		MLP-&gt;save(nestedarchive);
		MLPArchive.write(std::to_string(i), nestedarchive);
	}
	MLPArchive.save_to(FN);
        torch::serialize::InputArchive IA;
	IA.load_from(FN);	
</code></pre>

cheggars: <p>Just in case it may help somebody, I found that nested archive must be initiated with the same compilation unit as the main archive. For the above example:</p>
<pre><code class="lang-auto">torch::serialize::OutputArchive nestedarchive(MLPArchive.compilation_unit());
</code></pre>
Excessive memory consumption while using DistributedDataParallel

Sentient07: <p>Hello,</p>
<p>I have been using FasterRCNN from Torchvision’s <a href="https://github.com/pytorch/vision/blob/v0.4.0/torchvision/models/detection/transform.py" rel="nofollow noopener">models</a> that uses <code>DistributedDataParallel</code> for training. However, I find that while using multiple GPU, the memory consumption is far more than without multiple GPU. Here is my code</p>
<pre><code class="lang-auto">  kwargs = {}
  kwargs['min_size'] = args.min_size
  kwargs['max_size'] = args.max_size
  model = ModifiedFRCNN(cfg=cfg, custom_anchor=args.custom_anchor,
                        use_def=args.use_def, cpm=args.cpm,
                        default_filter=args.default_filter,
                        soft_nms=args.soft_nms,
                        upscale_r=args.upscale_r, **kwargs).cuda().eval()
  model = restore_network(model)
  model_without_ddp = model
  dataset = GenData(args.test_dataset,
                    args.base_path,
                    dataset_param=None,
                    train=False)

  if args.n_gpu &gt; 1:
      init_distributed_mode(args)
      model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu],
                                                        find_unused_parameters=True)
      model_without_ddp = model.module
      sampler = torch.utils.data.distributed.DistributedSampler(dataset)
      batch_sampler = torch.utils.data.BatchSampler(sampler,
                                                    args.batch_size,
                                                    drop_last=True)
      data_loader = torch.utils.data.DataLoader(dataset,
                                                batch_sampler=batch_sampler,
                                                num_workers=args.num_workers,
                                                collate_fn=coco_collate)
      metric_logger = MetricLogger(delimiter="  ")
      header = 'Valid:'
      batch_iterator = metric_logger.log_every(data_loader, 100, header)
  else:
      model = model.cuda()
      data_loader =  iter(data.DataLoader(dataset, args.batch_size, shuffle=False,
                                                num_workers=args.num_workers,
                                                collate_fn=coco_collate))
      batch_iterator = iter(data_loader)

</code></pre>
<p><code>ModifiedFRCNN</code> is a class that inherits FRCNN to make trivial changes, such as parameter, postprocessing etc.<br>
Case 1 : When  <code>n_gpu=1</code>, I am able to use a batch size of upto 8.<br>
Case 2 : When <code>n_gpu=4</code>, I am unable to even use a batch size of 1.</p>
<p>Both the above mentioned cases are on same the GPU, 2080Ti. Can someone please help me understand what causes this? Here is the command I use to launch the script</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=4 --use_env test.py &lt;other_arguments&gt; --world_size 4 --n_gpu 4
</code></pre>
<p>Thank you and Regards,</p>
RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'other' in call to _th_max

Sangwon_Jake: <pre><code class="lang-auto"> File "/home/jake/venv/lib/python3.7/site-packages/torchvision/ops/boxes.py", line 153, in box_iou
    lt = torch.max(boxes1[:, None, :2], boxes2[:, :2])  # [N,M,2]
RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'other' in call to _th_max
</code></pre>
<pre><code class="lang-auto">annotations-&gt; [{'boxes': tensor([[131.5124, 169.4232, 165.8235, 209.7217],
        [157.2659, 185.7604, 199.0850, 210.4478],
        [184.4726, 171.3594, 211.5178, 210.3267],
        [170.5867, 154.1751, 211.7600, 188.7858],
        [162.8364, 156.2324, 182.1313, 186.7285],
        [131.2702, 149.8185, 149.4349, 172.3276],
        [192.0614, 140.5002, 208.4500, 155.2642],
        [165.2583, 140.5002, 188.2670, 158.0476],
        [169.5371, 131.9080, 193.3531, 151.1497],
        [131.1895, 133.2392, 140.1507, 156.3534]], device='cuda:0',
       dtype=torch.float64), 'labels': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], device='cuda:0'), 'area': tensor([1382.6857, 1032.4053, 1053.8795, 1425.0382,  588.4218,  408.8705,
         241.9614,  403.7413,  458.2574,  207.1317], device='cuda:0'), 'iscrowd': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')}]
</code></pre>

ptrblck: <p>Based on the error message it seems that <code>boxes1</code> contains <code>float64</code> values, while <code>boxes2</code> uses <code>float32</code> values.<br>
You would have to make sure to use the same data type for both tensors.<br>
Depending which type you would like to keep you should either use:</p>
<pre><code class="lang-python">boxes1 = boxes1.float()
</code></pre>
<p>or</p>
<pre><code class="lang-python">boxes2 = boxes2.double()
</code></pre>
Applying cross entropy loss on one-hot targets

887574002: <p>Hi, I have labels in one-hot format with size [bsz, bsz<em>2]. My input also is a matrix of shape [bsz,bsz</em>2]. I want to use cross-entropy loss. I searched the pytorch doc and I found that we can’t apply cross-entropy loss on one hot except in the following way:</p>
<pre><code class="lang-auto">out = torch.FloatTensor([[0.05, 0.9, 0.05], [0.05, 0.05, 0.9], [0.9, 0.05, 0.05]])

y1 = torch.FloatTensor([[0, 1, 0], [0, 0, 1], [1, 0, 0]])

_, targets = y1.max(dim=0)
loss = nn.CrossEntropyLoss()(out, Variable(targets))
print(loss.item())

################################

a = torch.randn(3,3)
print(a.shape)

b = torch.randn(3,3)

c = torch.cat((a,b),dim=1)

lable = torch.randn(3,3)

_,target =lable.max(dim=0)

criterion = torch.nn.CrossEntropyLoss(reduction="sum")

loss = criterion(Variable(target),c)

IndexError: Dimension out of range (expected to be in the range of [-1, 0], but got 1)
</code></pre>
<p>I can’t understand why I get an error in the second case.  Any help is really appreciated.</p>

ptrblck: <p>Try to apply the <code>max</code> operation on dim1:</p>
<pre><code class="lang-python">target = torch.argmax(lable, 1)
</code></pre>
<p>as dim0 is the batch dimension, while dim1 should correspond to the “class dimension”.</p>
How do I show a GIF of tensors converted into images?

costa98: <p>I have a neural network whose goal is to create a small video. Each time i run the nn it return an image (tensor converted in image using plt.imgshow()), and i have to create a GIF using those images</p>

ptrblck: <p>You could store each image separately and use an external program to create the GIF or use <code>OpenCV</code> (or another lib) in Python.<br>
<a href="https://stackoverflow.com/questions/753190/programmatically-generate-video-or-animated-gif-in-python">This StackOverflow thread</a> has some suggestions.</p>
RuntimeError: bool value of Tensor with more than one value is ambiguous - image classification

Sarah_W: <p>Hi all together,</p>
<h2>I have problems with using the nn.MSELoss method for image classification: RuntimeError: bool value of Tensor with more than one value is ambiguous. Size of image and label are both (8, 104) - Do you have any ideas what the problem could be?</h2>
<p>model = EasyNet()<br>
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.5)</p>
<p>def train(epoch):<br>
# Training<br>
model.train()<br>
for image, label in train_loader:<br>
# Transfer to GPU<br>
# image, label = image.to(device), label.to(device)<br>
if image is None:<br>
continue</p>
<pre><code>    # Forward
    image = Variable(image)
    label = Variable(label)
    pred = model(image)
    print(pred.size())
    print(label.size())
    # print(pred)
</code></pre>
<p>optimizer.zero_grad()<br>
loss = nn.MSELoss(pred, label)</p>
<pre><code>    # Backward
    loss.backward()
    optimizer.step()
    global train_iter</code></pre>

ptrblck: <p>You are currently trying to pass <code>pred</code> and <code>label</code> to the initialization of <code>nn.MSELoss</code>.<br>
Use the functional API instead (<code>F.mse_loss</code>) or initialize the criterion before using it:</p>
<pre><code class="lang-python">criterion = nn.MSELoss()
...
optimizer.zero_grad()
loss = criterion(pred, label)
...
</code></pre>

Sarah_W: <p>Thank you!! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

saba: <p>HI Ptrblck,</p>
<p>I chnge my loss unction from binary cross entropy to the torch.nn.CrossEntropyLoss</p>
<p>but it givee me this error "RuntimeError: bool value of Tensor with more than one value is ambiguous<br>
"<br>
The code is "</p>
<pre><code class="lang-auto">  output = netD(real_cpu).view(-1)
 label = torch.full((b_size,), real_label, device=device)
  label=torch.mul(label,0.9)      
  errD_real = criterion(output, label)
</code></pre>

ptrblck: <p>I don’t know, which line of code raises this particular error message, but <code>nn.CrossEntropyLoss</code> expects class indices as the target, while you seem to use soft targets.</p>
<p>You could use <a href="https://github.com/pytorch/pytorch/issues/7455#issuecomment-513062631">this implementation</a> for label smoothing.</p>
Torchsummary for LSTM

Abheesht_Sharma1: <p>I’ve been trying to use torchsummary to print the summary of an LSTM model. However, it shows an error:<br>
“tuple’ object has no attribute 'size”. Is there a way to print the summary of an LSTM model using torchsummary?</p>

shuhongwang-zzu: <p>I ran into the same problem as you, how did you solve it</p>
Background segmentation

Adria_Ciurana: <p>I do not know if it is the right forum, if not, I apologize.</p>
<p>You see I’m trying to find some architecture to segment foreground / background. Initially I thought of a multiclass segmentation model learned with COCO + CRF (DeepLabv3 +). The problem is that, in my opinion, this approach is not good enough. Well, the model has learned a different problem.</p>
<p>The medium in which the algorithm would work works in static camera and I can perfectly obtain a reference image of the background. The question is: Is there an architecture that only separates foreground/background using a background image as a reference? That the algorithm is capable of segmenting objects that it does not know?</p>
<p>Obviously you could use classic background subtraction: MOG2, KNN, etc … The problem is that they are too sensitive. Is there a more robust analog in Deep Learning?</p>
<p>Thank you very much!</p>
Transformer model of Language Model Example performs worse much

Yoshiharu_Sato: <p>I tried laguage model sample of Example at <a href="https://github.com/pytorch/examples/tree/master/word_language_model" rel="nofollow noopener">https://github.com/pytorch/examples/tree/master/word_language_model</a>.</p>
<p>LSTM setting performs well. It shows the perplexity of around 100. And generated English sentences look good. However Transformer setting performs bad. It shows  the perplexity of around 1000. Transformer is highly appreciated these days, and those results look strange.</p>
<p>I may miss something, but this example doesn’t make sense if it tried to show a viability of the PyTorch implementation of transformer. Are there any suggestions on any changes of hyper parameter or something for Transformer model to perform comparatively with RNNs?</p>
Lstm multi layer help

imrankhan_pythonLang: <p><strong>Anyone, Please Help how can I use multiple LSTM layer</strong><br>
[NOTE: LSTM 1 and 2 are commented because when I try to add I face dimension problem ]</p>
<pre><code class="lang-auto">
class LSTMnetwork(nn.Module):
    def __init__(self,input_size=1,hidden_size=100,output_size=1):
        super().__init__()
        self.hidden_size = hidden_size
        
        # Add an LSTM layer:
        self.lstm = nn.LSTM(input_size,hidden_size)
        #self.lstm1 = nn.LSTM(hidden_size,200)
        #self.lstm2 = nn.LSTM(200,300)
        
        self.linear = nn.Linear(hidden_size,output_size)
        
        self.hidden = (torch.zeros(1,1,self.hidden_size),
                       torch.zeros(1,1,self.hidden_size))

    def forward(self,seq):
        lstm_out, self.hidden = self.lstm(seq.view(len(seq),1,-1), self.hidden)
        #lstm_out, self.hidden = self.lstm1(lstm_out, self.hidden)
        #lstm_out, self.hidden = self.lstm2(lstm_out, self.hidden)
        pred = self.linear(lstm_out.view(len(seq),-1))
        return pred[-1]  # we only want the last value




</code></pre>

harsha_g: <p>Are you aware of the <code>num_layers</code> argument for <code>nn.LSTM</code>? <a href="https://pytorch.org/docs/master/generated/torch.nn.LSTM.html" rel="nofollow noopener">https://pytorch.org/docs/master/generated/torch.nn.LSTM.html</a></p>

imrankhan_pythonLang: <p><code>num_layers</code> layer just repeat same input and hidden. I want a custom hidden.</p>

vdw: <p>Well, you probably need multiple hidden states, one for each LSTM, since the last dimension (i.e., hidden dimension) is different (in your case 100, 200, and 300).</p>
<p>By the way, I also feel that</p>
<pre><code>pred = self.linear(lstm_out.view(len(seq),-1))
return pred[-1]  # we only want the last value
</code></pre>
<p>doesn’t make sense.</p>

harsha_g: <p>Or you can simply use <code>nn.LSTMCell</code>. Here’s an <a href="https://github.com/pytorch/examples/blob/master/time_sequence_prediction/train.py" rel="nofollow noopener">example</a>.</p>

vdw: <p><a class="mention" href="/u/harsha_g">@harsha_g</a> This example also uses one hidden state for each of the 2 LSTM cells: <code>(h_t, c_t)</code> and <code>(h_t2, c_t2)</code>.</p>
How does pytorch import itself in internal implementation

zhuzilin: <p>Hi all, I’m new to the pytorch community. And when I’m reading the source code, I found that pytorch always import itself in the python code, for example:</p>
<pre><code class="lang-python"># torch/tensor.py
import torch
import torch._C as _C
...
</code></pre>
<p>Could anyone tell me why won’t this cause import cycle? Thank you!</p>
Unexpected behavior

nnresearch: <p>Hi,<br>
i am trying to run inference with a CNN (retinanet in this case) and i see very unexpected behavior:</p>
<ul>
<li>when running the network with bmp files everything works as expected</li>
<li>when running the network on a specific mp4 video i got the following crash for the second frame:<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/_functions.py”, line 54, in forward<br>
assert all(map(lambda i: i.is_cuda, inputs))</li>
<li>when i examine the errors i see that for the video i must run before the inference:<br>
os.environ[“CUDA_VISIBLE_DEVICES”] = device<br>
adding this line helps with the video while it is not needed at all with the images.</li>
<li>if i run this line:<br>
print(‘CUDA available: {}’.format(torch.cuda.is_available()))<br>
before the line mentioned above i get the same crash , but when i put it after the setting of cuda visible devices it runs ok.<br>
again this happens only for the video and not for the images.</li>
</ul>
<p>when checking the image - it is h * w * 8 for the images and h * w * 24 for the video.</p>
<p>is there any reason why it fails to load always the second image?</p>
Torch.cuda.is_available() returns false even with compatible graphics card

faizul: <p>I am trying to install PyTorch through anaconda on my machine (runs on Linux, my graphics card is NVIDIA GTX 1080 Ti) using this command:<br>
<strong>conda install pytorch torchvision cudatoolkit=10.2 -c pytorch</strong><br>
However, even though the installation seems right, the <strong>torch.cuda.is_available()</strong> command returns <strong>False</strong>. I checked my driver version and card and they are supposed to be compatible with CUDA 10.2. Here’s what I my <em>nvidia-smi</em> command returns:</p>
<p>±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 440.36       Driver Version: 440.36       CUDA Version: 10.2     |<br>
±----------------------------------------------------------------------------+</p>
<p>I even tried to downgrade and use earlier version of PyTorch using this command:<br>
<strong>conda install pytorch==1.2.0 torchvision==0.4.0 cudatoolkit=10.0 -c pytorch</strong></p>
<p>However, downgrading didn’t work either. After searching stuffs online and trying different methods, I am lost at this point. Any help will be highly appreciated.</p>

Rahul_R: <p>I’ve seen a similar problem when I had an incorrect version of CUDA installed/multiple pytorch installations. I am not sure if the same problem applies to you.</p>
<p>First uninstall all existing versions of torch (<code>pip uninstall torch</code>). It’s possible that multiple torch installations exist in the same conda environment (installed through pip and conda). Ensure that you remove all of them first. If you don’t mind installing <code>torch==1.5.1,</code>  installl pytorch using:</p>
<blockquote>
<p>conda install pytorch torchvision cudatoolkit=10.1 -c pytorch</p>
</blockquote>
<p>The above worked for me in a python 3.6 conda environment.</p>
<p>There are other threads that report the same problem: <a href="https://github.com/pytorch/pytorch/issues/6041" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.is_available() returns False · Issue #6041 · pytorch/pytorch · GitHub</a></p>

faizul: <p>The conda environment is a new installation in this machine, so there should not be any PyTorch, except my own installed one. The <strong>conda list</strong> command shows only one PyTorch.</p>
<p>However, I removed my PyTorch and run your suggested command. The problem still exists.</p>
Attention model in seq2seq tutorial vis a vis Bahdanau et al 2014

kakrafoon: <p>While working through the tutorial, I got into some difficulties aligning [sic] the tutorial material with the original paper by Bahdanau et al. It is possible that I might be misinterpreting, but it seems to me that the original paper combines computes alignment of the  encoder hidden unit (h) with the decoder hidden unit (s) [they call it a(s_{i-1}.h_j), while the tutorial seems to do a(s_{i-1}, y_{j-1}) where y is the output.</p>
<p>The alignments that the code produces are obviously very good, so it’s obviously quite flexible in the way we interpret it, but I was wondering if it would help resolve confusion if the tutorial were implemented more or less according to the paper’s architecture (together with the Bidirectional RNNs and all that).</p>

Amy12xx: <aside class="quote no-group" data-username="kakrafoon" data-post="1" data-topic="6651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kakrafoon/48/6976_2.png" class="avatar"> kakrafoon:</div>
<blockquote>
<p>the original paper combines computes alignment of the encoder hidden unit (h) with the decoder hidden unit (s) [they call it a(s_{i-1}.h_j), while the tutorial seems to do a(s_{i-1}, y_{j-1}) where y is the output</p>
</blockquote>
</aside>
<p>I had the same question. It looks like it is different, but I’d love some confirmation, and also reason for doing it that way (is that from a different paper/implementation).</p>
"transforms.functional" Broken When Using Images With Alpha

Huud: <p>Some “torchvision.transforms.functional” transformations such as “TF.rotate” and “TF.resize” break the image when the image is in RGBA format, as you can see:</p>
<p>                    <a href="https://i.imgur.com/KNvaCc2.jpg" target="_blank" rel="nofollow noopener" class="onebox">
            <img src="https://i.imgur.com/KNvaCc2.jpg" width="690" height="396">
          </a>

</p>
<p>The blue channel before the rotation is left, and after it in the right, everything in where the alpha channel is black was also made black in the blue channel.</p>
<p>Reproduction code:</p>
<pre><code class="lang-auto">import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
import torchvision.transforms.functional as TF

img = Image.open(r'C:\your\folder\test.png')

plt.imshow(np.asarray(img)[:,:,2:3].squeeze(), cmap='gray'); plt.show()

img   = TF.rotate(img, 30) # or img  = TF.resize(img, (512, 512))

plt.imshow(np.asarray(img)[:,:,2:3].squeeze(), cmap='gray'); plt.show()
</code></pre>
<p>Test image:</p>
<p>                    <a href="https://i.imgur.com/lJsVxLg.png" target="_blank" rel="nofollow noopener" class="onebox">
            <img src="https://i.imgur.com/lJsVxLg.png" width="690" height="388">
          </a>

</p>

Huud: <p>Workaround is to use opencv_transforms instead which works properly, it is a rewrite of Pytorch’s transforms to be OpenCV based instead of PIL, so it’s also faster:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/jbohnslav/opencv_transforms" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/17788259?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/jbohnslav/opencv_transforms" target="_blank" rel="nofollow noopener">jbohnslav/opencv_transforms</a></h3>

<p>OpenCV implementation of Torchvision's image augmentations - jbohnslav/opencv_transforms</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Int8 quantization of the resnet18 model, the results of each quantization are inconsistent

blueskywwc: <p>class quantizeModel(object):<br>
“”“docstring for quantizePytorchModel”""</p>
<pre><code>def __init__(self):
    super(quantizeModel, self).__init__()
    self.device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    self.train_loader, self.test_loader = get_imagenet()
    self.quant()

def quant(self):

    model = self.load_model()
    model.eval()
    self.print_size_of_model(model)
    self.validate(model, "original_resnet18", self.test_loader)
    model.fuse_model()

    self.print_size_of_model(model)
    self.quantize(model)

def load_model(self):
    model = resnet18()
    state_dict = torch.load("CIFAR10_resnet18.pth", map_location=self.device)
    model.load_state_dict(state_dict)
    model.to(self.device)
    return model

def print_size_of_model(self, model):
    torch.save(model.state_dict(), "temp.p")
    print('Size (MB):', os.path.getsize("temp.p") / 1e6)
    os.remove('temp.p')

def validate(self, model, name, data_loader):
    with torch.no_grad():
        correct = 0
        total = 0
        acc = 0
        for data in data_loader:
            images, labels = data
            images, labels = images.to(self.device), labels.to(self.device)
            output = model(images)

            _, predicted = torch.max(output, dim=1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
            if total == 1024: 
                break
        acc = round(100 * correct / total, 3)
        print('{{"metric": "{}_val_accuracy", "value": {}%}}'.format(name, acc))
        return acc

def quantize(self, model):
    #model.qconfig = torch.quantization.default_qconfig
    #model.qconfig = torch.quantization.default_per_channel_qconfig

    model.qconfig = torch.quantization.QConfig(
        activation=torch.quantization.observer.MinMaxObserver.with_args(reduce_range=True),
        weight=torch.quantization.observer.PerChannelMinMaxObserver.with_args(dtype=torch.qint8,
                                                                              qscheme=torch.per_channel_affine))
    pmodel = torch.quantization.prepare(model)

    #calibration
    self.validate(pmodel, "quntize_per_channel_resent18_train", self.train_loader)
    qmodel = torch.quantization.convert(pmodel)

    self.validate(qmodel, "quntize_per_chaannel_resent18_test", self.test_loader)
    self.print_size_of_model(qmodel)

    torch.jit.save(torch.jit.script(qmodel), "quantization_per_channel_model18.pth")</code></pre>
How to optimize tensor indexing

kairos03: <p>Hi, I’m facing indexing problem. How can I optimize this problem.<br>
I want to select the indices for each batch.</p>
<p>For Example,</p>
<pre><code class="lang-auto">x = [[[1, 2], [3, 4], [5, 6]], [[7, 8], [9, 10], [11, 12]]]
idx = [[0, 2], [2, 1]]
...
doing indexing
...
result = [ [[1, 2], [5, 6]], [[11, 12], [9, 10]] ]
</code></pre>
<p>I coded indexing with for loop. But it is very slow.<br>
So, the real problem is optimize this for loop.</p>
<pre><code class="lang-auto"># x is 4 dim tensor.
x = torch.empty((B, C, W, H))
idx = torch.tensor([[3, 5, 10],[2, 4, 1]])

result = []
for row, i in enumerate(idx):
    result += [x[row, i, :, :]]
</code></pre>
<p>Thank you!</p>

harsha_g: <aside class="quote no-group quote-modified" data-username="kairos03" data-post="1" data-topic="87374">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kairos03/48/26172_2.png" class="avatar"> kairos03:</div>
<blockquote>
<pre><code class="lang-auto">x = [[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]
idx = [[0, 1, 2], [1, 4, 2]]
</code></pre>
</blockquote>
</aside>
<pre><code class="lang-auto">x = torch.tensor(x)
idx = torch.tensor(idx)
result = torch.gather(x, 1, idx)
</code></pre>

kairos03: <p>Yes, in that case I can solve with <code>gather</code> but below case(x is 4-dim) I can’t solve with gather.<br>
I think the old example is not correct to this problem, I changed it.</p>

Hartbook: <p>For the moment I use this solution : <a href="https://discuss.pytorch.org/t/batched-index-select/9115/11" class="inline-onebox">Batched index_select</a> but I’m also curious if there is a better alternative.</p>

kairos03: <p>One of my friend solve this problem.</p>
<p>The solution is <code>expanding index matrix</code> and use <code>gather</code></p>
<pre><code class="lang-auto"># for 3-dim input
X = torch.tensor([[[1, 2], [3, 4], [5, 6]], [[7, 8], [9, 10], [11, 12]]])
I = torch.tensor([[0, 2], [2, 1]])
eI = I[..., None].expand(-1, -1, X.size(2))    ## expanding index
Y = torch.gather(X, dim=1, index=eI).squeeze()

# for 4-dim input
X = torch.tensor([[[[0, 0], [1, 1]], [[2, 2], [3, 3]], [[4, 4], [5, 5]]], [[[6, 6], [7, 7]], [[8, 8], [9, 9]], [[10, 10], [11, 11]]]])
I = torch.tensor([[0, 2], [2, 1]])
eI = I[..., None, None].expand(-1, -1, X.size(2), X.size(3))    ## expanding index
Y = torch.gather(X, dim=1, index=eI).squeeze()
</code></pre>
Calculate the gradients w.r.t non-zero values in a sparse matrix

Kai_Eiji: <p>Hi,</p>
<p>I’m using a sparse matrix in part of my neural network and I was wondering if there is a way to get the gradients with respect to only the non zero values of a sparse matrix. If I convert the sparse matrix into a dense one, the matrix is going to be too large and it’s going to take too much time to compute the gradient so I want to keep my matrix sparse.</p>
<p>I would appreciate it if someone can help me with this.</p>

albanD: <p>Hi,</p>
<p>Unfortunately, at the moment, very few of the backward formulas formulas support sparse gradients <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
But you can call <code>autograd.grad(out, inp, sparse_grad_out)</code> to try and propagate sparse gradient in the backward pass.<br>
If you want to create them during the backward, I think a custom Function can be used to do the conversion you want.</p>

Kai_Eiji: <p>Ok, Thanks for the help!</p>
How to count images

jasg: <p>How can I count the agave plants in this field? what technique do you recommend??<br>
<a href="https://drive.google.com/file/d/1UWf1p2juWVudpG7-I2oVI2B-M4xO3Izz/view?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://drive.google.com/file/d/1UWf1p2juWVudpG7-I2oVI2B-M4xO3Izz/view?usp=sharing</a></p>
<p>thanks</p>

ptrblck: <p>You could start with a naive approach of trying to predict the target value directly or alternatively use an object detection task, where you could count the predicted bounding boxes for each plant.</p>

jasg: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bb0d714d53e40bee560638da067743656e60475.png" alt="Captura de pantalla 2020-06-30 a la(s) 9.32.30" data-base62-sha1="hEdBDWdNi5Vja42AeOgu1JtGWe9" width="479" height="406"></p>
<p>as you see not being able to identify only the plants:<br>
my code is</p>
<pre data-code-wrap="img"><code class="lang-img">
#conversion a escala de grises
gray=cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
gray = cv2.GaussianBlur(gray,(25,25),0)


ret, threshed_img = cv2.threshold(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY),
                127, 255, cv2.THRESH_BINARY)
# find contours and get the external one

dilated = cv2.dilate(thresh,kernel,iterations = 13) # dilate
contours, hierarchy = cv2.findContours(dilated,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)

for cnt in contours:
    x,y,w,h = cv2.boundingRect(cnt)
    if h&lt;30 or w&lt;30:
        continue
    else:
        
        cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2)
            # get the min area rect
        rect = cv2.minAreaRect(cnt)
        box = cv2.boxPoints(rect)
        # convert all coordinates floating point values to int
        box = np.int0(box)
        # draw a red 'nghien' rectangle
        #cv2.drawContours(img, [box], 0, (0, 0, 255))
    
        # finally, get the min enclosing circle
        (x, y), radius = cv2.minEnclosingCircle(cnt)
        # convert all values to int
        center = (int(x), int(y))
        radius = int(radius)
        # and draw the circle in blue
        img = cv2.circle(img, center, radius, (255, 0, 0), 2)
    
    

print(len(contours))</code></pre>

ptrblck: <p>If you don’t have the labels and would like to use a classical computer vision approach, I would recommend to post the question in an OpenCV-specific discussion board or on StackOverflow, as your use case doesn’t seem to be PyTorch-specific. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
RuntimeError: CUDA error: unknown error

ojh3636: <p>I faced to something weird error when I try to move Tensor to GPU.<br>
First my machine spec is</p>
<p>Ubuntu 16.04<br>
nvidia driver: 390.77<br>
GPU: TitanX (Pascal)<br>
<br>
Torch 0.4.1 (with cuda9.0)</p>
<p>I didn’t install any cuda Toolkit in my local, just download pytorch via anaconda to my virtualenv. (because I saw some posts that downloading pytorch via anaconda will install automatically cuda and what I need to do is just install proper nvidia driver in my machine)</p>
<p>After install pytorch and run some simple code</p>
<p>import torch<br>
a = torch.rand(5, 3)<br>
device = torch.device(‘cuda’)<br>
a.to(device)</p>
<p>Then I encountered above error. (I also checked torch.cuda.is_available() and that returned True)</p>
<p>But when I run the python in sudo permission (like  sudo ./~~/env/bin/python3) Then the above simple code perfectly worked. &lt;some posts said that the cuda initialization once in sudo then normal user permission can run  cuda normaly but in my case, it didn’t work. Every time I need to run python as sudo to make perform above code&gt;</p>
<p>Please help.</p>

Jk749: <p>One problem I guess you might be having is cuda installation.<br>
When you installed nvidia driver and cuda toolkit, do you have cuda toolkit path available and loaded in your environment variables.</p>
<p>Try setting, LD_LIBRARY_PATH to <code>/usr/local/cuda/lib64</code></p>

ojh3636: <p>I didn’t install cudaToolkit in my /usr/local. I just installed pytorch in conda virtualenv, and when I looked the “~/anaconda3/envs/myenv/lib” There are all the cuda library (eg libcudart.so*). I think activating my virtualenv is enough rather than setting ld_library_path. And when I inspect my gpu memory by nvidia-smi, I can see the gpu memory is allocated every time I run the “a.to(device)” code (and not be free, and raise out of memory when I run this code  about15times.) So I think cuda library successfully loaded but some error (that caused by permission) occur. I don’t know why…this error occur</p>
<p>p.s<br>
I’ve reinstalled nvidia driver after posting this post, but It didn’t help…</p>

ptrblck: <p>Did you restart the machine after reinstalling the drivers?</p>
<p>If so, could you try to install PyTorch via conda with CUDA9.1 or 9.2?</p>

ojh3636: <p>Yes, I do restart after reinstall nvidia driver (First reinstall 390.77 &amp; reboot &amp; cuda9.0 -&gt; didn’t work, reinstall 384.xx &amp; reboot &amp; cuda9.0 -&gt; also didn’t work. ) I can’t use the machine now, So I’ll add info after doing with cuda9.2 tmr(I don’t know why but I can’t find pytorch-cuda9.1 in conda repository…) . Thank you for reply</p>

ojh3636: <p>Yeah~ I solved it!! I just upgrade my nvidia driver to 396 version and do with cuda92. Thank you so much. It works well!! But still don’t know what was problem in previous version.</p>

pinocchio: <p>What is the exact error message you are getting? Can you paste that please?</p>
Cuda: Initialization Error

jsuit: <p>RuntimeError: cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch_1502008109146/work/torch/lib/THC/generic/THCTensorCopy.c:65</p>
<p>I keep getting the above error randomly. What ran fine a couple of days ago, is now throwing the above error. What is causing this? I’m not using DataParallel.</p>
<p>Edit: Noticed that it happens only when going from the 1st epoch to the 2nd. The problem is that nothing changes between the first epoch and the second.</p>

pinocchio: <p>Did you manage to solve your issue?</p>
<p>Is it related to:</p>
<pre><code class="lang-auto">"RuntimeError: CUDA driver error: initialization error"
</code></pre>
<aside class="quote" data-post="1" data-topic="87505">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-does-runtimeerror-cuda-driver-error-initialization-error-mean/87505">What does "RuntimeError: CUDA driver error: initialization error" mean?</a> 
  </div>
  <blockquote>
    I’m getting this error: 
"RuntimeError: CUDA driver error: initialization error"

But I can’t find what it means (or why it happens). 
Of course I want to solve it but I want to know what the error means in the first place first. 

This error happened once I used pin_memory=True 

related: 

<a href="https://discuss.pytorch.org/t/what-does-runtimeerror-cuda-driver-error-initialization-error-mean/87505" class="inline-onebox">What does "RuntimeError: CUDA driver error: initialization error" mean?</a>
<a href="https://stackoverflow.com/questions/62723569/does-creating-a-data-loader-inside-another-data-loader-in-pytorch-slow-things-do" rel="noopener nofollow ugc">https://stackoverflow.com/questions/62723569/does-creating-a-data-loader-inside-another-data-loader-in-pytorch-slow-things-do</a>
  </blockquote>
</aside>
Classification of moving pixels

Alen: <p>I have a data set with videos of moving pixels. Each video contains 32 frames, each frame is 32x32 with two pixels in white and the rest in black. I have binary labels for 800 of these obtained by human labeling. I now want to construct a classifier in order to generalize from this set. I have access to the algorithm generating the pixel movements. The pixels move 1 pixel in any direction each frame, but they never overlap.</p>
<p>My understanding is that 3DCNNs is a good match for this type of problem, but I have mainly seen this used for “dense” videos, my data is very sparse in comparison. I have also seen suggestions of using LSTMs or dual stream solutions.</p>
<p>What is the best or more optimal approach given my type of data?</p>
<p>I would very much appreciate any suggestions, thank you.</p>
Get parameters gradient after every layer's backward computation

yzz: <p>Hi,</p>
<p>I wonder if there is any way that can directly report parameters gradient after each layer’s backward computation. Through the hook function on tensor or module, we are only able to get the intermediate results of the current layer but not the parameters’ gradient.</p>
<p>Thanks in advance.</p>

albanD: <p>Hi,</p>
<p>You can add a hook on the Parameter itself with <code>register_hook()</code>. This will be used when the gradients are computed (and just before the .grad field is populated).</p>

yzz: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="87477">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Parameter</p>
</blockquote>
</aside>
<p>Thanks for the prompt reply. I register the hook on the weights (e.g., module.weight.register_hook()). However, I got nothing when trying to print weight.grad in this hook function. I think this is because the hook function is called before .grad field is populated as you mentioned. I wonder if the computed gradient is the input of this tensor hook operation.</p>
<p>Thanks again!</p>

albanD: <p>Yes, the hook is given the grad as the argument.<br>
Note that if you want to modify the gradient that is saved in the .grad field, you can return the new value from the hook and it will be used.</p>
Counting the number of layers in a neural network

maureyes: <p>Why does it have 23 layers?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/053886abc44e0d0a1ce344c5dfa9210c248622c2.png" data-download-href="https://discuss.pytorch.org/uploads/default/053886abc44e0d0a1ce344c5dfa9210c248622c2" title="Screen Shot 2020-06-30 at 1.08.51 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/053886abc44e0d0a1ce344c5dfa9210c248622c2_2_690x430.png" alt="Screen Shot 2020-06-30 at 1.08.51 PM" data-base62-sha1="KbuCrZlK1FMC8AdOITyyJj2P3Y" width="690" height="430" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/053886abc44e0d0a1ce344c5dfa9210c248622c2_2_690x430.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/053886abc44e0d0a1ce344c5dfa9210c248622c2_2_1035x645.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/5/053886abc44e0d0a1ce344c5dfa9210c248622c2.png 2x" data-dominant-color="F5F7F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-06-30 at 1.08.51 PM</span><span class="informations">1272×794 61.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>[figure from <a href="https://arxiv.org/pdf/1505.04597.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1505.04597.pdf</a>]</p>

harsha_g: <p>18 blue arrows, 4 green arrows and 1 cyan?? arrow --&gt; 23 <code>convolutional</code> layers and that’s exactly what the paper says.</p>
Efficient way to mask a tesnor s according tensor t

Chen-Cai-OSU: <p>Given 2 1-dim tensor s, t, I would like to mark all entries of s whose value belong to tensor t as 1, and 0 otherwise. Right now, I am implementing it in the following way</p>
<pre><code class="lang-auto">s = torch.tensor([0,3,2,1])
t = torch.tensor([0,2])

i = torch.nonzero(s[..., None]==t)[:, 0]
new_s = torch.zeros(s.size())
new_s[i] = 1 // tensor([1., 0., 1., 0.])
</code></pre>
<p>However, torch.nonzero(s[…, None]==t) is quite slow for large s, t. It is also memory intensive since it will output as a tensor of shape (len(s), len(t)). (therefore out of memory for gpu) I was wondering is there a better way to achieve this? Thank you very much!</p>
Recources for computer vision tasks

887574002: <p>Hi, I am looking for some good resources to learn computer vision tasks in-depth with enough coding examples. I have already finished PyTorch tutorials and one book in computer vision with PyTorch and one online course. But I require to work on coding. Specifically, I want to find a structured way to learn and practice computer vision tasks, since I need to have a good coding ability to do these tasks for my research. I would apprecitae it if someone can help me.</p>
How to input fractional score labels for each input image while training a CNN in pytorch for regression problem

spaul13: <p>During a convolution neural network training, each input image is assigned to a score (floating point).  <strong>How to feed that fraction value corresponding to each image as a label while training a cnn?</strong>  Then I want to use the MSELoss.</p>
<p>For normal classification one have to create one folder for each classes then store the respective images to that class folder but for a continuous fraction score label how to do that and what torch API to use? Any suggestions on this will be highly helpful.</p>
How to access particular vector or row in embedding?

Ajay_Madhavan: <p>Hi,</p>
<p>I am creating a single layer network for graph embedding. I am trying to do a fast training with no_grad(). My embedding dimension is 35,128. My requirement is to select a random row from embedding.weight.data, calculation score, and multiply the scalar score to weight data of that particular vector or row.</p>

ptrblck: <p>You could sample a random number and index the <code>embedding.weight</code> directly via:</p>
<pre><code class="lang-python">w = emb.weight[index]
</code></pre>
<p>Afterwards, you could calculate some score and reassign the new value to <code>emb.weight[index]</code>.</p>
<p>I’m unsure, how the training works using random indices and skipping Autograd, so could you add a bit more information about your use case?</p>

Ajay_Madhavan: <p>I am working on converting a c++ program which was used in a paper to convert nodes to vectors, so it can be used for ML task. We basically initialize the weight vector of size V x d (V - number of nodes, d - embedding size) with values between -0.5 to 0.5. Then we sample nodes as u, v and neg_v, where u &amp; v has edge and u &amp; neg_v does not have edge.</p>
<p>The calculation selects the weight of respective nodes and uses the below logic for calculating scores.<br>
if we sample u, v:<br>
score = -bias<br>
score += weight[u] * weight[v]<br>
score = (1 - fast_Sigmoid(score)) * lr<br>
weight[u] += weight[v] * score<br>
weight[v] += weight[u] * score</p>
<p>if we sample u, neg_v:<br>
score = -bias<br>
score += weight[u] * weight[neg_v]<br>
score = - fast_Sigmoid(score) * lr<br>
weight[u] += weight[neg_v] * score<br>
weight[neg_v] += weight[u] * score</p>
<p>since all this are performed in single go and we want to have faster calculation, we came across with torch.no_grad() (" Disabling gradient calculation is useful for inference, when you are sure that you will not call backward(). it will reduce memory consumption") and we are using no_grad(). I just wanted to know if there is any way to select particular row of weight matrix and update it. I guess i got my answer from your reply. I am still trying to figure out how to multiple process this training along with memory sharing. If you can provide me some details it will be big help!!</p>
<p>Hope i gave a clear picture what i am trying to do!!</p>

Ajay_Madhavan: <p>Is it possible to give your inputs for the above comment??</p>