akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
OrderedDict:move_to_end not recognized

username: StrawVulcan
cooked: <p>Hey,</p>
<p>I tried to use <code>torch.jit.script</code> on code that used <code>OrderedDict</code>'s <code>move_to_end</code>, and got an error that suggested it was interpreted as a <code>Dict</code>. Am I doing something wrong, or are <code>OrderedDicts</code> not quite fully supported in TorchScript?</p>
<p>I’m on PyTorch 1.6.0.</p>
<p>The instantiation looks like this:</p>
<pre><code class="lang-auto">self.cache:typing.OrderedDict[str,Tensor] = OrderedDict()
</code></pre>
<p>Here’s the error:</p>
<pre><code class="lang-auto">RuntimeError:
Tried to access nonexistent attribute or method 'move_to_end' of type 'Dict[str, Tensor]'.:
  File "/home/strawvulcan/lad/lad/util.py", line 162
            return -1
        else:
            self.cache.move_to_end(key)
            ~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
            return self.cache[key]
'LRUCache.get' is being compiled since it was called from 'LRUCache.__getitem__'
  File "/home/strawvulcan/lad/lad/util.py", line 175
    def __getitem__(self, key:str):
        return self.get(key)
                        ~~~ &lt;--- HERE
'LRUCache.__getitem__' is being compiled since it was called from '__torch__.lad.util.LRUCache'
  File "/home/strawvulcan/lad/lad/util.py", line 188
        "`batch` is a list of strings"
        if not hasattr(self, 'item_cache'):
            self.item_cache = LRUCache(max_cache_size)
                              ~~~~~~~~ &lt;--- HERE
        hit_idxs = []
        hits = []
'__torch__.lad.util.LRUCache' is being compiled since it was called from 'Autoencoder.forward'
  File "/home/strawvulcan/lad/lad/util.py", line 188
        "`batch` is a list of strings"
        if not hasattr(self, 'item_cache'):
            self.item_cache = LRUCache(max_cache_size)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
        hit_idxs = []
        hits = []
</code></pre>
Torch.cat along negative dimension

username: SAI_VARSHITTHA
cooked: <blockquote>
<p>x = torch.randn(1,1,5)<br>
torch.cat((x,x),dim=-3)</p>
</blockquote>
<p>What exactly does this mean? I could not understand</p>

username: SAI_VARSHITTHA
cooked: <p>My problem is resolved,thanks</p>
Dataloader initiating data set upon call of each batch

username: Kuppharish
cooked: <p>I am trying to load data into my Dataset class in the <strong>get_item</strong> function rather than <strong>init</strong> function because of the data is very large and it cannot be loaded all at once to memory. Since the index of DataLoader keeps I am keeping a record of length of previously loaded part of data but this length goes to 0 which was initiated in <strong>init</strong> every time a new batch is loaded. Is there a way to not call init function of DataSet?</p>
<pre><code class="lang-auto">`	def __init__(self, data_path, graph_args={}, train_val_test='train'):
		'''
		train_val_test: (train, val, test)
		'''
		self.data_path = data_path
		self.path_list = sorted(glob.glob(os.path.join(self.data_path,'*.txt')))
		self.all_feature=[]
		self.all_adjacency=[]
		self.all_mean_xy=[]
		self.it = 0
		#self.load_data()
		#total_num = len(self.all_feature)
		# equally choose validation set
		self.feature_num=0
		self.prev=0
	def __getitem__(self, idx):
		# C = 11: [frame_id, object_id, object_type, position_x, position_y, position_z, object_length, pbject_width, pbject_height, heading] + [mask]
		#if(idx&gt;=self.feature_num):
		try:
			now_feature = self.all_feature[idx-self.prev].copy()
		except:
			path = self.path_list[self.it]
			self.it = self.it+1
			self.all_feature, self.all_adjacency, self.all_mean_xy = generate_data(path)
			self.prev = self.feature_num
			self.feature_num = self.feature_num+len(self.all_feature)	
			now_feature = self.all_feature[idx-self.prev].copy()`
</code></pre>

username: ptrblck
cooked: <p>Be careful with trying to manipulate the <code>Dataset</code> in a <code>DataLoader</code>, if you are using multiple workers.<br>
Each worker will use a clone of the <code>Dataset</code>, so that changes to the internal states of the <code>Dataset</code> will not be reflected.</p>
<p>Could this be the case for your issue?</p>

username: VitalyFedyunin
cooked: <p>1.7 release and current master will have reset functionality, see <a href="https://github.com/pytorch/pytorch/pull/35795" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/35795</a></p>

username: Kuppharish
cooked: <p>Thank you so much, making the num_workers to 0 solved the issue, But I guess I’m restricting the capabilities of machine by loading the data sequentially.</p>
Energy consumtion per layer

username: Luan_Goncalves
cooked: <p>Hi all!</p>
<p>I would like to estimate the energy consumption of each layer in a model. Does anyone knows how to do it?</p>

username: tom
cooked: <p>You would probably need to set up some model for what consumes energy.<br>
Going after FLOPS is the most common metric, I would wonder whether memory accesses (which have a huge impact on the time something takes) also play a role.<br>
After that, you would need to estimate the explanatory variables for each block you want to measure.<br>
Finally you can take the overall power consumption and then allocate that to the layers using the above.</p>
<p>An alternative to all this could be to gradually “shorten” the model and measure the total power per 1 epoch or so. The difference between two runs could then be attributed to the part the shortened model leaves out.</p>
<p>Best regards</p>
<p>Thomas</p>

username: mohit117
cooked: <p>Hello,<br>
I understood the FLOPS argument but not what is EXPLANATORY variables and how to go about it.</p>
<aside class="quote no-group" data-username="tom" data-post="2" data-topic="71928">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>After that, you would need to estimate the explanatory variables for each block you want to measure.<br>
Finally you can take the overall power consumption and then allocate that to the layers using the above.</p>
</blockquote>
</aside>
<p><em><strong>I am only interested in total energy consumption of the model and do not require layerwise diagnostic.</strong></em><br>
Is there an easy way to do this using PyTorch. Some rough estimates will also do. Even external links to GITHUB repos doing this will help</p>
Is loss calculation typically done on GPU or on CPU?

username: BramVanroy
cooked: <p>I can’t find the post anymore, but I read on these forums that it is better (or at least faster) to calculate loss on the CPU. I wonder whether that is true. What are the pros and cons of calculating loss on the GPU vs CPU. Particular concerns are speed and memory consumption. My assumption is also that if there is a notable difference, that this is magnified with larger batches.</p>

username: googlebot
cooked: <p>People often measure cuda times incorrectly (in this case it is even more likely as loss.item() is often a cuda synchronization point). In practice, loss calculation times should be too insignificant to be concerned. Same with memory.</p>
Extending torchscript a custom class in torch c++

username: Kumar_Govindam
cooked: <p>How can I create a custom class in c++ and import the output in python? Riight now, the example given is:</p>
<pre><code class="lang-auto">torch::Tensor warp_perspective(torch::Tensor image, torch::Tensor warp) {
  cv::Mat image_mat(/*rows=*/image.size(0),
                    /*cols=*/image.size(1),
                    /*type=*/CV_32FC1,
                    /*data=*/image.data_ptr&lt;float&gt;());
  cv::Mat warp_mat(/*rows=*/warp.size(0),
                   /*cols=*/warp.size(1),
                   /*type=*/CV_32FC1,
                   /*data=*/warp.data_ptr&lt;float&gt;());
  cv::Mat output_mat;
  cv::warpPerspective(image_mat, output_mat, warp_mat, /*dsize=*/{8, 8});
  torch::Tensor output = torch::from_blob(output_mat.ptr&lt;float&gt;(), /*sizes=*/{8, 8});
  return output.clone();

}

TORCH_LIBRARY(my_ops, m) {
  m.def("warp_perspective", warp_perspective);
}
</code></pre>
<p>But  suppose I want to create a class in c++, initialise the class and call a method from that in python, like:</p>
<pre><code class="lang-auto">Class imageTransform()
{

     initialise()
        {
               /////initialise some values
         }

     torch::Tensor warp_perspective () {
               ////use the initialised value here
               ////here do the transform and return the tensor
        }
}
</code></pre>
<p>How can I write the python binding for this case?</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/kumar_govindam">@Kumar_Govindam</a><br>
Please take a look at this:<br>
‘<a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>’<br>
You can create a cpp extension module and bind your own functions. You can not define a real c++ class, but this is similar, you have a module name and functions under the module, the only thing is that you have no constructor.</p>

username: Kumar_Govindam
cooked: <p>Actually I have tried those things, and its kind of working after some modifications, but its generating<br>
.so file like classifier.cpython-37m-x86_64-linux-gnu.so, instead of classifier.so. So, wanted to know if there is a cleaner way to do that. Anyway thanks for the reply.</p>
Are you initialising Linear (nn) bias to zeros?

username: RoySadaka
cooked: <p><code>nn.Linaer</code> with <code>bias=True</code> is initializing the bias using uniform:<br>
<code>init.uniform_(self.bias, -bound, bound)</code></p>
<p>I usually override it by initializing it to zeros like so:</p>
<pre><code class="lang-auto">fc = nn.Linear(in_dim, out_dim, bias=True)
nn.init.zeros_(fc.bias)
</code></pre>
<p>Is there any benefit to the default uniform initialization?</p>
<p>Thank you in advance<br>
Roy</p>
How to set a separate learnable learning rate for each parameter in Pytorch?

username: dirk
cooked: <p>I am a little bit confused about the learnable learning rate for each parameter in Pytorch.</p>
<p>I want to set a separate learnable learning rate for each parameter. The learning rate and model parameters will be optimized together during the training process.</p>
<p>I find some code：</p>
<pre><code class="lang-python">    optimizer = optim.Adam([
            {'params': net.layer1.weight},
            {'params': net.layer1.bias, 'lr': 0.01},
            {'params': net.layer2.weight, 'lr': 0.001}
        ], lr=0.1, weight_decay=0.0001)
</code></pre>
<p>This method solves the problem of setting a fixed learning rate for each parameter, but it cannot allow the learning rate to be trained along with the model.</p>
<p>I tried to define a class with learnable learning rate:</p>
<pre><code class="lang-python">    class Learner(nn.Module):

        def __init__(self, net, in_channels, num_classes):
            super(Learner, self).__init__()
            self.learner = net(in_channels, num_classes)
            self.task_lr = OrderedDict()

        def forward(self, X):
            out = self.learner(X)
            return out

        def define_lr(self):
            for key, val in self.named_parameters():
                self.task_lr[key] = nn.Parameter(
                    1e-4 * torch.ones_like(val, requires_grad=True))
</code></pre>
<p>and then:</p>
<pre><code class="lang-python">    model_params = list(model.parameters()) + list(model.task_lr.values())
    optimizer = optim.Adam(model_params, lr=args.lr)
    adapted_state_dict = model.cloned_state_dict()

    #####compute loss#####
    #......
    ######################
    
    model.zero_grads()
    loss.backward()
    for name, param in model.named_parameters():
        if param.grad is not None:
            adapted_state_dict[name] -= model.task_lr[name] * param.grad.data
    model.load_state_dict(adapted_state_dict)
    optimizer.step()
</code></pre>
<p>But this method still cannot train the learning rate.</p>
<p>Feel free to ask if more code is needed to explain the problem.</p>
Implement three models as one general model

username: OuisYasser
cooked: <p>Hello, I’m a little bit new to pytorch, cause I used to work with keras almost the time.<br>
I have a question concerning Cnn’s.Let’s say for example I have image input data and then I’ll feed this data to Net01() and then I’ll apply some calculations to the output and then feed this data to Net2(), apply calculations and feed it to Net03().<br>
So please can someone explain how will be the process or if someone can share some code to get an intuition and thank you.</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>In PyTorch, every model/layer can be defined by extending <code>nn.Module</code> class.<br>
For instance, you can see layers like <code>nn.Linear</code> or <code>nn.Conv2d</code>. When you define a new model, you need to follow same pattern which will be based on <code>nn.Module</code>. This leads to cases where we can have complete models such as resnet, vgg, etc (you can find <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="noopener nofollow ugc">available models here</a> and their <a href="https://github.com/pytorch/vision/tree/master/torchvision/models" rel="noopener nofollow ugc">source code</a>).</p>
<p>Here is an example:<br>
Let’s say you have 3 models called <code>Net1</code>, <code>Net2</code> and <code>Net3</code>. Now we want to have a model <code>Combined</code> that uses all three models in it.</p>
<pre><code class="lang-auto">class Combined(nn.Module):
  def __init__(self, # your args such as num classes):
    super(Combined, self).__init__()
    self.model1 = Net1()
    self.model2 = Net2()
    self.model3 = Net3()

  def forward(self, x):
    model1_output = self.model1(x) 
    # do your calc with model1_output 
    model2_output = self.model2(model1_output)  # or you can pass `x` 
    # do your calc with model2_output 
    model3_output = self.model3(x)  # or you can pass `x` or `model1_output`, etc
    # do your calc with model1_output 
    return model3_output  # or anything you want as the final output of your `Combined` model


</code></pre>
<p>Bests</p>

username: OuisYasser
cooked: <p>Wow ! Thanks a lot<br>
I started liking pytorch because of this forums !!</p>

username: Nikronic
cooked: <p>That’s great. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
You will enjoy it much more when you get comfortable with PyTorch itself. Literally, you feel there is no limit! And of course, I learned almost everything about PyTorch by reading many of discussions here.</p>
Lpd - PyTorch Trainer and Callbacks

username: RoySadaka
cooked: <p>Hey there,<br>
My name is Roy Sadaka, and I used to design and develop a DL model with TensorFlow <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=9" title=":heart:" class="emoji" alt=":heart:">, few months ago I moved to PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/fire.png?v=9" title=":fire:" class="emoji" alt=":fire:">  and love it so far.</p>
<p>I’ve recently open-sourced my little project - <a href="https://github.com/RoySadaka/lpd" rel="noopener nofollow ugc">lpd</a><br>
<img src="https://raw.githubusercontent.com/RoySadaka/ReposMedia/main/lpd/images/logo.png" alt="lpd" width="50" height="50"><br>
It’s a structured trainer with some common callbacks support, I use it for production needs so it’s fully maintained.<br>
I’m well aware that there are other, more mature alternatives out there, I’ve created this project to give myself the highest level of control over the training process, now you can use it too <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>lpd provides a structured trainer where you can <strong>train/validate/test/predict</strong> your model, it also <strong>collects statistics</strong> during training and prints them at the end of every epoch (you have full control when/if to print them). you can <strong>save/load</strong> the whole trainer and continue training at a later time.<br>
It also exposes some common callbacks such as <strong>ModelCheckPoint</strong>, <strong>EarlyStopping</strong>, <strong>SchedulerStep</strong>, and <strong>Tensorboard</strong>, and you can create <strong>custom callbacks</strong> with ease while still maintain full control.</p>
<p>lpd is currently available on pip or you can visit the <a href="https://github.com/RoySadaka/lpd" rel="noopener nofollow ugc">source code</a></p>
<p>Feel free to contact me if you think something is missing in lpd<br>
Roy</p>
Replacing tensro values with a list of index and corresponding features?

username: peepeepoopoo
cooked: <p>I’m currently trying to replace values AxBxC matrix with given array of indexes and corresponding values.</p>
<p>for example I have an index array [[0,1], [1,0], [3,4]] for a 6x6x4 matrix, with corresponding value [[1,1,1,1],[2,2,2,2],[3,3,3,3]]. I’d like to replace the values in the 6x6x4 matrix at [0,1,:], [1,0,:], and [3,4,:] with [1,1,1,1], [2,2,2,2] and [3,3,3,3].  extracting is possible with given indices, but not the other way around it seems</p>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">A, B, C = 6, 6, 4
x = torch.zeros(A, B, C)
idx = torch.tensor([[0,1], [1,0], [3,4]])
val = torch.tensor([[1,1,1,1],[2,2,2,2],[3,3,3,3]]).float()

x[idx[:, 0], idx[:, 1]] = val
print(x)
&gt; tensor([[[0., 0., 0., 0.],
           [1., 1., 1., 1.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.]],

          [[2., 2., 2., 2.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.]],

          [[0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.]],

          [[0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [3., 3., 3., 3.],
           [0., 0., 0., 0.]],

          [[0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.]],

          [[0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.]]])
</code></pre>

username: peepeepoopoo
cooked: <p>Thank you! I actually forgot to convert the tensor(in this case val) from numpy matrix, so that’s why it didn’t end up working haha.</p>
Help on using stream in multithread to accelerate

username: litttl
cooked: <p>I need some help about using stream in multithread to accelerate my computation. Could you help me and give me some advise?</p>
<p>my problem:<br>
the time killed by using multithread and multistreams is seem to using serial loop. it doesn’t speed up.</p>
<p>my codes:<br>
I used a thread-pool with 10 threads. I add my task into threads. In thread task funciton, I get a free GPUStream and do some job in this GPUStream.</p>
<p>the thread task is as follow:<br>
void* task_thread(void** arg)<br>
{</p>
<pre><code>auto options = torch::TensorOptions().device(torch::kCUDA, 0);
torch::Device device(torch::kCUDA, 0);

at::cuda::CUDAStream mystream = at::cuda::getStreamFromPool();

{
    at::cuda::CUDAStreamGuard guard(mystream);
    std::cout &lt;&lt; "Stream ID: " &lt;&lt; mystream.id() &lt;&lt; std::endl;

    torch::Tensor* pt_base_feature_cpu = (torch::Tensor*) arg[0];
    torch::Tensor* pt_match_feature_cpu = (torch::Tensor*) arg[1];

    for(int i = 0; i &lt; 10; i++)
    {
        torch::Tensor base_feature = (pt_base_feature_cpu-&gt;slice(0, i*50000, (i+1)*50000, 1)).to(device);
        torch::Tensor match_feature = (*pt_match_feature_cpu).to(device);

        torch::Tensor tensor_tmp;
        torch::Tensor tensor_sum;
        std::tuple&lt;torch::Tensor, torch::Tensor&gt; sort_ret;

        tensor_tmp = torch::sub(base_feature, match_feature);
        tensor_tmp = torch::pow(tensor_tmp, 2);
        tensor_sum = torch::sum(tensor_tmp, 1);
        sort_ret = torch::topk(tensor_sum, 1);
    }
}
</code></pre>
<p>}</p>
<p>my test:<br>
I test the time killed in serial way and in parallel way.<br>
in serial way:<br>
task function I use default stream and do the same job. I use for loop to add task in serial way.<br>
for(i=0; i&lt;taskcnt; ++i)<br>
{<br>
task_serial(arg);<br>
}</p>
<p>in parallel way:<br>
I use for loop to add task in multi thread. in thread I use a new stream.<br>
for(i=0; i&lt;taskcnt; ++i)<br>
{<br>
tp_threadpool_add_task(&amp;pool, task_thread, arg);<br>
}</p>
<p>However, I found that the way in serial is faster run the way in parallel.</p>
<p>Could you give me some advise to improve in parrel?</p>
Help on read in MNIST dataset in label and data in libtorch

username: Chen0729
cooked: <p>HI, All</p>
<p>I would like to read in libtorch with MNIST dataset, where the ptorch code is doing<br>
raw_dataset = datasets.MNIST(’…/data’, train=True, download=True,<br>
transform=transforms.Compose([<br>
transforms.ToTensor(),<br>
]))<br>
perm = np.random.permutation(raw_dataset.<strong>len</strong>())<br>
for i in range(raw_dataset.<strong>len</strong>()):<br>
datum, label = raw_dataset.<strong>getitem</strong>(perm[i])</p>
<p>In libtorch code, I write:<br>
auto raw_dataset= torch::data::datasets::MNIST("…/data").map(torch::data::transforms::Normalize&lt;&gt;(0.13707,0.3081)).map(torch::data::transforms::Stack&lt;&gt;());</p>
<pre><code>torch::Tensor perm=torch::randperm(sizeof(raw_dataset));
</code></pre>
<p>for (int i=0; i&lt;sizeof(raw_dataset);i++)<br>
{<br>
torch::Tensor datum=raw_dataset.get_batch(perm([i])).data;<br>
torch::Tensor label1=raw_dataset.get_batch(perm([i])).target;<br>
//torch::Tensor datum=raw_dataset.targets;<br>
}</p>
<p>I get the error “expected body of lambda expression”  in (perm[i]), any suggestions to fix it please ?<br>
Thanks a lot .</p>
<p>Much appreciated.</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/chen0729">@Chen0729</a><br>
Hi, your code is perm([i]), you should use perm[i] in c++, no bracket.  Can u try?</p>

username: Chen0729
cooked: <p>Thanks. That works.</p>
<p>Another inquiry is about:<br>
return Tensordataset=(torch.FloatTensor(np.array(data)), torch.LongTensor(np.array(labels)))</p>
<p>I convert it in C++<br>
As</p>
<p>Tensordataset[0]=torch::Floattensor(data)</p>
<p>No member named Floattensor</p>
<p>Any suggestion to modify that please ?</p>
<p>Thanks a lot.</p>
BinaryCrossEntropy gives better results than CrossEntropy for multi-class classification

username: Fatimah
cooked: <p>Hi,</p>
<p>When using BinaryCrossEntropy I get good better results than we I use Crossentropy.</p>
<p>Here is the code that I am using:</p>
<pre><code class="lang-auto">class BERT(nn.Module):
    def __init__(self):
        super(BERT, self).__init__()

        options_name = "bert-base-uncased"
        self.encoder = BertForSequenceClassification.from_pretrained(options_name, num_labels=3)
        
        for param in  self.encoder.bert.parameters():
            param.requires_grad = True
    def forward(self, text, label):
        text_fea = self.encoder(text, labels=label)[0]

        return text_fea
def train(model,
          optimizer,
          criterion = nn.CrossEntropyLoss(),
          train_loader = train_iter,
          valid_loader = valid_iter,
          num_epochs = 5,
          eval_every = len(train_iter) // 2,
          file_path = destination_folder):
   
    
    # training loop
    model.train()
    for epoch in range(num_epochs):
        for titletext, labels in train_loader:
            labels = labels.type(torch.LongTensor)           
            labels = labels.to(device)
            titletext = titletext.type(torch.LongTensor)  
            titletext = titletext.to(device)
            
            optimizer.zero_grad()
            
            y_train_pred = model(titletext, None)
            loss = criterion(y_train_pred, labels)
            
            loss.backward()

            optimizer.step()
</code></pre>
<p>Any idea what could be wrong?</p>
<p>thank you</p>

username: KFrank
cooked: <p>Hi Fatimah!</p>
<aside class="quote no-group" data-username="Fatimah" data-post="1" data-topic="99740" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/b9e5f3/48.png" class="avatar"> Fatimah:</div>
<blockquote>
<p>When using BinaryCrossEntropy I get good better results than we I use Crossentropy.</p>
</blockquote>
</aside>
<p>If you are performing a <em>single-label,</em> multi-class classification<br>
problem, your first (and almost certainly best) choice for your<br>
loss function is <code>CrossEntropyLoss</code>.</p>
<p>If you are performing a <em>multi-label,</em> multi-class classification<br>
problem, you should use <code>BCEWithLogitsLoss</code> as your loss<br>
function.</p>
<p>If you want to switch between <code>CrossEntropyLoss</code> and<br>
<code>BCEWithLogitsLoss</code>, you do have to change the type, shape<br>
and values of the <code>target</code> (labels) you pass your loss function.</p>
<p>Good luck.</p>
<p>K. Frank</p>
DistributedDataParallel

username: slavavs
cooked: <p>I took. for rent a server with 8 GPU and 8 CPU. I want to use DistributedDataParallel. Please give me a simple example of how to do this. I found a lot of different examples, but I do not fully understand them.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/slavavs">@slavavs</a></p>
<p><a href="https://pytorch.org/docs/stable/notes/ddp.html#example" rel="noopener nofollow ugc">This</a> is the minimum example.</p>
<p>For more advanced usages, please see <a href="https://pytorch.org/tutorials/beginner/dist_overview.html#torch-nn-parallel-distributeddataparallel" rel="noopener nofollow ugc">this overview</a>.</p>
<p>BTW, could you please add a “distributed” tag for questions related to distributed training, so that people work on that can get back to you promptly. Thx!</p>
Is trained resnet 50 a deterministic model

username: flipflop
cooked: <p>I have built a resnet 50 for classification task .I saved the model after training is completed . However when I load back the model and test the accuracy using the test set . It can give me a overall result 89% accuracy . But different accuracy can be obtained if I retest the test set .(test set is the always the same ).is it a problem about the model . I assume the accuracy should be always the same for same input .And the accuracy is actually fluctuating from 89 to 89.5 .(not a big difference) .is this expected ?  I am pretty sure my coding and algorithm is absolutely right .</p>

username: ptrblck
cooked: <p>You might get non-deterministic outputs based on some operations such as non-deterministic cudnn kernels as explained in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a>.<br>
If you follow these docs, you should get deterministic outputs.</p>
Why the default inputs' dimensions of LSTM are [sequence_length, batch_size, feature_size]?

username: Chance
cooked: <p>Why the default inputs’ dimensions of LSTM are [sequence_length, batch_size, feature_size]?<br>
I think it is more naturally to use the data in the shape of [batch_size, sequence_length, feature_size]</p>

username: jpeg729
cooked: <p>Before I comment on the principle, if your input_data is of shape [batch_size, sequence_length, feature_size], then input_data.permute(1, 0, 2) will transform it into shape [sequence_length, batch_size, feature_size].</p>
<p>I believe that permute doesn’t copy the data, it just alters the strides used for the underlying array, so it is very efficient.</p>
<p>I am running some analyses on some really long time-series data and I wanted to create sequential batches. I found that if my data was of shape [batch_size, sequence_length, feature_size], then selecting slices of the form [:, start:end, :] gave me non-contiguous tensors and the model couldn’t use them directly. So to avoid having to copy the tensor in order to make it contiguous, I first made sure my data was of shape [sequence_length, batch_size, feature_size] and then it all worked.</p>

username: jeremysalwen
cooked: <p>I also saw this with <code>nn.MultiheadAttention</code>.  It’s still not clear to me why you would not have the first dimension be the batch size like for <code>nn.Linear</code>.</p>

username: ptrblck
cooked: <p>The layout is chosen for performance reasons as also said <a href="https://discuss.pytorch.org/t/why-are-the-dimensions-of-input-output-in-lstm-oriented-in-the-way-they-are/98826/2">here</a>.<br>
Also, for RNNs you can use <code>batch_first=True</code> to change the shapes.</p>
Understanding Convolution with an Even Sized Filter

username: JamesDickens
cooked: <p>If I have an even sized convolutional filter, say 2 by 2, f_{i,j} for 1&lt;= i &lt;=2,  1&lt;= j &lt; = 2</p>
<p>Then if I apply it to a window centered at (x,y) (assuming appropriate padding),<br>
then will it apply something like<br>
f_{1,1} (x,y) + f_{1,2} (x+1, y) + f_{2, 1}(x, y+1) + f_{2,2}(x+1, y+1) in Pytorch?<br>
I’m not sure what the formula would be for even sized filters basically.<br>
Thanks.</p>

username: ptrblck
cooked: <p>I’m not sure what your notation means exactly, but you could use a simple example as given here:</p>
<pre><code class="lang-python">weight = torch.ones(1, 1, 2, 2)
x = torch.arange(4*4).view(1, 1, 4, 4).float()
print(weight)
&gt; tensor([[[[1., 1.],
            [1., 1.]]]])

print(x)
&gt; tensor([[[[ 0.,  1.,  2.,  3.],
            [ 4.,  5.,  6.,  7.],
            [ 8.,  9., 10., 11.],
            [12., 13., 14., 15.]]]])

out = F.conv2d(x, weight, stride=1, padding=0)
print(out)
&gt; tensor([[[[10., 14., 18.],
            [26., 30., 34.],
            [42., 46., 50.]]]])

out = F.conv2d(x, weight, stride=1, padding=1)
print(out)
&gt; tensor([[[[ 0.,  1.,  3.,  5.,  3.],
            [ 4., 10., 14., 18., 10.],
            [12., 26., 30., 34., 18.],
            [20., 42., 46., 50., 26.],
            [12., 25., 27., 29., 15.]]]])
</code></pre>
<p>As you can see, the kernel will use windows of <code>2x2</code> to create its output.</p>
<p>Let me know, if this example makes the output clear or if you would need more information.</p>
How to map a vector to a skew symmetric (or upper triangular) matrix?

username: jjbouza
cooked: <p>Hi,</p>
<p>I need to generate a skew symmetric matrix from some weights. It would suffice to generate an upper triangular matrix A from the weights, since then</p>
<pre><code class="lang-auto">S = A - A.t()
</code></pre>
<p>would do the trick. The hard part is generating the matrix A from a vector, i.e. [x,y,z] to</p>
<p>0,x,y<br>
0,0,z<br>
0,0,0</p>
<p>and similarly for longer vectors.</p>
<p>Any ideas for how to do this?</p>

username: Zimeng_Zhao
cooked: <p>def skewmat(x_vec):</p>
<pre><code>'''

torch.matrix_exp(a)

Eigen::Matrix3f mat = Eigen::Matrix3f::Zero();

mat(0, 1) = -v[2]; mat(0, 2) = +v[1];

mat(1, 0) = +v[2]; mat(1, 2) = -v[0];

mat(2, 0) = -v[1]; mat(2, 1) = +v[0];

return mat;

input : (*, 3)

output : (*, 3, 3)

'''

W_row0 = torch.tensor([0,0,0,  0,0,1,  0,-1,0]).view(3,3).to(x_vec.device)

W_row1  = torch.tensor([0,0,-1,  0,0,0,  1,0,0]).view(3,3).to(x_vec.device)

W_row2  = torch.tensor([0,1,0,  -1,0,0,  0,0,0]).view(3,3).to(x_vec.device)



x_skewmat = torch.stack([torch.matmul(x_vec, W_row0.t()) , torch.matmul(x_vec, W_row1.t()), torch.matmul(x_vec, W_row2.t())] , dim = -1)

return x_skewmat</code></pre>
Suggestion on improving accuracy and evaluation metrics on a deep learning model

username: Aniruddh
cooked: <p>Hi there! I am currently working on a skin cancer classification problem. The data consists of 9 target values and the model is trained with densenet for 20 epochs. The model file has been saved and the evaluation metrics are observed. But I want to have confidence on what I am seeing and proceed in creating a flask application and go straight to testing. The highest training accuracy is 97% and validation accuracy is 87%. Does that mean my model is evaluating well? This is the value counts of each label after sampling weights.</p>
<pre><code class="lang-auto">AK          8503
VASC        8496
unknown     8493
nevus       8450
melanoma    8410
DF          8400
SCC         8245
BCC         8208
BKL         7443
</code></pre>
<p>This is the best record of my model file.</p>
<pre><code class="lang-auto">best record: [epoch 11], [val loss 0.40036], [val acc 0.87697]
</code></pre>
<p>Here is the confusion matrix, the classification report</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3becc0b663f419404716e57389879a358278d4a5.png" alt="image" data-base62-sha1="8y7puRxuHSLtySOLohzjUbw7ABT" width="349" height="294"></p>
<pre><code class="lang-auto">              precision    recall  f1-score   support

          AK       0.66      0.66      0.66       204
         BCC       0.79      0.91      0.85       700
         BKL       0.67      0.69      0.68       657
          DF       0.66      0.75      0.70        56
       nevus       0.68      0.73      0.70       126
        VASC       0.90      0.86      0.88        63
    melanoma       0.62      0.69      0.65      1067
         SCC       0.92      0.85      0.88      4497
     unknown       0.95      0.96      0.95      5425

    accuracy                           0.87     12795
   macro avg       0.76      0.79      0.77     12795
weighted avg       0.88      0.87      0.87     12795
</code></pre>
<p>Notice in the confusion matrix, the SCC label is incorrectly predicted as melanoma 291 times! This made me take a step back and question my model again as it was a little freaky.<br>
I would really appreciate anyone who can give me a general observation on the model. This would help me make any changes for better results! Thanks a lot for the help.</p>
Adapting Sequence to sequence for time series

username: bibekx
cooked: <p>I am trying to implement a “Sequence to sequence prediction model for time series” using either LSTM or GRU but could not find good tutorials for the exact problem.</p>
<p>However, I found this, which is implemented for a NLP problem (Machine Translation)</p>
<p><a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/1%20-%20Sequence%20to%20Sequence%20Learning%20with%20Neural%20Networks.ipynb" rel="noopener nofollow ugc">Link to Notebook </a></p>
<p>After giving this a thorough read once, I intuitively figured out these things below (Which I am not sure are correct) :</p>
<ul>
<li>For time series data (1 dimensional data which has been converted to chunks of 12 time steps using moving window of stride 1), we do not need a embedding layer</li>
<li>We do not need tokens (, ) at the beginning and end of each sequence</li>
<li>In the NLP data, they use vocab size to represent each word in 1xvocav_size dimensions however, we just need a single dimension for each time step</li>
</ul>
<p>It would be great, If anyone could confirm these<br>
Or if you have any notebook I can follow for sequence to sequence prediction of time series</p>
<p>Thank you</p>
Prediction error : RuntimeError: CUDA error: an illegal memory access was encountered

username: TTyRAeLL
cooked: <p>I encountered this error when computing MSE using a trained model. The related codes are as follows:</p>
<pre><code class="lang-auto">for j in range(5):

    a = torch.randn(40, 8)
    pred = net(a)
    loss = ((pred - y_train)**2).mean()

    train_and_eval(trainloader, testloader, lr, optimizer, loss_func, net, y_test, Epoch = 1)
</code></pre>
<p>It seems that when I use pred, it will throw an error(even when I just print it). How can I fix this?</p>

username: TTyRAeLL
cooked: <p>When I run the code in debug mode, I found that when j is 0, pred is a normal tensor, but when j is 1, pred turned to become <code>torch.Tensor object at ...</code>, what does it imply?</p>

username: ptrblck
cooked: <p>Could you run your code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stack trace here, please?</p>

username: TTyRAeLL
cooked: <p>I tried and the results were the same as before</p>

username: ptrblck
cooked: <p>Could you then post the stack trace, please?</p>

username: TTyRAeLL
cooked: <p>When I set pred to cpu the problem was solved. Thanks for the help anyway.</p>
Downloading Pytorch 3D

username: Kiran_Ikram
cooked: <p>Hi, I’m trying to implement Mesh RCNN, but after running the demo file Im getting the error:<br>
“ModuleNotFoundError: No module named ‘pytorch3d’” , even though I have downloaded it.</p>

username: Reho_Princewill
cooked: <p>From the official tutorials on git, <a href="https://github.com/facebookresearch/pytorch3d/tree/master/docs/tutorials" rel="nofollow noopener">Pytorch3D Tutorials </a>, there wasn’t an <code>import pytorch3d</code> statement rather there were <code>from ... import ...</code>.</p>
<p>i think this to streamline the workflow</p>

username: albertotono
cooked: <p>are you running directly from the demo.py file and where are do you see this error?<br>
using the config provided it should work.</p>

username: albertotono
cooked: <aside class="quote no-group" data-username="Kiran_Ikram" data-post="1" data-topic="70984">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kiran_ikram/48/18473_2.png" class="avatar"> Kiran_Ikram:</div>
<blockquote>
<p>trying to implement Mesh RCNN, but after running the demo file Im getting the error:<br>
“ModuleNotFoundError: No mod</p>
</blockquote>
</aside>
<p>If you want we developed the google colab to try it <a href="https://github.com/CDInstitute/CompoNET" class="inline-onebox" rel="noopener nofollow ugc">GitHub - CDInstitute/CompoNET: CompoNET: geometric deep learning approach in architecture. From a single-image generates a building with all its components</a></p>
How to optimize camera and light parameters in pytorch3d?

username: Tony_Lee
cooked: <p>I want to optimize the light and camera parameters in pytorch3d, but find them to be defined at initial state, so how can I optimize them iteratively ?<br>
for example, the camera is defined as follows:</p>
<pre><code class="lang-auto">def SfMPerspectiveCameras(
    focal_length=1.0, principal_point=((0.0, 0.0),), R=_R, T=_R, device="cpu"
):
    """
    SfMPerspectiveCameras has been DEPRECATED. Use PerspectiveCameras instead.
    Preserving SfMPerspectiveCameras for backward compatibility.
    """

    warnings.warn(
        """SfMPerspectiveCameras is deprecated,
        Use PerspectiveCameras instead.
        SfMPerspectiveCameras will be removed in future releases.""",
        PendingDeprecationWarning,
    )

    return PerspectiveCameras(
        focal_length=focal_length,
        principal_point=principal_point,
        R=R,
        T=T,
        device=device,
    )
</code></pre>
<p>then the focal_length and principal_point can not be learned by optimization.</p>

username: ptrblck
cooked: <p>I don’t know, if any PyTorch3D devs are here in this board (I cannot find Nikhila or Jeremy), so I would recommend to create an issue on <a href="https://github.com/facebookresearch/pytorch3d/issues">their github</a>.</p>

username: albertotono
cooked: <p><a class="mention" href="/u/tony_lee">@Tony_Lee</a>  for SfMPerspectiveCameras their wanted to ensure that  <code>R_absolute</code>  had valid rotation matrix, so they used an exponential map (implemented with  <code>so3_exponential_map</code> ) of the axis-angle representation of the rotation  <code>log_R_absolute</code> .</p>
<p>The boundle adjustment could be solved by recovered up to an unknown global rigid transformation g glob∈SE(3). Thus, for simplicity, they assumed knowledge of the absolute extrinsics of the first camera g0. They set g0 as a trivial camera g0=(I,0⃗ ) to work for simple rigid tranformation, but now that will change soon.</p>
<p>In order to be able to initialize the parameters you will need to use <strong>init</strong> for the class, .self and all the proper constructors. I am sure it will be fixed soon as they mentioned.</p>
<p>change to this class</p>
<pre><code class="lang-auto">class SfMPerspectiveCameras(CamerasBase): 
     def __init__(
        self,
        focal_length=1.0,
        principal_point=((0.0, 0.0),),
        R=_R,
        T=_T,
        device="cpu",
        image_size=((-1, -1),),
    ):
</code></pre>
<p>Hope it helps.</p>
Get size of random layer output encoder-decoder network

username: Flock1
cooked: <p>Hi.</p>
<p>I am using a <code>resnet34</code> model to create an encoder-decoder network. My problem is when I reach I decoder side, the output shape is not the same as the input. I am defining my network like this:</p>
<pre><code class="lang-auto">m = models.resnet34(pretrained = False).cuda()
m = nn.Sequential(*list(m.children())[:-3])
conv = nn.Conv2d(256, code_sz, kernel_size=(2,2), stride=2).cuda()
m.add_module('CodeIn',conv)
conv = nn.Conv2d(32, 8, kernel_size=(2,2), stride=2).cuda()
m.add_module('Conv_2',conv)
conv = nn.Conv2d(8, 2, kernel_size=(2,2), stride=2).cuda()
m.add_module('Conv_3',conv)
</code></pre>
<p>This is the decoder:</p>
<pre><code class="lang-auto">add_layer(m,1,2,'CodeOut_1',scale=2)
add_layer(m,2,8,'CodeOut_2')
add_layer(m,8,32,'CodeOut_3')
add_layer(m,32,256,'CodeOut_4',scale=2)
add_layer(m,256,128,'Upsample0')
add_layer(m,128,64,'Upsample1')
add_layer(m,64,32,'Upsample2')
add_layer(m,32,3,'Upsample3',act='sig', scale=2)
</code></pre>
<p>Where <code>add_layer</code> just adds a new layer. One argument that I can assign to the function is <code>output_size</code> but I don’t know how to get this value or dynamically assign it while training because the size of my input images is also different. Please let me know what should I do.</p>
<p>Is there some way I can use <code>m._module</code> to get the size?</p>
Pytorch Geometric

username: snehith_routhu
cooked: <p>Hi, I’m new to Pytorch-geometric, I want to do mesh refining using graph cnn’s. I have an initial coarse mesh generated from a point cloud where the number of neighbours of each node is not same.I would like update the vertex(node) positions using graph network. I would like to know which type of graph convolution  or graph network would be better for my experiment.</p>

username: a_d
cooked: <p>Hello,<br>
This survey might help you - <a href="https://arxiv.org/abs/1901.00596" rel="noopener nofollow ugc">A comprehensive survey on graph neural networks </a></p>
Pytorch trace model vs ONXX, which one is better for deploying web app in front or back, or running in other languages, such as Java but trained in Python?

username: JonathanSum
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/parasol_on_ground.png?v=9" title=":parasol_on_ground:" class="emoji" alt=":parasol_on_ground:">Pytorch trace model vs ONXX, which one is better for deploying web app in front or back, or running in other languages, such as Java but trained in Python?</p>