akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
[newbie] how to copy the value of torch::Tensor to another torch::Tensor

username: Kallinteris-Andreas
cooked: <pre><code class="lang-auto">torch::Tensor t0 = torch::zeros(1);
torch::Tensor t1 = torch::ones(1);
t1 = t0.detach().clone();//does not work
t1 = t0.clone();//does not work
t1 = t0.detach();//does not work
t1 = t0.clone().detach();//does not work
t1.data().copy(t0.data());//does not compile
t1.data() = t0.data();//does not compile
t1.data_ptr() = t0.data();//does not compile
t1.data_ptr() = *t0.data_ptr();//does not compile

t1 = t0 //works but i can't verify if it copies the data or the reference		
assert((t1 == t0).item&lt;bool&gt;());//Verify that copy was succefull
</code></pre>

username: Matej_Kompanek
cooked: <p>this code:</p>
<pre><code class="lang-auto">torch::Tensor t0 = torch::zeros(1);
torch::Tensor t1 = torch::ones(1);
t1 = t0.detach().clone();

std::cout &lt;&lt; (t1 == t0).item&lt;bool&gt;() &lt;&lt; std::endl;
</code></pre>
<p>outputs: 1<br>
Doesn’t it mean it works ?</p>

username: Kallinteris-Andreas
cooked: <p>it does …<br>
must have made some typo along the way</p>

username: Matej_Kompanek
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> sure, friday night, we’re all drunk</p>

username: Kallinteris-Andreas
cooked: <p>if that works why doesn’t this</p>
<pre><code class="lang-auto">		const int input_nodes = 20, output_nodes=1, hidden_count = 1;
		Net nn (input_nodes, output_nodes, hidden_count);
		Net nn1 (input_nodes, output_nodes, hidden_count);
		for (size_t i = 0; i &lt; nn.parameters().size(); i++ ){
			//torch::Tensor t = nn.parameters()[i].detach().clone();
			//nn1.parameters()[i].set_data(t);
			nn1.parameters()[i] = nn.parameters()[i].detach().clone();
		}
		torch::Tensor input = torch::rand({1,input_nodes});
		assert((nn.forward(input) == nn1.forward(input)).item&lt;bool&gt;());

</code></pre>
<p>Net is a standard torch::nn::module</p>
<pre><code class="lang-auto">struct Net : torch::nn::Module {
	Net(int numIn, int numOut, int numHid, const size_t hid_count=1) {
		assert(hid_count &gt;= 1);
		first = register_parameter("inputW", torch::rand({numIn, numHid}))/numHid;
		middle = new torch::Tensor[hid_count-1];
		for (int i = 1; i != hid_count; i++)
			middle[i] = register_parameter("hidW"+std::to_string(i), torch::rand({numHid, numHid}))/numHid;
		last = register_parameter("outputW", torch::rand({numHid, numOut}))/numOut;
		h_c = hid_count;
		n_h = numHid;
	}
	torch::Tensor forward(torch::Tensor input) {
		torch::Tensor output_layer,h;
		h = (torch::mm(input, first));
		for (int i = 1; i != h_c; i++)
			h = torch::sigmoid(torch::mm(h, middle[i]));
		output_layer = (torch::mm(h, last));
		return output_layer;
	}
	torch::Tensor first, last, *middle;
	size_t h_c, n_h;
};
</code></pre>
<p>I create a new thread for this topic: <a href="https://discuss.pytorch.org/t/why-do-2-identical-neural-network-fowarding-a-different-result/117687/2" class="inline-onebox">Why do 2 identical Neural Network fowarding a different result</a></p>
Batchnorm downgrades performance with different batchsizes

username: AliAmiri
cooked: <p>Hi,<br>
I have a problem with batchnorm(1d) there are some discussions on that but did’t helped<br>
I train a network (with batchnorm1d) with batch_size 512. during eval and train it works fine with same number of batches but by changing batchsize from 512 to another value accuracy decreases</p>
<p>network:</p>
<pre><code class="lang-auto">class Model(nn.Module):
	def __init__(self, outdim, momentum, track=False):
		super(Model, self).__init__()
		self.inp = nn.Sequential(
              nn.BatchNorm1d(256, track_running_stats=track, momentum=momentum),
        )
		self.rnn1 = nn.GRU(7, 32, batch_first=True, bidirectional=True)
		self.attn1 = nn.MultiheadAttention(32*2, 8, dropout=.2)
		self.model = nn.Sequential(
			nn.BatchNorm1d(256*32*2, track_running_stats=track, momentum=momentum),
			nn.Linear(256*32*2, 64), nn.ReLU(),
			nn.BatchNorm1d(64, track_running_stats=track, momentum=momentum),
			nn.Linear(64, 128), nn.ReLU(),
			nn.BatchNorm1d(128, track_running_stats=track, momentum=momentum),
			nn.Linear(128, 128), nn.ReLU(),
		)
		self.out = nn.Sequential(
			nn.BatchNorm1d(128, track_running_stats=track, momentum=momentum),
			nn.Linear(128, outdim)
		)
		self.opt = AdamW(self.parameters(), .0001)
		self.loss = nn.CrossEntropyLoss()
		self.to(device)

	def forward(self, x):
		x = T.as_tensor(x).to(device).float()# x: (-1, 256, 7)
		x = self.inp(x)
		rnn1 = self.rnn1(x)[0]
		attn1 = self.attn1(*[rnn1.transpose(0,1)]*3)[0].transpose(0,1)
		model = self.model(attn1.flatten(1))
		return self.out(model)

</code></pre>
<p>results:<br>
after doing following:</p>
<pre><code class="lang-auto">model2 = Model(2, 0)
model2.load_state_dict(model.state_dict())
with T.no_grad():
    model2.forward(x_train[:32])
    model2.forward(x_train[32:32*2])
    model2.forward(x_train[32*2:32*3])
    model2.forward(x_train[32*3:32*4])
model2.eval()
with T.no_grad():
    accs = []
    for j,(x,y) in enumerate(data.batch(x=x_train, y=y_train, batch_size=32)):
        output = model2(x).cpu().detach().numpy().argmax(-1).reshape(-1)
        accs.append((output == y.astype(int)).sum() / len(y))
        print(str(j/(len(y_train) / x.shape[0]))[:7])
        display.clear_output(1)
print(str(np.mean(accs))[:7])
</code></pre>
<p>Accuracy: 85.5%<br>
and if I change batch_size from 512 to 32 results in:<br>
Acuraccy: 47.5%</p>
<p>btw I also tried changing momentum in <code>model2</code> but no effects</p>
Split some classes from Tiny-ImageNet dataset

username: S_M
cooked: <p>Hello all,<br>
I am trying to split class labels 0 to 9 of the Tiny-imagenet dataset so I tried the following code</p>
<pre><code class="lang-auto">    train_dataset = TinyImageNet('tiny-imagenet-200', 'train', transform=transform)
    train_labels_np=np.array(list(train_dataset.labels.values()))
    train_indexes=np.where((train_labels_np==0)|(train_labels_np==1)|
                            (train_labels_np==2)|(train_labels_np==3)|
                            (train_labels_np==4)|(train_labels_np==5)|
                            (train_labels_np==6)|(train_labels_np==7)|
                            (train_labels_np==8)|(train_labels_np==9))[0]
    np.random.shuffle(train_indexes)    #Shuffle   
    training_split_indexes = train_indexes[0:2048]    #select some examples
    sampler_train= torch.utils.data.SubsetRandomSampler(training_split_indexes)
 
    train_set = torch.utils.data.DataLoader(train_dataset, batch_size=args.train_batch_size, sampler=sampler_train, **kwargs)
    test_dataset = TinyImageNet('tiny-imagenet-200', 'val', transform=transform)
    test_labels_np=np.array(list(test_dataset.labels.values()))
    test_indexes=np.where((test_labels_np==0)|(test_labels_np==1)|
                            (test_labels_np==2)|(test_labels_np==3)|
                            (test_labels_np==4)|(test_labels_np==5)|
                            (test_labels_np==6)|(test_labels_np==7)|
                            (test_labels_np==8)|(test_labels_np==9))[0]
  
    sampler_test = torch.utils.data.SubsetRandomSampler(test_indexes)
    test_set = torch.utils.data.DataLoader(test_dataset, batch_size=args.test_batch_size, shuffle=False, sampler=sampler_test, **kwargs)
</code></pre>
<p>Error <code>IndexError: Target 68 is out of bounds.</code> has occurred from<br>
<code>               loss= self.cross_entropy_loss(output, target)</code><br>
in test function.</p>
<p>how can I handle it?<br>
thanks.</p>
Have specific examples not back propagate

username: Ben_Nicholl
cooked: <p>Apologies if this is simple, but I can’t find the answer anywhere.<br>
I’m running a 2 class classification NER task with a transformer, and I would like to have specific words in a segment not back propagate. For example, take the sentence, “the dog ran”, and say I don’t want “ran” to be back propagated. If I could use one hot encoded vectors, I could simply have the corresponding label vectors to “ran” be [0,0]. But to my knowledge, PyTorch does not support one hot encoded vectors, and I need to put some integer as the class from 0 to C-1. So is there another way to achieve this?</p>

username: ptrblck
cooked: <p>You could create an additional class index, which can be ignored in the loss calculation by specifying <code>ignore_index</code> in the criterion, and set the desired words to this index.</p>
Combining two input images and tabular data

username: mck97
cooked: <p>Hi everyone,</p>
<p>I’m a beginner with PyTorch and doing my first DL project.</p>
<p>I have created my own dataset, which is made of a collection of:</p>
<ul>
<li>one image</li>
<li>another image</li>
<li>x-coordinate location</li>
<li>y-coordinate location</li>
</ul>
<p>I want to “combine” those 4 data points in one, so that I can feed that my neural net. In the end I want to predict x, y locations as well, so this is a multi-output regression problem if I’m correct.</p>
<p>What would be the approach to combine the 4 data points in one so I can feed to a CNN?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>To “combine” values you could either use <code>torch.cat</code> or <code>torch.stack</code>, which will concatenate the subtensors in a specified dimension or a new one.<br>
I’m not sure, if this is what you are looking for exactly.</p>
Dataparallel and network with custom forward function

username: Vinson2233
cooked: <p>I’m working on retraining <a href="https://github.com/openai/CLIP" rel="noopener nofollow ugc">CLIP</a> to train on several new tasks.<br>
I’m still new to dataparallel(I’m not using the DDP since DP seems simpler to implement).<br>
This question is related to this thread</p><aside class="quote quote-modified" data-post="2" data-topic="58528">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-create-a-network-with-multiple-forward-routines/58528/2">How to create a network with multiple forward routines</a> 
  </div>
  <blockquote>
    Hi, 
I would do the following: 
class H:
  def __init__(self, *args):
    # init H

  def forward(self, inp):
    # forward of H
    return out

class G:
  def __init__(self, *args):
    # init G

  def forward(self, inp):
    # forward of G
    return out

class F:
  def __init__(self, *args):
    # init F
    self.g = G()
    self.h = H()

  def forward(self, inp):
    # forward of F
    return self.g(self.h(inp))

model = F()

# forward the whole model
model(inp)

# foward only g
model.g(inp)…
  </blockquote>
</aside>
<p>
Since I want to use data parallel on non-forward method of the network, I decide to write this :</p>
<pre><code class="lang-auto">class CustomModel1(nn.Module):
  def __init__(self, model) :
        super(CustomModel1, self).__init__()
        self.model = model
  def forward(self,x,y):
    output1 = self.model.custom_methodA(x)
    output2 = self.model.custom_methodA(y)
    &lt;do_something&gt;
    return result1,result2

class CustomModel2(nn.Module):
  def __init__(self, model) :
        super(CustomModel2, self).__init__()
        self.model = model
  def forward(self,x,y):
    output1 = self.model.custom_methodB(x)
    output2 = self.model.custom_methodB(y)
    &lt;do_something&gt;
    return result1,result2

model = CLIP()
model1 = CustomModel1(model)
model2 = CustomModel2(model)

model1 = nn.DataParallel(model1)
model2 = nn.DataParallel(model2)

optimizer = torch.optim.Adam(model.parameters())
loss1 = nn.BCELoss()
loss2 = nn.BCELoss()

for batch in dataloader:
  a,b,c,d,y = batch
  result1 = model1(a,b)
  result2 = model2(c,d)
  loss = loss1(result1,y) + loss2(result2,y)
  loss.backward()
  optimizer.step()
  optimizer.zero_grad()
</code></pre>
<p>This code will give <code>RuntimeError: CUDA error: device-side assert triggered</code> at the <code>loss.backward()</code>.</p>
<p>Which step is wrong in my code? I feel some stuff is wrong but can’t figure out how to handle it the right way.  Like, is the computational graph of CLIP inside CustomModel2 and 1 still connected to the original model? Should my optimizer connected to model, or should I create separate optimizer for model1 and model2?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Could you rerun the script via <code>CUDA_LAUNCH_BLOCKING=1 python script.pt args</code> and post the complete stack trace here, please?</p>
Train one model with another pre trained model

username: UCDuan
cooked: <p>Hi all,<br>
I’m wondering how to train a model with another pre-trained model.</p>
<p>Say I have a pre-trained model (model2) which takes input as the shape of (batch, 365, 15), and I have another model (model1) generate the output as (batch, 1, 3). I wanna use these outputs to serve as the input to my pre-trained model. What is the proper way to concatenate the inputs to my pre-trained model? Now I have zero gradients with the following code.</p>
<pre><code class="lang-auto">out = model_1(someinputs)
input[:, :1, 0] = out[:, :, 0]
input[:, :1, 5] = out[:, :, 1]
input[:, :1, 10] = out[:, :, 2]
y_hat = model_2(input)
calculate loss
gradient turns None.
</code></pre>
<p>I’m feeling this is not a proper code but could someone suggest a better method?<br>
Thank you all.</p>
Resizing Seismic Data for deep learning

username: sgautam666
cooked: <p>Hello,<br>
I am trying to build a UNET model for some fault prediction using seismic data. So, my training datasets contains 2 sets:<br>
A seismic and fault mask: Both are 3D volumes with following shapes:101, 589, 751<br>
Here, 101 is number of inlines, 589 is number of xlines<br>
and 751 is vertical sample.<br>
So, the idea is to treat the volume as stack of 2D arrays, in either xline or inline direction.</p>
<p>There is no RGB, so we treat them as single channel.<br>
i.e, along Inline direction: shape : 589 x 751 and there is 101 inlines stacked. Or we can look into xline in similar manner.</p>
<p>Now, the main issue I am having is to resize this into proper sample intervals of 256, which I believe is required for U-Net.<br>
Can anyone help me how I can achieve this with pytorch transforms.<br>
Thanks and much appreciated.</p>
<p>Below is an example of way the data is loaded:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89eb3686bf1281e3293ef56d5d2516d9dfb75c3f.png" data-download-href="https://discuss.pytorch.org/uploads/default/89eb3686bf1281e3293ef56d5d2516d9dfb75c3f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89eb3686bf1281e3293ef56d5d2516d9dfb75c3f.png" alt="image" data-base62-sha1="jG5m0um74Hyx5w9va9UcBgiigWz" width="690" height="382" data-dominant-color="F2F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">808×448 19.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: a_d
cooked: <p>You could pad your volume with <code>F.pad</code>. I am not familiar with the seismic data, so you would have to see the padding value and make sure it represents the background. For example in image segmentation I always pad with zeros giving it a thick black boundary which can be considered as the background.<br>
You could read about <code>F.pad</code> here-  <a href="https://pytorch.org/docs/stable/nn.functional.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional — PyTorch 1.8.1 documentation</a></p>
Adding the same parameter to optimisation multiple times

username: Joe_Harrison
cooked: <p>Does it matter if you add the same parameter multiple times to the vector of parameters in an optimiser? Example:</p>
<pre><code class="lang-auto">std::vector&lt;torch::Tensor&gt; differentiable_nodes;

differentiable_nodes.emplace_back(some_node.parameters()[0]);
differentiable_nodes.emplace_back(some_node.parameters()[0]);

torch::optim::SGD optimizer(differentiable_nodes, 0.001);
</code></pre>
<p>Does this in any way affect the training of these parameters? I want to avoid checking for duplicate nodes.</p>

username: ptrblck
cooked: <p>You should get a warning, as duplicated parameters will cause an error in the future:</p>
<pre><code class="lang-python">x = nn.Parameter(torch.randn(1))
optimizer = torch.optim.SGD([x, x], lr=1.)
&gt; UserWarning: optimizer contains a parameter group with duplicate parameters; in future, this will cause an error; see github.com/pytorch/pytorch/issues/40967 for more information
</code></pre>

username: Joe_Harrison
cooked: <p>I’m not getting this warning in Libtorch 1.8. Seen that it gives a warning in python I probably better avoid duplicate parameters</p>

username: ptrblck
cooked: <p>I’m not sure how these warning are used in <code>libtorch</code>, but I agree that you should rather avoid it, as you would run into errors in future releases.</p>
Gradient Computation as Inference

username: Blake_Camp
cooked: <p>I have a rather obscure Q I need help with (which i’ll be very surprised if someone can help me with, so no worries if it doesnt make sense).</p>
<p>Suppose I seek a model f_(cur_x), parameterized by 2 groups of variables theta and phi, which inherently induces correct gradients when backpropagating loss_(cur_x).  Said another way, suppose I had access to target gradients, and suppose I viewed computation of the gradient (backward) as a type of inference procedure (just a forward function). Suppose this is possible due to the inclusion of params (phi) during the backward pass, which are not present in a standard forward pass.</p>
<p>Is there a way to frame this so that I can minimize the error btw the gradients induced by my model, and the target gradients?  Said another way, its as if I need to call backward on the hypothetical output of my backward function. To be clear, I am not interested in using a hyper-network to generate gradients. I seek params (phi) which induce specific gradients for another set of params (theta) when they are included in the model. Am I simply talking about second order derivatives here?</p>
<p>What am I missing?</p>
<p>I’ve reviewed some of the posts regarding extending autograd, and custom backwards functions, but im wondering if there is an easy way to do this:  frame the backward call as a forward call, which returns gradients.  I should be able to measure the error btw those grads and target grads, and use that err to compute a gradient for phi, thereby minimizing the difference.</p>
<p>make sense, or will it not work?  Any feedback would be very much appreciated. Thanks.</p>
How to load a huge dataset

username: moh_soba
cooked: <p>hi guys , I’m new in pytorch , I am wondering if 3GB image dataset is huge to load in my CPU memory , should I crop or resize them (50<em>50 → 32</em>32) if that’s possible</p>

username: aza
cooked: <p>it moreso depends on your RAM. Assuming you have enough overhead → there it shouldn’t be a problem. Cropping and resizing is a great way to speed things up, but you risk losing information</p>
I have one server w/ Two type GPU(RTX 3090, TITAN RTX)

username: jaejun
cooked: <p>Hello<br>
I’m newbie user on server<br>
first of all, I introduce my server spec<br>
Power : 1600Wx3<br>
CPU : 2 x Intel® Xeon® Silver 4210R CPU @ 2.40GHz<br>
GPU : TITAN RTX x5  +  RTX3090 x3</p>
<p>Now i’m trying to my server for deeplearning(object detection)<br>
i use <a href="https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch" rel="noopener nofollow ugc">EfficientDet pytorch version code</a><br>
At first I had only TITAN RTX x5<br>
i could train w/ All graphics card</p>
<p>but i bought another RTX3090 x3 and mount on the server yesterday<br>
so i just changed ./projects/proeject.yml     num_gpu=8<br>
and then it did not train<br>
i checked using nvidia-smi<br>
but it did not changed after allocated almost 1.5GB each<br>
and pytorch recognize RTX3090 for cuda:0 even though RTX3090 is not PCI bus number 0<br>
RTX3090 PCI bus number=1,2,3<br>
TITAN RTX PCI bus number=0,4,5,6,7</p>
<p>so I doubted 2 things</p>
<ol>
<li>CUDA 11.1 does not recognize 2 types of graphic card(3090,titan)<br>
==&gt; but CUDA 11.1 can recognize 2types of graphic card according to NVIDIA Developer Forum</li>
<li>Dataparallel module does not have compatibility for 2 types of graphics card</li>
</ol>
<p>How can i use all the GPU for one Task</p>

username: ptrblck
cooked: <p>CUDA11.1 does recognize Turing and Ampere GPUs and it seems that they are indeed used, but your code seems to hang?<br>
<code>nn.DataParallel</code> should also be compatible with different architectures, but note that the slower GPUs would most likely be the bottleneck and the faster ones would have to wait.</p>
<aside class="quote no-group" data-username="jaejun" data-post="1" data-topic="117562">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/edb3f5/48.png" class="avatar"> jaejun:</div>
<blockquote>
<p>and pytorch recognize RTX3090 for cuda:0 even though RTX3090 is not PCI bus number 0</p>
</blockquote>
</aside>
<p>That’s expected and you can change it via <code>export CUDA_DEVICE_ORDER=PCI_BUS_ID</code>.</p>

username: jaejun
cooked: <p>Thanks for reply</p>
<p>Yes, my situation is hang</p>
<blockquote>
<p>os.environ[“CUDA_DEVICE_ORDER”] = “PCI_BUS_ID”<br>
os.environ[“CUDA_VISIBLE_DEVICES”] = “0,1,2,3,4,5,6,7”</p>
</blockquote>
<p>I used upper code in my script<br>
i waited more than 10m  but it did not work<br>
so if i wait more time, does it work???</p>

username: agolynski
cooked: <p>Could you try to see where it hangs, maybe add logging or see which GPUs are busy and idle during training?</p>
<p>You can also do an experiment where you run on 1 TITAN and 1 RTX3090 only and see if such training completes.</p>
<p>If the issue is some GPUs are faster and it leads to work imbalance as the result, maybe try to decrease batch size and see if it helps.</p>
JIT enabled indicator

username: Monne
cooked: <p>Hi,<br>
is there any method that can show model are using JIT or not? i mean, with this call <code>torch.jit._state.disable()</code> it will disable jit model initialization after that.</p>
<p>Just want to make sure model are loaded using JIT.</p>

username: SplitInfinity
cooked: <p><code>torch.jit._state.disable()</code> and <code>torch.jit._state.enable()</code> are not meant to be user-facing APIs and should not be called. All user-facing APIs are exposed in <code>torch/jit/__init__.py</code> as <code>torch.jit.XXX</code>.</p>
<p>One thing you can do is check the type of the <code>Module</code> or function you have loaded. TorchScript modules are of type <code>RecursiveScriptModule</code> in Python, and functions are of type <code>ScriptFunction</code>. In addition, the serialization formats used by regular Pytorch and TorchScript are not the same, so if a model is successfully loaded using <code>torch.jit.load</code>, then it is a TorchScript model.</p>
Scripting vs tracing

username: daver
cooked: <p>Hi folks. I am pretty new to pytorch, and torchscript, and I am very confused by the tutorial material on the subject. For instance, the " Using Scripting to Convert Modules" section of this doc: <a href="https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Introduction to TorchScript — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>
<p>The first two “Out:” blocks of that section seem identical to me, aside from variable/parameter names, even though the second one supposedly is contains the branching logic that the first one is missing. So, am I going crazy, or is there something subtle happening there, or is that doc incorrect?</p>

username: ptrblck
cooked: <p>The outputs don’t make the difference obvious, but the warning raised by tracing the model would give you some information, what could fail using this approach:</p>
<pre><code class="lang-python">TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if x.sum() &gt; 0:
</code></pre>
<p>This warning should be raised while tracing the <code>MyDecisionGate</code>.</p>

username: SplitInfinity
cooked: <p>The documentation is incorrect because the control flow is in the innermost module and that is not the one for which code is being printed. I have had a <a href="https://github.com/pytorch/tutorials/pull/866" rel="noopener nofollow ugc">PR</a> up to fix this for a while but couldn’t get a review for a while and I haven’t merged it since it’s been accepted. I will get this merged today.</p>
Torch.hub.load vs torchvision.model

username: AMellinger
cooked: <p>Greetings!  These two commands result in the “same” model if I dump them with torchsummary, but end up causing different downloads (cached files).</p>
<p>import torch.hub<br>
hub_model = torch.hub.load(‘pytorch/vision’, ‘resnet18’, pretrained=True)</p>
<p>import torchvision<br>
made_model = torchvision.models.resnet18(pretrained=True)</p>
<p>I am preloading caches for a container for distribution and I want to undestand why they are different and how. Also, how to determine the download name from each call would be useful.</p>
<p>Thanks for any help!</p>
How to optimize two sets of parameters and jump out local minima?

username: maskjp
cooked: <p>I have a function that has a structure like this: <img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/380b1a7776e699697377b2d4b9abd726fa02885d.png" alt="download (3)" data-base62-sha1="7ZMwGHIX3KoBYmmKd1la0bpkJcN" width="175" height="24"><br>
x are input, theta_1 and theta_2 are two sets of trainable parameters. g and h are two defined differentiable functions. h outputs a scalar. F is a fully connected neural network. The number of parameters theta_1 is much smaller than the number of parameters theta_2. (say theta_1 represents 10 trainable parameters and theta_2 represents 1000 trainable parameters) My goal is to minimize this function and get the corresponding value of theta_1.</p>
<p>I initialize theta_1 based on prior information and initialize theta_2 randomly. From my experiments, the function stays in local minima if I have a bad initial of theta_1. I tried to use <code>torch.optim.lr_scheduler.</code> <code>CosineAnnealingLR</code> and a large learning rate to jump out of the local minimum. But it doesn’t always work.<br>
I wonder whether there are any methods that can help me to avoid local minimal. Like learning rate scheduler strategy or design different structure of F.</p>
<p>Thanks!</p>
How to check if there is an AMD GPU

username: Pablo
cooked: <p>Hello everyone.</p>
<p>I would like to ask how to check whether there is an AMD GPU installed. Does <code>torch.cuda.is_available()</code> work for this case?</p>
<p>In my case it returns <code>False</code> so I am not sure whether I have not properly installed some library or it’s just that this is not the command.</p>
<p>Thank you!</p>
#Time series dataset #Anomaly detection #Stateful LSTM

username: cookieandcook
cooked: <p>Hello<br>
I want to make  anomaly detection model.<br>
so  I’m trying to train normal data pattern with Stateful LSTM.<br>
I find the code about stateful lstm predictor but the code is coded with tensorflow.<br>
It’s very easy to implement stateful lstm mode.<br>
I have to only implement an hyperparameter when coding to LSTM model.</p>
<p>Now I want to make the model to pytorch code.<br>
How can I make the model?<br>
Is there anything to know before starting?</p>
<p>(ps. what is repeat vector?<br>
whate is unroll in lstm hyperparameter?)</p>
<p>the stateful lstm model with tensorflow is below.</p>
<blockquote>
<p>class StatefulMultiStepLSTM(object):</p>
</blockquote>
<pre><code>def __init__(self,batch_size, look_back, look_ahead, layers, dropout, loss, learning_rate):
    self.batch_size = batch_size
    self.look_back = look_back
    self.look_ahead = look_ahead
    self.n_hidden = len(layers) - 2
    self.model = Sequential()
    self.layers = layers
    self.loss = loss
    # self.optimizer = optimizer
    self.learning_rate = learning_rate
    self.dropout = dropout
    logging.info("StatefulMultiStepLSTM LSTM Model Info: %s" % (locals()))

def build_model(self):
    # first add input to hidden1  
    self.model.add(LSTM( units=self.layers['hidden1'],
        batch_input_shape=(self.batch_size,self.look_back,self.layers['input']),
        #batch_size=self.batch_size,
        stateful=True))
        # unroll=True,
        # return_sequences= True if self.n_hidden &gt; 1 else False)) 
    self.model.add(Dropout(self.dropout)) #dropout= 0.1  10% drop

    # add hidden layers
    for i in range(2, self.n_hidden + 1):
        return_sequences = True
        if i == self.n_hidden:
            return_sequences = False
        self.model.add(LSTM(units = self.layers["hidden" + str(i)], stateful=True,return_sequences=return_sequences,unroll=True))
        self.model.add(Dropout(self.dropout))


    # add dense layer with output dimension to get output for one time_step
    self.model.add(Dense(units=self.layers['output']))

    # Repeat for look_ahead steps to get outputs for look_ahead timesteps.

    self.model.add(RepeatVector(self.look_ahead)) 

    # add activation
    self.model.add(Activation("linear"))

    # compile model and print summary
    start = time.time()
    self.model.compile(loss=self.loss, optimizer=Adam(lr=self.learning_rate,decay= .99))
    #self.model.compile(loss=self.loss, optimizer=Adam(lr=self.learning_rate))
    logging.info("Compilation Time : %s" % str(time.time() - start))
    self.model.summary()
    return self.model
</code></pre>

username: deividbotina
cooked: <p>Hello, I think <a href="https://discuss.pytorch.org/t/confusion-regarding-pytorch-lstms-compared-to-keras-stateful-lstm/44502/3">this</a> can be helpful.</p>
How does metric learning work for content-based image retrieval problems?

username: ElisonSherton
cooked: <p>I was doing some computer vision experiments and recently I have started learning about metric learning and the image retrieval problem. I was experimenting with the inshop image retrieval dataset to build a content based image recommendation system.</p>
<p><img src="https://i.stack.imgur.com/byVh2.png" alt="enter image description here" width="690" height="321"></p>
<p>The dataset is structured as shown above, <a href="https://drive.google.com/drive/folders/0B7EVK8r0v71pWVBJelFmMW5EWnM" rel="noopener nofollow ugc">here</a> is the complete file.</p>
<p>I thought of using <a href="https://arxiv.org/abs/1904.06627" rel="noopener nofollow ugc">Multi-Similarity Loss Function</a> with Batch Hard Triplet Mining for training. What I could understand from the way any variant of triplet loss works; is that it tries to bring the items from same class closer to each other and those from different classes away from one another. I <code>[referred this post][4]</code> for the same.</p>
<blockquote>
<p>I wondered then, how will it learn global concepts or will it learn only local semantic information? I thought it should be the latter because we’re explicitly framing the problem in this manner. But the results which I got confirmed that it also captures global level semantic information.</p>
</blockquote>
<p>For eg. If there’s a blue floral dress, a green floral dress and a red floral dress in the training data, it will try to bind the products of blue, green, red floral dresses closer whereas pushing the inter-class members as far as they can. But when I try doing a similarity search in the test set; if I have a blue floral dress &amp; if I have 10 dresses in my inventory (let’s say 8 solid print dresses and 2 floral dresses (green and red)), then the algorithm gives a higher ranking to the two floral dresses than and puts them above the ones with the solid print.</p>
<p>What’s confounding here is that I didn’t give any tag related information or try to weight one class/product set more over the other when training the model with the triplet loss function but it still captured the similarity across different product sets pretty well. So, how did it achieve that? How come a byproduct so good is achieved that captures the semantic info so well; when our main focus was to separate the images of different classes as far apart as we could (As per the definition of the triplet loss function)?</p>
<p>I would be highly obliged if anyone could provide me some insight into the working of metric learning using contrastive/triplet loss functions and if you think there’s any better way to actualy do it I would be delighted to hear about that from you as well</p>
<p>Thanks!</p>
Evaluating Barlow Twins on CIFAR10 datasets using Linear protocol and pretrained weights available

username: israrbacha
cooked: <p>I use the pre-trained weights provided by FAIR for Barlow twins a new self-supervised learning approached. My top1 validation accuracy is almost static(68%)  and top5 accuracy is higher(97%) than the official results.</p>
<p>model = torch.hub.load(‘facebookresearch/barlowtwins:main’, ‘resnet50’)<br>
model.fc=nn.Identity()</p>
<p>model.requires_grad_(False).to(device)</p>
<p>classifier = nn.Linear(2048, 10).to(device)</p>
<p>classifier.weight.data.normal_(mean=0.0, std=0.01)</p>
<p>classifier.bias.data.zero_()<br>
criterion = nn.CrossEntropyLoss().to(device)</p>
<p>optimizer = optim.Adam(classifier.parameters(), lr=1e-3, weight_decay=1e-6)![baa|690x373]</p>
<p>and the rest is training loop …(upload://o35oIvJFn8e8dEbEuhV0fkntTjh.png)</p>
PyTorch C++ FrontEnd returning multiple Tensors

username: sharonaim
cooked: <p>I was wonder how can I return a std::vector<a>torch::Tensor</a> in my forward pass of a Module Class,<br>
I read about the Macro of FORWARD_HAS_DEFAULT_ARGS in the docs, but didn’t really<br>
understand how to use it, and also how to use it for making it possible to return a vector in return.<br>
Thank you in advance.</p>

username: Ajinkya_Ambatwar
cooked: <p>I am also wondering how is it possible to do this.<br>
I was thinking about creating a strcuture like</p>
<pre><code class="lang-auto">typedef struct
{	
  at::Tensor index;
	at::Tensor mask;
} Ret;
</code></pre>
<p>and then define the function as</p>
<pre><code class="lang-auto">Ret *function(args)
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">at::Tensor function(args)
</code></pre>
<p>I wish to use this c++ implementation with a python wrapper for pytorch. How can I reflect this modification <code>at::Tensor</code> to <code>Ret *</code> in PyTorch implementation?</p>
Tiling image, getting predictions and assembling output in order

username: TWman
cooked: <p>Hi everybody,</p>
<p>I have recently build an algorithm that makes use of ResNet 50 and trained it on a custom database for class prediction of image tiles. It works like a charm and produces predictions of high accuracy.<br>
I am very pleased.</p>
<p>Now I am trying to upgrade the program. The tiles are taken from larger images and I produce the tiles with a small script. I then use ImageFolder and Dataloader to pipeline the tiles into the model.<br>
<strong>However, I would much rather automate the process of creating the tiles to reduce the manual inputs.</strong></p>
<p>For this I have been looking at something like:<br>
tiles = image.data.unfold(0, 3, 3).unfold(1, 50, 50).unfold(2, 50, 50)  # tile size 50 by 50,<br>
but I am at a bit of a loss as to how to apply transformations to this and feed the tiles to the model.</p>
<p><strong>The other end of the program needs to produce a heatmap of the results that is reassembled with the results in the correct place of the respective tile.</strong> I can produce color coded tiles based on the predictions easy enough, but I am not sure how to keep track of the correct order of the tiles through ImageFolder and Dataloader.</p>
<p>Any comments are appreciated. If there are examples where similar things have been done that people are aware of, great.</p>
<p>Many thanks.</p>

username: TWman
cooked: <p>I have figured out a solution where I split the larger image into a list of tiles and then run one big loop in which I pass one tile at a time to the model.<br>
Maybe not pretty, but it works pretty well.</p>

username: yasmineaitouares
cooked: <p>Hello,<br>
I had to tile my images to train my object detection model. For prediction I was wondering how you did reassembled the output into one complete image ?<br>
Thank you</p>
Is this training graph looks okay?

username: krishna511
cooked: <p>Hi there, I am training my PyTorch (1D CNN )model for 50 epochs on an audio dataset. the first graph is without using any validation loss scheduler while for the second one I have used a loss scheduler having patience equal to 9. The second graph is quite improved as per my observation, But still, I want some good comments on this, what else I can do to improve my classification accuracy here.<br>
Plz, suggest.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b08fe250f499a7e6d9e51ab99a473d1d9a81dec.png" alt="image" data-base62-sha1="jPXDo0fWQIng2WkVRsi13U3AB08" width="386" height="262"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38782fd6ce454060b007a93151bff4e164232a8a.png" alt="image" data-base62-sha1="83yeJ3XdAJ8Rmfkl8t3o5xXMOLg" width="386" height="262"></p>
<p>Regards</p>

username: Kapil_Rana
cooked: <p>It is not necessary that you will always get good test accuracy.</p>
<p>Although there are many things you can try</p>
<ol>
<li>Better scaling of the dataset (Normalization)</li>
<li>Drop out or batch normalization (I guess you should try this, like in your cases, overfitting)</li>
<li>By varying scheduler learning rate.</li>
</ol>

username: krishna511
cooked: <p>Thanks for reply <a class="mention" href="/u/kapil_rana">@Kapil_Rana</a> sir I have changed the first graph though , now its correct.<br>
Also I am already using step number 2.<br>
and graph two is the result of step 3, So I that’s shown the results is improved.</p>

username: Kapil_Rana
cooked: <p>Training is okay in the graph, but validation accuracy is not improving, it is the case of overfitting.</p>
<p>You can try a smaller model, a simpler one.<br>
start with a small model and gradually make it bigger. and check on results.</p>
Sickit Learn data split with torch data issue

username: Mohamed_Ragab
cooked: <p>I am trying to split my data using “train_test_split” of sklearn, but I got this error</p>
<h1>“TypeError: take(): argument ‘index’ (position 1) must be Tensor, not numpy.ndarray”</h1>
<p>here is the code:<br>
train_0, X_test, y_train, y_test = train_test_split(zero_cond, data_labels, test_size=0.5)</p>
<p>====<br>
Type of data:<br>
print(zero_cond.type())<br>
torch.FloatTensor<br>
print(data_labels.type())<br>
torch.LongTensor</p>

username: yunusemre
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/631cc4bb764670cd919aad23454c8ac5d1b42cf6.png" data-download-href="https://discuss.pytorch.org/uploads/default/631cc4bb764670cd919aad23454c8ac5d1b42cf6" title="capture_20190102201002"><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/631cc4bb764670cd919aad23454c8ac5d1b42cf6.png" alt="capture_20190102201002" data-base62-sha1="e8MU9MpDfcQAqaUDYnBNgjH3isK" width="690" height="201" data-dominant-color="F2F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">capture_20190102201002</span><span class="informations">805×235 9.58 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If we read documentation, input type of the function is list or numpy array. Hence, you have 2 choice.</p>
<p>1 - If you read the dataset as a numpy array and then convert it to torch.Tensor, first split it, then convert it.<br>
2 - If you read the dataset as a tensor (i don’t know how to read as a tensor <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> anyway), just convert to numpy array first (<code>zero_cond.numpy()</code>), then give to the function. Of course, you sould convert to tensor again.</p>

username: Fang_Wu
cooked: <p>It seems this problem no longer appears in a higher version of Pytorch (&gt;=1.8.0).</p>
torch.nn.BCELoss are unsafe to autocast while working with cosine similarity

username: Vinson2233
cooked: <p>I’m trying to modify CLIP network <a href="https://github.com/openai/CLIP" class="inline-onebox" rel="noopener nofollow ugc">GitHub - openai/CLIP: Contrastive Language-Image Pretraining</a><br>
This network receive pair of images and texts and return matrix of cosine similarity between each text and each image.</p>
<p>The training code is something like this :</p>
<pre><code class="lang-auto"># define BS as batch_size, optimizer 
loss_per_img = nn.BCELoss()
loss_per_txt = nn.BCELoss()

for batch in dataloader :
  images,text = batch # Image size is (BS,) and text also (BS,)
  model = CLIP() #Actually not like this but just assume this
  cosine_per_image,cosine_per_text = model(images,texts) # cosine_per_image dimension is (BS,BS) where each value represent cosine similarity.cosine_per_text just transpose of cosine_per_image
  loss_total= (loss_per_img(cosine_per_image,ground_truth ) + loss_per_txt(cosine_per_text ,ground_truth ))/2
  loss_total.backward()
  optimizer.step()
  optimizer.zero_grad()
</code></pre>
<p>And this code works fine. Now I want to improve the speed by introducing mixed precision training. Here’s my implementation</p>
<pre><code class="lang-auto">use_amp=True
scaler = torch.cuda.amp.GradScaler(enabled=use_amp)
for batch in dataloader : 
  with torch.cuda.amp.autocast(enabled=use_amp):
  ...&lt;same as before&gt;
 loss_total= (loss_per_img(logits_per_image,ground_truth ) + logits_per_text(loss_per_txt ,ground_truth ))/2
scaler.scale(total_loss).backward()
scaler.step(optimizer)
scaler.update()
</code></pre>
<p>and this code will give error about <code>torch.nn.BCELoss are unsafe to autocast.</code> and ask me to use <code>BCEWithLogitsLoss</code> instead. The problem is, the cosine similarity from CLIP is calculated from vectors product, not from sigmoid or softmax layer. That’s why I choose BCELoss directly.</p>
<p>Why not sigmoid? Because the value is cosine similarity, not logits for sigmoid.<br>
Also, I can’t use softmax to the cosine similarity matrix since the task is multi-target classification. I don’t want the probability to overly saturated to the class wit highest probability.</p>
<p>Any idea to use the mixed precision training without involving the sigmoid/softmax ?</p>

username: ptrblck
cooked: <p>You could either disable <code>autocast</code> for the loss calculation or implement a custom loss method, which you could make “autocast-safe”.</p>
How to pass module from Python to C++?

username: Stonepia
cooked: <p>I want to pass the traced module to C++ and get its input tensors’ shape. To do this, I use the following code :</p>
<pre><code class="lang-auto"># test.py
import torch
import custom_cpp_func
import torchvision.models as models

rn18 = models.resnet18()
traced_rn18 = torch.jit.trace(rn18, (torch.randn(5, 3, 224, 224),))
torch._C._jit_pass_inline(traced_rn18.graph)
custom_cpp_func.print_shape(traced_rn18._c)
</code></pre>
<pre><code class="lang-auto">// custom_op.cpp
int64_t print_shape(const torch::jit::Module &amp;mod)
{
  mod.eval(); // Error here
  auto forward_method = mod.get_method("forward");
  assert(forward_method.graph());
  //...
  std::vector&lt;torch::jit::IValue&gt; new_inputs;
  new_inputs.push_back(torch::randn({5, 3, 224, 224}));

  const c10::IValue result = mod.forward(new_inputs).toTensor(); //Error here
}
</code></pre>
<p>The above code could not compile, it has the following error:</p>
<pre><code class="lang-auto">error: 'this' argument to member function 'eval' has type 'const torch::jit::Module', but function is not marked const
  mod.eval();
  ^~~
.../torch/csrc/jit/api/module.h:182:8: note: 'eval' declared here
  void eval() {

 error: 'this' argument to member function 'forward' has type 'const torch::jit::Module', but function is not marked const
  const c10::IValue result = mod.forward(new_inputs).toTensor();

torch/csrc/jit/api/module.h:111:10: note: 'forward' declared here
  IValue forward(std::vector&lt;IValue&gt; inputs) {
</code></pre>
<p>Could someone tell me what’s wrong with this? Why can’t I just pass the module as an argument ? Must I store the module and re-load it into the C++?</p>
<p>Secondly , how could I run the module to make sure the incomplete tensor could get correct shape information?</p>

username: Stonepia
cooked: <p>Well, I found that this is a stupid question. I wrongly passed the module as <code>const</code>, remove it to</p>
<pre><code class="lang-auto">int64_t print_shape(torch::jit::Module &amp;mod)
</code></pre>
<p>would solve the problem.</p>
GPU shared memory for link of processes

username: litvinich
cooked: <p>Hi! I want to ask if you have plans to support GPU tensor shared memory for the link of processes (when a consumer sends tensor to next consumer) without copying of tensor?</p>

username: ptrblck
cooked: <p>Could you explain your use case a bit? Are you planning to share data between different processes or in a single application somehow?</p>

username: litvinich
cooked: <p>Yes. Each process runs specific model on the same batch of images, and sometimes there is a sequence of processes, so one process sends shared tensors to another process, at this place everything crashes and we need to clone this batch</p>
Unet training loss is blowing up

username: ron1
cooked: <p>Trying to train a single channel segmentation, but the loss keeps blowing up. Images are normalized.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff4e335432a25b1e72feeceb2c78688e7ceb9554.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff4e335432a25b1e72feeceb2c78688e7ceb9554" title="Screenshot 2021-04-08 11.40.24"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff4e335432a25b1e72feeceb2c78688e7ceb9554_2_487x500.png" alt="Screenshot 2021-04-08 11.40.24" data-base62-sha1="AqxiwtohGbAjkE9WV63CBFdzLbm" width="487" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff4e335432a25b1e72feeceb2c78688e7ceb9554_2_487x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff4e335432a25b1e72feeceb2c78688e7ceb9554_2_730x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff4e335432a25b1e72feeceb2c78688e7ceb9554_2_974x1000.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-04-08 11.40.24</span><span class="informations">1137×1165 63 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The loss doesn’t seem to blow up, but is decreasing into negative values steadily.<br>
I don’t know which criterion you are using and if a negative loss is expected, but note that you usually want to decrease the loss, so from this point of view your training seems to “work”.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>