akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to get data member and member function of torch._C.Value and torch._C.Graph?

icyhearts: <p>By torchscript tracing(I see this in torch.onnx.utils.py, in _trace_and_get_graph_from_model() function), I can get an object instance of torch._C.Graph, but how can know the data members  and available member functions of class torch._C.Graph?<br>
If the instance name of torch._C.Graph  is graph, by calling list(graph.inputs()) (just as shown in torch/onnx/utils.py), we can get a list of torch._C.Value, so, how can I know  the data members  and available member functions of torch._C.Value?</p>
<p>I think there should be some cpp codes for torch._C.Graph and torch._C.Value,  but I am not sure which class  is corresponding to torch._C.Value.</p>
<p>For example, there is struct Graph defined in caffe2/core/graph.h, third_party/onnx/onnx/common/ir.h, torch/csrc/jit/ir/ir.h, I don’t know which file defineds torch._C.Graph.</p>

ptrblck: <p>You can get attributes via <code>print(dir(torch._C.Graph))</code>, but note that <code>torch._C</code> is an internal function and thus there is no guarantee it won’t break or attributes will be added or removed without any warning.</p>
Subtraction along axis

AndreaRossi: <p>Hi everyone, I am a bit of a newbie in PyTorch and I have a very basic issue that I am having trouble with.</p>
<p>I have two matrices A and B, with different number of rows, but same number of columns.<br>
Basically, A and B are different collections of same-sized vectors.<br>
What I am trying to do is to subtract each vector in B from each vector in A.</p>
<p>This minimal example does exactly what I’m trying to accomplish:</p>
<pre><code class="lang-python">a = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
b = torch.tensor([[1, 1, 1], [2, 2, 2]])

c = torch.stack([a-x for x in b])
</code></pre>
<p>but of course I would like to get rid of that ugly “for” cycle, which I guess is not good for performances.<br>
I think I need some sort of <code>torch.diff(a, b, axis=1)</code>.<br>
Does something like that exist?</p>

ptrblck: <p>Broadcasting should work, but would trade memory for a potential speedup:</p>
<pre><code class="lang-python">c = a.unsqueeze(0) - b.unsqueeze(1)
</code></pre>
[RESOLVED] DataSet class takes 1 positional argument but 3 were given

CogentCoder73: <p>[EDIT] It turns out that I accidentally made TrainDataSet a function, instead of a class. This simple fix resolved the issue. My apologies</p>
<p>I have recently learned how to create a custom DataSet class. The class and its <strong>init</strong> method are shown below:</p>
<pre><code class="lang-auto">def TrainDataSet(Dataset):
    def __init__(self, data, labels=None, transforms=None):
        self.data = data.values()
        self.labels = labels.values()
        self.transforms = transforms
</code></pre>
<p>However, when I try to define a TrainDataSet as below:</p>
<pre><code class="lang-auto">dataset = TrainDataSet(data, labels, transform)
</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">TypeError: TrainDataSet() takes 1 positional argument but 3 were given
</code></pre>
<p>Could someone please help me with this?</p>
<p>The full stack trace is below</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/Users/~/PythonProjects/MLProject/preprocess.py", line 39, in &lt;module&gt;
    dataset = TrainDataSet(data, labels, transform)
TypeError: TrainDataSet() takes 1 positional argument but 3 were given

</code></pre>

hhaoao: <p>This problem has nothing to do with pytorch, this is the basic concept of python, I think you should lay the foundation.<br>
<a href="https://docs.python.org/3/glossary.html#term-argument" class="onebox" target="_blank" rel="nofollow noopener">https://docs.python.org/3/glossary.html#term-argument</a></p>

CogentCoder73: <p>Thanks for the advice. My foundations are clear, I had just made a simple error - I accidentally defined the DataSet as a function, instead of a class. Thank you!</p>
Is there any easy way to make a predefined classification modal multioutput

flyingpig: <p>I am trying to use some open source neural network architecture written in pytorch. The original model cannot handle multi output, the situation where one example is associated with multiple labels. I wonder whether there is any easy way one can adapt it to be able to handle multi output. I can certainly define a class that inherits the model and then change the fit and predict method, but it seems quite complicated.</p>

tom: <p>If you don’t have the (log_)softmax at the end (more often one has not these days), it should be trivial to use sigmoid + thresholding on the scores (the raw model outputs).</p>
Train Test Split with separate transforms?

Arohan_Ajit: <p>I’m a beginner in PyTorch but I’ve made a data pipeline a couple of time. The way I know to split the data is, by taking indices and separating them into train and test.:</p>
<pre><code class="lang-auto">data_transforms = transforms.Compose([
    transforms.Resize((50,50)),
    transforms.RandomRotation(degrees=30),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                        std=[0.229, 0.224, 0.225])
    ])

data = ImageFolder('breast-histopathology/',transform=data_transforms)
valid_size = 0.15
test_size = 0.15

num_train = len(data)
indices = list(range(num_train))
np.random.shuffle(indices)
valid_split = int(np.floor((valid_size) * num_train))
test_split = int(np.floor((valid_size+test_size) * num_train))
valid_idx, test_idx, train_idx = indices[:valid_split], indices[valid_split:test_split], indices[test_split:]

print(len(valid_idx), len(test_idx), len(train_idx))

# define samplers for obtaining training and validation batches
train_sampler = SubsetRandomSampler(train_idx)
valid_sampler = SubsetRandomSampler(valid_idx)
test_sampler = SubsetRandomSampler(test_idx)


loaders = {
    'train': torch.utils.data.DataLoader(data, batch_size=128, sampler=train_sampler),
    'test': torch.utils.data.DataLoader(data, batch_size=32, sampler=test_sampler),
    'valid': torch.utils.data.DataLoader(data, batch_size=32, sampler=valid_sampler),
}
</code></pre>
<p>However I don’t know how to do it incase I want separate transforms. This method provides one data transform for the whole dataset. Is there a way to divide dataset and specify separate transforms for each subset(eg. augmented data for train and original for validation).<br>
P.S it can be done by making separate train and test folders using <code>shutil</code> or <code>os</code> but I was thinking if there’s a method in pytorch for doing so.</p>

tom: <p>For best practice:</p>
<ul>
<li>I would advise to separate train/val/test earlier, possibly even at the file system level.</li>
<li>I know nothing about your dataset, but you have checked that just splitting at a image level is the right thing to do (in medical imaging, you typically want to have all images of a given patient on the same side of the split and the like)?</li>
<li>After the above two, it is natural to have separate datasets for train/val/test. At that point, having different transforms is easy. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">
</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>
Converting a Tensor to List in C++

bhasin: <p>Hi,</p>
<p>How to convert a torch::Tensor to a list in C++? I am looking for a C++ equivalent of the following python code:</p>
<pre><code class="lang-auto">x
tensor([[0., 6.],
        [1., 2.]])
x.tolist()
[[0.0, 6.0], [1.0, 2.0]]
</code></pre>
<p>Thanks!</p>

kaihsinwu: <p>The list in C++ is different than the list in python.</p>
<p>I would assume you probably want to convert tensor to some C++ standard container (like vector etc.)?</p>
<p>I don’t think it is possible now using libtorch’s API,</p>
<p>but using Tensor.data_ptr&lt;float&gt;() will return the pointer float*. Then you can start from here to convert the internal data to the C++ container, for example, naive traversal like <a href="https://discuss.pytorch.org/t/convert-tensor-into-an-array/56721" class="inline-onebox">Convert tensor into an array</a> or else.</p>
Text generation - input best practice

vince120: <p>I am currently working on a project: I want to generate text with a LSTM using Pytorch. My model is working but I have a question about the methodology:</p>
<p>I’m using the BPTTIterator and something seems weird to me: you have to give one example with all your text in it then it will provide your neural network with the current word and the target word for each step. In that way, I’m not using SOS and EOS tokens.</p>
<p>I’m wondering the impact on my model. Will it changes something if I don’t use those tokens? I know It won’t be able to stop the sentence automatically but will it change the performance of my model? What is the best practice?</p>
<p>Thanks</p>
OpenCV to Libtorch incorrect conversion

Narendra_Patwardhan: <p>I have two functions, one to load image data to tensor and one to convert back the tensor into the image form.</p>
<pre><code class="lang-auto">at::Tensor image2tensor(std::string image_path){
    cv::Mat image = cv::imread(image_path); 
    image.convertTo(image,CV_32FC3);
    std::vector&lt;int64_t&gt; sizes = {1, 3, image.rows, image.cols};
    c10::TensorOptions options = torch::TensorOptions().dtype(torch::kFloat32);
    at::Tensor tensor_image = torch::from_blob(image.data, at::IntList(sizes), options);
    tensor_image = tensor_image/tensor_image.max();
    return tensor_image;
}

cv::Mat tensor2image(at::Tensor tensor){
    tensor = tensor.cpu().squeeze().detach().permute({1, 2, 0});
    tensor = tensor.mul(255).toType(torch::kU8).clamp(0, 255);
    cv::Mat resultImg(tensor.size(0), tensor.size(1),CV_8UC3);
    std::memcpy((void *) resultImg.data, tensor.data_ptr(), sizeof(torch::kU8) * tensor.numel());
    return resultImg;
}
</code></pre>
<p>However the reconstructed image looks garbage. How does from_blob maps a mat to tensor? as I feel tensor2mat is exact copy of python code and works well.</p>
Mass prediction

anna1234: <p>Hi, I am currently working on my first classification task. However, I can’t seem to figure out how to predict on new, unlabeled images in a folder - only on single images. I have been stuck for days with this. Ideally, I want to use my trained model to classify all images in a folder, and then write the predictions to a csv file (as image-name, label). Any hints, ideas?</p>
<p>Thanks!!</p>

narain1: <p>you can either get a list of files as a list using os.listdir() and create a dataset</p>
A bunch of dead weight in model

coll-j: <p>I plotted out the parameters in the model and I got this figure.<br>
Is it normal to have a lot of parameters on/near 0 like this? If not, are there any ways to prevent these dead weights?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b2d8ee36c9cf3644320be4b48f51eaf6c91f4e5a.png" data-download-href="https://discuss.pytorch.org/uploads/default/b2d8ee36c9cf3644320be4b48f51eaf6c91f4e5a" title="dead weight 2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b2d8ee36c9cf3644320be4b48f51eaf6c91f4e5a.png" alt="dead weight 2" data-base62-sha1="pw9LcdwQO41aPUGEyZuH14hTwKu" width="501" height="500" data-dominant-color="E5E5E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dead weight 2</span><span class="informations">628×626 30.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Element 0 of tensor does not require grad_fn

a_d: <p>Hello all,<br>
I am experimenting with a custom IoU and trying to propagate the sum of IoU backward and the implementation is as follows -</p>
<pre><code class="lang-auto">
def sumIoU(anchors, gt):
    num_anchors = anchors.shape[1]
    num_gts = gt.shape[1]
    net_IoU = 0
    for i in range(num_anchors):
        anchor = anchors[:, i, :]
        for j in range(num_gts):
            j_gt = gt[:, j, :]
            xi = torch.max(anchor[:, 0],j_gt[:, 0])
            yi = torch.max(anchor[:, 1], j_gt[:, 1])
            wi = torch.clamp(torch.min(anchor[:,2], j_gt[:, 2])-xi, min=0)
            hi = torch.clamp(torch.min(anchor[:,3], j_gt[:, 3])-yi, min=0)
            area_i = wi*hi
            area_u = ((anchor[:,2]-anchor[:,0])*(anchor[:,3]-anchor[:,1])) + \
                     ((j_gt[:,2]-j_gt[:,0])*(j_gt[:,3]-j_gt[:,1])) - area_i

            IoU = torch.clamp(area_i, 1e-5)/torch.clamp(area_u, 1e-7)
            net_IoU+=IoU
    return net_IoU
</code></pre>
<p>and Propagating it as -</p>
<pre><code class="lang-auto">IoU = sumIoU(anchors=anchors, gt=gt)
loss.backward()
</code></pre>
<p>I run into the following error -</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>The Returned value i.e <code>IoU</code> is <code>tensor([[466.2932]], device='cuda:0')</code><br>
I had once read that till we use torch functions we would be fine, but I dont know what is going wrong since I am not understanding the error properly. Could someone please tell what I should do to get rid of it<br>
Sorry for the noob question…<br>
TIA</p>
How to choose a part of data for train and test?

Sumanta_Ghosh: <p>Hi everyone<br>
I am new in pytorch. I have a large datafile size. Suppose I want to take 20% data (randomly choosen from the file), out of that 20% I want 75% for training and 25% for testing.<br>
How can I take the 20% data from the original data file ?</p>

Henry_Chibueze: <p>U can use train_test_split from sk-learn or u can code it ur self<br>
If u r coding it ur self all u need to do is get the length of ur training data and multiply it by 0.2, after that use np.random.shuffle( ) to shuffle the data and the perform a slice of the shuffled data like this: test_data = data[-splice:]<br>
where splice = int(len(data)*0.2)</p>
How to stack 2 tensors

Parlak_Sair: <p>Hi everyone;<br>
I’m trying to stuck 2 tensors A.shape=(64,16,16) and B.shape=(64,16,16) in a tensor of shape C.shape=(1,128,16,16) and non of functions i’ve tried work where<br>
torch.stack =&gt; C.shape=(2,64,16,16) and torch.cat =&gt; C.shape=(128,16,16)<br>
can enyones help me</p>

LeviViana: <p>torch.cat and then you unsqueeze(0) the result<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29b3a800fd2c09fdc059a167f627267bed1b79c5.png" alt="image" data-base62-sha1="5WUug6DSyfkNGIxPQO7HmZhC0zb" width="294" height="93"></p>
How to calculate entropy of each class to measure model uncertainty

Kudo_AI: <p>I am trying to calculate Entropy to measure model’s uncertainty using MC Dropout for image classification task ,I have calculated the Entropy for each sample using the samples mean output_mean as shown in the code below</p>
<pre><code class="lang-auto">for images,labels in testloader:
        images = images.to(device)
        labels = labels.to(device)
            
        with torch.no_grad():
          output_list = []                               
          for i in range(T):
            output_list.append(torch.unsqueeze(model(images), 0)) 
            
        
        
        #calculating samples mean
        output_mean = torch.cat(output_list, 0).mean(0) #shape (n_samples, n_classes)     
        output_mean = np.asarray(output_mean.cpu())
        
        epsilon = sys.float_info.min
        # Calculating entropy across multiple MCD forward passes 
        entropy = -np.sum(output_mean*np.log(output_mean + epsilon), axis=-1) #shape (n_samples, n_classes)
</code></pre>
<p>After calculating Entropy of each sample, I am trying to calculate Entropy for each class to get the model uncertainty about each one of them. Can anyone help me to get the right formula to calculate entropy</p>
[Torchscript] Serialization - Loss of Information

Knight3: <p>Hello everyone.<br>
I am working with Torchscript on PyTorch 1.5.0.<br>
I want to automatically grab input shape &amp; dtype information from a deserialized, traced torchscript model with the following code line:</p>
<pre><code class="lang-auto">[(i.debugName().split('.')[0], i.type().sizes(), i.type().scalarType()) for i in 
list(reloaded_module.graph.inputs())[1:]]
</code></pre>
<p>The Problem is that this gives me None for both the size and the scalarType when i deserialize the model.<br>
On the other hand, when i execute the same line of code on a non-deserialized model (e.g. a just-in-time traced model that has not been serialized&amp;deserialized) i get the desired results.</p>
<p>You should be able to reproduce this through just a few lines of code.</p>
<pre><code class="lang-auto">model = torch.hub.load('pytorch/vision:v0.6.0', 'resnet18', pretrained=True)
input_image = "get an image from somewhere"

traced_path = "./test.pt"
traced_module = torch.jit.trace(model,input_batch)
traced_module.save(traced_path)

reloaded_module = torch.jit.load(traced_path)

print([(i.debugName().split('.')[0], i.type().sizes(), i.type().scalarType()) for i in  list(traced_module.graph.inputs())[1:]])
print([(i.debugName().split('.')[0], i.type().sizes(), i.type().scalarType()) for i in  list(reloaded_module.graph.inputs())[1:]])

</code></pre>
<p>Any suggestions on solutions? Am i missing something or is this actually a bug in Torch-Serialization/Deserialization?</p>
<p>Best regards,<br>
Knight3</p>
Can nn.quantized.FloatFunctional().cat() be used multiple times in one module?

chuanqi305: <p>When I was reading the source code of torchvision.models.quantization.inception_v3, I found self.myop.cat is used 3 times in QuantizableInceptionE, so when I finished the training, there is only one group of quantization params (min_val/max_val/scale/zeros_point). If I understand correctly, we need 3 different group of quantization params for each concat operation.<br>
Can any one help to explain whether it’s a bug here or I misunderstood it?</p>

hx89: <p>You are right, looks like we need 3 different self.myop.cat. Could you file an issue for it? We will take a look.</p>

chuanqi305: <p>Thanks! I saw it has been solved in the newest version by this <a href="https://github.com/pytorch/vision/commit/ae4012e27287466c6250235daa5ac3b74c26fbf2#diff-8309d86ffb3a63402883ab29f4c6b0f2" rel="nofollow noopener">commit</a>.</p>
The GPU memory is not enough when computing feature centers

ddguoll: <p>I want use the class feature centers in the loss function. But when I compute the centers,<br>
the GPU memory is not enough. How can I solve it? The code is as follows. Thank you very much!</p>
<pre><code class="lang-auto">for epoch in range((args.start_epoch+1), args.epochs):
  Center= computer_Center(model,dataloader, classnum)
	for input, target in train_loader:
        target = target.cuda()
        input = input.cuda()
        input_var = torch.autograd.Variable(input)
        target_var = torch.autograd.Variable(target)
        outputs, feature = model(input_var)
        l = criterion(feature,target_var, Center) .forward()
        l.backward(retain_graph=True)

def computer_Center(model,dataloader, classnum):
    model.train()
    for i in range(classnum):
            j=0
            for input,target in dataloader:
                target=target.cuda()
                input = input.cuda()
                input_var = torch.autograd.Variable(input)
                target=torch.autograd.Variable(target)
                _, feature_ext = model(input_var)
                ind=torch.where(target==i)[0]
                if ind.shape[0]&gt;0:
                    if j==0:
                        feature_mid = feature_ext[ind, :]
                        feature_sum_mid=feature_mid.sum(0)
                    else:
                        feature_mid = feature_ext[ind, :]
                        feature_sum_mid = feature_sum_mid+feature_mid.sum(0)
                    j=j+1

            feature_sum_mid=feature_sum_mid.unsqueeze(0)
            if i==0:
                feature_sum=feature_sum_mid
            else:
                feature_sum=torch.cat([feature_sum,feature_sum_mid],dim=0)

    Center=feature_sum
    for i in range(7):
            Center[i,:]=feature_sum[i,:]/ClaSamNum[i]

return Center
</code></pre>
WSL cuda not detected

acmilannesta: <p>wsl 4.19.121<br>
ubuntu 18.04<br>
cuda-toolkit 11.0<br>
nvidia driver: 460.15<br>
Pytorch: 1.6.0<br>
Python: 3.6.8</p>
<p>I’ve followed the steps in this tutorial<br>
<a href="https://docs.nvidia.com/cuda/wsl-user-guide/index.html" class="onebox" target="_blank" rel="nofollow noopener">https://docs.nvidia.com/cuda/wsl-user-guide/index.html</a></p>
<p>But when I run pytorch  <code>torch.cuda.is_available()</code> , it returns False.</p>
Correlations and NN's

Jona: <p>Hi,</p>
<p>After doing and EDA … ¿it’s a good practice to get rid of the correlated features before feeding in a fully connected NN? … or ¿should I let the NN try to figure out the relationships between the features, being present those correlated? …</p>
<p>EDA stands for Exploratory Data Analysis<br>
</p><aside class="onebox wikipedia">
  <header class="source">
      <a href="https://en.wikipedia.org/wiki/Exploratory_data_analysis" target="_blank" rel="nofollow noopener">en.wikipedia.org</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:350/263;"><img src="//upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Data_visualization_process_v1.png/350px-Data_visualization_process_v1.png" class="thumbnail" width="350" height="263"></div>

<h3><a href="https://en.wikipedia.org/wiki/Exploratory_data_analysis" target="_blank" rel="nofollow noopener">Exploratory data analysis</a></h3>

<p>In statistics, exploratory data analysis is an approach to analyzing data sets to summarize their main characteristics, often with visual methods. A statistical model can be used or not, but primarily EDA is for seeing what the data can tell us beyond the formal modeling or hypothesis testing task. Exploratory data analysis was promoted by John Tukey to encourage statisticians to explore the data, and possibly formulate hypotheses that could lead to new data collection and experiments. EDA is dif...</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Best regards</p>

googlebot: <p>Input preprocessing can be useful, but in vision they usually do different things than decorrelation.</p>
<p>In general, I’d say that transformations can help with some hard to train models (you’d basically scale first layer’s gradients to help training), otherwise NN will “figure it out”, especially with adaptive learning rate optimizers.</p>

Jona: <p>My data set is not about vision o images … it’s a tabular dataset with numerical and categorical features … So its nature is different</p>
<p>Best regards</p>
Equivalent of tf.embedding_lookup()

jubick: <p>Let’s say i have this code in tf. I want to rewrite it in pytorch</p>
<pre><code class="lang-auto">self.W = tf.v1.get_variable("W",dtype=tf.v1.float32,
                                            initializer=tf.v1.constant(self.params['wordvectors'].astype('float32')))
self.input_support_set_sents = tf.v1.placeholder(tf.v1.int32,
                                                         [None, None, sen_len],
                                                         'support_set_sents')
self.support_set_sents = tf.v1.nn.embedding_lookup(self.W, self.input_support_set_sents)
</code></pre>
<p>So far i have this:</p>
<pre><code class="lang-auto">  self.W = torch.tensor(config['wordvectors'], dtype=torch.float32)
   # input_support_set_sents passed by arg
  self.support_set_sents = None
</code></pre>
<p>embedding_lookup() in tf basically takes all words from second parameter and returns their emedding valeus from first argument.</p>

ptrblck: <p>Based on the description I would assume you can simply index your embedding matrix with the word indices.</p>

jubick: <p>Can you provide code for that?</p>

ptrblck: <p>This would create an embedding and use <code>x</code> to get the corresponding embedding vector at index 0:</p>
<pre><code class="lang-python">emb = nn.Embedding(10, 100)
x = torch.tensor([0])
out = emb(x)
</code></pre>

jubick: <p>But i already have my embeddings stored in W.<br>
Also my problem is about placeholders in tf. I understand difference in this frameworks but i don’t really know how to rewrite it in pytorch</p>

ptrblck: <p>In that case you can directly index the weight matrix:</p>
<pre><code class="lang-python">emb = nn.Embedding(10, 100)
x = torch.tensor([0])
out = emb(x)
out_manual = emb.weight[x, :]
print((out_manual == out).all())
&gt; tensor(True)
</code></pre>
<p>I’m not 100% sure, what placeholders in TF are, but aren’t they used to feed data into the graph?</p>

sqlkrad: <p><a href="https://pytorch.org/docs/stable/torch.html#torch.index_select" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.index_select</a></p>

russellkim: <p><a href="https://pytorch.org/docs/stable/generated/torch.index_select.html#torch.index_select" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/generated/torch.index_select.html#torch.index_select</a></p>
Exactly same networks in PyTorch and Tensorflow

sajjad: <p>Hi all<br>
I am working on adversarial attacks. Some of the attacks I am interested in are written using PyTorch and some are based on TensorFlow. Thus I need to have the trained network in both PyTorch and TensorFlow and they must be exactly similar to have a fair comparison.<br>
How can I have a PyTorch network in TensorFlow while I can do differentiating to generate adversary.<br>
Thanks for your help in advance</p>

ptrblck: <p>You would have to reimplement the model in the desired framework, as I don’t think there is a way to use TF models in PyTorch and vice versa.</p>

sajjad: <p>Thanks for your response.</p>
Problem in loading the model

Safa_Mathlouthi: <p>I want to load the model after saving it but i got this error</p>
<pre><code class="lang-auto">NotADirectoryError: [Errno 20] Not a directory: './module/net9-1.957-0.991.pth/'
</code></pre>
<p><strong>code</strong></p>
<pre><code class="lang-auto">import parameter as para
#  Define network and load parameters  test.py
net = torch.nn.DataParallel(ResUNet(training=False)).cuda()
net.load_state_dict(torch.load(para.module_path))
net.eval()

#para parameter.py
module_path = './module/net9-1.957-0.991.pth/'  #   Test model path

</code></pre>

tiwari-ai-harsh: <aside class="quote no-group" data-username="Safa_Mathlouthi" data-post="1" data-topic="95079">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/ac91a4/48.png" class="avatar"> Safa_Mathlouthi:</div>
<blockquote>
<p><code>module_path = './module/net9-1.957-0.991.pth/'  </code></p>
</blockquote>
</aside>
<p>Remove the “/” and try with<br>
<code>module_path = './module/net9-1.957-0.991.pth' </code></p>

Safa_Mathlouthi: <p>i tried it but tstill same error</p>
Pointer network implementation for transformers

amarazad: <p>Is there any implementation of pointer network for transformers. I found an implementation in TF at <a href="https://github.com/xiongma/transformer-pointer-generator" rel="nofollow noopener">https://github.com/xiongma/transformer-pointer-generator</a><br>
I tried converting but not able to find a counter part of the following function in pytrorch.<br>
The function is<br>
loss = tf.map_fn(fn=lambda x: tf.gather_nd(x[1], x[0]), elems=(indices, final_dists), dtype=tf.float32)</p>
<p>It is at<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/xiongma/transformer-pointer-generator/blob/master/model.py#L152" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/xiongma/transformer-pointer-generator/blob/master/model.py#L152" target="_blank" rel="nofollow noopener">xiongma/transformer-pointer-generator/blob/master/model.py#L152</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="142" style="counter-reset: li-counter 141 ;">
<li>        dec = tf.range(0, limit=dec_t) # [dec]</li>
<li>        dec = tf.expand_dims(dec, axis=-1) # [dec, 1]</li>
<li>        dec = tf.tile(dec, [1, attn_len]) # [dec, atten_len]</li>
<li>        dec = tf.expand_dims(dec, axis=0) # [1, dec, atten_len]</li>
<li>        dec = tf.tile(dec, [batch_size, 1, 1]) # [batch_size, dec, atten_len]</li>
<li>
</li><li>        x = tf.expand_dims(x, axis=1) # [batch_size, 1, atten_len]</li>
<li>        x = tf.tile(x, [1, dec_t, 1]) # [batch_size, dec, atten_len]</li>
<li>        x = tf.stack([dec, x], axis=3)</li>
<li>
</li><li class="selected">        attn_dists_projected = tf.map_fn(fn=lambda y: tf.scatter_nd(y[0], y[1], [dec_t, self.hp.vocab_size]),</li>
<li>                                         elems=(x, attn_dists), dtype=tf.float32)</li>
<li>
</li><li>        final_dists = attn_dists_projected + vocab_dists</li>
<li>
</li><li>    return final_dists</li>
<li>
</li><li>def _calc_loss(self, targets, final_dists):</li>
<li>    """</li>
<li>    calculate loss</li>
<li>    :param targets: reference</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Faster RCNN evaluation

mr_joe: <p>Hi eveyone,</p>
<p>I’m working with the Faster RCNN version provided by pytorch (<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">Here</a>). I’m training the model with my own custom dataset but I have some difficulties on understanding the evaluation metrics. I couldn’t find any good explanation on internet.</p>
<p>Here is an example showing my results:</p>
<pre><code class="lang-auto">IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.658
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 1.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.827
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.663
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.658
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.687
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.687
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.687
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.700
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.685
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
</code></pre>
<p>In this example what does the -1 means for large areas? and why it happens only for that?<br>
I have just started object detection for a few day so it’s all new for me. Please, can you help me? thank you for your support</p>
Measuring uncertainty using Entropy

Kudo_AI: <p>I have implemented the MC Dropout method on pytorch ,the main idea of this method is to set the dropout layers of the model to train mode. This allows for different dropout masks to be used during the different various forward passes. Below is the implementation of MC Dropout  illustrating how multiple predictions from the various forward passes are stacked together</p>
<pre><code class="lang-auto">def mcdropout_test(batch_size,model):

    #set non-dropout layers to eval mode
    model.eval()

    #set dropout layers to train mode
    enable_dropout(model)
    
    test_loss = 0
    correct = 0
    n_samples = 0
    T = 100
    
       
    for images,labels in testloader:
        images = images.to(device)
        labels = labels.to(device)
            
        with torch.no_grad():
          output_list = []
          #getting outputs for T forward passes
          for i in range(T):
            output_list.append(torch.unsqueeze(model(images), 0))
        
        #calculating mean
        output_mean = torch.cat(output_list, 0).mean(0)
            
        
        test_loss += F.nll_loss(F.log_softmax(output_mean,dim=1), labels, reduction='sum').data  # sum up batch loss       
        _, predicted  = torch.max(output_mean, 1) 
        correct += (predicted == labels).sum().item() #sum up correct predictions
        n_samples += labels.size(0)    

    acc = 100.0 * correct / n_samples
    print(f'the mcdropout score test is : {acc} %')
      
</code></pre>
<p>To mesure uncertainty for classification tasks we need to calculate entropy<br>
and I am trying to implement this on the code above (should i calculate Entropy after the T forward passes? and how exactly i can do that ?and how exactly should i to choose the samples or classes (high/low entropy) ?)<br>
the code for the test_loader</p>
<pre><code class="lang-auto">  #downloading the test set
  testset = torchvision.datasets.CIFAR10(root='./data', train=False,download=True, transform=test_transform)

  #loading the test set
  testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=4)
</code></pre>