akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What ML model is optimal for this situation?

ML_Motivation: <p><strong>What ML model is optimal for the situation my situation (explained below):</strong><br>
The ML model should decide if to drill a hole in the ground in an effort to get an oil reserve. It should be rewarded if it correctly found oil and punished if it destroys a pipe. At any given moment it can either say to either drill or not drill.</p>
<p>Note:<br>
<strong>Rewarded if</strong>: decided to drill and found oil, or decided not to drill and had a pipe below it<br>
<strong>Punished id</strong>: decided to drill and had pipe under it or, decided not to drill had oil below it (missed opportunity)</p>
<p>I tried to create a Deep Q Learning model but this isn’t the relevant model since the current action doesn’t NOT affect the next state. <strong>Please comment on which model would work best for the situation!</strong></p>

vmoens: <p>hello <a class="mention" href="/u/ml_motivation">@ML_Motivation</a><br>
I don’t really get why the action doesn’t affect the next state.<br>
IMO it does, it’s just that it’s a very short trajectory (just one step). In other words your value function is identical to your reward, and you should be able to use some sort of tabular q-learning.<br>
Hope that helps</p>

ML_Motivation: <p>There are millions of possible states for this situation so an Artificial Neural Network (ANN) must be used; I think it would be inefficient to use a Q-table.<br>
We check the location of the agent individually so that its choice does not influence the next state.<br>
What model would be best for the situation? We tried DeepQ Learning but are not sure its the best model</p>
Don't do autodiff when ReLU it's not differentiable

ryanbty: <p>Hi everyone,</p>
<p>Do you know if it’s possible to train a model with the condition to not do the back propagation when it’s not differentiable ? For exemple, I want to throw away the input if at some point I do ReLU’(0).</p>
<p>class MyReLU(torch.autograd.Function):</p>
<pre><code>@staticmethod
def forward(ctx, input):
    if input == 0:
        ctx.mark_non_differentiable(input)
        return input.clamp(min=0)
    else: 
        ctx.save_for_backward(input)
        return input.clamp(min=0)

@staticmethod
def backward(ctx, grad_output):
    input, = ctx.saved_tensors 
    grad_input = grad_output.clone()
    grad_input[input &lt; 0] = 0
    return grad_input  
</code></pre>
<p>Do you think it’s possible to do it ?</p>
<p>Thanks</p>
Reproducibility and Number of DataLoader Workers

AttackPenguin: <p>For reproducibility in the experiments I’m running, I’m seeding at the beginning of my code and calling torch.use_deterministic_algorithms. However, if I use a different number of dataloader workers on two different runs, will I get repeatable results for the same seed, or do I need to ensure that the number of workers is also consistent?</p>
<p>I’m using pytorch lightning, so I’m calling it’s pl.seed_everything method for seeding - if that makes a difference.</p>
TypeError: object.__new__() .... while creating a Custom Dataset

Emrah: <p>Hi!<br>
I’m trying to  custom my own dataset as  in the pytorch tutorial (<a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files" class="inline-onebox" rel="noopener nofollow ugc">Datasets &amp; DataLoaders — PyTorch Tutorials 1.12.1+cu102 documentation</a>)</p>
<pre><code class="lang-auto">class CustomEmoDataset(Dataset):
    def __int__(self, annotations_file, audio_dir):
        self.annotations = pd.read_csv(annotations_file)
        self.audio_dir = audio_dir

    # i defined __len__ and __getitem__ functions.

ANNOTATIONS_FILE = ".....";
AUDIO_DIR = "...";
    # create object.. 
ied = CustomEmoDataset(ANNOTATIONS_FILE, AUDIO_DIR )
</code></pre>
<p>And i’m getting the following error message:</p>
<p><em>Traceback (most recent call last):</em><br>
…<br>
<em>File “/usr/lib/python3.8/typing.py”, line 875, in <strong>new</strong></em><br>
*obj = super().<strong>new</strong>(cls, *args, *<em>kwds)</em><br>
<em>TypeError: object.<strong>new</strong>() takes exactly one argument (the type to instantiate)</em></p>
<p>But when I use <strong>new</strong> instead of <strong>init</strong> as below, it works without error.</p>
<pre><code class="lang-auto">    def __new__(cls, annotations_file, audio_dir):
        obj = super().__new__(cls)
        obj.annotations = pd.read_csv(annotations_file)
        obj.audio_dir = audio_dir
        return obj
</code></pre>
<p>what could be the reason for this error?</p>

InnovArul: <aside class="quote no-group" data-username="Emrah" data-post="1" data-topic="163003">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/emrah/48/53512_2.png" class="avatar"> Emrah:</div>
<blockquote>
<p><code>    def __int__(self, annotations_file, audio_dir):</code></p>
</blockquote>
</aside>
<p>I think you have a typo?<br>
Instead of <code>__int__</code>, try with <code>__init__</code>.</p>

Emrah: <p><em>it was a good typo</em> )) !<img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming/2.png?v=12" title=":man_facepalming:t2:" class="emoji" alt=":man_facepalming:t2:" loading="lazy" width="20" height="20"><br>
Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/raising_hand_man/2.png?v=12" title=":raising_hand_man:t2:" class="emoji" alt=":raising_hand_man:t2:" loading="lazy" width="20" height="20"></p>
Fetching data for the next epoch

DiXcipuli: <p>Hi. I would need some explanation concerning the behavior of the <strong>Dataloader</strong>.<br>
This is my setup: I have a <strong>batch size of 96</strong>, and given my data set, I need <strong>27 iterations for 1 epoch</strong> to be performed. So having many cores to my disposal, I set <strong>num_workers to 27</strong> (I know that it is not particularly optimized to have so many workers, but it’s for the sake of the demonstration).</p>
<p>I can then see through prints, that all 27 processes are fetching images, <strong>until they reach 96</strong>. At that time, the data are then copied to the GPU with .to(), and processes by the model, which takes time. But while the GPU is working at 90%, <strong>all my CPU cores are idle</strong>, and not doing anything, until the epoch is complete. I would have expected them to load the data for the next epoch, while the GPU makes the calculations, but they are not. I tried messing around with the prefetch_factor, persistent_workers or pin_memory, but from what I understand, it is not really related.</p>
<p>Could someone tell me what is the limitation preventing from loading the next batches for the next epoch, while the previous epoch is not over yet? As far as I understand, the loading process from <strong>Dataset.<strong>get_item</strong>()</strong> happens on the CPU RAM, and should not conflict? Thanks!</p>
Batch size and cache

Samuel_Lewis: <p>Hi,</p>
<p>I have remote sensing images that I am training a simsiam model on. My issue is these images are large (11, 10980, 10980). my current workflow is to load an image into the dataset, obtain a crop, augment it and return two augmented versions. Then load the image, or another random image again and obtain a different random crop.</p>
<p>My issue is that the current workflow takes a long time to load the entire image, which is repeated for every batch. ie I need to load 32 images for each epoch, only to grab a small 256x256x11 crop for each in the batch.</p>
<p>My question is, is there a better way to do this, without needing to pre-process all the images into smaller crops? I was thinking there could be a way to cache a subset of images for each epoch. I will post my dataclass and data loader code below: currently I am only testing this code with two images, but for final training there will be 100’s.</p>

Samuel_Lewis: <pre><code class="lang-auto">```{}
class CustomImageDataset(Dataset):
    def __init__(self, folder_path,  valid_exts: List[str] = ['tif', 'tiff']):
       
        # Get image list
        self.files = []
        for file in os.listdir(folder_path):
          ext = file.split('.')[-1]
          if ext in valid_exts:
            file = join(folder_path, file)
            for i in range(16):
              self.files.append(file) 

        self.transforms = create_simsiam_transforms(size=256)
    
    def __getitem__(self, i: int):
        single_img_path = self.files[i]
        #single_img_path = os.path.join(self.image_dir, self.images[index])
        #print(single_img_path)
        image = gdal.Open(single_img_path).ReadAsArray().astype(np.float32) ## gdal read in CHW
        array = np.zeros((11,256,256)) ## intialise empty array
        width = image.shape[2] ## get dimensions of image
        height = image.shape[1]
        ## set the number of patches here
        xstart = randrange(width-256) ## make sure you dont go over edge of image
        ystart = randrange(height-256)
        array[:,:,:] = image[:,xstart:xstart+256, ystart:ystart+256]
        array = np.transpose(array, (1,2,0)).astype(np.float32)


        x1 = self.transforms(image=array)
        x2 = self.transforms(image=array)
        
        return x1, x2
        
    def __len__(self):
        return len(self.files)

def create_simsiam_dataloader(folder_path,
                              batch_size: int = 32, 
                              num_workers: int = 8):
    """
    Returns DataLoader from SimSiamDataset
    Args:
     
    """
    dataset = CustomImageDataset(img_dir)
                             
    dataloader = DataLoader(dataset=dataset, batch_size=batch_size, 
                            shuffle=True, num_workers=num_workers)
    return dataloader</code></pre>
AttributeError: 'Series' object has no attribute 'labelNames'

Berny: <p>Hello. I have some matlab files containing some matrix (x1, x2,…) and a table (called labelNames). With the following code I’m trying to access to labelNames contained in the matlab files:</p>
<pre><code>Folder = 'dataCubes/'
# Listing all .mat files contained in dataCubes folder; hence, put all mat
# files in one folder without subfolders
mat_files = glob(f'{Folder}/*.mat')
numberOfDbExamples = len(mat_files)
Data = {}
for i in range(0, numberOfDbExamples):
    Data[i] = read_mat(mat_files[i])
    Data.keys()
    my_df = pd.DataFrame(Data)
    newLabelNames, newMask = changeLabelNames(my_df[i].labelNames, my_df[i].mask)
</code></pre>
<p>changeLabelNames is a function I created to change some names contained in labelNames. But it seems that when I read the matlab files in python, the table labelNames is not present. Can you help me to find the problem?</p>

ptrblck: <p>This error is raised by <code>pandas</code> as the <code>DataFrame</code> doesn’t seem to contain the wanted attribute. Check the <code>DataFrame</code> content and make sure you are indexing it in the right way. Since this issue is unrelated to PyTorch you might get a faster and better answer in a pandas-specific discussion board.</p>

Berny: <p>I checked the content of the Dataframe and doesn’t seem to contain the table labelNames, do you have an idea of how to extract it?</p>

ptrblck: <p>If the <code>DataFrame</code> doesn’t contain the <code>labelNames</code> attribute you won’t be able to “extract” it and you would have to check why it’s missing and/or why you expect it to be there. Maybe you are loading the wrong <code>DataFrame</code>?</p>

srishti-git1110: <p>Hi,<br>
As <a class="mention" href="/u/ptrblck">@ptrblck</a> has already stated, this is a pandas related question - Pandas documentation will be helpful.</p>
<p>As I can see, you’ve created a dataframe using a dictionary. Then you index it <code>[i]</code> which will yield columns (Series to be more specific). I don’t quite understand how you are trying to access its <code>labelNames</code> attribute that isn’t a pre-defined one for Series objects.</p>

Berny: <p>I think the problem is that the dataframe contains array data and a table (labelNames). I can load the array data correctly, while the table in the dataframe is called None. So the question is: how to read table correctly?</p>
Dataloader usage

Samuel_Lewis: <p>Hi,<br>
I am struggling to wrap my head around using the data loader.</p>
<p>Currently I have a data loader which returns a dictionary for x1, x2 objects.</p>
<p>I can call x1, x2 = next(iter(dataloader), which returns the two dictionary objects.</p>
<p>However, my train script it is set up to run the following.</p>
<p>for x1, x2 in dataloader:<br>
x1 = x1[‘image’].cuda()</p>
<p>With this however I get a none type error when reading in the image from the dataset class. I am quite confused why this happens, as i know the dataloader is working from the above example?</p>

Samuel_Lewis: <p>I have added</p>
<p>for x1, x2 in next(iter(train_dataloader)):<br>
x1 = x1[‘image’].cuda()<br>
x2 = x2[‘image’].cuda()</p>
<p>Which solves, this first issue, however I now get a</p>
<p>ValueError: not enough values to unpack (expected 2, got 1)</p>

ptrblck: <aside class="quote no-group" data-username="Samuel_Lewis" data-post="2" data-topic="162984">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samuel_lewis/48/44762_2.png" class="avatar"> Samuel_Lewis:</div>
<blockquote>
<p>for x1, x2 in next(iter(train_dataloader)):<br>
x1 = x1[‘image’].cuda()<br>
x2 = x2[‘image’].cuda()</p>
</blockquote>
</aside>
<p>That doesn’t look right and you should iterate the <code>DataLoader</code> directly:</p>
<pre data-code-wrap="python"><code class="lang-python">dataset = TensorDataset(torch.randn(10, 1), torch.randn(10, 1))
loader = DataLoader(dataset, batch_size=1)

for x, y in loader:
    print(x, y)
</code></pre>
<p>Could you post the <code>Dataset</code> definition, please?</p>
Channels last with Conv1d

iiSeymour: <p>Is possible to use a channel last memory layout with libtorch? I see it’s possible with PyTorch <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Channels Last Memory Format in PyTorch — PyTorch Tutorials 1.12.1+cu102 documentation</a> but I don’t see a <code>torch::channels_last</code> or anything with <a href="https://pytorch.org/cppdocs/api/structtorch_1_1nn_1_1functional_1_1_conv_func_options.html#_CPPv4I_6size_tEN5torch2nn10functional15ConvFuncOptionsE" rel="noopener nofollow ugc">ConvFuncOption</a>.</p>

ptrblck: <p>You could use <code>at::MemoryFormat::ChannelsLast</code> in C++.</p>

iiSeymour: <p>Do you have an example <a class="mention" href="/u/ptrblck">@ptrblck</a> <code>conv-&gt;to</code> takes either a <code>torch::Dtype</code> or a <code>torch::Device</code>?</p>

ptrblck: <p>I think you could use <code>x.contiguous(at::MemoryFormat::ChannelsLast);</code></p>

iiSeymour: <p>That works for input tensors <code>x</code> but the module parameters are still in <code>at::MemoryFormat::Contigous</code> and I’d really like to avoid the cost of <code>cudnn::ops::nchwToNhwcKernel</code> and <code>cudnn::ops::nhwcToNchwKernel</code>.</p>

ptrblck: <p>If the inputs are already in channels last then checks should also transform the parameters (if available) to the same memory format in intermediate layers. However, you should also be able to explicitly transform these parameters with the same call in a no_grad context.</p>
ML training time increases after first epoch when using two gpus

simines: <p>I am using U-net with input image size 256, and two gpus. I have set batch size to 32, and used dataparallel in pytorch. The training time for the first epoch is 11 min. However, for the second epoch the time increases to 58 min. Does anyone know why this problem happen?<br>
Thank you for your help.</p>

ptrblck: <p>Could you check if your system might be overheating and could reduce the clocks during the training?</p>

simines: <p>Thank you. It may be the reason.</p>
Good tutorials for PyTorch?

VlSkill: <p>What is the best way to learn PyTorch?</p>
<p>I looked at the tutorials on the PyTorch site and didn’t feel they did a good job explaining what everything would be used for/ Also I checked some online courses on this <a href="https://skillcombo.com/topic/pytorch/" rel="noopener nofollow ugc">platform</a>. I understand the general concept of propagation and backpropagation in an attempt to reduce errors.</p>
Backprop an empty tensor does not save time

11178: <pre><code class="lang-auto">import time
from transformers import AutoModelForCausalLM

torch.manual_seed(10)

num_tokens = 500
input = torch.randint(0, 30000, torch.Size([8, num_tokens])).cuda()

model = AutoModelForCausalLM.from_pretrained("gpt2").cuda()

o = model(input_ids=input).logits

##################################
# o[:, :500].detach()
# o = o[:, 500:]
##################################

output = o.sum()

torch.cuda.synchronize()
s = time.time()
output.backward()
torch.cuda.synchronize()
e = time.time()
print(e-s)
</code></pre>
<p>I am surprised that uncommenting the two lines between ### does not result in speeding up the backward pass, even though o ended up being an empty tensor. I wonder why that is the case. Thanks!</p>

ptrblck: <p>Calling <code>backward</code> on an empty tensor should raise an error:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10)
x = torch.randn(1, 10)

out = lin(x)
out = out[:, 10:]
print(out)
# tensor([], size=(1, 0), grad_fn=&lt;SliceBackward0&gt;)
out.backward()
# RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>
<p>so your <code>output</code> seems to be valid.</p>

11178: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="162969">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">lin = nn.Linear(10, 10)
x = torch.randn(1, 10)

out = lin(x)
out = out[:, 10:]
print(out)
</code></pre>
</blockquote>
</aside>
<p>My output is the sum of an empty tensor. bp with it takes the same time as back-propagating through the sum of a non-empty tensor and I wonder why that is the case.</p>

ptrblck: <aside class="quote no-group" data-username="11178" data-post="3" data-topic="162969">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/1/d26b3c/48.png" class="avatar"> 11178:</div>
<blockquote>
<p>My output is the sum of an empty tensor.</p>
</blockquote>
</aside>
<p>Ah, that’s interesting as it is creating a valid tensor:</p>
<pre data-code-wrap="python"><code class="lang-python">print(out.sum())
tensor(0., grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>which will then allow you to backpropagate and I would guess calculate all gradients to zero.</p>

11178: <p>Yeah, but I wonder why this happens and if there is a way to not calculate the gradients to speed up backprop.</p>

ptrblck: <p>Let me add <a class="mention" href="/u/alband">@albanD</a> as he might know if and how this optimization could be done (I’m unsure if it’s possible or would break other valid use cases).</p>
<p>For now you might want to skip the <code>backward</code> call if an empty tensor is detected.</p>
Storing element in tensor?

donparker1: <p>I am trying to store an element in a tensor ; however, I believe the syntax for doing so is incorrect. Can anyone provide the correct syntax for storing an element into a tensor  ?</p>
<pre><code class="lang-auto">std::vector &lt;at::Tensor&gt; ten_vec;
ten_vec.push_back( torch::randn({10}));
at::Tensor idxs;
      for (int x = 0; x &lt; 10; x++){
              idxs[x] =  ten_vec[0][x].item&lt;float&gt;() ;
    
    }
</code></pre>
Some linear layers have a valid grad_sample, others don't

Anirban_Nath: <p>I have two linear layers defined in the model as such: -</p>
<pre><code class="lang-auto">self.feature_reduction = nn.Linear(in_features = 512, out_features = 256)
self.class_embed = nn.Linear(in_features = self.hidden_dim, out_features = 2)
</code></pre>
<p>They both work fine in a non-DP setting but with Opacus, for some reason, param.grad_sample for self.class_embed shows up as None. And consequently, in the optimizer.step() part, I get a “Per sample gradient is not initialized. Not updated in backward pass?” error.</p>
<p>I saw an issue on the Pytorch forum that if a layer is initialized but not used, it’s grad_sample comes up as None and we get the aforementioned error. But I know for sure that the self.class_embed layer is called every time, so how is it possible that it doesn’t have a valid gradient?</p>

tls430: <p>I cannot see the rest of your code, however I assume there is the error as these two lines seem correct.<br>
Are you certain you use the layer in the forward pass?</p>

ashkan_software: <p>Hi <a class="mention" href="/u/anirban_nath">@Anirban_Nath</a></p>
<p>Do you mind sharing a reproducible code here so we can debug this error?  <a href="https://colab.research.google.com/drive/1R-dnKipK9LOVV4_oKbuHoq4VvGKGbDnd?usp=sharing" rel="noopener nofollow ugc">template Colab</a></p>

Anirban_Nath: <p><a class="mention" href="/u/tls430">@tls430</a> <a class="mention" href="/u/ashkan_software">@ashkan_software</a> Hi. I don’t think I will be able to reproduce my code through the Colab template because the error is oddly specific to one LayerNorm and one linear layer among hundreds of other layers that are there in my code. My code is too large to be reproduced but I have made sure that the layers in question are being called using their respective forward function. It is pretty weird that these two layers are the only ones that are causing problems. Is there any other way I can communicate my problem?</p>

ashkan_software: <p>Hi <a class="mention" href="/u/anirban_nath">@Anirban_Nath</a></p>
<p>Do you think you can simply your code and send? Maybe instead of having hundreds of layers, just stick to a few (including the ones you think are causing the problem), to make sure we have a small example. In order for us to help, we should see the code and see where the issue is happening <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Also, a small example may even give you some clue and you may even find the bug!</p>
Is Bidirectional LSTM cheating?

Gabriele_Gennaro: <p>I am confused about how a bidirectional lstm works. I will post a link of the architecture of a bidirectional lstm. You can see that for each time-step the network produces an output based on past and future information wrt the time step. If I have a sequence of N elements and I pass the sequence to the lstm, for each time step I will produce an output that would be something like output(N-c) = [output_from_the_past(0:c),output_from_the_future(N:N-c)] . So each hidden state will have a part coming from the forward direction ( from the past to the future) and a part coming from the backward (from the future to the past ) . Now, supposed I have a sequence of N elements and I want to predict for each time-step the next value, I will have this configuration:</p>
<ul>
<li>SEQUENCE = [0,1,…,N]</li>
<li>INPUT = [0,1,…,N-1]</li>
<li>TARGET= [1,2,…,N]</li>
<li>OUTPUT = [1_tilde,2_tilde,…,N_tilde] My network will produce an output of length equal to the length of the input because for each time step I will get an output(the hidden_state produced with that time step input ), but each of them will contain information from the past and the future ( as demonstrated before ) . So my question is: to predict the next value I am using both information from the past and the future, including information about the value I want to predict. Is the Bidirectional LSTM Cheating? Or maybe there is something I didn’t get ?</li>
</ul>
<p>Here are some related links:</p>
<ul>
<li>Bidirectional lstm architecture : <a href="https://www.google.com/imgres?imgurl=https%3A%2F%2Fmiro.medium.com%2Fmax%2F764%2F1*6QnPUSv_t9BY9Fv8_aLb-Q.png&amp;imgrefurl=https%3A%2F%2Ftowardsdatascience.com%2Funderstanding-bidirectional-rnn-in-pytorch-5bd25a5dd66&amp;tbnid=dTcj5EWeqc27gM&amp;vet=12ahUKEwji0qjZjsz6AhWswwIHHSChAY0QMygDegUIARDIAQ..i&amp;docid=cuVJyCbuXd5BHM&amp;w=764&amp;h=270&amp;q=bidirectional%20lstm%20architecture&amp;hl=en&amp;ved=2ahUKEwji0qjZjsz6AhWswwIHHSChAY0QMygDegUIARDIAQ" rel="noopener nofollow ugc">https://www.google.com/imgres?imgurl=https%3A%2F%2Fmiro.medium.com%2Fmax%2F764%2F1*6QnPUSv_t9BY9Fv8_aLb-Q.png&amp;imgrefurl=https%3A%2F%2Ftowardsdatascience.com%2Funderstanding-bidirectional-rnn-in-pytorch-5bd25a5dd66&amp;tbnid=dTcj5EWeqc27gM&amp;vet=12ahUKEwji0qjZjsz6AhWswwIHHSChAY0QMygDegUIARDIAQ..i&amp;docid=cuVJyCbuXd5BHM&amp;w=764&amp;h=270&amp;q=bidirectional%20lstm%20architecture&amp;hl=en&amp;ved=2ahUKEwji0qjZjsz6AhWswwIHHSChAY0QMygDegUIARDIAQ</a>
</li>
<li>Pytorch Documentation : <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" class="inline-onebox" rel="noopener nofollow ugc">LSTM — PyTorch 1.12 documentation</a>
</li>
</ul>
LSTM with multiple classes

Mariam: <p>I want to make LSTM prediction model where I have multiple classes in one .csv file (with many X axis and Y axis), I don’t know how to implement it to define that  X1 is going for Y1 class and X2 is for Y2 class and etc.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/213dddfaa3088c78bc400a25b1fd8b48c7b335cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/213dddfaa3088c78bc400a25b1fd8b48c7b335cc" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/1/213dddfaa3088c78bc400a25b1fd8b48c7b335cc.png" alt="image" data-base62-sha1="4K4iYZ1NAfls0MTqm6c06yMNpQo" width="690" height="244" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">749×265 3.98 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Any interest in DeepNash?

Ryan_Keathley1: <p>I’m trying to implement DeepNash <a href="https://arxiv.org/abs/2206.15378" rel="noopener nofollow ugc">https://arxiv.org/abs/2206.15378</a>.</p>
<p>Essentially, it is a way to get a policy model to converge to nash equilibrium in Imperfect Information Games without search, using only regularization. It is clear to me that this has myriad applications.</p>
<p>I am fairly certain I understand the paper and its prerequisites enough. I have already written a toy matrix-tree-game environment that is vectorized on the GPU, allowing for very fast simulation. I am also implementing a large grid based game that is visually intuitive and quite large in its number of information sets.</p>
<p>Would anybody like to hop on board?</p>

Balaji_AI: <p>I’m interested.<br>
This is my Github profile - <a href="https://github.com/BalajiAI" class="inline-onebox" rel="noopener nofollow ugc">BalajiAI (Balaji) · GitHub</a></p>
[Autodiff] Custom autograd Maxpooling2d

ryanbty: <p>Hi, I’m trying to custom the Maxpool2d autograd using module torch.autograd.Function.</p>
<p>I want to change the backward output. For my gradient and I want to keep all the maximum indexes and take the mean of this values, not only the first one index. At the end, I have to compute the product with my_grad and the grad_outputs.</p>
<p>This is my code :</p>
<p>class myMaxpool(torch.autograd.Function):<br>
“”"<br>
This autograd function implements a MaxPooling function.<br>
“”"</p>
<pre><code>@staticmethod
def forward(ctx, input, kernel_size, stride):

    out = None
    pool_height, pool_width = kernel_size
    ctx.pool_height, ctx.pool_width = pool_height, pool_width

    ctx.stride = stride

    N, C, H, W = input.size()
    indexes = torch.zeros(N, C, H, W, device = input.device)
    grad_input = torch.zeros(N, C, H, W, device = input.device, dtype = input.dtype)

    out_H = 1 + (H-pool_height)//stride
    out_W = 1 + (W-pool_width)//stride
    out = torch.zeros(N, C, out_H, out_W, device = input.device, dtype = input.dtype)

    for n in range(N):
        for c in range(C):
            for i in range(out_H):
                for j in range(out_W):
                    out[n, c, i, j] = torch.max(input[n, c, i*stride:i*stride+pool_height, j*stride:j*stride+pool_width])
                    tmp = ((input[n, c, i*stride:i*stride+pool_height, j*stride:j*stride+pool_width] == out[n, c, i, j])*1)
                    indexes[n, c, i*stride:i*stride+pool_height, j*stride:j*stride+pool_width] = tmp
                    grad_input[n, c, i*stride:i*stride+pool_height, j*stride:j*stride+pool_width] = torch.div(indexes[n, c, i*stride:i*stride+pool_height, j*stride:j*stride+pool_width] ,indexes[n, c, i*stride:i*stride+pool_height, j*stride:j*stride+pool_width].sum())
    
    indexes =  indexes.view(-1)
    print(grad_input)
    ctx.save_for_backward(input,indexes,grad_input)
    return out

@staticmethod
def backward(ctx, grad_outputs):

    input,indexes,grad_input = ctx.saved_tensors

    grad_x, grad_kernel_size, grad_stride = grad_input, None, None
    N, C, H, W = grad_outputs.size()
    grad_x = torch.zeros(grad_x.size())

    for n in range(N):
        for c in range(C):
            for i in range(H):
                for j in range(W):
                    grad_x[n, c, i*ctx.stride:i*ctx.stride+ctx.pool_height, j*ctx.stride:j*ctx.stride+ctx.pool_width] = grad_input[n, c, i*ctx.stride:i*ctx.stride+ctx.pool_height, j*ctx.stride:j*ctx.stride+ctx.pool_width]*grad_outputs[n, c, i, j]
   

    return grad_x, grad_kernel_size, grad_stride
</code></pre>
<p>It’s possible to do the same without re-compute all the loops in the backward function ?</p>
<p>Thanks.</p>
How to calculate binary segmentation metrics?

vinita: <p>I am working on a binary segmentation problem using Pytorch. I want to know the correct way of calculating metrics like precision, recall, f1-score, mIOU, etc for my test set. From many of the online codes available, I found different ways of calculation, which I have mentioned below:</p>
<h5>
<a name="method-1-1" class="anchor" href="#method-1-1"></a>Method 1</h5>
<ul>
<li>Calculate metrics for individual images separately.</li>
<li>Compute the sum of each metric of all test images.</li>
<li>Divide the metrics by the number of test images.</li>
</ul>
<p>Example:</p>
<pre><code class="lang-auto">for i, (x, y) in enumerate(zip(test_x, test_y)):
           ...
           Score += calc_metrics(mask1, pred1)
         # Score could be any of the metrics 
Final_Score = Score/ len(test_x)

</code></pre>
<h5>
<a name="method-2-2" class="anchor" href="#method-2-2"></a>Method 2</h5>
<ul>
<li>Add TP, FP, TN, and FN pixel count of all the images and prepare a confusion matrix.</li>
<li>Calculate all metrics at the end using the total TP, FP, TN, and FN pixel count and confusion matrix.</li>
</ul>
<p>Example:</p>
<pre><code class="lang-auto">for i, (x, y) in enumerate(zip(test_x, test_y)):
         ...
        FP += np.float(np.sum((pred1==1) &amp; (mask1==0)))
        FN += np.float(np.sum((pred1==0) &amp; (mask1==1)))
        TP += np.float(np.sum((pred1==1) &amp; (mask1==1)))
        TN += np.float(np.sum((pred1==0) &amp; (mask1==0)))
Score = calc_metrics(TP, FP, TN, TP)
</code></pre>
<h5>
<a name="method-3-3" class="anchor" href="#method-3-3"></a>Method 3</h5>
<p>Calculate batch-wise metrics and divide them by the number of test images at the end.</p>
<h5>
<a name="method-4-4" class="anchor" href="#method-4-4"></a>Method 4</h5>
<p>Unlike all the above methods, which use 0.5 as the threshold on the predicted mask (after applying sigmoid), this method uses a range of thresholds and computes metrics on different thresholds for each metric and takes the mean of these values at the end.<br>
Example:</p>
<pre><code class="lang-auto">for i, (x, y) in enumerate(zip(test_x, test_y)):
           ...
	for t in range(len(threshold)):
        thresh = threshold(t)
		thresh_metric(t) = calc_metrics(mask1, pred1, thresh)	
    thresh_final(i,:) = thresh_metric
Score = np.sum(mean(thresh_final))/len(test_x)

</code></pre>
<p>I am confused about which way to use to report my model’s results on the test set.</p>
Get current epoch inside process function of evaluator

AfonsoSalgadoSousa: <p>Hi. I have an evaluation decorated function like:</p>
<pre><code class="lang-auto">  @trainer.on(Events.EPOCH_COMPLETED(every=cfg.train.validation_interval))
  def run_validation():
      epoch = trainer.state.epoch
      state = evaluator.run(val_dataloader)
      log_metrics(
          logger, state.times["COMPLETED"], "Validation", state.metrics, epoch
      )
</code></pre>
<p>Inside the <code>evaluator</code> I do some text generations and would like to save them to new files named after the current epoch. However, I do not know how to get the current epoch inside this process function (engine.state.epoch always returns 1).<br>
How can I get the current epoch inside the evaluation engine? Thanks in advance for any help you can provide.</p>

vfdev-5: <p>Hi <a class="mention" href="/u/afonsosalgadosousa">@AfonsoSalgadoSousa</a></p>
<p>You can add <code>trainer.state.epoch</code> to <code>evaluator.state</code> such that you’ll get easily trainer’s epoch:</p>
<pre><code class="lang-python">  @trainer.on(Events.EPOCH_COMPLETED(every=cfg.train.validation_interval))
  def run_validation():
      epoch = trainer.state.epoch
      evaluator.state.trainer_epoch = epoch
      state = evaluator.run(val_dataloader)
      log_metrics(
          logger, state.times["COMPLETED"], "Validation", state.metrics, epoch
      )
</code></pre>
<p>so in the handler where you woudl like to save files you can fetch trainer’s epoch as</p>
<pre><code class="lang-python">def eval_fn(...):
    ...
    epoch = evaluator.state.trainer_epoch
    ...
</code></pre>
<p>Hope this helps</p>
Tensorboard HParams Metric creates unwanted runs

Ignacio_Hernandez: <p>Hello,</p>
<p>I’ve been using Tensorboard for a while to plot different scalars during training. Now I want to do hyperparameter tuning in an organised way, so I’m trying to use HParams correctly.</p>
<p>I plot scalars like this and it works well:</p>
<pre><code class="lang-auto">writer = SummaryWriter('runs/' + run_name)
...
...
# I do this every epoch
writer.add_scalar('Training/CE_Loss', loss_CE, epoch)
writer.add_scalar('Validation/Accuracy', validation_acc, epoch)
</code></pre>
<p>However, I’m seeing some weird behaviour after adding the hyperparameters. I have a dictionary with all the parameters of my experiment, plus an extra key containing the best validation accuracy. I use that dictionary to create the HParams table and looks fine on TensorBoard:</p>
<p><code>writer.add_hparams(experiment_data, {'Validation/Accuracy': experiment_data['VALIDATION_ACCURACY_BEST']})</code></p>
<p>It also adds a scalar (the green dot) to the existing plot:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c00de90d720c2ae39285231450eb1c27c360e37d.png" alt="hparams1" data-base62-sha1="roZtA7mNARUzs3xMh4OMY2TKM8d" width="285" height="191"></p>
<p>The problem is that it also creates something that looks like a new run, which is very annoying because I get many repeated colours very quickly and makes it difficult to compare between experiments:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd880d0945f3710c04844ac5677d5dff1a8e303c.png" alt="hparams2" data-base62-sha1="r2FJeRovFNPmRMyYZnX7U5DYFSk" width="268" height="84"></p>
<p>I see a LOT of tutorials on TB+Pytorch, but none of them (not even the official one!) talks about HParams.</p>
<p>Am I doing something wrong? How could I avoid this?</p>
<p>Thanks</p>

alpha_omega_alpha: <p>Hi <a class="mention" href="/u/ignacio_hernandez">@Ignacio_Hernandez</a>, I’m also getting the exact same issue.<br>
Have you managed to solve it, yet?</p>

Ignacio_Hernandez: <p>Hi, I haven’t managed to solve it yet… I’m sorry.</p>

el-hult: <p>I just want to report that I have exactly the same issue. I hoped that the 1.7.0 option <code>run_name=""</code> would solve it. It doesnt. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

mrTsjolder: <p>For anyone who is still looking for a workaround: using <code>run_name="."</code> does work!</p>
Mixed precision training and FP16 weights

SM19: <p>Hello,</p>
<p>My objectives are:</p>
<ol>
<li>
<strong>Train a network using mixed precision and then get weights as FP16</strong> - I need a smaller model so that inference using Tensorrt can be optimized.</li>
</ol>
<p>I know we can compile with FP16 weights using Torch-Tensorrt, but, with the recent releases of Torch-Tensorrt I have observed performance at FP16 to be unacceptable.<br>
I also get warnings during compiling as - Subnormal FP16 values encountered.</p>
<p>Hence, my assumption is - if I train using mixed precision and also have FP16 weights to start with, hopefully tensorrt compiling with fp16 should work better.</p>
<p>Therefore, please advice how may I do that.</p>
<ol start="2">
<li>
<strong>Train a pytorch model using fake quanitze with INT8 weights and also have weights as INT8 to use with Torch-Tensorrt</strong><br>
I am aware of this - <a href="https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/tutorials/quant_resnet50.html" class="inline-onebox" rel="noopener nofollow ugc">Quantizing Resnet50 — pytorch-quantization master documentation</a>
</li>
</ol>
<p>Unfortunately, the example ends at export to onnx and does not explain how to use with Tensorrt or Torch-Tensorrt.</p>
<p>Could you please elaborate on that?</p>
<p>Basically, I expect to get INT8 model weights after this fake-quantize training and wish to use these with Torch-Tensorrt for final deployment.</p>
<p>Please provide an example to show how.</p>
<p>Best Regards</p>
Memory mapped Tensor?

mbp28: <p>Hello,</p>
<p>I was wondering whether something like <code>numpy.memmap</code> exists in PyTorch? Or should I just save as a numpy array and use <code>numpy.memmap</code> instead, because conversion to torch tensors is cheap?</p>
<p>Numpy: <a href="https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.memmap.html" rel="nofollow noopener">https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.memmap.html</a></p>

colesbury: <p>You can use <code>torch.Storage.from_file</code> (<a href="http://pytorch.org/docs/master/storage.html?highlight=from_file#torch.FloatStorage.from_file">docs</a>):</p>
<p>e.g.</p>
<pre><code class="lang-python">x = torch.IntTensor(torch.IntStorage.from_file('file.bin', size=100))</code></pre>

Santosh-Gupta: <p>Is there a way to store the data to memory mapped files as well. Does Pytorch have a method for this?</p>

vmoens: <p>Check this feature from TorchRL if you’re interested in it:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/rl/blob/main/torchrl/data/tensordict/memmap.py">
  <header class="source">

      <a href="https://github.com/pytorch/rl/blob/main/torchrl/data/tensordict/memmap.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/rl/blob/main/torchrl/data/tensordict/memmap.py" target="_blank" rel="noopener nofollow ugc">pytorch/rl/blob/main/torchrl/data/tensordict/memmap.py</a></h4>


      <pre><code class="lang-py"># Copyright (c) Meta Platforms, Inc. and affiliates.
#
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

from __future__ import annotations

import functools
import os
import tempfile
from typing import Any, Callable, List, Optional, Tuple, Union, Dict

import numpy as np
import torch

from torchrl._utils import prod
from torchrl.data.tensordict.utils import _getitem_batch_size
from torchrl.data.utils import (
    DEVICE_TYPING,
    INDEX_TYPING,
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/rl/blob/main/torchrl/data/tensordict/memmap.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>