akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA running out of memory when using float("-inf")

username: douglasrizzo
cooked: <p>Hi everyone. I’m implementing an RL algorithm and I’m masking unavailable actions by setting their Q-values really low, so when I apply <code>max()</code> I never select them. Usually, I do that with <code>float("-inf")</code> in pure Python. The thing is, in all my experiments, I’m getting a <code>RuntimeError: CUDA out of memory</code> in the lines where I use this trick, like the example below:</p>
<pre><code class="lang-auto">live_next_q_vals = current_q_vals.clone().detach()[:, 1:]
live_next_q_vals[~av_actions_batch[:, 1:]] = float('-inf')
</code></pre>
<p>Does anyone know if using <code>float("inf")</code> or <code>float("-inf")</code> is not recommended with PyTorch tensors? I saw other implementations use large negative numbers, like <code>-999999</code>, but I didn’t see a reason to do that and it looks like I made the wrong choice, lol.</p>

username: douglasrizzo
cooked: <p>I believe it’s worth clarifying that this actually happens after several thousand iterations, not in the first one, but always in lines in which I perform masking with <code>float("-inf")</code>.</p>

username: ptrblck
cooked: <p>I don’t think the error is raised due to the specific value and assume you would get the same OOM error, if you are using any other value.<br>
The OOM might be raised by the indexing, since temp. tensors might need to be created.<br>
Are you seeing an increased memory usage during training or are you close to an OOM from the beginning?</p>

username: douglasrizzo
cooked: <p>You’re right <a class="mention" href="/u/ptrblck">@ptrblck</a>, I got an OOM error even when using a large negative value instead of <code>gloat("-inf")</code>. The tensor I am manipulating in that code snippet has dimensions 32x60x10x16. I don’t know if that’s big in PyTorch terms.<br>
To answer your question, my training script does not consume all of my GPU memory from the beginning, it slowly increases the memory usage with time. It usually starts with 1 GB and goes up until it reaches 3.5 GB. My GPU has 8 GB of memory.<br>
The thing is, because processing is actually low (training a network usually uses 5~20% of my GPU processing), I was being greedy and training multiple networks at the same time in the same GPU. So it’s not like a single process actually consumed all my GPU memory.</p>
<p>I now implemented a few measures to limit GPU memory usage, such as deleting tensors once they are used (especially the big ones), limiting the scope of some tensors with the use of functions and calling <code>torch.cuda.empty_cache()</code> periodically to free memory to other processes. I am still trying to figure out if these things are making a difference.</p>
LibTorch in iOS message filtering extension

username: jiayu
cooked: <p>Hi,</p>
<p>I am trying to build an iOS message filtering extension, and I am using LibTorch 1.6, however after integrating with the LibTorch library, I find that my message extension no longer works: it compiles, runs, can be installed and enabled on my device, but the filtering just stops working (i.e. all messages are no longer filtered).</p>
<p>Debugging this is really difficult, because there are no error messages. I wonder if anyone has run into similar issues before, or if anyone has successfully run LibTorch in this setup. I can confirm that my main iOS App runs perfectly, it’s just the extension (.apex) that doesn’t.</p>

username: jiayu
cooked: <p>After digging through the log an device console I found the cause: it is hitting memory limit: on my device it is only 6MB.</p>
<p>Guess that renders LibTorch hardly usable now…</p>
<p>Hope this is helpful to somebody else.</p>
LSTM+Dense has different training and validation output with different evaluation interval

username: mintin97
cooked: <p>Hello! I am developing a LSTM network with a Dense layer on it for stock prediction. I want to calculate the validation loss, so I call model.eval() and “with torch.no_grad()” before the evaluation. This can happen every epoch, every 2 epochs etc. If this epoch interval changes, the training and validation output (acc and loss) change for the same epoch.<br>
I have also tried to do param.requires_grad=False for every parameter before evaluation and set it back to True before training, but it didn’t work.<br>
Any explanation?<br>
Thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="mintin97" data-post="1" data-topic="93757">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mintin97/48/59386_2.png" class="avatar"> mintin97:</div>
<blockquote>
<p>If this epoch interval changes, the training and validation output (acc and loss) change for the same epoch.</p>
</blockquote>
</aside>
<p>Do you mean the training losses change for the following epoch, if you’ve used the validation loop before?<br>
If so, you might call into the pseudo-random number generator in your validation loop, as explained <a href="https://discuss.pytorch.org/t/does-a-dataloader-change-random-state-even-when-shuffle-argument-is-false/92569/4">here</a>, which might change the next training epoch. If your training losses change by a large margin, it seems that your training might not be really stable for different seeds.</p>
"Correct" way deal with hidden state, .detach() and retain graph in Recurrent scenarios

username: mayool
cooked: <p>Hi!<br>
I am working with video data (individual frames) and want to improve the performance using a Recurrent Unit  (Convolutional LSTM, to be precise). My idea is to use information from previous timeframes [t-2, t-1] to improve the prediction of [t]</p>
<p>I read through <a href="https://discuss.pytorch.org/t/solved-why-we-need-to-detach-variable-which-contains-hidden-representation/1426/14">this thread</a> and got confused about the different ways to deal with recurrency in Pytorch.</p>
<p>What is the “correct” way implement the scenario given above?</p>
<ol>
<li>Let the timesteps flow through the LSTM and then detach it:</li>
</ol>
<pre><code class="lang-auto">input.shape()  #[bs, #channels, height, width]

output, hidden_state = model(input, hidden_state)
hidden_state = tuple(state.detach() for state in hidden_state) 
# detach so loss.backward() can be called without any retain_graph error
</code></pre>
<p>inside the forward() method, the previous model predictions are saved and stacked with the current input:</p>
<pre><code class="lang-auto">input = combine_with_old_predictions() #[bs, 3, #channels, height, width] with [t-2, t-1, t] stacked in dim1
h, c = previous_hidden
output_inner = []
for t in range(num_timesteps): # num_timesteps=3
    # feed the timesteps one by through the lstm cell
    h, c = lstm_cells(input_tensor=input[:, t, :, :, :], cur_state=[h, c])
    output_inner.append(h)
layer_output = torch.stack(output_inner, dim=1)
</code></pre>
<ol start="2">
<li>only feed in 1 timestep but let gradient flow back until 2 timesteps:</li>
</ol>
<pre><code class="lang-auto">detach_interval = 3
for i, batch in enumerate(data_loader):
    input, ground_truth = batch
    output, hidden_state = model(input, hidden_state)
    # input stays the same in the forward method of the model and is not stacked 
    # with the previous 2 timesteps [t-2, t-1]
    loss = criterion(output, ground_truth)
    if i % detach_interval == 0:
        loss.backward(retain_graph=False)
    else:
        loss.backward(retain_graph=True)
</code></pre>
<p>What is the “correct LSTM-way” to deal with this scenario?<br>
Any help is much appreciated!<br>
Cheers,<br>
Sven</p>
Why do we need to do model.to(device) and input.to(device)

username: puneeth2001
cooked: <p>I understood that we need to move to gpu, but why do we need to move to GPU it’s not the case with keras and all right?</p>
<p>I am learning to to transfer learning from a tutorial and I found this.</p>

username: ptrblck
cooked: <p>Keras might transfer the model parameters and data automatically for you behind the scenes, while you are responsible to do it in PyTorch.<br>
I guess it comes down to balance “convenience vs. flexibility” and I personally like to manually specify which tensor is places in which device, which e.g. enables models sharing easily.</p>
Will `torch.manual_seed` in program A on cuda:A affect program B on cuda:B?

username: Richard_Wang
cooked: <p>I recently know inside <code>torch.manual_seed(seed)</code>, it calls <code>torch.cuda.manual_seed_all(seed)</code>, which sets the seed for generating random numbers on all GPUs.</p>
<p>This brings me a question.<br>
If I launch program A (use <code>torch.manual_seed(3)</code>) on cuda:0 and then program B (use <code>torch.manual_seed(5)</code>) on cuda:1, (Both are on the same machine), will program A end up in using seed 3 or 5 ?</p>

username: ptrblck
cooked: <p>If both scripts are started independently (e.g. in other terminals), the seeds shouldn’t have any influence, as the python programs won’t interact with each other.</p>
ModuleNotFoundError: No module named 'torch' in IDLE

username: Jay
cooked: <p>So I installed pytorch through anaconda successfully and when I try importing torch in IDLE I get this error</p>
<p>I did what I found on stack exchange “print(sys.executable)”</p>
<p>Conda’s result: C:\Users\readz\anaconda3\python.exe<br>
IDLE’s result: C:\Users\readz\AppData\Local\Programs\Python\Python38-32\pythonw.exe</p>
<p>Problem is there they didn’t explain what to do with this info or how to solve the problem…</p>

username: ptrblck
cooked: <p>Did you install <code>IDLE</code> in the same conda environment and executed it there?<br>
If not, could you do it, as currently <code>IDLE</code> seems to use another Python environment.</p>
Error in Gaussian Process implementation

username: RR_1
cooked: <p>I create a new version of Expected Improvement for my problem.<br>
When I run it I obtain the next configuration to analyse in the model.<br>
Then an error occur in this line :<br>
<em>Algorithm.update_model(next_config)</em><br>
where my update_model function is defined like this:<br>
<em>def update_model(self, cell_config):<br>
self.model = self.model.condition_on_observations(cell_config.get_tilt(), cell_config.get_reward())<br>
mll = ExactMarginalLogLikelihood(self.model.likelihood, self.model)<br>
fit_gpytorch_model(mll)<br>
return self.model</em><br>
The error is this one:<br>
RuntimeError: Fantasy observations can only be added after making predictions with a model so that all test independent caches exist. Call the model on some data first!</p>

username: ptrblck
cooked: <p>The error message seems to be raised by a method, which is specific to the repository you are using, so I would recommend to ask the authors or create an issue in the repository directly. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>That being said, based on the error message it seems you would have to call the model with data before calling the code snippet, which is raising this error.</p>
Gradients Flow tensorboard error

username: Maicol_Polvere
cooked: <p>Good Morning,<br>
I would like to visualize the flow of the network parameters gradients during training using tensorboar.</p>
<p>my code is:</p>
<pre><code class="lang-auto">def training(epoch, net, device, criterion, train_data, optimizer,tb):
    """
    Run one training epoch
    :param epoch: Current epoch
    :param net: Network
    :param device: Torch device
    :param criterion: function to evaluate the loss
    :param train_data: Training Dataset
    :param optimizer: Optimizer
    :return:  Average Losses for Epoch
    """
    
    results = 0
    batch_idx = 0
    # change flag of training
    net.train()
    
    for  sample in train_data:        
        
        batch_idx +=1

        # load of the normalized image
        img = sample['image'].to(device)
        
        # load of the ground bounding boxes
        ground_bb = sample['bb'].to(device)
        
        # forward propagation
        pred_bb = net(img)
        
        # evaluate the loss
        loss = criterion(ground_bb,pred_bb)
        
        results += loss.item()
        
        #  gradients are zeroed
        optimizer.zero_grad()
        
        # backward propagation
        loss.backward()
        
        # optimization of the parameters
        optimizer.step()        
        
        for param in net.parameters():
            tb.add_histogram('gradient',param.grad,param)

        logging.info('Epoch: {}, Batch: {}, Loss: {:0.4f}'.format(epoch, batch_idx, loss))
    
    return results/batch_idx
</code></pre>
<p>However it returns error. Do you have any ideas on how to display the gradients?</p>

username: ptrblck
cooked: <p>What kind of error are you getting?</p>
Creating an algotithm Dataloader to load .mat files

username: adam_kossemtini
cooked: <p>Hello, Am a beginner in deep-learning, i have 2 .mat (2 classes) file and both of them contain a number images … So i need a hand in creating an algorithm for loading files in a DataLoader,<br>
Thank you.</p>

username: ptrblck
cooked: <p>Have a look at <a href="https://discuss.pytorch.org/t/how-to-convert-mat-file-struct-into-pytorch-tensors/38094/4">this post</a> for an example code snippet. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Resnext101 backbone faster rcnn train occur loss is Nan

username: yunsang.ju
cooked: <p>Hello<br>
Loss is nan error occurs when I learn fast rcnn with resnext101 backbone<br>
My code is as follows</p>
<pre><code class="lang-python">backbone = resnet_fpn_backbone('resnext101_32x8d', pretrained=True)
model = FasterRCNN(backbone, num_classes)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
</code></pre>
<p>error message</p>
<pre><code class="lang-auto">Epoch: [0]  [   0/7208]  eta: 1:27:42  lr: 0.000040  loss: 40613806080.0000 (40613806080.0000)  loss_box_reg: 7979147264.0000 (7979147264.0000)  loss_classifier: 11993160704.0000 (11993160704.0000)  loss_objectness: 9486380032.0000 (9486380032.0000)  loss_rpn_box_reg: 11155118080.0000 (11155118080.0000)  time: 0.7301  data: 0.4106  max mem: 1241
Loss is nan, stopping training
</code></pre>
<p>When i change the backbone to resnet50 and resnet152, no error occrus.</p>
<p>i try to reduce lr but error still occur</p>
<pre><code class="lang-auto">Epoch: [0]  [   0/7208]  eta: 1:37:48  lr: 0.000000  loss: 34383663104.0000 (34383663104.0000)  loss_box_reg: 8708817920.0000 (8708817920.0000)  loss_classifier: 8583784448.0000 (8583784448.0000)  loss_objectness: 7424135168.0000 (7424135168.0000)  loss_rpn_box_reg: 9666924544.0000 (9666924544.0000)  time: 0.8142  data: 0.4027  max mem: 1242
Epoch: [0]  [  10/7208]  eta: 0:44:46  lr: 0.000000  loss: nan (nan)  loss_box_reg: nan (nan)  loss_classifier: nan (nan)  loss_objectness: nan (nan)  loss_rpn_box_reg: inf (nan)  time: 0.3732  data: 0.0429  max mem: 1919

</code></pre>

username: ptrblck
cooked: <p>The loss value seems to be huge, which could yield to an overflow in the second iteration and thus a NaN output.<br>
Could you disable the training completely and check the loss values for each batch?</p>
Weight factor for the BCEwithlogitloss

username: Mughees
cooked: <p>I have unbalanced data  and i want to use BCEwithLogitLoss with weight factor for minority classes. Total samples in each classes are,<br>
<code>samples = torch.FloatTensor([74891,11865,194148,98997,29350,104203,130637,30649,141300,164775,176567,16267,148950,1536,12022,22100,1566,67277,74877])</code><br>
and I have devised a formula for the calculation of weight factor as <code>max(1,log2(majority_class/class_samples))</code> which gives somehow the values which seems fine to me as well. but when I am trying to implement this with pytorch it gives me error as,</p>
<pre><code class="lang-auto">step1 = torch.log2(torch.max(samples)/samples)
step2 = torch.max(1,step1)
</code></pre>
<p>error at step2 as <code>Dimension out of range (expected to be in range of [-1, 0], but got 1)</code></p>
<p>final weights should be<br>
<code>weights= torch.FloatTensor([1.37429258667969,4.03237283987156,1,1,2.7257244370681,1, 1,2.66324493724027,1,1,1,3.57713673021635,1,6.98183481923172, 4.01340801523125,3.13503857078672, 6.95392882266183,1.52897156569056,1.37456230692208])</code></p>
<p>Lastly this formula is fine? or can anyone suggest me any better formula or improvement to this? Thanks alot in advance,</p>

username: Usama_Hasan
cooked: <p>Hy <a class="mention" href="/u/mughees">@Mughees</a> , I hope I understand this correctly,<br>
When I run this.</p>
<pre><code class="lang-auto">samples = torch.FloatTensor([74891,11865,194148,98997,29350,104203,130637,30649,141300,164775,176567,16267,148950,1536,12022,22100,1566,67277,74877])
a = torch.tensor(1).float() 
step1 = torch.log2(torch.max(samples)/samples)
step2 = torch.max(a,step1)
print(step2)
tensor([1.3743, 4.0324, 1.0000, 1.0000, 2.7257, 1.0000, 1.0000, 2.6632, 1.0000,
        1.0000, 1.0000, 3.5771, 1.0000, 6.9818, 4.0134, 3.1350, 6.9539, 1.5290,
        1.3746])

</code></pre>
<p>It gives the desired output.</p>

username: Mughees
cooked: <p>I guess I was missing the part to convert 1 into tensors. Thanks alot.</p>
Network trains in Pytorch 1.4 but doesn't in Pytorch 1.6

username: zshn25
cooked: <p>I am training an unsupervised CNN encoder decoder network for dense output (optical flow). The network trains perfectly fine in version 1.4 and gives proper output. However, in Pytorch 1.6, with the same configuration, the trained network gives constant output (one value in the whole image)</p>

username: albanD
cooked: <p>Hi,</p>
<p>It is very hard to say without more information. There are almost ten thousand commits between 1.4 and 1.6 <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Can you explain what you did change when doing the upgrade?<br>
What have you tried to identify the difference?<br>
How stable was the training in 1.4?</p>

username: zshn25
cooked: <p>Well, first I thought the problem was in distributed training. So, I disabled that and ran on single GPU. Then I thought it was the network architecture. I tried other architectures but same issue.</p>
<p>Finally, I concluded that this is not because of any of the above but because of the L2 regularization I was using with the Adam optimizer. I was using weight decay with 0.0005 as the weight but I think it was too large weight.</p>
Video clip data

username: pcshih
cooked: <p>Dear Pytorch users,</p>
<p>I have a dataset. It contains a bunch of video clips and labels.</p>
<p>e.g.<br>
video clips -&gt; clip0, clip1, clip2<br>
labels         -&gt;       0,       1<br>
label 0 means clip0 and clip1 should not be merged<br>
label 1 means clip1 and clip2 should be merge</p>
<p>Can any model/architecture fit in this dataset? (the amount of data and label are not equal)</p>
<p>Thank you.</p>
Performance of LU-factorization, can it be improved?

username: Anders_G
cooked: <p>Dear Pytorch users,</p>
<p>I want to do batched LU factorisation as fast as possible for some medium sized matricies (around 200 x 200). However, when I run a simple benchmark it is much slower than expected. On a GTX 1080Ti I get about 30 GFlop/s on single precision. Considering that it has a peak computational power of 10 TFlops/s I find it a bit disappointing. Is there something I can do to improve the situation or is this the performance to be expected? For instance does anyone have experience of using the MAGMA version of Pytorch and knows if that improves performance? (Some improvement should be possible I already know since Tensorflow is about twice as fast.)</p>
<p>Best<br>
Anders</p>
<p>Bench-marking code used:</p>
<pre><code class="lang-auto">import torch
import time

# Repeated operations
rep = 1000 # Batch size
loop = 25 # Number of repeats
m = 200 # Matrix size (m x m)

# cuda:0 corresponds to a GTX 1080 Ti
settings = {'dtype': torch.float32, 'device': 'cuda:0'}


def print_no_flops(t, m):
    lu_ops_total = 2/3*m**3
    GFlops = 1024**3
    print(f"Performance: {lu_ops_total / t / GFlops} GFlops")


# Create data (1 system)
A = torch.randn((m, m), **settings)
ATA = A.t().mm(A)

# Make it a batch
ATA = (ATA).unsqueeze(0) * torch.ones((rep, 1, 1),**settings)


# Time the calculation
tic = time.perf_counter()
for i in range(loop):
    LU = torch.lu(ATA)
toc = time.perf_counter()

dt = toc-tic
dt_per_system = dt / (rep * loop)
print(f"Solve time per system = {dt_per_system * 1e6} us")
print_no_flops(dt_per_system, m)
</code></pre>
Optimizing network architecture

username: Talmaj
cooked: <p>Hi, I was benchmarking different convolutional layers on CPU. It seems that less output channels does not mean faster runtime. This is my test:</p>
<pre><code class="lang-auto">def benchmark(model, inp_size=(1,1,28,28), n=1000):
    model.eval()
    dummy_input = torch.rand(*inp_size)

    func = lambda: model(dummy_input)
    runtimes = timeit.repeat(func, repeat=10, number=n, )
    print(model)
    print(min(runtimes))
    print()
</code></pre>
<p>The result:</p>
<blockquote>
<p>Conv2d(1, 64, kernel_size=(7, 7), stride=(1, 1), padding=(1, 1))<br>
0.13669140800000035</p>
<p>Conv2d(1, 61, kernel_size=(7, 7), stride=(1, 1), padding=(1, 1))<br>
0.19433658299999967</p>
<p>Conv2d(1, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
0.1496824219999997</p>
<p>Conv2d(1, 125, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
0.2669210919999996</p>
<p>Conv2d(1, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
0.501409099</p>
<p>Conv2d(1, 509, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
1.0796660310000021</p>
<p>Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
0.666838663</p>
<p>Conv2d(61, 61, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
0.8703990200000007</p>
<p>Conv2d(33, 33, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
0.5053342729999954</p>
</blockquote>
<p>Questions:</p>
<ol>
<li>Is my benchmarking correct?</li>
<li>Why is using less channels not faster?</li>
<li>Would a network compilers like glow accelerate layers with fewer channels?</li>
</ol>
Unable to load the model_weights from S3 Bucket

username: Shakir_Khurshid
cooked: <p>Im working on S3 , I ve my model saved in S3 as PTH file , but when i try to load it  I get the following error</p>
<pre><code class="lang-auto">TypeError: new(): invalid data type 'bytes'
</code></pre>
<p>My code is:</p>
<pre><code class="lang-auto">s3 = boto3.resource('s3')
bucket_name = 'bucket_name'
key = 'network/model/model.pth'
obj = s3.Object(bucket_name,key)
body = obj.get()['Body'].read()

state = torch.as_tensor(body)

</code></pre>
<p>I am assuming torch cant load model as bytes …I am new to AWS , so anyone can help me with a solution how to load the model weights from S3 bucket</p>
How to take gradients with respect to model parameters on a loss function already defined in terms of gradients of inputs?

username: Ashok_Vardhan
cooked: <p>The problem I am trying to solve has the following setup. Let <code>f(x; theta)</code> and <code>g(y; phi)</code> be two neural networks whose outputs are scalar and <code>theta, phi</code> are the parameters I want to learn. <code>x,y</code> are vector valued inputs to these networks. In order to train the parameters, the loss function I have is the following:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3c9312f7a2c5aef8031104421cb86aace058aa98.gif" alt="CodeCogsEqn" data-base62-sha1="8DRKGAiMNkT5NJQV4MdvaWt5W9q" width="565" height="22"></p>
<p>The issue I have is that since the loss <code>L</code> already has gradients with respect to input <code>y</code> in the form of <code>nabla g(y)</code>, I am wondering if it’s possible to compute the gradient again with respect to both <code>theta</code> and <code>phi</code>. The tentative code I have is the following but I am not sure if it’ll work:</p>
<pre><code class="lang-auto">x, y = Variable(torch.randn(1,2), requires_grad= True), Variable(torch.randn(1,2), requires_grad= True)

# Assuming that f_theta and g_phi are predefined neural networks

g_y = g_phi(y)

g_phi(y).backward()

grad_g_y = y.grad

f_grad_g_y = f_theta(grad_g_y)

f_x = f_theta(x)

loss = f_grad_g_y - f_x - torch.sum(y*grad_g_y ,dim=1)


# Computing gradient with respect to theta and phi

loss.backward()
print(theta.grad, phi.grad)
</code></pre>

username: MariosOreo
cooked: <p>Hello,</p>
<p>In my shallow view, it will work, but we should fix something manaully, see <a href="https://discuss.pytorch.org/t/convert-tuple-to-tensor-without-breaking-graph/36568">this thread</a> which computing Hessian product and do backward, I think it is similar to you, do backward on a gradient.</p>
<p>So, did you have a try on the snippet above?</p>

username: KAISER1997
cooked: <p>Hi ,so is your tentative code working correctly?</p>
Optimal transport metric

username: kl_divergence
cooked: <p>Hi,<br>
I’m new to Optimal transport theory. I wish to minimize the domain gap in two data distributions (two datasets) using a metric which should be differentiable as I wish to use it as loss func. I found Sinkhorn distance (Wasserstein) to be interesting but couldn’t find an implementation outside GANs. I’m training my networks in an adversarial manner but still it’s not efficient. How can I use something from OT to minimize the gap between two distributions as loss ?</p>

username: vv111y
cooked: <p>This comes recommended for pytorch <a href="https://github.com/jeanfeydy/geomloss" rel="nofollow noopener">https://github.com/jeanfeydy/geomloss</a></p>
<p>A more mature and comprehensive implementation of all things OT related is <a href="https://github.com/rflamary/POT" rel="nofollow noopener">https://github.com/rflamary/POT</a>, however they do not yet have pytorch support. That on their todo list</p>

username: tux
cooked: <p>Hello,</p>
<p>Indeed the Geomloss package is really efficient to compute entropic variants of Optimal Transport. You can have access to the entropic regularized OT or the Sinkhorn divergence. They also provide typical ground cost but you can compute your own. I would suggest to use the Sinkhorn divergence instead of the entropic regularized OT. The later interpolates between OT and MMD losses while the former does not have the separability axiom (i.e OT_eps(a,a) &gt; 0). You can find more details in : <a href="https://arxiv.org/abs/1810.08278" rel="nofollow noopener">https://arxiv.org/abs/1810.08278</a></p>

username: edgarriba
cooked: <p>the library below also implements OT through the sinkhorn algorithm<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/fwilliams/fml" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/1343232?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/fwilliams/fml" target="_blank" rel="nofollow noopener">fwilliams/fml</a></h3>

<p>FML (Francis' Machine-Learnin' Library) - A collection of utilities for machine learning tasks - fwilliams/fml</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: saba
cooked: <p>Hi,<br>
I am using the Geomloss package to compute the Sinkhorn  distance. my inputs are two distributionS(DATA1 AND DATA2) with batch size of 64 and 9<em>9 dimension. how I can pass them into the loss to compute , should I pass them as a vector with size of 64</em>9*9?</p>
<pre><code class="lang-auto">    loss = SamplesLoss(loss="sinkhorn", p=2, blur=.5)
    L = loss(DATA1.float(),DATA2.float())</code></pre>
How could I configure `setup.py` to use opencv to write pytorch cpp extension?

username: coincheung
cooked: <p>Hi,</p>
<p>Following <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">this</a> tutorial, I got to know how to write simple cpp extension for pytorch. However, in my extension, I need to use opencv as a third party library. When I have the line <code>#include &lt;opencv2/opencv.hpp&gt;</code> in my code, I got the error message of <code>opencv2/opencv.hpp: No such file or directory</code>. Would you please tell me how to configure the <code>setup.py</code> file to let the extension support opencv ?</p>
Different results from sum() and torch.sum()

username: zhoulukuan
cooked: <p>I want to find out how many numbers which are nonzero in the tensor：</p>
<pre><code class="lang-python">output = model(input)
output = output.reshape(-1)
result1 = sum(output != 0)
result2 = torch.sum(output != 0)
</code></pre>
<p>However, I found sometimes result1 and result2 are different and the result1 is zero even though there are many nonzero elements in the output. Why?</p>

username: vainaijr
cooked: <p>maybe,</p>
<pre><code class="lang-auto">len(output.nonzero())
</code></pre>
<p>also, what case did you get different result1 and result2?</p>

username: zhoulukuan
cooked: <p>Thanks for your advice!<br>
I feel very confused because I cannot reproduce it stably. This is the problem I encountered when reproducing the transformer.</p>

username: vainaijr
cooked: <p>it might have to do with dropout or with seed,<br>
for example,</p>
<pre><code class="lang-auto">x = torch.randn(1, 1, 28, 28)
c = nn.Sequential(nn.Conv2d(1, 10, 1), nn.Dropout())
c(x).sum(), c(x).sum()
</code></pre>
<p>would give different output<br>
whereas if model is set to eval mode, that is,</p>
<pre><code class="lang-auto">c.eval()
</code></pre>
<p>then,</p>
<pre><code class="lang-auto">c(x).sum(), c(x).sum()
</code></pre>
<p>would give same output<br>
for seed, if you use something like,</p>
<pre><code class="lang-auto">seed = 0
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.random.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic=True
</code></pre>
<p>then it would be fine</p>

username: zhoulukuan
cooked: <p>Thank you, I will have a try.</p>
Using a single threads on ARM CPU

username: Woosung
cooked: <p>Hi guys,</p>
<p>I’m trying to inference my model with a single core. but it seems not working</p>
<p>Pytorch : 1.5.0<br>
Platform: Nvidia Jetson Xavier<br>
Tried:  os.sched_setaffinity, torch.set_num_threads</p>
<p>when I get the result with os.sched_getaffinity and torch.get_num_threads, both shows only 1threads and 1 core affinity output but with htop, it shows using 8 cores.</p>
<p>can I get any advice?</p>

username: Woosung
cooked: <p>Hi guys,</p>
<p>I think I found the problem.<br>
Setting threads number was working correctly</p>
<p>The reason why htop tell me that 8 cores been utilized is because of DataLoader<br>
I didn’t set num_workers as 1.</p>
<p>so after set num_workers as 1 it works fine.</p>
How to extract data from Dataloader?

username: shadowkiller33
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5efec194830827d59d844a60719b9419e80a23f.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5efec194830827d59d844a60719b9419e80a23f" title="a"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5efec194830827d59d844a60719b9419e80a23f.png" alt="a" data-base62-sha1="z5F0HyYLxBwRKReoyXcgep8vpgj" width="690" height="231" data-dominant-color="EEEEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">a</span><span class="informations">1339×450 32.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c5e3782a21f8aa05ecaf4eaf0b9a7fa97d2e9da.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c5e3782a21f8aa05ecaf4eaf0b9a7fa97d2e9da" title="aa"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c5e3782a21f8aa05ecaf4eaf0b9a7fa97d2e9da.png" alt="aa" data-base62-sha1="1LpAtnstrK8h3whw9EnBZM63XAK" width="690" height="384" data-dominant-color="F9DAD9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">aa</span><span class="informations">1341×748 49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The dataloader is bulit with the AG_NEWS, I tried to demonstrate the labels and tensors in it in many ways but ended with the similar error which is shown in the Figure 2.</p>
<p>Can’t be more appreciated if you can give me a hand.</p>

username: BaruchG
cooked: <p>It looks like your <code>data</code> variable in your for loop is your incrementer and the <code>label</code> variable in there contains the inputs and labels.  That’s what is being assigned in <code>for (data, labels) in enumerate(full_train_loader)</code> and I doubt that is what you want.</p>

username: shadowkiller33
cooked: <p>Still can’t fix it…</p>

username: BaruchG
cooked: <p>Do you get the same error if you change that line to <code>for i, (data, labels) in enumerate(full_train_loader):</code> ? That should be at least part of the problem.  I can’t see the entire stack trace or code, so I don’t know if there are other issues as well.</p>
How to concat datasets for LSTM properly?

username: Mango_Freeze
cooked: <p>I have a list of files that contains equal length data. To simplify let’s say the two files contain<br>
file1: [1, 2, 3, 4, 5]<br>
file2: [6, 7, 8, 9, 10]</p>
<p>Currently I create a dataset for each file and use ConcatDataset() in my dataloader. Let’s assume a sequence length of 2.<br>
Now if I use a batch size of 2, will the dataloader return [[1, 2], [3, 4]] or [[1, 2], [6, 7]]?</p>
<p>My desired batch is [[1, 2], [6, 7]], so that I can feed the LSTM with proper start point and the next batch [[3, 4], [8, 9]] will make sense for the LSTM state.</p>
<p>How can I achieve that?</p>
Training Word Embedding with Categorical Vector

username: John_Watkins
cooked: <p>As far as I am aware, the Embedding class “forward-pass” converts each word-index in a LongTensor into its corresponding word embedding. However, I am wondering if there is a method to adapt Embedding to accept a categorical vector instead of word-index? Specifically, each vector represents a distribution over the entire vocabulary, and the Embedding would return a linear combination of the word embedding based on the vector. So for example:</p>
<p>If the vector is [0.7, 0.2, 0.0, 0.1], the embedding would be 0.7<em>emb0+0.2</em>emb1+0.0<em>emb2+0.1</em>emb3.</p>
<p>While I suppose one way to do this is to create a matrix of dimension [vocab_size, embedding_size] to substitute the Embedding, I have some pretrained Embeddings that I would prefer to (continue) using.</p>
<p>Any help is appreciated.</p>

username: vdw
cooked: <aside class="quote no-group" data-username="John_Watkins" data-post="1" data-topic="93602">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_watkins/48/15400_2.png" class="avatar"> John_Watkins:</div>
<blockquote>
<p>While I suppose one way to do this is to create a matrix of dimension [vocab_size, embedding_size] to substitute the Embedding, I have some pretrained Embeddings that I would prefer to (continue) using.</p>
</blockquote>
</aside>
<p>I haven’t tried it, but this should work just fine since both the embedding layer or your “manual” linear layer have the same shape <code>(vocab_size, embed_dim)</code>. And it should be possible to set the weights of the linear layer to the pretrained word vectors.</p>

username: John_Watkins
cooked: <p>Thank you for your reply. I have a follow-up question:</p>
<p>For Embedding, there is a padding_idx argument. As I understand, the row of weights corresponding to the padding_idx is always zeroed, and never updated during backpropagation. This causes the padding vector to return an all-zero vector.</p>
<p>Is it possible to do something similar for the weights of a Linear module? Specifically, is there a way to define only a certain row of weights that is initialized to zero, and never update by backpropagation?</p>

username: vdw
cooked: <p>Well, I guess you can set the weights manually to zero in your forward method. Before pushing the tensor to the linear layer, you first set the weights as position <code>padding_idx</code> to zero. In this case, you don’t need to worry if there’s any update through backpropagation. I mean, there is, but you overwrite it each time before “using” the linear layer.</p>
Per epoch loss equation

username: Vinayak_Mikkal
cooked: <pre><code class="lang-auto">from torch import optim
def fit(train_dl,model,test_dl=None,epoch=1000,lr=0.001):

  criterion = nn.CrossEntropyLoss()
  optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

  for epoch in range(epoch):  
      running_loss = 0.0
      model.train()
      for i,(inputs, labels) in enumerate(train_dl):

          optimizer.zero_grad()

          outputs = model(inputs)
          loss = criterion(outputs, labels)

          loss.backward()
          optimizer.step()
          running_loss += loss.item()*inputs.size(0)

          # if i % 2000 == 1999:    
          #     print('[%d, %5d] loss: %.3f' %
          #           (epoch + 1, i + 1, running_loss/2000))
          #     running_loss = 0.0
      print("Train Loss: ",running_loss/len(train_dl.sampler))


      if test_dl is not None:
          model.eval()
          running_loss = 0
          for i,(inputs, labels) in enumerate(test_dl):
              outputs = model(inputs)
              loss = criterion(outputs, labels)             
              running_loss += loss.item()*inputs.size(0)
          print("Validation Loss: ",running_loss/len(test_dl.sampler))

  print('Finished Training')
</code></pre>
<p>Is the equation for the per epoch loss correct?</p>

username: AladdinPerzon
cooked: <p>Could you explain some more of what it is you want to do? Is there any particular reason as to why you are not simply calculating the mean loss of every epoch and doing something like:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    losses = []
    for idx, (inputs, labels) in enumerate(train_dl):
        ...
        loss = criterion(outputs, labels)
        losses += loss.item()
    
        print(
            f"Mean loss at epoch [{epoch}/{num_epochs}] was {sum(losses)/len(losses)}"
        )


</code></pre>
Building a custom Cifar10 or any image dataset by including their augmented images

username: Kalyan
cooked: <p>I would like to build a custom image dataset. Let’s say my dataset contains 100 images and its labels. I would like to create a new dataset with 2 augmentations for every image and include its labels too.</p>
<p>so, the dataset should contain 100 original images and labels + 100 images of some augmentation and labels + 100 images of some other augmentation and labels.</p>
<p>so, each label is repeated 3 times i.e. one for original and 2 for augmented images.</p>

username: AladdinPerzon
cooked: <p>In PyTorch (and TensorFlow for that matter) how data augmentation works is that it’s done on the fly as part of image preprocessing (and preferably run in parallel on the CPU while the model is training on the GPU). This means that you do not need to run any augmentation and then store on disk, actually this is not recommended, although I guess this would be possible to do if you really want to.</p>
<p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">This</a> PyTorch tutorial is oftentimes referenced on this forum that could be worth checking out, it’s really great both to get a deeper understanding of custom dataloading but also insight in best practices to perform data augmentation. If you would like to see a minimal example of how performing data augmentation on the fly in PyTorch looks like you can see a code example I’ve created <a href="https://github.com/AladdinPerzon/Machine-Learning-Collection/blob/804c45e83b27c59defb12f0ea5117de30fe25289/ML/Pytorch/Basics/pytorch_transforms.py#L56-L72" rel="nofollow noopener">here</a></p>
Weird behaviors when calling model.train() and model.eval() alternatively

username: antran96
cooked: <p>Hi,</p>
<p>I noticed that my model yields poor results when trying to do this:</p>
<pre><code class="lang-auto">for epoch in num_epochs:
   model.train()
   # training scripts
   model.eval()
   #evaluate scripts
</code></pre>
<p>However, this script works perfectly fine:</p>
<pre><code class="lang-auto">for epoch in num_epochs:
   model.train()
   # training scripts

model.eval()
#evaluate scripts
</code></pre>
<p>So in the 2nd script, I wait for my model to finish training and evaluate, and in the 1st one, I evaluate in every epoch.</p>
<p>Why does the 1st method result in worse results?</p>
<p>Thanks</p>

username: AladdinPerzon
cooked: <p>I don’t think there should any difference running evaluation inside the training loop as opposed to running it afterward expect for the obvious that the model has trained longer if you run it after. Could you expand a little on what weird behaviors you encounter? Could you give some example code (preferably not too large) that produces this behavior, that would help to debug <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
ReLU vs LeakyReLU vs PReLU

username: Topsoil
cooked: <p>What are the advantages and disadvantages of using each of them?</p>
<p>Is general formula of ReLU &lt; LeakyReLU &lt; PReLU correct?</p>

username: Lin_Jia
cooked: <p>ReLU will have the value to be zero when the input is below zero. This “flat line” zero will make gradient descent algorithm difficult, because the gradient of a “flat line” is zero. LeakReLU is introduced to resolve this problem, cause the output of a LeakReLU will be a nearly flat line, but not exactly flat.</p>

username: AladdinPerzon
cooked: <p>From <a href="https://arxiv.org/pdf/1710.05941.pdf" rel="nofollow noopener">this</a> paper they compared various activation functions and the results are quite varied. There doesn’t seem to be one obvious superior activation function. Because of that I don’t think that general formula is entirely accurate. If you also look at this <a href="https://youtu.be/lGbQlr1Ts7w?t=1651" rel="nofollow noopener">lecture</a> by Justin Johnson you can see a nice summarization graph of that paper I referenced and I recommend listening to the part of the lecture about activation functions to gain some deeper insights.</p>
<p>But to be brief, in case you don’t have the time to watch the lecture and to summarize: from my understanding there is no superior activation function. It varies which one is best on a case by case basis so people just stick with relu.</p>
How handle variable size sequences in LSTM module of PyTorch?

username: Farhad_Dalirani
cooked: <p>I know how to handle variable size sequence data in PyTorch for most cases. For example presume we want to predict next position of a person given its past locations. In this case we can pad different sequences with  <code>pad_sequence</code>  function. Then we can use  <code>pack_padded_sequence</code>  and  <code>pad_packed_sequence</code>  to calculate a batch of sequences of diverse lengths. Finally, We can use a mask on output of LSTM and ground truth positions and calculate loss and do back propagation.</p>
<p>In case of above example I’m aware of handling variable size sequences during training.</p>
<p>But assume another application of LSTM that we use hidden state of LSTM in last time step and concatenate it with other features and pass it to a deep neural network. Finally the deep neural network gives us a loss. I want to train all networks in an end to end fashion. But in this case I don’t know how to handle variable size sequences. In this case we don’t have ground truth for LSTM for different sequences, so we can’t use masking after  <code>pack_padded_sequence</code>  and  <code>pad_packed_sequence</code> . Instead we have a loss that back propagated from deep neural network.</p>
AttributeError: module 'torch.utils' has no attribute 'mobile_optimizer'

username: zshn25
cooked: <p>I have Pytorch 1.6 installed and according to this link <a href="https://pytorch.org/docs/stable/mobile_optimizer.html" rel="nofollow noopener">https://pytorch.org/docs/stable/mobile_optimizer.html</a>, there is a mobile optimizer util. But, I cannot find it. I get the <code>AttributeError: module 'torch.utils' has no attribute 'mobile_optimizer'</code> when I try <code>torch.utils.mobile_optimizer.optimize_for_mobile(traced_model)</code></p>

username: user_123454321
cooked: <p>Can you try</p>
<pre><code class="lang-auto">import torch.utils.mobile_optimizer as mobile_optimizer
mobile_optimizer.optimize_for_mobile(traced_model)
</code></pre>
BertTokenClassifier

username: sandeep1
cooked: <p>What is the significance of  ignore_label=[‘O’] in the following code?</p>
<p>from bert_sklearn import BertTokenClassifier<br>
model = BertTokenClassifier(bert_model=‘bert-base-cased’,<br>
max_seq_length=178,<br>
epochs=3,<br>
gradient_accumulation_steps=4,<br>
learning_rate=5e-5,<br>
train_batch_size=16,<br>
eval_batch_size=16,<br>
validation_fraction=0.,<br>
label_list=label_list,<br>
ignore_label=[‘O’])</p>