akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
A worker instance holds down other workers when processing load is unevenly distributed

username: DarQ
cooked: <p>I have an iterable dataset and let’s say I have 2 workers: worker 0 generates an audio clip and combines it with other pre-generated audio clips to create a data point, and workers 1 only combines pre-generated audio clips to create a data point. Naturally, worker 0 takes more time then worker 1 (audio clip generation and saving to the disk takes ≈ 0.8s while loading 12 audio clips takes 0.18s).</p>
<p>The workers obviously work in parallel so I expect the average loading time to be equal to 0.18 seconds at most (while worker 0 is generating the audio clip, worker 1 will take on the task of generating data points) that is not the case however, since I’m getting avg loading time of about 0.52s (almost thrice as slow). How can I solve this issue? is that even possible?<br>
here is some pseudo code:</p>
<pre><code class="lang-auto">class GeneratedDataset(IterableDataset):
    def __init__(self, note_num):
        super(GeneratedDataset).__init__()
        self.note_num = note_num
    
    def __iter__(self):
        worker = torch.utils.data.get_worker_info()
        if worker:
            id = worker.id
            np.random.seed(int(2**31*torch.rand(1)))
        else:
            id = 0
        while True:
            if id==0: # this is the code that only runs on worker 0 
                
                # GENERATE AUDIO CLIP
                # GENERATE RESPECTIVE LABELS
                # SAVE BOTH
                
                ------------------------
                
                # LOADING AUDIO CLIP
                # GENERATING SPECTROGRAM
                # TENSOR MANIPULATION (padding, normalizing, etc.)
                # SAVING THE TENSOR
            else:
                # LOAD RANDOM (pre-saved) TENSOR
                # LOAD RESPECTIVE LABELS
            for _ in range(12):
                # MORE TENSOR MANIPULATION
                
                # PROCESSING THE LABELS
                
                # CHOOSE ANOTHER (TENSOR, LABELS) PAIR FOR NEXT LOOP

            # CONCATENATE THE LOADED TENSORS. SAME FOR THE LABELS
            yield # TENSOR, LABELS
</code></pre>
<p>Any help would be greatly appreciated.</p>
Help understanding how to release GPU memory / avoid leaks

username: LuisValerio
cooked: <p>Hello!</p>
<p>I’ve looked around online but I still haven’t been able to figure out how to properly free GPU memory, here’s a link to a simple Colab demo explaining the situation [make sure to change the <strong>Runtime</strong> type to use a GPU]:<br>
<a href="https://colab.research.google.com/drive/1i-1rIbcsxwm90krtPyP5px6P6owKUKYH?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1i-1rIbcsxwm90krtPyP5px6P6owKUKYH?usp=sharing</a></p>
<p>I basically start by allocating a random tensor, move it to the GPU, report the GPU memory usage, then move the tensor back to the CPU, report the GPU memory usage, then delete the reference to the tensor and report the GPU memory usage once more. On the last two memory reports I’d expect the usage to go back to zero, but I keep getting a non zero answer (e.g. the usage goes down from 37% to 4%, but not zero).</p>
<p>How can I get the GPU memory back to zero in this simple case? Is this an issue with Google Colab specifically?</p>
<p>Thanks in advance!</p>

username: tom
cooked: <p>This likely isn’t a leak.<br>
PyTorch initializes CUDA “on demand” when you first use it and as part of this initialization, some global GPU memory is allocated. You would not expect this to be freed before terminating the Python process.</p>

username: Zinuo
cooked: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="126160">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>some global GPU memory is allocated</p>
</blockquote>
</aside>
<p>I wonder what global memory it is and its functions.</p>

username: tom
cooked: <aside class="quote no-group" data-username="Zinuo" data-post="3" data-topic="126160">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zinuo/48/40378_2.png" class="avatar"> Zinuo:</div>
<blockquote>
<p>I wonder what global memory it is and its functions.</p>
</blockquote>
</aside>
<p>These maintain state of the device and also work areas for various libraries I think.</p>
<p>You can poke around in the relevant PyTorch <a href="https://github.com/pytorch/pytorch/tree/master/aten/src/ATen/cuda" rel="noopener nofollow ugc">source</a> <a href="https://github.com/pytorch/pytorch/tree/master/aten/src/ATen/cuda" rel="noopener nofollow ugc">directories</a> and read up on <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#context" rel="noopener nofollow ugc">context</a> in the CUDA docs and the libraries like cuDNN, cuBLAS etc. (Seems that NVidia+cloudfare helpfully decided I should not link to the NVidia docs more, so you’ll have to find those yourself. Typically the workflow is that you get a global handle).</p>
About clone() and modify value in-place after clone()

username: dhkleung
cooked: <ol>
<li>
</li>
</ol>
<pre><code class="lang-auto">x = torch.tensor([6., 9.], requires_grad=True)
y = torch.tensor([0.1, 0.01], requires_grad=True)
y_clone = y.clone()
z = torch.sum(x * y_clone + y_clone)
z.sum().backward()
print(x.grad)
print(y.grad)
print(y_clone.grad)
</code></pre>
<p>will give</p>
<pre><code class="lang-auto">tensor([0.1000, 0.0100])
tensor([ 7., 10.])
None
</code></pre>
<p>I am not sure why y_clone.grad is None, but I know that clone will still pass the gradient to y. Since it is a clone, so the working on gradient flow for y_clone is the same as y. I expect the y_clone.grad will be the same as y.grad but not.</p>
<ol start="2">
<li>
</li>
</ol>
<pre><code class="lang-auto">x = torch.tensor([6., 9.], requires_grad=True)
y = torch.tensor([0.1, 0.01], requires_grad=True)
y_clone = y.clone()
y_clone[0] = 2.99
z = torch.sum(x * y_clone + y_clone)
z.sum().backward()
print(x.grad)
print(y.grad)
print(y_clone.grad)
</code></pre>
<p>will give</p>
<pre><code class="lang-auto">tensor([2.9900, 0.0100])
tensor([ 0., 10.])
None
/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:9: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more information.
  if __name__ == '__main__':
</code></pre>
<p>I have no idea how to interpret this.</p>
<p>Thanks!</p>

username: JuanFMontesinos
cooked: <p>It is None because y_clone is not a leaf tensor, thus, gradients for that tensor are not saved.</p>
<pre><code class="lang-auto">import torch
x = torch.tensor([6., 9.], requires_grad=True)
y = torch.tensor([0.1, 0.01], requires_grad=True)
y_clone = y.clone()
print(f'clone leaf? &gt;{y_clone.is_leaf}')
print(f'Y leaf? &gt;{y.is_leaf}')
z = torch.sum(x * y_clone + y_clone)
z.sum().backward()
print(x.grad)
print(y.grad)
print(y_clone.grad)
</code></pre>
<p>This seems to return a warning in my version</p>
<pre><code class="lang-auto">clone leaf? &gt;False
Y leaf? &gt;True
tensor([0.1000, 0.0100])
tensor([ 7., 10.])
/home/jfm/.pycharm_helpers/pydev/pydevconsole.py:11: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more information.
  from code import InteractiveConsole
None

</code></pre>
Pytorch transformers implementation vs other repos

username: Mika_S
cooked: <p>Is pytorch’s implementation of transformers state of the art or it is better to use a different implementation?</p>

username: JuanFMontesinos
cooked: <p>Well, anything you can find “oficially implemented” is not SOTA.<br>
Think that by the time they port it to pytorch researchers relese new transformers.<br>
A different thing is whether it is worth to spend time on a coarser code or not.<br>
Pytorch transformers are straight-forward.</p>
Are min() and max() operations O(n) on PyTorch tensors?

username: savvas17
cooked: <p>I have experimented with some example tensors of reasonably large size, and fetching the <code>min</code> and <code>max</code> values from these tensors seems suspiciously fast. Are these operations O(n) or does Pytorch perhaps keep track of them behind the scenes for quick retrieval? Of course the time taken will be system dependent, as well as GPU dependent. The below example is running with tensors on a CPU.</p>
<pre><code class="lang-auto">import torch
import time

large_tensor = torch.randn(100,500,10000)
start = time.time()
print(large_tensor.min())
print(large_tensor.max())
end = time.time() - start 
print(f'Time taken to find both min and max: {end} seconds')
</code></pre>
<p>Output:<br>
<code>Time taken to find both min and max: 0.6569840908050537 seconds</code><br>
Thank you in advance for considering my question!</p>

username: tom
cooked: <aside class="quote no-group" data-username="savvas17" data-post="1" data-topic="126756">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/savvas17/48/31639_2.png" class="avatar"> savvas17:</div>
<blockquote>
<p>Are these operations O(n)</p>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group">
<blockquote>
<p>or does Pytorch perhaps keep track of them behind the scenes for quick retrieval?</p>
</blockquote>
</aside>
<p>No.</p>
<ul>
<li>You want warmup, probably.</li>
<li>You don’t want to use time.time for timing. Use time.perf_counter. Or use Jupyter/IPython and %timeit</li>
<li>One thing to keep in mind with GPU benchmarking is to do the syncing before looking at the wall clock (before start and before end).</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

username: savvas17
cooked: <p>Thank you <a class="mention" href="/u/tom">@tom</a> . One more question about this: if I want to get both the <code>min</code> <strong>and</strong> the <code>max</code> as fast as possible which way is most efficient:</p>
<ul>
<li>Use <code>Tensor.min()</code> and then <code>Tensor.max()</code> (separately)</li>
<li>Or, sort the entire tensor <strong>once</strong> to and then look at the first and last element in the result</li>
</ul>
<p>Kind regards</p>

username: tom
cooked: <p>It probably depends on the problem size, sorting is a much more expensive operation than min/max, and you will feel that with large inputs.<br>
On the other hand, for small problems, you would probably not overthink optimizing this, anyways.<br>
You might also see if the advanced jit fusers optimize this, I think they are capable of this, but I don’t know if they do it.</p>
<p>If you don’t need gradients, there also is the relatively new, undocumented <code>torch._aminmax</code>.</p>
How to know the memory that is allocated from your model on gpu?

username: seyeeet
cooked: <p>is there a pytorchic way (command/code) to know the memory that the model is taking on gpu instead of doing nvidia-smi?</p>

username: ptrblck
cooked: <p>Yes, you could use <code>print(torch.cuda.memory_summary())</code>.</p>
How to extract learned weights by channel

username: James_Lee
cooked: <p>Follow this issue,</p><aside class="quote quote-modified" data-post="2" data-topic="4295">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexis-jacq/48/180_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-extract-learned-weights-correctly/4295/2">How to extract learned weights correctly</a> 
  </div>
  <blockquote>
    I can’t repeat your error, my idea is that your problem comes from your data set: 
model = Net()
optimizer = optim.Adam(model.parameters(), 0.01)
criterion = nn.MSELoss()

def train(epochs):
    model.train()
    for epoch in range(1, epochs+1):
        # I dont have your data set:
        data = Variable(torch.rand(1,100))
        target = Variable(torch.ones(1,100))
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
     …
  </blockquote>
</aside>

<p>Is there a way to get the sum of weights.data by channel dim? Thanks!</p>

username: ptrblck
cooked: <p>I assume you mean the <code>in_channel</code> dimension of a conv layer by “channel dim”?<br>
If so, this would work:</p>
<pre><code class="lang-python">conv = nn.Conv2d(3, 6, 3, 1, 1)
conv.weight.sum([0, 2, 3])
&gt; tensor([-0.2552,  0.2478,  0.3706], grad_fn=&lt;SumBackward1&gt;)
</code></pre>

username: James_Lee
cooked: <p>Thanks for your reply. Actually, I would like to explore the importance of the channel-wise features by summing up the channel-wise weights for a  group=3 Conv.</p>
<p>I tried to calculate a group=3 3x3 conv’s sum of weights by this,</p>
<pre><code class="lang-auto">            mid1_1 = net.conv1[-3].weight.data.shape[1]//3
            mid1_2 = net.conv1[-3].weight.data.shape[1]//3*2
            conv1_g1_ws = torch.sum(net.conv1[-3].weight.data[:mid1_1, :mid1_1, :, :])
            conv1_g2_ws = torch.sum(net.conv1[-3].weight.data[mid1_1:mid1_2, mid1_1:mid1_2, :, :])
            conv1_g3_ws = torch.sum(net.conv1[-3].weight.data[mid1_2:, mid1_2:, :, :])
</code></pre>
<p>Am I right? Many thanks for your advice.</p>
CrossEntropyLoss only calculates for the node of the class of the label but not others?

username: dhkleung
cooked: <p>I know that the CrossEntropyLoss requires logits.</p>
<p>My problem is that from what I learnt from textbook or from courses, for each data sample, cross entropy should calculate for each output node then sum them up.</p>
<p>The following assumes that we have one data sample passed to the network and we have the output logit z vector. We have three classes. The label for this data sample is 0.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af2b613b80b4e8ae82662225d948545c44e99f77.png" data-download-href="https://discuss.pytorch.org/uploads/default/af2b613b80b4e8ae82662225d948545c44e99f77" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af2b613b80b4e8ae82662225d948545c44e99f77.png" alt="Capture" data-base62-sha1="oZCuSTx5gYeJXKunznGF9Ww6S9x" width="690" height="420" data-dominant-color="F6F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1020×621 33.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why it only calculates for the node of the label’s class? but not summing up all the losses of each output node?</p>

username: ptrblck
cooked: <p>It seems you are trying to apply the binary cross entropy loss formula to the multi-class <code>nn.CrossEntropyLoss</code>, which won’t work.<br>
Instead of using the <code>-[1 * (...) + (1 - y) * (...)]</code> approach (which would be correct for the binary use case) you would have to multiply the targets directly as <code>[y[0] * (...) + y[1] * (...) + y[2] * (...)]</code>, which would then reduce to <code>y[0] * (...)</code> only since only a single class is active.</p>
Does the Python virtual environment not support C ++ extensions

username: nobody
cooked: <p>I write a cpp extension demo like <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">Custom C++ and CUDA Extensions</a>, but much simpler.</p>
<p>In a physical python environment, it work well, but when I switched to a python virtual environment (<code>python -m venv venv</code>), it raise error. <code>setuptools</code> and <code>Just in Time (JIT)</code> are both. The former will report an error during installation, while the latter will report an error during runtime.</p>
<pre><code class="lang-auto">ImportError: No module named 'test_cpp'
</code></pre>
<p>I setting <code>verbose=True</code> in JIT mode. I see the directory where the cpp extension was exists, but there is no <code>.o</code> file in the directory.</p>
<p>I don’t know why. Is it that the Python virtual environment does not support cpp extensions? or do I need some extra settings?</p>

username: ptrblck
cooked: <p>I don’t believe the virtual environment is itself causing the issue, but I guess packages are not installed correctly.<br>
Could you post a minimal code snippet reproducing the issue raised only in a virtual env?</p>

username: nobody
cooked: <p>Thanks for your answer. I have located the problem, mainly because some environment variables are not set correctly, and I made the wrong modification on <code>cpp_extension.py</code> according to a blog on the Internet.</p>
How to divide a batch of matrices by a batch of scalars?

username: AlphaBetaGamma96
cooked: <p>Hi everyone,</p>
<p>I just have a general question. Let’s say I have a batch of matrices of shape <code>matrices = [1000, 4, 8]</code> and I have a batch of scalars <code>scalars = [1000]</code>, how can I divide the i-th matrix in <code>matrices</code> with the i-th scalar in <code>scalars</code>?</p>
<p>I could see a potential way to use <code>torch.stack</code> but it’d be way too slow as I need the operation  to be vectorized! I’m sure I’ve missed something here, any help is apprecitated! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: AlphaBetaGamma96
cooked: <p>I’ve just tried this and I think I’ve done it. Can someone confirm this is correct?</p>
<pre><code class="lang-auto">x = torch.randn(1000, 4, 8)
y = torch.randn(1000)

xnew = x/y[:, None, None]
</code></pre>

username: ptrblck
cooked: <p>Yes, your approach should work fine and you could double check it with a defined examples, such as:</p>
<pre><code class="lang-python">x = torch.arange(1, 5)[:, None, None].expand(-1, 4, 8).float()
y = torch.arange(1, 5).float()

xnew = x/y[:, None, None]
print(xnew)
</code></pre>
Copying pytorch weights to tensorflow weights for deconvolution

username: hcleung3325
cooked: <p>Hi! I convert weights from a pytorch model to tf2 model. For the conv2d, each feature map from tensorflow model is as same as that of pytorch. Thus, the conv2d are ok. However, for the deconvolution, the output of tensorflow deconvolution layer has black bars on the top and left hand side, whereas the pytorch one does not have these black bars. And it seems that the image is shift to right and down by 2 pixels.<br>
Is that the mechanism of devolution in pytorch and tensorflow are different?<br>
Here is the pytorch code:<br>
<code>nn.ConvTranspose2d(56, 1, kernel_size=9, stride=4, padding=9//2, output_padding=4-1)</code></p>
<p>tensorflow code:<br>
tf.keras.layers.Conv2DTranspose(1, 9, strides=4, padding=‘same’)(x)</p>
<p>The conversion weights code from pytorch to tensorflow:</p>
<pre><code class="lang-auto">onnx_1_w_num = onnx_l.weight.data.permute(2,3,1,0).numpy()
onnx_1_b_num = onnx_l.bias.data.numpy()
tf_l.set_weights([onnx_1_w_num,onnx_1_b_num])
</code></pre>
<p>Please help. Thanks.</p>
Dataloader for multi label data in Pytorch

username: Myailab
cooked: <p>Hi,<br>
My data has multi labels in range of 1 to 4 labels per image.<br>
I have been using one hot encoding of labels to obtain dataloader.</p>
<p>I am after few customised loss functions now, such as → <br>
class ArcFaceLoss(nn.modules.Module):<br>
def <strong>init</strong>(self, s=30.0, m=0.5):<br>
super().<strong>init</strong>()<br>
self.crit = nn.CrossEntropyLoss()<span class="hashtag">#DenseCrossEntropy</span>()<span class="hashtag">#BCEWithLogitsLoss</span>()#<br>
self.s = s<br>
self.cos_m = math.cos(m)<br>
self.sin_m = math.sin(m)<br>
self.th = math.cos(math.pi - m)<br>
self.mm = math.sin(math.pi - m) * m</p>
<pre><code>def forward(self, logits, labels):
    logits = logits.float()
    cosine = logits
    sine = torch.sqrt(1.0 - torch.pow(cosine, 2))
    phi = cosine * self.cos_m - sine * self.sin_m
    phi = torch.where(cosine &gt; self.th, phi, cosine - self.mm)

    output = (labels * phi) + ((1.0 - labels) * cosine)
    output *= self.s
    loss = self.crit(output, labels)
    return loss / 2
</code></pre>
<p>It seems, this loss function doesn’t take labels in one hot encoding, if i drop it, then i get an error<br>
‘RuntimeError: stack expects each tensor to be equal size, but got [1] at entry 0 and [3] at entry 1’</p>
<p>May i know if there is any other way to obtain dataloader for multilabel data. Thank you.</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects class indices in the range <code>[0, nb_classes-1]</code> instead of one-hot encoded tensors, so <code>loss = self.crit(output, labels)</code> would raise an error.<br>
However, I guess the posted error message is raised even before reaching this line of code, so could you check the line of code raising it and then the shapes of all used tensors in this operation?</p>

username: Myailab
cooked: <p>Thanks for the inputs.<br>
This error appears at the execution of  def <strong>getitem</strong>(self,index) function for the dataloader.<br>
It is because images in the data set has different number of labels, and therefore they have different length. I am looking for the alternate ways to encode labels so that i can use other loss functions than nn.BCEWithLogitsLoss().<br>
Thanks</p>

username: ptrblck
cooked: <p>I’m not sure why the targets would have a different length, since also one-hot encoded targets should contain values for all classes (0 for inactive, 1 for active classes).<br>
Could you explain the targets and their shape a bit more?</p>

username: Myailab
cooked: <p>Yes, sure.<br>
If I use one hot encoding for targets, then there is no such error because the length of the vector is same.<br>
But if i drop this encoding and just use image labels as such, for example:<br>
Image Number   1,     Label: (0, 4)<br>
Image Number   2,    Label: (1, 3, 4)<br>
Image Number   3,    Label: (1, 5, 7, 8)</p>
<p>Then the error appears.</p>
<p>Thanks</p>
How can I reorder columns to put padding at the end?

username: erip
cooked: <p>I have a tensor with values I’d like to ignore. Sometimes these values appear in the middle of the tensor. For example, given</p>
<pre><code class="lang-python">tensor([[1, 2, 3, 4, 5],
        [1, 2, 3, 3, 5]])
</code></pre>
<p>I might want to ignore 3. Is there some way to filter out the 3s while padding as necessary? For instance,</p>
<pre><code class="lang-python">tensor([[1, 2, 4, 5],
        [1, 2, 5, -1]])
</code></pre>
<p>My thought is to use <code>torch.where</code>, but this only updates the tensor’s ignore-value without changing the shape.</p>

username: ptrblck
cooked: <p>This approach would need a loop, but unfortunately I cannot come up with another (better) approach right now:</p>
<pre><code class="lang-python"># setup
x = torch.tensor([[1, 2, 3, 4, 5],
                  [1, 2, 3, 3, 5]])

# create mask for valid values
mask = x!=3
# calculate max. valid elements per row
nb_elements = mask.sum(1).max()
# create output tensor with padding values
pad_val = -1
out = torch.full((x.size(0), nb_elements), fill_value=pad_val)

# assign values to out
for idx, (x_, m) in enumerate(zip(x, mask)):
    tmp = torch.masked_select(x_, m)
    out[idx, :len(tmp)] = tmp

print(out)
&gt; tensor([[ 1,  2,  4,  5],
          [ 1,  2,  5, -1]])
</code></pre>
How to retrain/fine-tune based on the pre-trained model

username: 111179
cooked: <p>Hello~</p>
<p>I try to initialize the parameters of my model with the pre-trained model.<br>
I am not sure how to retrain/fine-tune based on the pre-defined model.<br>
Is the following code correct?</p>
<pre><code class="lang-auto">def resnet18b_fq_lq_tern_tex_4(num_classes=1000):
    model = ResNet(BasicBlock, [2, 2, 2, 2], fp_fl=False, fp_ll=False)
    model.load_state_dict(model_zoo.load_url(model_urls['resnet18']))
    return model
</code></pre>
<p>Thank you so much.</p>

username: ptrblck
cooked: <p>I don’t know what the additional <code>fp_</code> arguments mean, but would assume that the parameter loading should work as long as you don’t get any errors pointing to missing or unexpected keys in the <code>state_dict</code>.</p>
Is it possible to use DistributedDataParalel inside of a class?

username: Konsthr
cooked: <p>Hi there!</p>
<p>I’m using PyTorch for gradient based inverse problem solving. My current architecture utilizes custom  <em>Trainer</em> class which takes model, dataset, optimizer, regularization function …etc. and handles all the training/optimization in its method.  It currently wraps the model into the <em>DataParallel</em> object to utilize multiple available GPUs.</p>
<p>Now, I want to switch it to the <em>DistributedDataParallel</em> to further speed up the process and maybe scale it to multiple nodes. However,  I’ve figured out that I cant do it in the class method, since <em>DistributedDataParallel</em> requires the process group creation, which can be don only in    __main__ and I’m basically need to write separate train script to use the <em>DistributedDataParallel</em>?</p>
<p>Am I understood this correctly ? ( I am not super experienced with torch.multiprocessing)<br>
Are there maybe some ways to use per-created processes group and/or <em>DistributedDataParallel</em> objects to unify the train process with <em>DistributedDataParallel</em> and <em>DataParallel</em>.</p>
<p>Thanks for your answer.</p>

username: pritamdamania87
cooked: <p>In general we recommend using DistributedDataParallel over DataParallel. Yes, the setup for DistributedDataParallel is indeed different from DataParallel since it requires setting up process group and also multiple processes. My recommendation would be to just have one version of your training script and only use DistributedDataParallel.</p>

username: seungjun
cooked: <p>Hi <a class="mention" href="/u/konsthr">@Konsthr</a>,</p>
<p>There is no restriction where DDP is stated. Whether using DDP in a class or outside is not a problem.<br>
You can refer to several examples including <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch" rel="noopener nofollow ugc">mine</a>.<br>
The following functions are related.<br>
<code>option.py</code> - <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch/blob/master/src/option.py#L245" rel="noopener nofollow ugc">setup</a><br>
<code>model/__init__.py</code> - <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch/blob/master/src/model/__init__.py#L38" rel="noopener nofollow ugc">parallelize</a></p>
<p>p.s. My implementation allows both DP and DDP.</p>
Unable To Install pytorch

username: Johnsprenger
cooked: <p>Hello friends. I can’t install pytorch on my Windows 10 computer. Just showing “error opening file for writing” error. I don’t understand why this error is showing and how to solve this problem. Any suggestions will be appreciated.</p>

username: ptrblck
cooked: <p>Could you check, if the entire error message mentions a specific folder and then check the permissions in it, as you might be running into a permission issue where Python isn’t allowed to move files to the desired location?</p>

username: William_Graham
cooked: <p>Hello. Just follow the link below:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://digicruncher.com/error-opening-file-for-writing/">
  <header class="source">
      <img src="https://digicruncher.com/wp-content/themes/outreach-pro/images/favicon.ico" class="site-icon" width="64" height="64">

      <a href="https://digicruncher.com/error-opening-file-for-writing/" target="_blank" rel="noopener nofollow ugc" title="02:41PM - 24 April 2020">DigiCruncher – 24 Apr 20</a>
  </header>

  <article class="onebox-body">
    <img src="https://digicruncher.com/wp-content/uploads/2020/03/Error_icon.png" class="thumbnail onebox-avatar" width="200" height="200">

<h3><a href="https://digicruncher.com/error-opening-file-for-writing/" target="_blank" rel="noopener nofollow ugc">Error Opening File For Writing – How To Fix ? [SOLVED]</a></h3>

  <p>If you are stuck with the issue error opening file for writing, then please go through this article to know about the various ways to get rid of the issue</p>

  <p>
    <span class="label1">Est. reading time: 9 minutes</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Decompose convolution of window size 5 by convolution filter window size 3

username: Hyeonuk_Woo
cooked: <p>Is it okay to use convolution filter of window size 3 twice instead of using convolution filter of window size 5 once? I think use convolution filter of window size 3 twice require less number of parameters…<br>
(3x3)x2. = 18 &lt; (5x5) =25</p>
<p>Below is example code.</p>
<p><code>model = nn.Sequential( nn.Conv2d(in_dim, out_dim, kernel_size =5, stride =1, padding=2), nn.BatchNorm2d(out_dim), nn.ReLU())</code></p>
<p>and</p>
<p><code> model = nn.Sequential( nn.Conv2d(in_dim, out_dim, kernel_size =3, stride =1, padding=1), nn.Conv2d(in_dim,out_dim, kernel_size=3, stride =1 ,padding=1), nn.BatchNorm2d(out_dim), nn.ReLU())</code></p>
How to convert a dense matrix to a sparse one?

username: chenchr
cooked: <p>Hello. I need to construct a sparse matrix, can anyone tell me how to convert a dense matrix to a sparse one, or getting the index from a dense matrix? Thanks!</p>

username: ezyang
cooked: <p>This should have a library function to handle this, but here’s how you can do it:</p>
<pre><code>dense = torch.randn(3,3)
dense[[0,0,1], [1,2,0]] = 0 # make sparse
indices = torch.nonzero(dense).t()
values = dense[indices[0], indices[1]] # modify this based on dimensionality
torch.sparse.FloatTensor(indices, values, dense.size())</code></pre>

username: justheuristic
cooked: <p>To reformat ezyang’s answer as a simple function,</p>
<pre><code class="lang-auto">def to_sparse(x):
    """ converts dense tensor x to sparse format """
    x_typename = torch.typename(x).split('.')[-1]
    sparse_tensortype = getattr(torch.sparse, x_typename)

    indices = torch.nonzero(x)
    if len(indices.shape) == 0:  # if all elements are zeros
        return sparse_tensortype(*x.shape)
    indices = indices.t()
    values = x[tuple(indices[i] for i in range(indices.shape[0]))]
    return sparse_tensortype(indices, values, x.size())
</code></pre>

username: Rakshit_Kothari
cooked: <p>This might seem like a silly question but is there anyway we could convert a tensor to sparse and retain it’s <code>requires_grad</code> property? This is to ensure it can be backpropped within a network <code>.backward()</code> call</p>

username: HONGJIN_LYU
cooked: <p>Why has no one mentioned the to_sparse method?</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.Tensor.to_sparse.html" rel="noopener nofollow ugc">torch.Tensor.to_sparse.html</a></p>
torch.sparse.FloatTensor not found

username: Anthony_Sirico
cooked: <p>I pulled a program from github, and the creator used torch.sparse.FloatTensor, and my versions are telling me that it cannot be found. Is there an alternative?</p>

username: ptrblck
cooked: <p>Which PyTorch version are you using? I can use this class in a pre-1.9.0 build as well as the current master.</p>

username: Anthony_Sirico
cooked: <p>My version is 1.8.1.</p>

username: ptrblck
cooked: <p>That’s really weird, since <code>torch.sparse.FloatTensor</code> was already used ~3 years ago as seen in e.g. <a href="https://github.com/pytorch/pytorch/pull/5464/files">this PR</a>.<br>
Could you try to reinstall PyTorch or update to the latest release?</p>

username: Anthony_Sirico
cooked: <p>I fixed it, it had to do with my data. Thanks though</p>
Does pytorch support calculate loss on model A and calculate grad on Model B

username: XJ_LI
cooked: <p>I want to achieve the function as:<br>
output = modelA(input)<br>
loss = loss_function(output, label)<br>
calculate gradient and update params on ModelB</p>
<p>modelA and modelB have the same architecture.<br>
Can I solve this? and how.</p>

username: ptrblck
cooked: <p>Yes, you could manually copy the <code>.grad</code> attributes from <code>modelA</code> to <code>modelB</code> and update <code>modelB</code>'s parameters with its <code>optimizer</code>.</p>

username: XJ_LI
cooked: <p>Thanks for your advice, I have tried that. Specifically, I wonder if there is a method that the loss is calculated on modelA , and the BP of loss can be calculated on modelB.</p>

username: ptrblck
cooked: <p>I don’t think it’s possible to “redirect” the computation graph to another model, so my guess would be to use the approach of copying the gradients.</p>

username: XJ_LI
cooked: <p>Thanks for your advice. If convenient, may you help me to reproduce the result in another my topics and solve it. <a href="https://discuss.pytorch.org/t/use-the-same-data-but-get-different-loss/126736" class="inline-onebox">Use the same data, but get different loss</a></p>
Esimate bounding box coordinates from the pictures with data and noise

username: SuperBald
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee831fa3f5b81edd19ff7b8a90e5eef1ccc2cbfd.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee831fa3f5b81edd19ff7b8a90e5eef1ccc2cbfd" title="(3)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee831fa3f5b81edd19ff7b8a90e5eef1ccc2cbfd_2_690x267.png" alt="(3)" data-base62-sha1="y1Yzh8CfW4rIDpVmiT31IjcKzgx" width="690" height="267" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee831fa3f5b81edd19ff7b8a90e5eef1ccc2cbfd_2_690x267.png, https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee831fa3f5b81edd19ff7b8a90e5eef1ccc2cbfd.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee831fa3f5b81edd19ff7b8a90e5eef1ccc2cbfd.png 2x" data-dominant-color="262621"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">(3)</span><span class="informations">824×320 73.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>A n00b question. Just start learning PyTorch. The left side of the picture is generated from electrical devices. From human’s eyes, two bounding boxes correspond to the actual data; this part of the picture has higher altitude and is more dense. In comparison, the noise corresponds to the part with lower altitude.</p>
<p>I would like to extract the coordinates of two bounding boxes. Wonder if this is doable in PyTorch? Any code pointers, similar examples, etc? Thanks a lot!</p>
How to make CPU->GPU faster?

username: amsword
cooked: <p>Setup:</p>
<ul>
<li>
<p>the project is vision problem to process images.</p>
</li>
<li>
<p>the data loader is based on the following implementation so that the data copying has already been overlapped with network forward/backward, as my understanding<br>
<a href="https://github.com/HenryJia/Lighter/blob/master/lighter/train/loaders.py" rel="noopener nofollow ugc">Lighter/loaders.py at master · HenryJia/Lighter (github.com)</a></p>
</li>
<li>
<p>if the image data is fp32, the speed is 2000 images per sec; if it is fp16, it is 2500 images per sec. Thus, I assume the data copying from cpu to gpu takes a lot of time. In the meanwhile, data loading time is super small, so the CPU data loading part should not be a problem. The batch size for each GPU is 32; image size is 640x640.</p>
</li>
</ul>
<p>Problem:</p>
<ul>
<li>
<p>One question is that as non_blocking=true is used, is there a way to check whether the data is ready for forward/backward in GPU so that I can confirm that cpu-&gt;gpu is indeed a bottleneck.</p>
</li>
<li>
<p>Another question is that whether there is a faster way to move data from CPU to GPU.</p>
</li>
</ul>

username: JuanFMontesinos
cooked: <p>Which image format are you using?<br>
You can think about saving raw arrays or using gpu decoding if you work with jpg images.<br>
Another suggestion is you to try DALI as it’s the fastest dataloader you can use.</p>

username: amsword
cooked: <p>it is jpg format. Thanks for pointing DALI, and I will look at this.</p>
Plot all loss values for wgan moedel

username: anoud_abdalziz
cooked: <p>I asked for how I can  plot  the model losses  with number of iterations<br>
that my train_losses.append([g_loss.item(), d_loss.item()-gp_loss.item(), gp_loss.item()])<br>
neither only plottting generator loss nor only disciminator  loss<br>
hope plot the loss values for all model loss .<br>
is that correct   plt.plot(np.array(train_losses), ‘r’)?</p>

username: ptrblck
cooked: <p>Yes, your approach might work, as <code>matplotlib</code> should be smart enough to be able to plot a <code>list</code> containing multiple values in each element:</p>
<pre><code class="lang-python">a = []
for _ in range(10):
    a.append([torch.randn(1).item(), torch.randn(1).item(), torch.randn(1).item()])

plt.plot(a)
</code></pre>

username: anoud_abdalziz
cooked: <p>Thanks, I realized and Solved my issue.</p>
Finite difference method in DARTS code

username: promach
cooked: <p>For my <a href="https://gist.github.com/promach/b6f526c56e20f029d68e6f9041c3f5c0#file-gdas-py-L373-L374" rel="noopener nofollow ugc">DARTS coding</a> , how to do the coding for <em>Ltrain(w+)</em> and <em>Ltrain(w-)</em> inside <a href="https://arxiv.org/pdf/1806.09055.pdf#page=4" rel="noopener nofollow ugc">equation (8) of DARTS paper</a> ?</p>
<p>Note: Upon checking the definition of finite difference method at <a href="https://mythrex.github.io/math_behind_darts/" class="inline-onebox" rel="noopener nofollow ugc">MATH behind DARTS</a> , it seems that the above 2 lines of code are wrong.</p>
<p>but the question is how to modify these 2 lines of code ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4efebbef821bc6f3641a26de42b2485725376d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/f4efebbef821bc6f3641a26de42b2485725376d9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4efebbef821bc6f3641a26de42b2485725376d9_2_466x500.png" alt="image" data-base62-sha1="yWOwUOK3vd4XPMYCHgaH0Npc2SJ" width="466" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4efebbef821bc6f3641a26de42b2485725376d9_2_466x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4efebbef821bc6f3641a26de42b2485725376d9_2_699x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4efebbef821bc6f3641a26de42b2485725376d9_2_932x1000.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1000×1072 205 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Achieving desired size with 2D Transpose Convolution

username: AfonsoSalgadoSousa
cooked: <p>Hi. I have a feature map with size [N, 64, 248, 216] and would like to upsample it using 2D Transpose Convolutions to the size of [N, 64, 496, 432] (double dims 2 and 3).<br>
I have the following code extract:</p>
<pre><code class="lang-auto">feats = torch.randn([2, 64, 248, 216]).cuda()
decoder_block = nn.Sequential(nn.ConvTranspose2d(
        feats.size(1),
        64,
        3,
        stride=2,
        padding=1,
        bias=False),
        nn.BatchNorm2d(64, eps=1e-3, momentum=0.01),
        nn.ReLU(inplace=True),
    ).cuda()
decoded_feats = decoder_block(feats).cuda()
</code></pre>
<p>The output I get is [2, 64, 495, 431] instead of [2, 64, 496, 432]. How can I fix this? Changing the stride to 1 as in <a href="https://discuss.pytorch.org/t/question-of-2d-transpose-convolution/99419/3">this issue</a> does not fix this problem. I really appreciate any help you can provide.</p>

username: AfonsoSalgadoSousa
cooked: <p>I have the same problem for a similar use case where I want to convert a feature map with size [2, 64, 128, 400] to [2, 3, 512, 1600]. Using the following code, I get an output shape of [2, 3, 511, 1599]:</p>
<pre><code class="lang-auto">img_decoder_block = nn.Sequential(nn.ConvTranspose2d(
        img_feats.size(1),
        int(img_feats.size(1) // 2),
        3,
        stride=2,
        padding=1,
        bias=False),
        nn.ConvTranspose2d(
        int(img_feats.size(1) // 2),
        3,
        3,
        stride=2,
        bias=False),
        nn.BatchNorm2d(3, eps=1e-3, momentum=0.01),
        nn.ReLU(inplace=True),
        nn.Tanh(),
    ).cuda()
decoded_img = img_decoder_block(img_feats).cuda()
</code></pre>

username: AfonsoSalgadoSousa
cooked: <p>The solution for my problem was to use the <em>output_padding</em> parameter. Setting it to 1 resolves the ambiguity of the mapping of multiple input shapes to the same output shape.</p>
Input of the ctc_loss

username: Jack7
cooked: <p>I have a model which outputs/predicts from an image (a word) to a label tensor<br>
[1,64] = 119,111,114,100,0,0,…0,0,0,0:  stands for “word”</p>
<p>How i transform that tensor for the input of the ctc_loss(input, …, …). It needs the number of the classes in the tensor i think.</p>
Should I use a list of object on GPU?

username: weiguowilliam
cooked: <p>I just started to use Pytorch. My question is that for object o_i that is already on GPU (which means I used <code>.to(device)</code> already.), should I use a list of o1…on? Is it efficient? Thanks in advance.</p>

username: AfonsoSalgadoSousa
cooked: <p>For your object to be in the GPU, it should be a tensor. If you have multiple objects you would like to pass to the GPU, you could stack them in a multi-dimensional tensor. A minimal example is below:</p>
<pre><code class="lang-auto">t1 = torch.tensor(1) # tensor(1)
t2 = torch.tensor(2) # tensor(2)
t3 = torch.tensor(3) # tensor(3)
t_list = torch.cat([t1.view(1), t2.view(1), t3.view(1)])
Out: tensor([1, 2, 3])
</code></pre>
<p>The .view serves for the unique case of 0-dimensional tensors to make them 1D. For higher dimensions it can be omitted.</p>

username: weiguowilliam
cooked: <p>Thank you. So what if the object is a neural network model? I use <code>to(device)</code> to the network model.</p>
nn.AdaptiveAvgPool2d reference

username: Monne
cooked: <p>How exactly <code>nn.AdaptiveAvgPool2d</code> is implemented on PyTorch? any paper or journal for references?</p>
<p>Thanks.</p>

username: AfonsoSalgadoSousa
cooked: <p>Hi. You can visit the <a href="https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html" rel="noopener nofollow ugc">official documentation</a> and check the “SOURCE” implementation on the upper right corner. Visit the source code for the Pytorch’s Functional package (<a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#adaptive_avg_pool2d" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional — PyTorch 1.9.0 documentation</a>).</p>

username: Monne
cooked: <p>Hi, thanks for reply. I’m already do that but i’m still curious about the referal of implementation. Like <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">LSTM documentation</a> it is stated where implementation come from, or in <a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html" rel="noopener nofollow ugc">Dropout documentation</a> it’s also stated. did i miss something? especially for AdaptiveAvgPool2d</p>
How to copy cuda memory to or from cuda tensor?

username: ItalianSC
cooked: <p>I create a cuda tensor use code like below:</p>
<p>auto my_tensor = torch::ones({1,3,512,512},torch::device(torch::kCUDA,0));</p>
<p>so how can I copy the data in a cuda memory to a cuda tensor ,or copy from cuda tensor to cuda memory directly?  What I want is to be able to complete the copy inside GPU without having to do GPU-&gt;CPU-&gt;GPU copy.</p>

username: ItalianSC
cooked: <p>Can I use cudaMemcpy() directly?</p>

username: tom
cooked: <p>I’m not sure what you want to do here, but there are several ways to do things:</p>
<ul>
<li>use <code>from_blob</code> with a pointer to cuda memory,</li>
<li>create the cuda tensor and use the <code>tensor.data_ptr()</code> as a destination,</li>
<li>create a cuda tensor and use <code>tensor.copy_(src)</code>
</li>
</ul>

username: ItalianSC
cooked: <p>thanks tom,you said I can create a cuda tensor and use the tensor.data_ptr() as a destination, so can I be sure the return of datr_ptr()  from a cuda tensor is a cuda memory address ?</p>
<p>I had tried to use the data_ptr() of a created cuda tensor as the destination or source pointer of cudaMemcpy() before, it works well, but I’m not sure it is valid;</p>

username: tom
cooked: <aside class="quote no-group" data-username="ItalianSC" data-post="4" data-topic="126650">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/91b2a8/48.png" class="avatar"> ItalianSC:</div>
<blockquote>
<p>thanks tom,you said I can create a cuda tensor and use the tensor.data_ptr() as a destination, so can I be sure the return of datr_ptr() from a cuda tensor is a cuda memory address ?</p>
</blockquote>
</aside>
<p>Yes, this is what PyTorch uses internally, too.</p>
How to make my models agnostic to FX

username: Alexander_Soare
cooked: <p>I’m working on an FX transform (or set of transforms) that I intend to be able to work with a whole library of CNN and transformer based models for the purpose of feature pyramid extraction.</p>
<p>As such, I want to <em>minimize</em> (so, a bit relaxed vs <em>totally avoid</em>) any overhead in changing the existing models, and ensuring future models are symbolically traceable.</p>
<p>I’m aware of the <a href="https://pytorch.org/docs/stable/fx.html#limitations-of-symbolic-tracing" rel="noopener nofollow ugc">limitations of symbolic tracing</a> and just wanted to see if there are ways I can get around them without touching the models (much).</p>
<p>For instance:</p>
<ul>
<li>
<p>Tensor constructors with dynamic args. Many models might expect a fixed input size, so I tried using the<code>concrete_args</code> argument but I found that this “concreteness” of the input gets lost after it passes through a non-custom module.</p>
</li>
<li>
<p>If I can get <code>concrete_args</code> working, could I just rerun the trace with multiple input sizes (whatever I might expect for a particular model) and thereby build up the various dynamic flow control paths that I would expect into the final IR graph?</p>
</li>
<li>
<p>Could I very specifically customize the tracer class for each model to just do whatever I need it to do at a particular problematic node?</p>
</li>
</ul>
<p>These are examples but overall my questions are: <strong>What types of things can I do to get around symbolic tracing limitations without modifying my models? Is it worth it or would I be better off putting more responsibility on the models? Will these answers change dramatically in the near future given developments to the FX library?</strong></p>
High Loss RPN Box Regression in Mask R-CNN

username: aliceinland
cooked: <p>I’m using the Mask R-CNN resnet-50 as backbone model. The value of the “loss_rpn_box_reg” is very high (around 4/5).<br>
There are 256 hidden layers, the value of the learning rate is low since the model converge better with low learning rate.<br>
The bounding box are correctly given to the network, in the format (x1, y1, x2, y2) .<br>
Any idea on how I could reduce this loss value?</p>