akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Difference between torch.sum and addition operator

username: samiksome92
cooked: <p>I have noticed that the results of <code>torch.sum</code> and the addition operator are not always equal.</p>
<p>For example:</p>
<pre><code class="lang-python">import torch

def f():
    a = torch.rand(100).view(10,10)
    s1 = torch.sum(a, dim=0)
    s2 = torch.zeros(10)
    for i in range(10):
        s2 += a[i]
    assert (s1 == s2).all()

for i in range(1000):
    f()
</code></pre>
<p>Running the above code results in assertion failure. If I check the difference between <code>s1</code> and <code>s2</code>, they are generally of the order of <code>1e-7</code></p>
<p>Similar numpy code however seems to not have this issue.</p>
<pre><code class="lang-python">import numpy as np

def f():
    a = np.random.rand(100).astype(np.float32).reshape((10,10))
    s1 = np.sum(a, axis=0)
    s2 = np.zeros(10, dtype=np.float32)
    for i in range(10):
        s2 += a[i]
    assert (s1 == s2).all()

for i in range(1000):
    f()
</code></pre>
<p>I have run this several times and haven’t had any assertion errors so far.</p>
<p>Is this behavior simply because of floating point precision or is there something else that is causing this issue?</p>

username: JuanFMontesinos
cooked: <p>Comparing 2 float numbers with the equality operator is a bad practice in any coding language.<br>
If you use a proper way of comparing floats such us <a href="https://pytorch.org/docs/stable/generated/torch.allclose.html" class="inline-onebox" rel="noopener nofollow ugc">torch.allclose — PyTorch 1.12 documentation</a> or <a href="https://numpy.org/doc/stable/reference/generated/numpy.allclose.html" class="inline-onebox" rel="noopener nofollow ugc">numpy.allclose — NumPy v1.23 Manual</a> you will find both are correct given the precision of fp32.</p>
Understanding ROI Pool of fast r-cnn

username: basingse
cooked: <p>Hello all,</p>
<p>I am facing difficulty when understanding the roi pool layer that was implemented in torchvision <code>torchvision.ops.roi_pool</code>. I have attached the sample code below</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torchvision

import numpy as np
feature_map = np.array([
    [0.70, 0.41, 0.38, 1.23, 0.24],
    [0.14, 0.45, 0.31, 0.73, 3.22],
    [0.11, 0.41, 0.79, 0.69, 0.44],
    [1.47, 0.25, 0.09, 0.32, 2.98],
    [0.48, 0.87, 0.77, 0.26, 0.11],
])


feature_map = torch.tensor(feature_map, requires_grad=False).float()

# (batch, channel, h, w) -&gt; (1, 1, 5, 5)
feature_map = feature_map.unsqueeze(0).unsqueeze(0)

# boxes -&gt; (1, 5)
boxes = np.array([
    [0, 0, 0, 4, 4],
])
boxes = torch.tensor(boxes, requires_grad=False).float()

# roi pooling layer of 3x3
pool = torchvision.ops.roi_pool(input=feature_map, boxes=boxes, output_size=3)
print(pool)
</code></pre>
<p>The output of the above code is</p>
<pre data-code-wrap="bash"><code class="lang-bash">tensor([[[[0.7000, 1.2300, 3.2200],
          [1.4700, 0.7900, 3.2200],
          [1.4700, 0.8700, 2.9800]]]])
</code></pre>
<p>So here the <code>5x5</code> map is divided in such a way that the output is <code>3x3</code>. So <code>5x5</code> region can be divided the following way to <code>3x3</code> (there are many ways but I only drew 2)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d525e5b614b150c8dc9e9629e89719799be518d.png" alt="image" data-base62-sha1="mrJr4NKRdHKnbCsItzKlUNUNVud" width="247" height="245"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79f814ac87b7a25e1a0b2f737486d09ca6e2a538.png" alt="image" data-base62-sha1="hoZhDBpn65YC9RFd6xdzT6kGRRu" width="238" height="254"></p>
<ul>
<li>How the subregions are decided in pytorch code (I checked the paper and there was no detail on this one). Is there any algorithm pytorch is referring to ?</li>
<li>My understanding says that there should be no overlap between sub-regions but that doesn’t explain the output of above code (as you can see <code>3.22</code> is present in (0, 2) and (1, 2) positions of output)</li>
</ul>
<p>The versions I am using are :</p>
<pre data-code-wrap="bash"><code class="lang-bash">torch = 1.10.2+cpu
torchvision = 0.11.3+cpu
</code></pre>
<p>Thanks.</p>
Same Noise vector to train generator and discriminator?

username: H-Sor
cooked: <p>Is there any advantage to using the same noise vector in the same iteration to train a GAN?</p>
<p>I’ve seen cases where 2 different noise vectors are created, one for disc forward pass and another for the generator.</p>
<p>I guess, aside from not having to detach the generator and not setting retain_graph=True,<br>
is there any impact on training?</p>
Trying to load a model in pytorch 1.6 results in _pickle.UnpicklingError: invalid load key, '\xad'

username: Shisho_Sama
cooked: <p>Hello everyone,<br>
I’m trying to load a model that was trained and saved in Pytorch1.5.1 in Pytorch1.6. the model loads just fine in 1.5.1 but it fails with the following error when I try to load it in Pytorch 1.6:</p>
<pre><code class="lang-auto">  File "d:\codes\fac_ver\python\FV\F_V.py", line 563, in _init_model
    checkpoint = torch.load(self.model_checkpoint_path, map_location=torch.device('cpu'))
  File "C:\Users\User\Anaconda3\Lib\site-packages\torch\serialization.py", line 585, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\User\Anaconda3\Lib\site-packages\torch\serialization.py", line 755, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\xad'.
</code></pre>
<p>Whats wrong here? how should I be loading this in Pytorch 1.6?<br>
Thanks a lot in advance</p>

username: ptrblck
cooked: <p>The error seems to be raised by <code>pickle</code> so you might need to change the encoding via:</p>
<pre><code class="lang-python">import pickle
pickle.load = partial(pickle.load, encoding="latin1")
pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1")
model = torch.load(model_file, map_location=lambda storage, loc: storage, pickle_module=pickle)
</code></pre>
<p>This would be necessary, if you’ve stored e.g. numpy arrays in Python2 and try to load it in Python3.<br>
However, PyTorch <code>1.5.1</code> didn’t support Python2.7 anymore, so I’m not sure if that’s the root cause of this issue.</p>

username: Shisho_Sama
cooked: <p>Thanks a lot really appreciate your kind help. The model was trained in 1.4 and not 1.5.1 and I have always used Python3.<br>
I’ll give that a try and see how it goes</p>

username: Shubham_kumar
cooked: <p>did you find a way to resolve the error</p>
Pytorch to keras

username: ldalf
cooked: <p>Hello everybody,<br>
I am sorry if the post is uncategorized but this is my first post in the forum and I am learning how it works.<br>
I need to convert a pytorch based CNN into a keras based one. In particular I need to convert the pytorch code</p>
<p>conv2 = torch.nn.Conv2d(32, 64, 3, 2, 1)<br>
x=torch.nn.functional.relu(self.conv2(x))</p>
<p>into a keras one. I tried with<br>
conv2 = keras.layers.Conv2D(64,3,strides=(2,2),padding=‘same’,activation=‘relu’,name=‘conv2’)</p>
<p>but when I call the keras layer and the pytorch one, starting from the same input data and with the same initialization for weights and biases, I obtain two tensors (one in pytorch and one in keras) with the same size but different values. Everything works if I set stride=1 in both the cases.</p>
<p>Does the stride work differently in keras and in pytorch?<br>
Does anyone have some suggestions for my issue?</p>
<p>Thank you.</p>

username: JWLee89
cooked: <p>Really late response, so you probably solved the problem but in case somebody else has similar problems</p>
<p>Could you try <code>pt2keras</code> and see if it works?<br>
To install <code>pt2keras</code>, type the following in the terminal:</p>
<pre data-code-wrap="shell"><code class="lang-nohighlight">pip install -U pt2keras
</code></pre>
<p>Below is an example for converting your model.</p>
<pre><code class="lang-auto">from copy import deepcopy

import numpy as np
import tensorflow as tf
import torch
from pt2keras import Pt2Keras


class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv2 = torch.nn.Conv2d(3, 64, (3, 3), (2, 2), 1)

    def forward(self, x):
        return torch.nn.functional.relu(self.conv2(x))


if __name__ == '__main__':
    input_shape = (1, 3, 224, 224)

    # Grab model
    model = Model().eval()

    # Create pt2keras object
    converter = Pt2Keras()

    # convert model
    keras_model: tf.keras.Model = converter.convert(model, input_shape, strict=True)

    x_pt = torch.randn(input_shape)
    # Generate dummy inputs
    x_keras = tf.convert_to_tensor(deepcopy(x_pt.numpy()))

    # input dimensions for PyTorch are BCHW, whereas TF / Keras default is BHWC
    if len(x_keras.shape) == 4:
        x_keras = tf.transpose(x_keras, (0, 2, 3, 1))

    print(f'pt shape: {x_pt.shape}, x_keras.shape: {x_keras.shape}')

    # Make PT model the same input dimension as Keras
    # If the output is &gt;= 4 dimensional
    pt_output = model(x_pt).cpu().detach().numpy()
    keras_output = keras_model(x_keras).numpy()
    if len(keras_output.shape) == 4:
        keras_output = keras_output.transpose(0, 3, 1, 2)
    # Mean average diff over all axis

    average_diff = np.mean(pt_output - keras_output)
    print(f'pytorch: {pt_output.shape}')
    print(f'keras: {keras_output.shape}')
    # The differences will be precision errors from multiplication / division
    print(f'Mean average diff: {average_diff}')
    print(f'Pytorch output tensor: {pt_output}')
    print(f'Keras output tensor: {keras_output}')
</code></pre>
MultiModal learning of losses - added losses of models of 2 different feature vector modalities stop converging after threshold

username: Tony363
cooked: <p>Sorry for the late/early day question. I’m trying to do multi-modal ensembling of losses from 2 different models. I get loss1 = criterion(x0,y0) , loss2 = criterion(x1,y1) and loss3 = loss1+loss2. I do optimizer.zero_grad();loss3.backward();optimizer.step()  …The loss function would stop converging after a threshold. Do you guys know any ways to continue reducing loss of these feature vectors of different modalities of data?<br>
I tried concatenating tensors of these different modalities of data, but the precision-recall isn’t performant.</p>
L2 regularization via torch.optim weight decay

username: JINGHAN_YANG
cooked: <p>When I add weight decay to achieve L2 regularization via torch.optim, what is the constant before alpha in the formula below, is it 2 * alpha or 1/2 * alpha or simply alpha? How can I find the official document for this?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/355f36f2f9598bb78bac5191372585094e5756ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/355f36f2f9598bb78bac5191372585094e5756ba" title="Screen Shot 2022-08-21 at 11.58.19 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/355f36f2f9598bb78bac5191372585094e5756ba_2_690x166.png" alt="Screen Shot 2022-08-21 at 11.58.19 AM" data-base62-sha1="7C9iKjcLvX2s8upcXBRb4A5VDU6" width="690" height="166" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/355f36f2f9598bb78bac5191372585094e5756ba_2_690x166.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/355f36f2f9598bb78bac5191372585094e5756ba_2_1035x249.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/5/355f36f2f9598bb78bac5191372585094e5756ba.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-08-21 at 11.58.19 AM</span><span class="informations">1108×268 52.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: JINGHAN_YANG
cooked: <p>I refer to the code, it seems is alpha<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9e8d7b65722197d9f95e6b5fd928a928b5947505.png" data-download-href="https://discuss.pytorch.org/uploads/default/9e8d7b65722197d9f95e6b5fd928a928b5947505" title="Screen Shot 2022-08-21 at 12.02.57 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9e8d7b65722197d9f95e6b5fd928a928b5947505_2_690x268.png" alt="Screen Shot 2022-08-21 at 12.02.57 PM" data-base62-sha1="mCCyYBCSXjcVpIcmuhgX04eZiAd" width="690" height="268" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9e8d7b65722197d9f95e6b5fd928a928b5947505_2_690x268.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9e8d7b65722197d9f95e6b5fd928a928b5947505_2_1035x402.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9e8d7b65722197d9f95e6b5fd928a928b5947505_2_1380x536.png 2x" data-dominant-color="323332"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-08-21 at 12.02.57 PM</span><span class="informations">1458×568 71.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
LSTM - multivariate prediction to predict single target

username: moot
cooked: <p>Hi</p>
<p>Does a sequence length of 1 make any sense on time series data, taking in multiple features to predict the output? Or should it be a sequence length of say a week followed by a day of prediction for one output?</p>
<p>If it is the latter, how does one go about implementing it in PyTorch? Any links would be nice to have - I have already gone through the docs for LSTM so please dont repost that.</p>
Can torch.jit.script model be reverse engineered?

username: Fei_Liu
cooked: <p>As title. The question regards source code security and IP protection. If I <code>torch.jit.script</code> a model and pass it to others, can the model be reverse engineered to python code (to a roughly-equivalent level, something like “the model called Linear(xx, xx) =&gt; ReLU() =&gt; Conv3d(xx, xx, xx) … in the forward step and the corresponding parameters are yy, zz, …” but not necessarily the exact original model source code) ? In other words, can I use it to encrypt my model so the others can only use it as a blackbox?</p>

username: marksaroufim
cooked: <p>Given a torchscripted model anyone can freely get the python code back for it by running <code>loaded = torch.jit.load("path/to/torchscripted-model.pt"</code> and then <code>model = loaded.code</code></p>
<p>I’ve seen a few encrypted ML packages pop up but haven’t personally used any to give a better recomemndation</p>
Why Pytorch can calculate the gradient of the loss function with quantiles?

username: Oliver_L
cooked: <p>Hi Guys</p>
<p>I am doing a project related to neural networks recently.</p>
<p>I was surprised to find that pytorch can calculate the gradient of loss function with quantiles, because the quantile calculation should be non differentiable.</p>
<p>I was asked to elaborate the exact principle of it, why Pytorch can calculate the gradient of loss function with quantiles?</p>
<p>The following is a screenshot of my customized loss function.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e77eccdd4dc325d353cf6d59e3613877f047f6c.png" data-download-href="https://discuss.pytorch.org/uploads/default/6e77eccdd4dc325d353cf6d59e3613877f047f6c" title="thumbnail_0e2409c8f4b33658c3208a4a053ea6e"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e77eccdd4dc325d353cf6d59e3613877f047f6c_2_690x183.png" alt="thumbnail_0e2409c8f4b33658c3208a4a053ea6e" data-base62-sha1="fLfsvTA0PUCnViqyZOnRYvQ6LkE" width="690" height="183" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e77eccdd4dc325d353cf6d59e3613877f047f6c_2_690x183.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e77eccdd4dc325d353cf6d59e3613877f047f6c_2_1035x274.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e77eccdd4dc325d353cf6d59e3613877f047f6c_2_1380x366.png 2x" data-dominant-color="EBEEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">thumbnail_0e2409c8f4b33658c3208a4a053ea6e</span><span class="informations">1564×416 56.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can anyone answer my question?</p>

username: ptrblck
cooked: <p>The implementation of <code>torch.quantile</code> should be defined <a href="https://github.com/pytorch/pytorch/blob/08c03c91d70a625f70487af81cd54edd5d16aa1a/aten/src/ATen/native/Sorting.cpp#L216-L337">here</a> and based on the used operations, I don’t see a reason why it should not be differentiable.<br>
It seems, internally a sorting and interpolation is used, which are both differentiable in PyTorch (in <code>torch.sort</code> only the sorted values with have a backward function, not the indices) so could you explain why it should not work?</p>
Why layer norm runs faster than var_mean

username: quancs
cooked: <p>Layer norm will first compute the var and mean of its input, so why the following code shows that layer norm is faster than <code>torch.var_mean</code>.</p>
<pre><code class="lang-python">from functools import partial
import timeit
import torch

x = torch.randn((257, 252, 192),dtype=torch.float32)
ln = torch.nn.LayerNorm(192)

ln.eval()
with torch.no_grad():
    var_mean_time = timeit.timeit(partial(torch.var_mean, input=x, dim=(2,)), number=100)
    ln_time = timeit.timeit(partial(ln, input=x), number=100)
    print(var_mean_time, ln_time) # 3.2 1.18
</code></pre>
Why does vectorization provide little to no speedup for matrix_exp?

username: SamRoss
cooked: <p>I’m trying to do some optimizations. My understanding is that, usually, it’s better to vectorize than to do loops. But this doesn’t seem to be the case for <code>matrix_exp</code>. Here are some tests:</p>
<pre><code class="lang-auto">matrices = [torch.rand(1000, 1000) for _ in range(16)]
</code></pre>
<p>Test using loops:</p>
<pre><code class="lang-auto">
%%timeit
for m in matrices:
    e = torch.matrix_exp(m)
2.24 s ± 48.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
</code></pre>
<p>Test with vectorization:</p>
<pre><code class="lang-auto">
%%timeit
torch.matrix_exp(torch.stack(matrices))
2.21 s ± 16.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
</code></pre>
<p>I would love to understand why there’s very little speed-up in this case, and if there’s anything else I can do.</p>

username: tom
cooked: <p>So I don’t know how matrix exp is implemented (it could just be looping over the items), but one general thought that applies to 1000x1000 matrices: Vectorization works best when the individual operations either cannot use all computational facilities (ie fully use the CPU/GPU) or the “administrative overhead” (e.g. creation of Tensor data structures with metadata) has a substantial computation. I would doubt that this is necessarily the case here.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Lezcano
cooked: <p>Late to the party here, but yeah, there’s a part of the implementation that’s basically sequential: <a href="https://github.com/pytorch/pytorch/blob/7f18ef14c1fed4e4376a75d626d98ba3c074809c/aten/src/ATen/native/LinearAlgebra.cpp#L2285-L2293" class="inline-onebox" rel="noopener nofollow ugc">pytorch/LinearAlgebra.cpp at 7f18ef14c1fed4e4376a75d626d98ba3c074809c · pytorch/pytorch · GitHub</a><br>
We could do better there by simply choosing the largest number in the whole batch and perform that many matmuls for the whole batch. I suspect this would work best in real world scenarios.</p>
Can't transfer numpy array to GPU

username: kbtorcher
cooked: <p>I have a numpy array which I would like to shift to gpu. I used the method: <code>from_numpy(numpy_array).float().to(device)</code>. However, I get the following error:</p>
<blockquote>
<p>To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor)</p>
</blockquote>
<p>This warning doesn’t make sense as <code>.clone()</code> method is not available for a numpy array. How can I move the said numpy array into a tensor on GPU?</p>

username: Lezcano
cooked: <p>The following code works for me:</p>
<pre><code class="lang-auto">import numpy as np
import torch
a = np.array([3., 2., 4.])
a_cuda = torch.from_numpy(a).to(device="cuda", dtype=torch.float32)
</code></pre>
<p>Note that I just use the <code>to</code> function to make the cast as well simply to avoid an extra copy.</p>
Local maximums of sub-tensors by index tensor

username: teshnizi
cooked: <p>Hello everyone!</p>
<p>I have a tensor <code>x</code> of shape <code>(1,n)</code>, and another index tensor <code>d</code> of shape <code>(1,k)</code>. I’m trying to find the maximums of <code>k</code> sub-tensors <code>x[0:d[0]], x[d[0]:d[1]], x[d[1]:d[2]], ..., x[d[-2]: d[-1]]</code>. So the output is a tensor of shape <code>(1,k)</code> with <code>k</code> local maximums. I can implement a for loop, but that’s too slow. Can I do it in parallel in PyTorch?</p>

username: teshnizi
cooked: <p>I found the answer. There is a <a href="https://pytorch-scatter.readthedocs.io/en/latest/functions/segment_csr.html" rel="noopener nofollow ugc">SegmentCSR</a> function in torch_scatter that does the job.</p>
How to get embeddings from MViT as encoder

username: over9k
cooked: <p>I am trying to use MViT as an encoder to get embeddings of an input to then pass into another model. I am looking at <a href="https://github.com/facebookresearch/pytorchvideo/blob/main/pytorchvideo/models/vision_transformers.py" rel="noopener nofollow ugc">https://github.com/facebookresearch/pytorchvideo/blob/main/pytorchvideo/models/vision_transformers.py</a> and the create_multiscale_vision_transformers function to build the mvit model. Something like this:</p>
<blockquote>
<p>model = create_multiscale_vision_transformers(spatial_size=100, temporal_size=10)</p>
</blockquote>
<p>Then, when I pass in a tensor, to the model like</p>
<blockquote>
<p>output = model(tensor)</p>
</blockquote>
<p>and get the shape</p>
<blockquote>
<p>output.shape</p>
</blockquote>
<p>I get a tensor of dim 400. This looks like the logits for the classification task.</p>
<p>The question is, I do I get the embeddings insteads of the logits. That way, I would be able to use the embeddings as features for another model. Is there a way to get the embedding projects to be of a certain dimension, like 1024?</p>
<p>Thank you for the help.</p>

username: azhanmohammed
cooked: <p>Going over the codebase for the MViT model you will see that the head layer towards the end is just a linear layer defined over <a href="https://github.com/facebookresearch/pytorchvideo/blob/5984809510df14c0eb37a8edc8d8f77e3fb4865e/pytorchvideo/models/head.py#L134" rel="noopener nofollow ugc">here</a>. This head layer can be directly used to create embeddings without making any changes in the model definition, just pass the embedding dimensions you want as <code>head_num_classes</code> here:</p>
<pre><code class="lang-auto">MViT_B = create_multiscale_vision_transformers(
            spatial_size=spatial_size,
            temporal_size=temporal_size,
            head_num_classes=head_num_classes,
        )
</code></pre>
<p>If you wish to apply certain normalization techniques to your output embeddings, that can be done either after the forward call, or you can modify the head definition above.</p>
<aside class="quote no-group" data-username="over9k" data-post="1" data-topic="159276">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/a87d85/48.png" class="avatar"> over9k:</div>
<blockquote>
<p>Is there a way to get the embedding projects to be of a certain dimension, like 1024?</p>
</blockquote>
</aside>
<p>To do this, just use:</p>
<pre><code class="lang-auto">MViT_B = create_multiscale_vision_transformers(
            spatial_size=spatial_size,
            temporal_size=temporal_size,
            head_num_classes=1024,
        )
</code></pre>
<p>And train the model, it should work well.</p>
Is there any examples for multi model system for RL?

username: Myelin_SheathXD
cooked: <p>Hey!<br>
I am a game developer. I am familiar with couple of Unity ML agents systems like PPO , SAC and other  systems. But having trouble with multi tasking systems .</p>
<p>As most of the compex 3d interactive tasks consist of several  small tasks, learning complex task  needs to be started  from small skills . For inctance,  in sports like soccer, basketboll, players do  start with just learning individual sport skills rather than playing big task playing with  basketball or soccer.<br>
First  players learn specific skills , run jump ,kick, then try use these skills in real match.</p>
<p>So  here brain should consist of several small models rather than large one NN model.</p>
<ol>
<li>tiny skills(during every day  training)<br>
2 . large skills that integrates tiny skill (during weekly matches)</li>
</ol>
<p>Is there any multi model RL examples that first starts to learn simlple tasks, then try to solve big task using individual skills.?</p>

username: Andrew_Edmondson
cooked: <p>What you want is called hierarchical RL. I doubt unity has a built in agent for it but hugging face may have one since they like unity, not sure <a href="https://huggingface.co/models?pipeline_tag=reinforcement-learning&amp;sort=downloads" class="inline-onebox" rel="noopener nofollow ugc">Models - Hugging Face</a>. The standard task for this is a soccer game simulation, exactly as you mention. This is a paper of a real robot doing this task on my “not got to that yet” list. <a href="https://arxiv.org/abs/2208.01160" class="inline-onebox" rel="noopener nofollow ugc">[2208.01160] Hierarchical Reinforcement Learning for Precise Soccer Shooting Skills using a Quadrupedal Robot</a></p>
Questions on making a shared library (.so) with libtorch

username: Fei_Liu
cooked: <p>I’m asking this question to make sure I’m doing the right thing, as my knowledge on cmake is very limited. My purpose is to make a shared library with libtorch, so I tried a toy model based on the <a href="https://pytorch.org/cppdocs/installing.html" rel="noopener nofollow ugc">install libtorch example</a>. After checking some <a href="https://stackoverflow.com/questions/17511496/how-to-create-a-shared-library-with-cmake" rel="noopener nofollow ugc">stackoverflow answers</a>, I found that the only thing I need to do is to change the following line in the  <a href="https://pytorch.org/cppdocs/installing.html#minimal-example" rel="noopener nofollow ugc"><code>CMakeLists.txt</code> - Minimal Example</a> from:</p>
<pre><code class="lang-auto">add_executable(example-app example-app.cpp)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">add_library(example-app SHARED example-app.cpp)
</code></pre>
<p>(<code>example-app.cpp</code> needs to be changed to a library code accordingly, say changing the <code>main</code> function to a lib function <code>f</code>).<br>
Then with the same build command in the <a href="https://pytorch.org/cppdocs/installing.html#minimal-example" rel="noopener nofollow ugc">Minimal Example</a>:</p>
<pre><code class="lang-auto">mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..
cmake --build . --config Release
</code></pre>
<p>I’ll get a <code>lib&lt;xxx&gt;.so</code> that can be readily used. After that, as long as I have the <code>libtorch/</code> shared libraries in my computer, I can use that shared object for the other binaries.</p>
<p>My question:</p>
<ol>
<li>Am I doing it in the right way?</li>
<li>Do I have any redundant or missing steps?</li>
</ol>
How to update epoch loss in a different way?

username: BanAckerman
cooked: <p>Currently I update my per epoch loss as such:</p>
<pre><code class="lang-auto">for i in range(epoch):
        print('epoch %d:' % i)
        epoch_loss = 0
        model.train()
        for graph1, graph2, target in train_loader:
            pred = torch.squeeze(model(graph1, graph2))
            pred = pred.detach().cpu().numpy()
            target = target.detach().cpu().numpy()
            pred = torch.tensor(pred, requires_grad=True)
            target = torch.LongTensor(target)
            loss = loss_fn(pred, target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            epoch_loss += loss.detach().item()
        epoch_loss /= (iter + 1)
        print('Epoch {}, loss {:.4f}'.format(epoch, epoch_loss))
        epoch_losses.append(epoch_loss)
</code></pre>
<p>However the line</p>
<pre><code class="lang-auto">epoch_loss /= (iter + 1)
</code></pre>
<p>gives me this error</p>
<pre><code class="lang-auto">TypeError: unsupported operand type(s) for +: 'builtin_function_or_method' and 'int'
</code></pre>
<p>How do I update the loss in a different way? Why is this giving an error?</p>

username: ptrblck
cooked: <p><code>iter</code> is a Python function defined as:</p>
<pre><code class="lang-python">iter(iterable) -&gt; iterator
iter(callable, sentinel) -&gt; iterator

Get an iterator from an object.  In the first form, the argument must
supply its own iterator, or be a sequence.
In the second form, the callable is called until it returns the sentinel.
Type:      builtin_function_or_method
</code></pre>
<p>and I guess you want to divide by <code>i</code> instead?</p>
Torch::from_blob failed to convert cv::Mat to torch::Tensor

username: duan22677
cooked: <p>I want to trace <code>cv2.getAffineTransform</code> by implementing the operator in c++, but there is a problem when converting <code>cv:Mat</code> to <code>torch::Tensor</code> by using <code>torch::from_blob</code> in my code:</p>
<p>C++ code:</p>
<pre><code class="lang-auto">// C++implementation
#include &lt;opencv2/opencv.hpp&gt;
#include "torch/script.h"

torch::Tensor get_affine_transform(at::Tensor src, at::Tensor dst) {

  cv::Mat src_mat(/*rows=*/src.size(0),
                  /*cols=*/src.size(1),
                  /*type=*/CV_32FC1,
                  /*data=*/src.data_ptr&lt;float&gt;());

  cv::Mat dst_mat(/*rows=*/dst.size(0),
                  /*cols=*/dst.size(1),
                  /*type=*/CV_32FC1,
                  /*data=*/dst.data_ptr&lt;float&gt;());

  std::vector&lt;cv::Point2f&gt; srcTri = cv::Mat_&lt;cv::Point2f&gt;(src_mat);
  std::vector&lt;cv::Point2f&gt; dstTri = cv::Mat_&lt;cv::Point2f&gt;(dst_mat);

  cv::Mat output_mat = cv::getAffineTransform(srcTri, dstTri);

  std::cout &lt;&lt; "output_mat: " &lt;&lt; output_mat &lt;&lt; std::endl;  // for debug

  torch::Tensor out_data =
      torch::from_blob(output_mat.ptr&lt;float&gt;(), /*sizes=*/{2, 3}, at::kFloat);

  std::cout &lt;&lt; "out_data: " &lt;&lt; out_data &lt;&lt; std::endl;  // for debug

  return out_data.clone();
}

// the op is successfully registered
static auto registry =
    torch::RegisterOperators("my_ops::get_affine_transform", &amp;get_affine_transform);
</code></pre>
<p>python code:</p>
<pre><code class="lang-auto">import cv2
import torch

pts1 = torch.tensor([[50, 50], [200, 50], [50, 200]], dtype=torch.float32)
pts2 = torch.tensor([[10, 100], [200, 50], [100, 250]], dtype=torch.float32)

M = cv2.getAffineTransform(pts1.numpy(), pts2.numpy())
print(f"M: {M}")

M2 = torch.ops.my_ops.get_affine_transform(pts1, pts2)
print(f"M2: {M2}")

</code></pre>
<p>but the outputs:</p>
<pre><code class="lang-auto">M: [[  1.26666667   0.6        -83.33333333]
 [ -0.33333333   1.          66.66666667]]

output_mat: [1.266666666666667, 0.6, -83.33333333333334;
  -0.3333333333333334, 1, 66.66666666666669]

out_data: 7.8507e+02  1.9083e+00  4.1723e-08
 1.7750e+00  1.4660e+13 -3.3255e+00

M2: tensor([[ 7.8507e+02,  1.9083e+00,  4.1723e-08],
        [ 1.7750e+00,  1.4660e+13, -3.3255e+00]])

</code></pre>
<p>It seems that the <code>torch::from_blob</code> not copy the data of <code>output_mat</code>(cv::Mat) to <code>out_data</code> (Tensor).<br>
Can someone help me to see where my problem is?</p>

username: Matej_Kompanek
cooked: <p>I suspect that ‘getAffineTransform’ returns matrix of doubles (float64), so you could try → </p>
<pre><code class="lang-auto"> torch::from_blob(output_mat.ptr&lt;double&gt;(), /*sizes=*/{2, 3}, at::kDouble)
</code></pre>

username: duan22677
cooked: <p>Thank you so much. it was indeed caused by double dtype, and after I use your code, the problem was solved.</p>
Two-phase training for imbalanced data

username: JessM
cooked: <p>Hi everyone,<br>
I’m training a basic feedforward NN on an imbalanced dataset and I wanted to implement a two-phase training approach:<br>
In the first training phase the model is trained on an undersampled version of the dataset so that all labels have equal probability.<br>
The second phase then accounts for the data imbalance by re-training only the output layer of the NN with the original distribution of the dataset.</p>
<p>What would be the workflow for something like this?<br>
Train entire model on under sampled dataset → set param.requires_grad = False for all layers except last (output layer) → save model parameters → re-load model and train on imbalanced dataset?</p>
<p>Is it possible to do all this in one run or would I do two separate runs?</p>
<p>Any tips/insight would be appreciated!</p>

username: ptrblck
cooked: <p>You wouldn’t need to save and reload the model and could do both training passes in a single script.<br>
However, if you think it’s more convenient to write different script, it’s also fine and I don’t see a reason this would not work, too.</p>

username: JessM
cooked: <p>If I use two scripts the first script will end by saving the model weights</p>
<pre><code class="lang-auto">torch.save(model.state_dict(), out_dir / "model_parameters.pth")
</code></pre>
<p>And then the second script starts by reloading those weights to a new model?</p>
<pre><code class="lang-auto">model2 = model.load_state_dict(torch.load("model_parameters.pth"))
</code></pre>
<p>And then I presume the freezing step will follow:</p>
<pre><code class="lang-auto">    for name, param in model2.named_parameters():
        if not '12' in name:
            param.requires_grad = False
</code></pre>
<p>I’m using ‘12’ since the names in named_parameters are as follows:<br>
[‘linear_relu_stack.0.weight’, ‘linear_relu_stack.0.bias’, ‘linear_relu_stack.3.weight’, ‘linear_relu_stack.3.bias’, ‘linear_relu_stack.6.weight’, ‘linear_relu_stack.6.bias’, ‘linear_relu_stack.9.weight’, ‘linear_relu_stack.9.bias’, ‘linear_relu_stack.12.weight’, ‘linear_relu_stack.12.bias’]</p>
<p>Is my thinking here correct?</p>

username: ptrblck
cooked: <p>Yes, your approach sounds correct.</p>
The samples in one batch are loaded in parallel while enumerating DataLoader?

username: asif_hanif
cooked: <blockquote>
<p>for idx, batch in enumerate(Data_Loader):<br>
…</p>
</blockquote>
<p>When we iterate over data_loader, do all samples in the current (idx_th) batch get loaded in parallel or they are loaded one-by-one and then collated by collate_fn ?</p>

username: ptrblck
cooked: <p>Each worker (specified by <code>num_workers</code> in the <code>DataLoader</code>) will load a full batch by loading and processing the samples sequentially. The workers work in parallel.</p>

username: asif_hanif
cooked: <p>Isn’t there a method to use multi-processing to load all samples of one batch in parallel?<br>
I am using map-style dataloader with batch_size of 512 images. Loading of one batch (i.e. 512 images) takes around 20 seconds which is acting as a bottleneck in my training.</p>

username: ptrblck
cooked: <p>I think you could try to use a <code>BatchSampler</code> and pass the indices of the entire batch to the <code>Dataset.__getitem__</code> as described <a href="https://discuss.pytorch.org/t/how-to-use-batchsampler-with-getitem-dataset/78788/4">here</a>. Inside the <code>__getitem__</code> you could then try to use multiple processes to load the samples in parallel.</p>

username: asif_hanif
cooked: <p>Alright. Do you think that initiating multiple processes in <code>__getitem__</code>  (in this particular case) will create issues for a scenario user sets <code>num_workers&gt;0</code>?</p>
<p>I have tried using parallel batch loading in conjunction with <code>num_workers&gt;0</code> , however, processes linked to <code>num_workers&gt;0</code> do not get initiated and get the following error:<br>
<code>AssertionError: daemonic processes are not allowed to have children</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/3826812f216f6ee4c8d08b2690e356e7e60a73d4.png" data-download-href="https://discuss.pytorch.org/uploads/default/3826812f216f6ee4c8d08b2690e356e7e60a73d4" title="Screen Shot 2022-08-19 at 3.57.47 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/3826812f216f6ee4c8d08b2690e356e7e60a73d4.png" alt="Screen Shot 2022-08-19 at 3.57.47 PM" data-base62-sha1="80JevYrWAJHANhO95Gwxr8gpKzG" width="690" height="199" data-dominant-color="3D3C3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-08-19 at 3.57.47 PM</span><span class="informations">1337×387 29.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You might want to try to use the <code>spawn</code> startmethod and see if this would fix the issue. However, note that I have not verified it as I wasn’t running in your use case yet.</p>
Is it possible to slice a tensor with another tensor like this?

username: Alio
cooked: <p>This is what I am trying to do.</p>
<p><code>tensor_A[:, tensor_B]</code></p>
<p>torch::Tensor cholesky_vector = torch::ones({5, 6});<br>
torch::Tensor cholesky_diag_index = torch::arange(da, torch::dtype(torch::kLong)) + 1;<br>
torch::Tensor cholesky_diag_index_2 = (cholesky_diag_index * (cholesky_diag_index + 1)) / 2 - 1;</p>
<p>int main () {<br>
std::cout &lt;&lt; cholesky_vector.index({ Slice(None, cholesky_diag_index_2.to(torch::kLong)) });<br>
}</p>
<p><code>cholesky_vector</code> is tensor_A and <code>cholesky_diag_index_2</code> is tensor_B.</p>
<p>I’m getting</p>
<pre><code class="lang-auto">/home/iii/tor/m_gym/multiv_normal.cpp:54:39: error: no matching function for call to ‘at::Tensor::index(&lt;brace-enclosed initializer list&gt;)’
   54 |     std::cout &lt;&lt; cholesky_vector.index({ Slice(None, cholesky_diag_index_2.to(torch::kLong)) });
      |                  ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/iii/tor/m_gym/libtorch/include/ATen/core/Tensor.h:3,
                 from /home/iii/tor/m_gym/libtorch/include/ATen/Tensor.h:3,
                 from /home/iii/tor/m_gym/libtorch/include/torch/csrc/autograd/function_hook.h:3,
                 from /home/iii/tor/m_gym/libtorch/include/torch/csrc/autograd/cpp_hook.h:2,
                 from /home/iii/tor/m_gym/libtorch/include/torch/csrc/autograd/variable.h:6,
</code></pre>
<p>etc.</p>
<p>I am able to index with the second tensor, but I am not able to use it at all with <code>Slice</code>. Would I be able to maybe convert the tensor to a list or array or something that the slice would not have a problem with?</p>

username: Andrew_Edmondson
cooked: <p>Slice wants an int there so I believe you need to .item&lt;int&gt;() the tensor location that contains the int.</p>