akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Mis match error in pytorch module

username: zeeshan2121
cooked: <p>[Before posting a query, check the <a href="https://discuss.pytorch.org/c/faq/30">FAQs</a> - it might already be answered!](<a href="https://colab.research.google.com/drive/1EVHqmOmiBxAUXla6cBjrfI5PT3EMp3B5#scrollTo=WqSivYLxiZ0G&amp;uniqifier=3" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a>)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a39d0b7758d43a19369f1db727509cdc9f97f99.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0a39d0b7758d43a19369f1db727509cdc9f97f99" title="IMG_20221103_143419"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0a39d0b7758d43a19369f1db727509cdc9f97f99_2_375x500.jpeg" alt="IMG_20221103_143419" data-base62-sha1="1ssDTh7qLhqMKIkMewkzBnAuZeh" width="375" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0a39d0b7758d43a19369f1db727509cdc9f97f99_2_375x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0a39d0b7758d43a19369f1db727509cdc9f97f99_2_562x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0a39d0b7758d43a19369f1db727509cdc9f97f99_2_750x1000.jpeg 2x" data-dominant-color="787875"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20221103_143419</span><span class="informations">1920×2560 349 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Your linear layer is raising the shape mismatch so make sure the input activation to this layer matches the <code>in_features</code>.<br>
PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>
How to load Pytorch model using the provided bin,config files

username: Niranj_Jyothish
cooked: <p>Hi all,<br>
Im new to Pytorch.<br>
I’m trying to load someone’s saved Pytorch model.<br>
The downloaded folder has the below contents:<br>
config.json<br>
pytorch_model.bin<br>
special_tokens_map.json<br>
tokenizer_config.json<br>
training_args.bin<br>
vocab.txt</p>
<p>I understand that a model can be loaded as such (from <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">PyTorch documentation</a>):</p>
<pre><code class="lang-auto">model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()
</code></pre>
<p>but how can I make use of the files in the output directory to do this?</p>
<p>I have tried<br>
model = torch.load(‘pytorch_model.bin’)<br>
This doesnt load the model but a dictionary thus, the eval(), predict() does not work.<br>
model.eval()<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
AttributeError: ‘collections.OrderedDict’ object has no attribute ‘eval’</p>

username: Niranj_Jyothish
cooked: <p>Hi,<br>
I was able to find the solution for above.<br>
<a href="https://huggingface.co/transformers/v1.2.0/serialization.html#serialization-best-practices" class="onebox" target="_blank" rel="noopener nofollow ugc">https://huggingface.co/transformers/v1.2.0/serialization.html#serialization-best-practices</a></p>
How can I process stack of frames

username: dato_nefaridze
cooked: <p>Hi, I am building a reinforcement learning environment and have a question about: How can I process stack of frames? I plan to do it with Transformers, but the problem is I don’t know how to summarize frames into one state vector, what I mean is: I am doing frame processing by using pre-trained model, after that i plan to input those features into transformer network so that I get one vector which summarizes sequence of frames</p>

username: desert_ranger
cooked: <p>Usually function approximators like CNNs and Transformers take in a stack of frames as input. Therefore, you don’t need to do any pre-processing.</p>

username: dato_nefaridze
cooked: <p>What i meant was: i am featurizing images into vectors so that’s why i want to use resnet, it’s necessary. I am doing same project as CURL (contrastive unsupervised reinforcement learning).<br>
input is: [B, C, F, H, W] batch size, channels, frames, height and width</p>

username: vmoens
cooked: <p>Hey <a class="mention" href="/u/dato_nefaridze">@dato_nefaridze</a> can you provide a code snippet of some (even buggy) implementation of what you’re trying to do? I’m struggling to understand what it is…</p>
Update ResNet18 Classifer according to the classes in the batch

username: papoo13
cooked: <p>If I want to compute the loss based on only the labels of the current batch, I can select those classes from the output logits by index, then create a new logits output and compute the loss using the unique relabeled [from 0 to n] targets of the current batch.<br>
The code is as follows:</p>
<pre><code class="lang-auto">if training:
    # Use labels trick
    # Get current batch labels (and sort them for reassignment)
    unq_lbls = labels.unique().sort()[0]
    # Assign new labels (0,1 ...)
    for lbl_idx, lbl in enumerate(unq_lbls):
        labels[labels == lbl] = lbl_idx
    # Calcualte loss only over the heads appear in the batch:
    loss = criterion(outputs[:, unq_lbls], labels)
else:
    loss = criterion(outputs, labels)
</code></pre>
<p>My question is how to use this loss to update the weights of the classifier and the model. I think I should freeze the weights of those classes that are not in the current batch and only update the relevant connections per batch. I would like to have a second opinion on this, whether it is correct or not. And if yes, a hint on the implementation would be great.</p>
How to use torch to achieve a search algorithm？

username: sev777
cooked: <p>Give two lists, <code>P=[p_1,p_2,p_3,..,p_i]</code> with i positive value, <code>N=[n_1,n_2,...,n_j]</code> with j negative value, give a target T, I want to select some elements from P and N so that their sum is T.</p>
<p>I want to use Torch to achieve this, So, the Goal is to minimize the function:<br>
<code>Loss= w_1*P + w_2*N - T,</code><br>
where w_1 and w_2 are trainable parameters.<br>
But the loss does not converge but decreases linearly, from positive to negative values.</p>
<pre><code class="lang-auto">data=torch.load('use_data.pkl')

delta = torch.zeros(data[0].shape, requires_grad=True, device="cuda") 
`#P and delta shape: 382 `

delta2 = torch.zeros(data[2].shape, requires_grad=True, device="cuda") # N
`#N and delta2 shape: 386 `
P=data[0]
N=data[2]

T=data[4][0]

opt = torch.optim.Adam([delta,delta2], lr=0.0001)

for s in range(10000):
    opt.zero_grad()
    main_loss=torch.sum(delta*P)+torch.sum(delta2*N)-T
    #loss = main_loss + (delta.norm(2) + delta2.norm(2))*0.01
    main_loss.backward()
    opt.step()

</code></pre>

username: tom
cooked: <p>It sounds a bit as if your goal might be to get to zero rather than minimize.<br>
In that case, you could, for example minimize the square of what you currently have, i.e.<br>
<code>loss = (w_1 * P + w_ 2 * N - T) ** 2</code></p>
<p>Best regards</p>
<p>Thomas</p>

username: sev777
cooked: <p>Thanks for your reply, and there has another problem: why are the element values of w_1 and w_2 the same? Each element in w_1 is close to 0.6, and w_2 is close to -0.6 ? <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
Cannot access parent functions from child class after using `TORCH_MODULE`

username: alireza_dizaji
cooked: <p>I am using TORCH_MODULE below both child and parent class definition, though I cannot access parent functions from child objects and if I only use the Impl version, it works.</p>

username: tom
cooked: <p>Hi,</p>
<p>I’m not sure I’d know, but I imagine that it’d be easier to say something if you had a short code snippet showing what you want to do and how you run into trouble with it.</p>
<p>Best regards</p>
<p>Thomas</p>
How to remove multiple elements from a tensor by index?

username: Tianming_Qu
cooked: <p>Hi all,</p>
<p>Is there anyway I can remove multiple elements from a tensor according to index?</p>
<p>For example, I have a tensor with 10000 elements, and I have the indexes of 1000 elements that I want to remove. In numpy I can do this simply by np.delete, is there any similar function I can use in torch?<br>
Thanks for any help!!</p>
<p>Best</p>

username: sev777
cooked: <p>Is it a way to use the index to select?</p>
<pre><code class="lang-auto">q=torch.randn(5)
retain_ind=[0,2]
q=q[retain_ind]
</code></pre>
<p>Then we remove the [1,3,4] elements.</p>
How to combine train forward function and torchscript forward function?

username: wtiandong
cooked: <p>Hi,<br>
I have a custom nn.Module which has custom c++ forward and backward function. I bind these function with pytorch extension:</p>
<pre><code class="lang-auto">PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
  m.def("my_forward_cpu", &amp;my_forward_cpu);
  m.def("my_backward_cpu", &amp;my_backward_cpu);
}
</code></pre>
<p>And now, I also want to use <code>my_forward_cpu</code> for inference using torchscript. But it seems that I need to write</p>
<pre><code class="lang-auto">TORCH_LIBRARY(my_ops, m) {
  m.def("my_forward_cpu", my_forward_cpu);
}
</code></pre>
<p>and insert</p>
<pre><code class="lang-auto">import torch
import myop.backend
torch.ops.load_library("build/libmyfunc.so")

class MyModule(nn.Module):
...
def forward(self, x):
    if torch.jit.tracing():
        return torch.ops.my_ops.my_forward_cpu(x)
    else:
        return myop.backend.my_forward_cpu(x)
</code></pre>
<p>Is this the way it should be? Or can I combine the my_forward_cpu for train and for torchscript?</p>

username: dancedpipi
cooked: <p>met the same problem,  is any update?</p>
Persisting benchmark cache

username: jiangzz
cooked: <p>When using torch.backends.cudnn.benchmark, we can select the fastest operator for conv.<br>
Inside the pytorch,  cache is used to store the pre-searched fastest algorithms.<br>
But the thing is that our model’s input shape changes a lot and will cause frequent cudnn search which slows down the entire speed. So we figure if we can persist some cache to improve the hit rates of cache while still using the fastest implementation.</p>
<p>A simple way is like this:</p>
<pre><code class="lang-auto">template &lt;typename T&gt;
struct BenchmarkCache {
  std::mutex mutex;
  std::unordered_map&lt;ConvolutionParams, T, ParamsHash&lt;ConvolutionParams&gt;, ParamsEqual&lt;ConvolutionParams&gt;&gt; map;
  std::string CACHE_STORED_PATH;
  BenchmarkCache(){}
  
  BenchmarkCache(std::string cache_stored_path): CACHE_STORED_PATH(cache_stored_path){
    load();
  }

  bool find(const ConvolutionParams&amp; params, T* results) {
    std::lock_guard&lt;std::mutex&gt; guard(mutex);
    auto it = map.find(params);
    if (it == map.end()) {
      return false;
    }
    *results = it-&gt;second;
    return true;
  }

  void insert(const ConvolutionParams&amp; params, const T&amp; results) {
    std::lock_guard&lt;std::mutex&gt; guard(mutex);
    map[params] = results;
  }

  int load(){
    std::lock_guard&lt;std::mutex&gt; guard(mutex);
    std::ifstream i_cache(CACHE_STORED_PATH, ios::in);
    //load the mapping
    return 0;
  }

  int save(){
    std::lock_guard&lt;std::mutex&gt; guard(mutex);
    std::ofstream o_cache(CACHE_STORED_PATH, ios::out);
    //save the mapping
    return 0;
  }

  ~BenchmarkCache(){
    save();
  }
};
</code></pre>
<p>Of  course, there are a few issues need to work out.</p>
<ul>
<li>The time of load and save, where should we put them. Are there some places we can insert the load() or save() call for efficiency.</li>
<li>One file per model or a global cache, other models can also use it.</li>
<li>The lock for load and save. we may have lots of threads try to read or write data.</li>
<li>How to load and save data with less time and space cost. e.g. It seems pytorch will calculate the hash of ConvolutionParams, can we store the hash value instead and make the cache map from a size_t value(acquired from ConvolutionParams) to cudnnConvolutionAlgoPerf.</li>
</ul>
Implementation of Squared Earth movers distance loss function for ordinal scale

username: Nolan_Cardozo
cooked: <p>Hi everyone, I recently came across the paper on “Squared earth mover’s distance-based loss for training deep neural networks.” (<a href="https://arxiv.org/abs/1611.05916" class="inline-onebox" rel="noopener nofollow ugc">[1611.05916] Squared Earth Mover's Distance-based Loss for Training Deep Neural Networks</a>). I want to use the squared EMD loss function for an ordinal classification problem .</p>
<p>However, I could not find a single implementation for the same. Though there is a tensorflow implementation which is as follows:</p>
<pre><code class="lang-auto">def earth_mover_distance(
        **kwargs
) -&gt; Callable:
    """
    Wrapper for earth_mover distance for unified interface with self-guided earth mover distance loss.
    """
    import tensorflow as tf
    from tensorflow.keras import backend as K

    def _earth_mover_distance(
            y_true: K.placeholder,
            y_pred: K.placeholder
    ) -&gt; K.placeholder:
        return tf.reduce_mean(tf.square(tf.cumsum(y_true, axis=-1) - tf.cumsum(y_pred, axis=-1)), axis=-1)

    return _earth_mover_distance
</code></pre>
<p>Can someone please help me with the implementation in pytorch?</p>

username: ptrblck
cooked: <p>You could try to translate this code by using the <code>torch</code> namespace.<br>
E.g this might work (untested):</p>
<pre><code class="lang-python">def earth_mover_distance(y_true, y_pred):
    return torch.mean(torch.square(torch.cumsum(y_true, dim=-1) - torch.cumsum(y_pred, dim=-1)), dim=-1)
</code></pre>

username: jyothi_sri
cooked: <p>Hello sir ,i tired the above code but i got the following problem<br>
TypeError: cumsum() received an invalid combination of arguments - got (list, dim=int), but expected one of:</p>
<ul>
<li>(Tensor input, int dim, *, torch.dtype dtype, Tensor out)</li>
<li>(Tensor input, name dim, *, torch.dtype dtype, Tensor out)<br>
please help me</li>
</ul>

username: ptrblck
cooked: <p>As the error message explains: pass a <code>tensor</code> to <code>torch.cumsum</code> instead of a <code>list</code>:</p>
<pre><code class="lang-python">torch.cumsum([1., 2.], dim=0)
# TypeError: cumsum() received an invalid combination of arguments - got (list, dim=int), but expected one of:
#  * (Tensor input, int dim, *, torch.dtype dtype, Tensor out)
#  * (Tensor input, name dim, *, torch.dtype dtype, Tensor out)

torch.cumsum(torch.randn(10), dim=0) # works
</code></pre>

username: jyothi_sri
cooked: <p>will try .Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Lazy backend comutation cache

username: lipracer
cooked: <p>I am reading the code of lazy beckend and has some question.</p>
<ul>
<li>We use the hash value of the computational graph to represent the uniqueness of the computational graph.How we avoid hash collisions？<pre><code class="lang-auto"> GetComputationCache()-&gt;Add(coll.hash, cached_computation);
</code></pre>
</li>
<li>LazyGraphExecutor::RunPostOrder<br>
We use topological sorting to reorder the computation graph, but this will lose the original computation order of the computation graph, sometimes we need to know the original computation order of the user, can we reorder based on the construction order of the tensor and then based on the postorder？</li>
</ul>
Tips on improving the ability to implement ideas into code

username: Xiaohao_Lin
cooked: <p>Hi everyone, I am new to pytorch. And I would like to ask for resources/guides that teach you to transform ideas into code. Now, I get a bit overwhelmed when I try to implement complex neural network architectures, due to lack of experience, and can get quite frustrated. When I read others’ code, which are mostly sparsely documented, I feel frustrated when I can’t understand, because every line is important.</p>
<p>Any tips to help would be deeply appreciated.</p>
<p>Best<br>
Xiaohao</p>

username: srishti-git1110
cooked: <p>Refer to the official documentation - it’s the best resource. And this <a href="https://www.manning.com/books/deep-learning-with-pytorch" rel="noopener nofollow ugc">book</a>.</p>

username: ptrblck
cooked: <p>Additionally to <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a>’s advice, you could also check some courses such as <a href="https://www.fast.ai/">FastAI</a>.</p>

username: Xiaohao_Lin
cooked: <p>hi thanks so much! it’s a great book!</p>

username: Xiaohao_Lin
cooked: <p>Hi i have actually bought the book after your recommendation, thanks mate!</p>
Torch.cuda.is_available() returns different value in different scripts

username: Xiaohao_Lin
cooked: <p>Dear all,</p>
<p>I have two projects in Pycharm, both using the same interpreter. I have checked and ensured that the cuda, cuda driver and pytorch (GPU version) versions are right.</p>
<p>Nvidia GEFORCE RTX 2080 TI<br>
CUDA:<br>
DRIVER VERSION: 516.94<br>
cudatoolkit: 10.2<br>
python: 3.8</p>
<p>The first project is able to use cuda, and torch.cuda.is_available() returns True.</p>
<p>But the second project returns False.</p>
<p>Then, I tried to create a new environment for the second project, and reinstall torch, cuda, and python.<br>
but torch.cuda.is_available() still False.</p>
<p>The strange thing is, when I check torch.cuda.is_avaiable() at the python console, I get True, but in the script, I get False.</p>
<p>Can anyone enlighten me on what’s the underlying issue?</p>

username: ptrblck
cooked: <p>Your IDE might use a different virtual environment or could set some env variables (e.g. <code>CUDA_VISIBLE_DEVICES</code> to a wrong values etc.), so check your PyCharm setup and which envs are used.</p>

username: Xiaohao_Lin
cooked: <p>hi thanks so much! indeed that’s the issue, problem solved.</p>
Set right cuDNN path when building torch from source

username: Tianyu9748
cooked: <p>I copy <code>cudnn</code> to my customized folder, since I don’t have enough right to cp them to <code>$CUDA_HOME</code>.</p>
<p>I tried to set the path in CMakeLists.txt</p>
<pre><code class="lang-auto">set(CUDNN_LIBRARY_PATH /usr/local/cudnn-8.4.1/lib)
set(CUDNN_INCLUDE_PATH /usr/local/cudnn-8.4.1/include)
</code></pre>
<p>However, it doesn’t work.</p>
<p>That is what I set in my .bashrc file</p>
<pre><code class="lang-auto"># CUDNN
export CUDNN_HOME=/usr/local/cudnn-8.4.1
export LD_LIBRARY_PATH=${CUDNN_HOME}/lib:$LD_LIBRARY_PATH
export CPLUS_INCLUDE_PATH=${CUDNN_HOME}/include:$CPLUS_INCLUDE_PATH
</code></pre>
<p>Does anyone know how could I set the proper path?</p>
<p>Best<br>
Max</p>
Setting custom CuDNN path

username: mahmoodn
cooked: <p>Hi<br>
I have extracted CuDNN files to a custom folder because of non-root access. As I saw this message in the config output:</p>
<pre><code class="lang-auto">-- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH)
</code></pre>
<p>I set the variables accordingly in .bashrc and sourced it</p>
<pre><code class="lang-auto">$ echo $CUDNN_LIBRARY_PATH
/home/mahmood/cudnn/lib64
$ ls /home/mahmood/cudnn/lib64
libcudnn_adv_infer.so        libcudnn_cnn_infer.so        libcudnn_ops_infer.so        libcudnn.so
libcudnn_adv_infer.so.8      libcudnn_cnn_infer.so.8      libcudnn_ops_infer.so.8      libcudnn.so.8
libcudnn_adv_infer.so.8.1.0  libcudnn_cnn_infer.so.8.1.0  libcudnn_ops_infer.so.8.1.0  libcudnn.so.8.1.0
libcudnn_adv_train.so        libcudnn_cnn_train.so        libcudnn_ops_train.so        libcudnn_static.a
libcudnn_adv_train.so.8      libcudnn_cnn_train.so.8      libcudnn_ops_train.so.8      libcudnn_static_v8.a
libcudnn_adv_train.so.8.1.0  libcudnn_cnn_train.so.8.1.0  libcudnn_ops_train.so.8.1.0
$ echo $CUDNN_INCLUDE_PATH
/home/mahmood/cudnn/include
$ ls /home/mahmood/cudnn/include
cudnn_adv_infer.h     cudnn_backend.h       cudnn_cnn_train.h     cudnn_ops_infer_v8.h  cudnn_version.h
cudnn_adv_infer_v8.h  cudnn_backend_v8.h    cudnn_cnn_train_v8.h  cudnn_ops_train.h     cudnn_version_v8.h
cudnn_adv_train.h     cudnn_cnn_infer.h     cudnn.h               cudnn_ops_train_v8.h
cudnn_adv_train_v8.h  cudnn_cnn_infer_v8.h  cudnn_ops_infer.h     cudnn_v8.h
</code></pre>
<p>Still I see the message that CuDNN has not been found. Should I do something more?</p>

username: mahmoodn
cooked: <p>OK. The solution is to set <code>CUDNN_ROOT</code>.</p>

username: Tianyu9748
cooked: <p>Sorry, could you explain it in more details?<br>
I have a similar issue here.<br>
What should I do to solve this, ‘cudnn not found’?</p>
<p>Best<br>
Max</p>
Can I compare CUDA and CPU operations in the hook

username: blime4
cooked: <p>Hey!<br>
I want to achieve an accuracy comparison in the forward_hook</p>
<pre><code class="lang-auto">torch.nn.modules.module.register_module_forward_hook(hook_forward_fn)
torch.nn.modules.module.register_module_full_backward_hook(hook_backward_fn)
</code></pre>
<p>In each hook, let all the nn.Module runs once in the cuda device and run once in the CPU device</p>
<p>Then compare whether their results are consistent (In the allowable margin of tolerance)</p>
<p>Please someone guide me, how to do it, can I do it use hooks (register_module_forward_hook, register_module_full_backward_hook)</p>
<p>Here’s what I’m thinking: (probably all wrong):</p>
<pre><code class="lang-auto"># 1. deepcopy the module ,input, output in the hook.
module_copy = copy.deepcopy(module)
input_copy = copy.deepcopy(input)
output_copy = copy.deepcopy(output)

# 2. to cpu
module_copy = module_copy.to('cpu')

#  3. call the function , maybe like that
module_copy.call

# 4. compare the output_copy the output


</code></pre>

username: soulitzer
cooked: <p>This sounds OK, are you running into any issues?</p>
RMSE results close to zero

username: Ghazaleh
cooked: <p>Hi everyone<br>
I have developed a pytorch code for video image recognition. I have developed it by 3DCNN and Regression. The code works but my problem is the RMSE and MAE are very low and close to zero which is not expected since other reseachers results is around 6 to 8 for the same data. I couldn’t find where is wrong. I would greatly appreciate it if someone could help me by reviewing my code. Please let me know if you are happy to do it and then I will share the code.<br>
Best<br>
Ghazaleh</p>
Freezing layers for transfer learning

username: dneprdva
cooked: <p>Hi, guys,<br>
My model is ShuffleNet_V2_X0_5_Weights.IMAGENET1K_V1<br>
I m trying to freeze layers using this code:</p>
<blockquote>
<p>for param in model.parameters():<br>
param.requires_grad = False</p>
</blockquote>
<p>But it doesn’t work. I recieve:</p>
<p>‘RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn’</p>
<p>So I found this code:</p>
<blockquote>
<p>for params in list(model.parameters())[0:-5]:<br>
params.requires_grad = False<br>
It works but I don’t understand what [0:-5] means in this case.</p>
</blockquote>
<p>And I tried:</p>
<blockquote>
<p>for param in model.features.parameters():<br>
param.requires_grad = False</p>
</blockquote>
<p>My model doesn’t have attribute ‘features’. So how should I freeze layers right way?</p>

username: ptrblck
cooked: <p>You cannot freeze all parameters are expect <code>backward()</code> to work and calculate gradients.<br>
The indexing e.g. via <code>[0:-5]</code> is used to freeze all parameters besides the last 5 (whatever these are depends on your model).<br>
The last code snippet expects your model to contain a <code>.features</code> attribute, which is again model-dependent.<br>
Check which layers are defined in your model, which should be frozen, and access them directly via their attributes.</p>
Create matrix by applying function to the mulitplication of to vectors

username: Thars75
cooked: <p>Hello,</p>
<p>I was wondering what the fastest way to compute the following is :<br>
Suppose two vectors (v1,v2) of dim (bs,n,c), (bs,m,c):</p>
<pre><code class="lang-auto">matrix = torch.zeros((bs,n,m))
For i in range(n):
     For j in range(m):
          matrix[:,i,j] = foo(v1[:,i,:],v2[:,j:,])
</code></pre>
<p>Thank you in advance</p>
Adding a constant to an activation function

username: smu226
cooked: <p>Hello! I would like to define an activation function which is the normal ELU + 1. Based on what I found online I decided to do this:</p>
<pre><code class="lang-auto">def SELU(input):
    return nn.ELU(input)+1+1e-15

class SELU(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, input):
        return SELU(input)
</code></pre>
<p>However when I try to call it, I am getting this error: <code> TypeError: unsupported operand type(s) for +: 'ELU' and 'int'</code>. How should I properly add 1 to the activation function? Thank you!</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/smu226">@smu226</a>,</p>
<p>When you do <code>nn.ELU()</code> you’re initializing a class, which represents the elu function. It, itself, it <strong>not</strong> a callable function, but its instance is. So, you end up trying to add a <code>ELU</code> class with an int (hence your error message).</p>
<p>The following code below works,</p>
<pre><code class="lang-auto">import torch
from torch import nn

class SELU(nn.Module):
    
    def __init__(self):
        super(SELU, self).__init__()

        self.elu = nn.ELU() 

    def forward(self, input):
        return self.elu(input) + 1.
      
      
        
x = torch.randn(1)
selu = SELU()
elu = torch.nn.ELU()

selu_y=selu(x)
elu_y = elu(x)

print(selu_y) #returns tensor([2.0493])
print(elu_y) #returns tensor([1.0493])

</code></pre>
Concat two template tensors multiple times in a specific order

username: fpicetti
cooked: <p>I want to concatenate two same-size tensors, let’s say A and B. The desired output would be something like<br>
C = [A, B, B, A, A, A, B] etc.</p>
<ul>
<li>the order is specified by a condition</li>
<li>the resulting tensor will not be attached to the autograd graph</li>
</ul>
<p>The following code is working:</p>
<pre><code class="lang-python">C = torch.stack([A if condition else B for condition in conditions])
</code></pre>
<p>Is there any other option to achieve this? Specifically, I’d like the idea of creating a “view” of the resulting tensor given <code>A</code>, <code>B</code>, and <code>conditions</code>.</p>

username: ptrblck
cooked: <p>Using indices or a mask should also work, but you wouldn’t get a view and I don’t know if any approach could satisfy a view in this case:</p>
<pre><code class="lang-python">A = torch.randn(4)
B = torch.randn(3)
C = torch.empty(len(A)+len(B))

mask = torch.tensor([True, False, False, True, True, True, False])
C[mask] = A
C[~mask] = B
</code></pre>

username: fpicetti
cooked: <p>This is interesting! Thank you for the suggestion!<br>
So in both these approaches we fill a new tensor C using A and B values.</p>
<p>In my use case, I need C for a later matrix multiplication so I’d like to use the additional dim resulting by <code>stack</code> (you can think of it as a batch dimension).</p>
<pre><code class="lang-auto">A = torch.randn(3,2)
B = torch.randn(3,2)
conditions = torch.tensor([True, False, False, True]) # size (4)

C = torch.stack([A if c.item() else B for c in conditions], dim=0) # size (4,3,2)

D = torch.randn(4,2,5)

E = torch.matmul(C, D) # size (4,3,5)
</code></pre>
<p>Is there a way to create an <em>ephemeral</em> C that uses A and B storages just once? It would be great avoiding the allocation of the whole C tensor.</p>