akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
1D convolution but before the sum operation

username: duyducvo4444
cooked: <p>Hello all~<br>
For the <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.functional.conv1d.html" rel="noopener nofollow ugc">F.conv1d</a> function. Suppose I have the input</p>
<pre><code class="lang-auto">input = [[a, b, c],
         [d, e, f]]
</code></pre>
<p>And I have the weight</p>
<pre><code class="lang-auto">weight = [[x, y],
          [z, w]]
</code></pre>
<p>Mathematically, the F.conv1d function will return</p>
<pre><code class="lang-auto">output = [[a, b, c] * [x, y]] + [[d, e, f] * [z, w]] # * is the cross-correlation operator
</code></pre>
<p>Now my problem is, I want to obtain the result before the sum operation, that is</p>
<pre><code class="lang-auto">desired_output = [[[a, b, c] * [x, y]],
                  [[d, e, f] * [z, w]]]
</code></pre>
<p>How should I do so? The simplest solution is I can create multiple weights and then conv1d each of them with the corresponding input channel but that will involve a loop that I really don’t want.</p>
<p>Best regards,</p>

username: ptrblck
cooked: <p>You could use the <code>groups</code> argument as described <a href="https://discuss.pytorch.org/t/convolution-operation-without-the-final-summation/56466">here</a>.</p>

username: duyducvo4444
cooked: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a>. I was suspecting the groups argument should do the job but the description on the official docs is so confusing.</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x128 and 2x128)

username: drey08
cooked: <p>“”"</p>
<p>class ActorCritic(nn.Module):<br>
def <strong>init</strong>(self, input_dims, n_actions, gamma=0.99):<br>
super(ActorCritic, self).<strong>init</strong>()<br>
self.gamma = gamma</p>
<pre><code>    self.policy1 = nn.Linear(*input_dims, 128)
    self.value1 = nn.Linear(*input_dims, 128)
    self.policy = nn.Linear(n_actions,128)
    self.value = nn.Linear(128, 1)

    self.rewards = []
    self.actions = []
    self.states = []

def remember(self,reward, action, state):
    self.rewards.append(reward)
    self.actions.append(action)
    self.states.append(state)

def clear_memory(self):
    self.rewards = []
    self.actions = []
    self.states = []

def forward(self, state):
    print('forward function state value: {} '.format(state.shape))
    pi1 = F.relu(self.policy1(state))
    v1 = F.relu(self.value1(state))

    pi = self.policy(pi1)
    v = self.value(v1)
    return pi, v
</code></pre>
<p>“”"</p>
<p>I tried logging the shape of the input in forward function below<br>
“”"<br>
forward function state value: torch.Size([1, 4])<br>
C:\Users\olufi\PycharmProjects\RL-A3C\actor_critic.py:68: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  …\torch\csrc\utils\tensor_new.cpp:201.)<br>
state = T.tensor([observation], dtype=T.float)<br>
C:\Users\olufi\PycharmProjects\RL-A3C\actor_critic.py:68: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  …\torch\csrc\utils\tensor_new.cpp:201.)<br>
state = T.tensor([observation], dtype=T.float)<br>
forward function state value: torch.Size([1, 4])<br>
C:\Users\olufi\PycharmProjects\RL-A3C\actor_critic.py:68: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  …\torch\csrc\utils\tensor_new.cpp:201.)<br>
state = T.tensor([observation], dtype=T.float)<br>
forward function state value: torch.Size([1, 4])<br>
C:\Users\olufi\PycharmProjects\RL-A3C\actor_critic.py:68: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  …\torch\csrc\utils\tensor_new.cpp:201.)<br>
state = T.tensor([observation], dtype=T.float)</p>
<p>“”"</p>

username: ptrblck
cooked: <p>I don’t know how you’ve initialized the model and in particular which shapes you’ve used to initialize the layers, but the shape mismatch is most likely raised by a linear layer.<br>
Print the shape of the input activation to each layer and make sure its feature dimension matches the expected <code>in_features</code> of the layer.</p>
What is PyTorch equivalent of embedding_lookup() function in TensorFlow (for 2D indices)?

username: ziba
cooked: <ul>
<li>Operating System: Windows 10</li>
<li>Python Version: 3.7.11</li>
<li>PyTorch Version: 1.10.1</li>
</ul>
<p>I have two below tensors:</p>
<pre><code class="lang-auto">import torch

embedding_vectors = torch.tensor([
    [0.01, 0.02, 0.03], 
    [0.07, 0.08, 0.04], 
    [0.05, 0.09, 0.06],
    [0.51, 0.92, 0.67],
    [0.55, 0.99, 0.64],
    [0.17, 0.23, 0.85],
    [0.45, 0.66, 0.31],
    [0.01, 0.07, 0.92],
    [0.25, 0.56, 0.32]
])

indices = torch.tensor([
        [0, 2], 
        [4, 5], 
        [6, 0]
])
</code></pre>
<p>I want to map the <strong>values</strong> in <code>indices</code> variable to <strong>rows</strong> in <code>embedding_vectors</code> variable, so I expect bellow tensor as <em>output</em> :</p>
<pre><code class="lang-auto">[
    [[0.01, 0.02, 0.03], [0.05, 0.09, 0.06]], 
    [[0.55, 0.99, 0.64], [0.17, 0.23, 0.85]], 
    [[0.45, 0.66, 0.31], [0.01, 0.02, 0.03]]
]
</code></pre>
<p>Question:</p>
<ul>
<li>Does PyTorch have built-in function to do this as same as <code>tf.nn.embedding_lookup(embedding_vectors, indices)</code> in tensorflow?</li>
<li>If not, how can I do this?</li>
</ul>
<p>I used <code>torch.index_select(embedding_vectors , 0, indices)</code> but it says that it expect a vector as indices while my <code>indices</code> variable has 2 dimension.</p>

username: ptrblck
cooked: <p>You could use the functional API via:</p>
<pre><code class="lang-python">F.embedding(indices, embedding_vectors)
</code></pre>
<p>or assign <code>embedding_vectors</code> to the <code>.weight</code> attribute of an <code>nn.Embedding</code> layer.</p>
Interpolate with parameters

username: sevagh
cooked: <p>Is there a form of interpolate with parameters? <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.interpolate.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.interpolate — PyTorch 1.10.1 documentation</a></p>
<p>I’m using the interpolate function to interpolate a small length tensor to a long length tensor, and I use the same to de-interpolate from long back to short. This results in a little bit of an error introduced (I’m using <code>mode='trilinear'</code>).</p>
<p>What sort of layers with learnable parameters could I be using for a learnable de-interpolation? Conv?</p>
<p>Either I can do:</p>
<pre><code class="lang-python">import torch.nn.functional as F

# strategy 1: parameter-free interpolate, parameter-free deinterpolate

x_long = F.interpolate(x_short, size=(long_size), mode='trilinear')
x_short_recon = F.interpolate(x_long, size=(short_size), mode='trilinear')

# strategy 2: parameter-free interpolate, learned deinterpolate

x_long = F.interpolate(x_short, size=(long_size), mode='trilinear')
x_short_recon = Conv(...)(x_long)
</code></pre>
Torchaudio; resampling; Performance Benchmarking

username: DDavid
cooked: <p>I’ve read the tutorial <a href="https://pytorch.org/audio/stable/tutorials/audio_resampling_tutorial.html#performance-benchmarking" rel="noopener nofollow ugc">here</a>, and tried to understand the sentence :<br>
“using <code>kaiser_window</code> results in longer computation times than the default <code>sinc_interpolation</code> because it is more complex to compute the intermediate window values - a large GCD between the sample and resample rate will result in a simplification that allows for a smaller kernel and faster kernel computation.”</p>
<p>The results down there say that<br>
“16 → 8 kHz” is slower than “48 → 44.1 kHz”. But I guess a GCD of 16,8 is 8 and that of 48, 44.1 is around 4 so GCD of 16,8 is larger and it should be faster according to the tutorial. Can anyone explain this?</p>
GAN - If we want to maximize the objective function, why we use backward?

username: laro
cooked: <p>I looked on pytorch tutorial on GAN models: <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">DCGAN TUTORIAL</a></p>
<p>According to this article we want to maximize the objective function (G).</p>
<p>In the code they used</p>
<blockquote>
<p>errG.backward()</p>
</blockquote>
<p>Now I’m confused.<br>
If we use backward() it will calculate the gradient down the hill,<br>
but we want to calculate the gradient up the hill (and get the maximum of the objective function)</p>
<p>What am I missing ?</p>
How to achieve efficient greedy Hamiltonian path construction?

username: 111414
cooked: <p>I have N C-dimensional points to form a tensor <code>Data</code> with a shape of <code>(C, N)</code>. Given a starting point with its index in <code>Data</code> (e.g. 0), I want to implement an efficient greedy Hamiltonian path construction algorithm as follows:</p>
<pre><code class="lang-auto">1. Initialize a list S = [], and the remaining set R = {all N data points};
2. Pick the starting point from R to S;
3. Using a loop to pick points one-by-one from R to S, in each round,
find the nearest point in R from the last point in S (measured by L2
distance), move it out from R and append to S;
4. When R is empty, return S and end.
</code></pre>
<p>where each point is represented by its corresponding index in <code>Data</code> (i.e. a number in {0,1, …, N-1}), and the returned S is actually a permutation which can be applied on <code>Data</code> to get the ordered result. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>It’s relatively easy to implement the above algorithm by for-loops, but I am pursuing a more efficient way based on PyTorch. The trivial implementation could be with O(N^2) time complexity, and I’m wondering if this can be achieved with O(N*log(N)) time complexity and O(N) memory. <img src="https://discuss.pytorch.org/images/emoji/apple/cold_sweat.png?v=10" title=":cold_sweat:" class="emoji" alt=":cold_sweat:"></p>
<p>Does there exists an API or a combination of fast operations to achieve this? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=10" title=":thinking:" class="emoji" alt=":thinking:"></p>
<p>(When the point dimensionality equals to 1, the result can be directly obtained by <code>torch.Tensor.sort().indices</code>, if I choose the largest value as the starting point)</p>
Expand the 'start' and 'end' parameter of torch arange

username: duyducvo4444
cooked: <p>Hello all~</p>
<p>With torch.arange(start=1, end=4) I will have a tensor [1, 2, 3]<br>
Now if I want to replace the ‘start’ parameter’s value with a tensor, how can I do that without doing the loop?<br>
for example if I have a=[[1], [2]]<br>
If I write torch.arange(start=a, end=a+3)<br>
I want to obtain<br>
[[1, 2, 3],<br>
[2, 3, 4]]</p>
<p>Is it possible to do that without doing the loop?</p>

username: KFrank
cooked: <p>Hi Duy!</p>
<aside class="quote no-group" data-username="duyducvo4444" data-post="1" data-topic="140162" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/duyducvo4444/48/44643_2.png" class="avatar"> duyducvo4444:</div>
<blockquote>
<p>Now if I want to replace the ‘start’ parameter’s value with a tensor, how can I do that without doing the loop?</p>
</blockquote>
</aside>
<p>You can add the desired start values to the result of <code>arange()</code>, and do<br>
so on a tensor basis:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; start_values = torch.tensor ([1, 2, 5, 7])
&gt;&gt;&gt; torch.arange (3).repeat (4, 1) + start_values.unsqueeze (1)
tensor([[1, 2, 3],
        [2, 3, 4],
        [5, 6, 7],
        [7, 8, 9]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: duyducvo4444
cooked: <p>Great solution! Thank you very much Frank!</p>
Convergence speed on 8 V100 vs 1 V100

username: Bika
cooked: <p>Hello everybody,<br>
I have a dataset composed of 1 million training data. I am using Distrubted Data parallel with 8 gpus with batch size of 1200 (150 per gpu). My training steps per GPU are 814 step nearly and the steps on 1 GPU is 6500.<br>
My question is , when the training reach 814 steps in multi gpu scenario and 6500 steps in 1 gpu scenario they should have the same loss meaning the multigpu will nearly converge 7 times faster ? or how do i understand more the difference. (I cannot experiment with 1 GPU because it takes 1 hour and 22 minutes per epoch).</p>
CapaGNN program: No module named 'torch_scatter.scatter_cuda'

username: Ndd
cooked: <p>ModuleNotFoundError: No module named ‘torch_scatter.scatter_cuda’</p>
<p>cuda operating environment is 11.1</p>
<p>How can I fix it without lowering the CUDA version?</p>

username: ptrblck
cooked: <p>Based on the error message it seems your code depends on the <code>torch_scatter</code> package, which is either not installed or does not provide the used method.</p>
Evaluating my object detection model using COCO metric shows 0 and -1 values

username: ixiondbz
cooked: <p>I’m currently trying to solve an object detection problem and decided to use faster RCNN on for this. I followed this <a href="https://www.youtube.com/watch?v=Egz4bXMlmDM" rel="noopener nofollow ugc">Youtube video</a>  and their <a href="https://colab.research.google.com/drive/1NziO_b-SW9KmWFh-6C8to9H_QAdpmCBZ?usp=sharing&amp;authuser=1#scrollTo=7K9qfPoTh_k1" rel="noopener nofollow ugc">Code</a>. The loss decreases but the big problem is it won’t evaluate correctly no matter how I try to. I’ve tried looking into the inputs, if there is any sort of size mismatch or missing information but it still doesn’t work. It’s always showing -1 and 0 values for all of its metrics like this</p>
<pre><code class="lang-auto">creating index...
index created!
Test:  [0/1]  eta: 0:00:08  model_time: 0.4803 (0.4803)  evaluator_time: 0.0304 (0.0304)  time: 8.4784  data: 7.9563  max mem: 7653
Test: Total time: 0:00:08 (8.6452 s / it)
Averaged stats: model_time: 0.4803 (0.4803)  evaluator_time: 0.0304 (0.0304)
Accumulating evaluation results...
DONE (t=0.01s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
&lt;coco_eval.CocoEvaluator at 0x7ff9989fea10&gt;
</code></pre>
how to merge many models to one model with shared backbone just use some code ,not a create a new model

username: designer_zhou
cooked: <p>I train some models with different datas ,these models’ some parameters are shared ,<br>
when i inference the models ,i need merge the models to one model ,i know the shared op ,so ,i want to merge these models<br>
shared op to one op with seperate head only when inference not train.<br>
i don’t want to write a new model ,could i write a function to merge the op with same dict name ,and create a new inference model<br>
automatic .<br>
or maybe any way else is ok<br>
thank you !</p>
Unable to use cuda with libtorch

username: tbhaxor
cooked: <p>Python is detecting GPU and CUDA but not the C++ API</p>
<pre><code class="lang-auto">$ uname -r ; python -c "import torch; print(torch.cuda.is_available())"
5.4.164-1-MANJARO
True
$ make &amp;&amp; ./bin/test
[ 50%] Linking CXX executable bin/test
[100%] Built target test
Cuda not available
</code></pre>
<p>Following is the CMake configuration</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.21)
project(test VERSION 0.1.0)
enable_language(CXX CUDA)

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_SOURCE_DIR}/bin)

# libtorch settings
if (DEFINED ENV{CMAKE_PREFIX_PATH})
    message("-- Setting the cmake prefix path to \"$ENV{CMAKE_PREFIX_PATH}\"")
    set(CMAKE_PREFIX_PATH "$ENV{CMAKE_PREFIX_PATH}")
else ()
    message("-- Setting default cmake prefix path to \"/opt/libtorch/share/cmake/Torch\"")
    set(CMAKE_PREFIX_PATH /opt/libtorch/share/cmake/Torch)
endif ()

# find packages
find_package(Torch REQUIRED QUIET)

add_executable(${PROJECT_NAME} main.cpp)
include_directories(${TORCH_INCLUDE_DIRS})
target_link_libraries(${PROJECT_NAME} ${TORCH_LIBRARIES})
</code></pre>
<p>Following is the main.cpp file contents</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;
#include &lt;torch/cuda.h&gt;

int main() {

    if (torch::cuda::is_available()) {
        std::cout &lt;&lt; "Cuda available\n";
    } else {
        std::cout &lt;&lt; "Cuda not available\n";
    }
}
</code></pre>
<p>Downloaded the libtorch from <a href="https://download.pytorch.org/libtorch/cu113/libtorch-cxx11-abi-shared-with-deps-1.10.1%2Bcu113.zip" rel="noopener nofollow ugc">here</a> and extracted in /opt/libtorch</p>
[TorchScript] PackedSequence issue

username: Bartlomiej_Roszak
cooked: <p>I want to export LSTM based pytorch model to TorchScript script but I failed because of such error:</p>
<pre><code class="lang-auto">RuntimeError: 
Arguments for call are not valid.
The following variants are available:
  
  forward__0(__torch__.torch.nn.modules.rnn.LSTM self, Tensor input, (Tensor, Tensor)? hx=None) -&gt; ((Tensor, (Tensor, Tensor))):
  Expected a value of type 'Tensor' for argument 'input' but instead found type '__torch__.torch.nn.utils.rnn.PackedSequence'.
  
  forward__1(__torch__.torch.nn.modules.rnn.LSTM self, __torch__.torch.nn.utils.rnn.PackedSequence input, (Tensor, Tensor)? hx=None) -&gt; ((__torch__.torch.nn.utils.rnn.PackedSequence, (Tensor, Tensor))):
  Expected a value of type 'Optional[Tuple[Tensor, Tensor]]' for argument 'hx' but instead found type 'Tensor (inferred)'.
  Inferred the value for argument 'hx' to be of type 'Tensor' because it was not annotated with an explicit type.

The original call is:
  File "", line 24
                                 enforce_sorted=False)

        output, feat = self.lstm(x, feat)
                       ~~~~~~~~~ &lt;--- HERE
        output = pad_packed_sequence(output,
                                     batch_first=True,
</code></pre>
<p>Script to reproduce:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence


class LSTMBlock(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size

        self.lstm = nn.LSTM(input_size,
                            hidden_size,
                            batch_first=True,
                            bidirectional=False)

    def forward(self, x, input_lengths, feat=None):
        total_length = x.shape[1]
        x = pack_padded_sequence(x,
                                 input_lengths,
                                 batch_first=True,
                                 enforce_sorted=False)

        output, feat = self.lstm(x, feat)
        output = pad_packed_sequence(output,
                                     batch_first=True,
                                     total_length=total_length)[0]

        return output, input_lengths, feat

model = LSTMBlock(100, 100)
script = torch.jit.script(model)
</code></pre>
<p>Is there a way how to handle models trained with packed_sequence?</p>

username: Rachel1
cooked: <p>Did you solve this issue? I met a very similar issue when I was trying to convert LSTM pytorch model to TorchScript.</p>

username: Bartlomiej_Roszak
cooked: <p>I had to change model architecture.</p>

username: Anjul_Sharma
cooked: <p>Hi <a class="mention" href="/u/bartlomiej_roszak">@Bartlomiej_Roszak</a> I’m also facing same issue.<br>
Can you please share what changes did you make exactly?</p>
Warning:grad and param do not obey the gradient layout contract

username: coco-99-coco
cooked: <p>When I try my network,the following warnings occurred:<br>
[W accumulate_grad.h:185] Warning: grad and param do not obey the gradient layout contract. This is not an error, but may impair performance.<br>
[1,2]:grad.sizes() = [128, 2, 1, 1], strides() = [2, 1, 1, 1]</p>
<p>What is the reason for this? How can I correct it?</p>
Second order gradient optimization

username: hieudd
cooked: <p>Hi, I am trying to optimize for the objective that includes gradients dot product:</p>
<pre><code class="lang-python">optimizer = optim(model.parameters())
regular_loss = criteria(model(batch), target)
loss = regular_loss - regular_loss.grad(batch[i]) * regular_loss.grad(batch[j])
</code></pre>
<p>where <code>regular_loss.grad(batch[i])</code> is gradient of <code>regular_loss</code> w.r.t sample <code>i</code> in <code>batch</code>.</p>
<p>The problem is, when I call <code>loss.backward()</code>, since the <code>regular_loss.grad(batch[i])</code> doesn’t have <code>requires_grad</code>, so it is meaningless in optimizer (or its grads is 0).</p>
<p>How can I optimize for my desired objective? Thanks.</p>
Feature extraction using EfficeintNet

username: Kapil_Rana
cooked: <p>I have seen multiple feature extraction network Alexnet, ResNet. And it is quite easy to extract features from specific module for all these networks using<br>
resnet1 = models.resnet50(pretrained=True)<br>
modules1 = list(resnet1.children())[:-1]</p>
<p>But in case of Effcientnet if you use the this command. Just at difference of 1, whole model is gone<br>
See the output of these two<br>
eff1 = EfficientNet.from_pretrained(‘efficientnet-b0’)<br>
modules1 = list(eff1.children())[:-6]<br>
modules1 = nn.Sequential(*modules1)<br>
print(modules1)</p>
<p>resnet1 = EfficientNet.from_pretrained(‘efficientnet-b0’)<br>
modules1 = list(eff1.children())[0:-7]<br>
modules1 = nn.Sequential(*modules1)<br>
print(modules1)</p>
<p>The whole network of MB Blocks is gone.<br>
Considering Table 1 in main paper <a href="https://arxiv.org/pdf/1905.11946.pdf" rel="noopener nofollow ugc">EfficientNet</a><br>
If i want to extract output of stage 5 from pretrained model and want to combine with mine separate CNN network for further classification(self.conv1, self.fc1, self.fc2,num_classes=10).<br>
How I can use that?</p>

username: mMagmer
cooked: <p>Base on your code, i think you are using this implementation of <a href="https://github.com/lukemelas/EfficientNet-PyTorch" rel="noopener nofollow ugc">EfficientNet</a><br>
check this <a href="https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py#L231" rel="noopener nofollow ugc">link</a> for the source code and more information.<br>
you can use <code>extract_endpoints</code> or <code>extract_features</code>.</p>
<blockquote>
<p>Dictionary of last intermediate features<br>
with reduction levels i in [1, 2, 3, 4, 5].<br>
Example:</p>
<blockquote>
<blockquote>
<p>import torch<br>
from efficientnet.model import EfficientNet<br>
inputs = torch.rand(1, 3, 224, 224)<br>
model = EfficientNet.from_pretrained(‘efficientnet-b0’)<br>
endpoints = model.extract_endpoints(inputs)<br>
print(endpoints[‘reduction_1’].shape)  # torch.Size([1, 16, 112, 112])<br>
print(endpoints[‘reduction_2’].shape)  # torch.Size([1, 24, 56, 56])<br>
print(endpoints[‘reduction_3’].shape)  # torch.Size([1, 40, 28, 28])<br>
print(endpoints[‘reduction_4’].shape)  # torch.Size([1, 112, 14, 14])<br>
print(endpoints[‘reduction_5’].shape)  # torch.Size([1, 320, 7, 7])<br>
print(endpoints[‘reduction_6’].shape)  # torch.Size([1, 1280, 7,</p>
</blockquote>
</blockquote>
</blockquote>
<pre><code class="lang-auto"></code></pre>
Epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible

username: ss007k
cooked: <p>I am using a LambdaLR scheduler as follows:</p>
<pre><code class="lang-auto">lmbda = lambda epoch: (1 - epoch/num_epochs)**0.9
scheduler = optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lmbda)

...
# training and validation loop
for epoch in range(1, num_epochs + 1):
    train_one_epoch(model, train_loader, train_criterion, optimizer, epoch=epoch)
    val_metric = validate(model, val_loader, val_criterion, epoch=epoch)
    scheduler.step(val_metric)

</code></pre>
<p>This is also how the documentation recommends <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.LambdaLR.html#torch.optim.lr_scheduler.LambdaLR" rel="noopener nofollow ugc">here</a>.</p>
<p>However, in PyTorch1.9.1, a Warning message is thrown as below:</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:154: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
  warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
</code></pre>
<p>I see the training loop is going through, but not sure if the lr is reducing in scheduler.step() as per the scheduler?</p>
<p>Any comments on the discrepancy between documentation and runtime Warning?</p>
How to achieve fast "vector clustering" sort?

username: 111414
cooked: <p>I have n c-dimensional vectors that formed into a matrix A with the shape of (n, c), how can I perform a quick sort such that the vectors with low Euclidean distances are as close as possible, and the vectors with high distances are as far as possible?<br>
For example, I have</p>
<pre><code class="lang-auto">A = [[0, 3], [0, 0], [0, 1]],
</code></pre>
<p>and the solution can be</p>
<pre><code class="lang-auto">A_sorted = [[0, 3], [0, 1], [0, 0]].
</code></pre>
<p>Explain: Because the original A has a total weighted distance sum of 3x1+1x1+2x2 = 7, and A_sorted has 2x1+1x1+3x2 = 8.<br>
Mathematically, the goal is to maximize the total weighted distance sum.<br>
For the 1-dimensional case, this can be achieved by torch.sort(), and my main concern is if there exists a fast implementation when c ≥ 2 with a time complexity of O(nlog(n))? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=10" title=":thinking:" class="emoji" alt=":thinking:"><br>
After a long struggle, I failed. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"> Could you please do me a favor?</p>

username: 111414
cooked: <p>It seems that this is as hard as Traveling Salesman Problem (I’m not sure), so sad. <img src="https://discuss.pytorch.org/images/emoji/apple/cold_sweat.png?v=10" title=":cold_sweat:" class="emoji" alt=":cold_sweat:"></p>

username: 111414
cooked: <p>I found that there is a similar question: <a href="https://stackoverflow.com/questions/37742358/sorting-points-to-form-a-continuous-line" rel="noopener nofollow ugc">Link</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
Does there exists a fast PyTorch implementation that supports batch processing? (I have B instances, each instance contains N C-dimensional points, i.e. the shape of input data is [B, C, N]) <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=10" title=":thinking:" class="emoji" alt=":thinking:"></p>

username: 111414
cooked: <p>I think it’s hard to directly find the optimal results, so I am pursuing a weaker solution which is fast and can find results that may not be optimal but better than random permutation or just calculating their L2 norm (distance from [0,0]) and sort, at least. <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=10" title=":thinking:" class="emoji" alt=":thinking:"></p>
How to load ImageNet from RAM using pytorch dataloader?

username: Yi_Ren
cooked: <p>Hi everyone, I am seeking help on how to effectively write a data loader for ImageNet. I store the ImageNet-1K dataset (with *.zip) format on my local disk. However, I cannot unzip it to my local disk due to the limitation of the number of files. I wrote a data loader that can get items from *.zip file, but it is too slow. Also, that data loader doesn’t allow me to do any distributional training.</p>
<p>The GPU node I can use has a big RAM (1TB), I am wondering whether could I unzip all the training data to the RAM, and then let the dataloader directly get data from RAM?</p>
<p>P.S.: Someone told me that I could mount a memory disk before training, but I do not have the right to do this.</p>
<p>Thanks very much.</p>
How to compose torchaudio transforms

username: Mohammad_Hassan_Soha
cooked: <p>Hi<br>
Is there an integrated solution to compose torchaudio transforms in the same style as torchvision do?</p>

username: vincentqb
cooked: <p>torchaudio doesn’t provide a dedicated compose transformation since 0.3.0 (see <a href="https://github.com/pytorch/audio/releases" rel="nofollow noopener">release notes</a>). Instead, one can simply apply them one after the other <code>x = transform1(x); x = transform2(x)</code>, or use <code>nn.Sequential(transform1, transform2)</code>.</p>

username: gabrielziegler
cooked: <p>Is there any specific reason to why there is no compose transformation for torchaudio anymore?</p>