akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to pass multiple inputs to forward() in such a way that DistributedDataParallel still hooks up to all of them?

username: Stephan
cooked: <p>I have a model where my forward() has multiple tensors as inputs. I have the model wrapped in DDP and to do this I created an input dictionary with all my input tensors in it. I know very clever.</p>
<p>My problem is that DDP only syncs the first weight matrix of the model so this approach is probably completely wrong.</p>
<p>How do I make DDP work with multiple input tensors?</p>
<p>EDIT: I tried removing the dictionary entirely and passing multiple tensors to forward() and again, only the first parameter is synced.</p>
<p>EDIT 1: As a temporary work around I am using</p>
<pre><code>                    for param in simulator.parameters():
                        dist.all_reduce(param.data, op=dist.ReduceOp.SUM)
                        param.data /= dist.get_world_size()
</code></pre>
<p>But this is obviously not ideal because for one, I have all_reduce called twice, once by DDP and once by me.</p>

username: ptrblck
cooked: <p>I’m not sure how the input arguments to the <code>forward</code> method are related to the parameter sync issue, but <code>DDP</code> should synchronize the entire <code>state_dict</code> at the beginning.<br>
Are all parameters properly registered and returned by <code>model.state_dict()</code> before you wrap it into <code>DDP</code>?</p>

username: Stephan
cooked: <p>So the input arguments in the forward method do not really make a difference. That was a wrong assumption by me.</p>
<p>But my problem remains, that DDP only syncs one weight tensor, but model.state_dict() shows the same tensors listed before and after wrapping in DDP. So I do not understand why the others are not synced. Is there a way to check which tensors have a DDP all_reduce hook attached to them?</p>

username: ptrblck
cooked: <p>Could you share a minimal, executable code snippet showing this behavior, please?</p>
Batching with seq2seq model

username: YashasSamaga
cooked: <p>I have a baseline seq2seq model with LSTM encoder and decoder. The encoder forwards the input sequence and the final hidden and cell state is presumed to contain the summary of the entire sequence. This summary is then fed as the initial hidden and cell state to the LSTM decoder which then generates the output sequence token by token.</p>
<p>I have batched sequences of different lengths by padding at the end (using <code>torch.utils.rnn.pack_padded_sequence</code>). Consider the case where a sequence has padding tokens towards the end. If we pass this sequence through the LSTM encoder, the final hidden/cell state retrieved will not be the real one as this state is reached after processing the padding tokens. Ideally, we would want the hidden state right after the last non-padding token was processed. How can this ideal situation be realized?</p>

username: Imbesat_Rizvi
cooked: <p>If you are using a <code>packed_sequence</code> e.g. from the <code>pack_padded_sequence</code> and pass it through the LSTM, the <code>(h_n, c_n)</code> that you obtain correspond to the last valid non-padded entry so you don’t have to worry about it. For example, see the discussion at <a href="https://discuss.pytorch.org/t/lstm-hidden-cell-outputs-and-packed-sequence-for-variable-length-sequence-inputs/1183">link</a>.</p>
<p>However, if you want to do it yourself, you can do the unpacking of the output and then access the corresponding last valid vectors/tensors using the length of unpacked tensors. E.g. from a <code>packed</code> sequence:</p>
<pre><code class="lang-auto">import torch
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

# create a BxTxD = 3x3x3 tensor with 
# first entry in batch having 2 valid Timesteps and third being a pad entry
# 2nd entry in batch being 1 valid timestep and the other two being pad entries
# and 3rd entry in batch with all valid timesteps
seq = torch.tensor([[[1,1,1],[2,2,2],[0,0,0]],[[3,3,3],[0,0,0],[0,0,0]],[[4,4,4],[5,5,5],[6,6,6]]])
lens = [2,1,3]
packed = pack_padded_sequence(seq, lens, batch_first=True, enforce_sorted=False)

# to get the last valid entries per row in such a packed sequence as above, 
# unpack and extract corresponding entries
seq_unpacked, lens_unpacked = pad_packed_sequence(packed, batch_first=True)

valid_last_entry = seq_unpacked[range(len(lens_unpacked)), lens_unpacked-1]
assert torch.all(valid_last_entry == torch.tensor([[2,2,2],[3,3,3],[6,6,6]]))
</code></pre>
Converting a list to tensor

username: Niki
cooked: <p>Any ways for converting a list as follows to tensor?</p>
<pre><code class="lang-auto">a = [[1,2,3],[4,5,6],[1]]
b = torch.tensor(a)
</code></pre>
<p>For this one, I am getting this error:</p>
<pre><code class="lang-auto">ValueError: expected sequence of length 3 at dim 1 (got 1)
</code></pre>

username: ptrblck
cooked: <p>This won’t work, as your input has varying shapes in dim1.<br>
You could pad the last row with some values:</p>
<pre><code class="lang-python">a = [[1,2,3],[4,5,6],[1, 0, 0]]
b = torch.tensor(a)
</code></pre>

username: Niki
cooked: <p>I see. Thank you very much for your help <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

username: insung3511
cooked: <p>Actually I’m stuck in same trouble but, in my case my list are so large that don’t know how to change that.<br>
My errors like this.</p>
<pre><code class="lang-auto">
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
C:\Users\HILAB_~1\AppData\Local\Temp/ipykernel_14512/3355998112.py in &lt;module&gt;
      1 for epoch in range(EPOCH):
----&gt; 2         X_train_ten, y_train_ten = Variable(torch.tensor(X_train)), Variable(torch.tensor(y_train, dtype=torch.float32, requires_grad=True))
      3         print(X_train_ten.size())
      4 
      5         # X_train_ten = X_train_ten.view(428, -1)

ValueError: expected sequence of length 1889 at dim 1 (got 2300)
</code></pre>
Retrieving forward operations Dynamically

username: JWLee89
cooked: <p>Hello,</p>
<p>I am trying to dynamically analyze the forward pass / operations performed on an input tensor X. Suppose we have the following code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class Model(nn.Module):
    def __init__(self):
        super().__init__()
        # These can all be found using named_modules() or children()
        self.conv = nn.Sequential(
            nn.Conv2d(3, 16, (1, 1), (2, 2), bias=True),
            nn.BatchNorm2d(16),
        )
        self.act = nn.SiLU()

    def forward(self, X):
        # we can retrieve these operations via .modules(), named_modules(), children(), etc.
        output = self.conv(X)
        output = self.act(output)
        # But not this
        final_output = torch.sigmoid(output)
        return final_output
</code></pre>
<p>To module evaluation is done below (for some reason, markdown is parsing the remaining code incorrectly, so I added it separately below)</p>
<pre><code class="lang-auto">
if __name__ == '__main__':
    model = Model()

    for name, module in model.named_children():
        print(f'Name: {name}, module: {module}')
</code></pre>
<p>As noted in the code above, since the <code>self.conv</code> and <code>self.act</code> are <code>nn.Module</code>, we can retrieve these layers using <code>model.named_children()</code>. However, for <code>torch.sigmoid</code>, since the operation is stateless, I cant think of a way to obtain the forward pass information without converting the model into <code>onnx</code>.</p>
<p>Is there a way to do this in PyTorch?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>I don’t know what exactly you are trying to do, but maybe <a href="https://pytorch.org/docs/stable/fx.html"><code>torch.fx</code></a> could help?</p>

username: JWLee89
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you it looks very promising. I will check it out right now.</p>
<p>I asked this question, because I am building a general conversion library that converts PyTorch models into Keras model (that output same values).</p>
<p>I have been moving weights and operations into the keras model by iterating over named_children() and writing appropriate converters, but I ran into a snag where stateless operations not extending <code>nn.Modules</code>  inside the <code>forward</code> function would not be added to the conversion.</p>
<p>The final resort would be to parse the onnx computational graph operations one by one, but that can get pretty ugly, so I don’t want to resort to it.</p>

username: JWLee89
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I played around with it a little bit and this was exactly what I was looking for!. Thank you so much for taking time to respond <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
torch.fx.proxy.TraceError: Proxy object cannot be iterated

username: MartinZhang
cooked: <p>Dear All.</p>
<p>Is there have a function or method to deal : Proxy object cannot be iterated ?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class M(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        """
        x.size() -&gt; (batch, seq, feature)
        """
        _, seq, _ = x.size()
        for i in range(seq):
            ...

m = M()
import torch.fx 
tracer = torch.fx.Tracer().trace(m)
</code></pre>
<p>and then,</p>
<pre><code class="lang-auto">type(range(seq)) -&gt; Proxy
</code></pre>
<p>So, the <code>seq</code>  type is <code>Proxy</code>, is there have a function change <code> Proxy</code> into <code>int</code> ?</p>
<p>Or not to trace the seq variable, which not to change it’s type.</p>
<p>Thx All.</p>
Masking along one dimension

username: anyfin22
cooked: <p>This topic has been discussed but wasn’t finished:</p><aside class="quote quote-modified" data-post="1" data-topic="103889">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anton_z/48/30069_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/masking-one-dimension/103889">Masking one dimension</a> 
  </div>
  <blockquote>
    Suppose I have a tensor A with the following shape: 
torch.Size([5, 16, 5000, 3])

I also have a mask of the same shape: 
torch.Size([5, 16, 5000, 3])

If I apply this mask M directly to the tensor A via 
A = A[M]

I end up with a flattened tensor with single dimension. 
However, I would like to mask out only along dimension 2. In other words, I would like to get a tensor of the shape 
torch.Size([5, 16, 5000 - N, 3])

where N is the number of entries for which mask M is False. 
What is the way …
  </blockquote>
</aside>

<p>I ran into a similar problem. In my case I have a tensor with [60000, 100, 400], that I need to do some operation along the third dimension. I don’t need to do any operation on most of the entries along third dimension. I was able to produce a mask of the same size that I hope reduce the tensor to something like [60000, 100, 150], but I was only able to either flatten the entire array, or keep the original dimension and make masked values zero, either way doesn’t help.</p>
<p>The number of unmasked values for the third dimension also varies from a few to roughly 150… for this I’m thinking making the extra values NaN would be fine.</p>
Where is fx2trt and fx_acc moved to?

username: Wenlong_Shi
cooked: <p>fx2trt and fx_acc have been removed from PyTorch core repository. Where are they moved to?</p>

username: frank-wei
cooked: <p>it will be open source again soon with <a href="https://github.com/pytorch/TensorRT" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/TensorRT: PyTorch/TorchScript compiler for NVIDIA GPUs using TensorRT</a>.</p>
F.pad edge padding

username: spyroot
cooked: <p>Hi Folks,</p>
<p>Anyknow can you do that in torch ? I need to that while tensor on GPU can you do that ?</p>
<blockquote>
<blockquote>
<blockquote>
<p>a = [1, 2, 3, 4, 5]<br>
np.pad(a, (2, 3), ‘edge’)<br>
array([1, 1, 1, 2, 3, 4, 5, 5, 5, 5])</p>
</blockquote>
</blockquote>
</blockquote>

username: ID56
cooked: <p>You can try using <code>F.pad</code> in <code>replicate</code> mode. However it works only for 3D, 4D and 5D tensor inputs, won’t work for 1D or 2D. Also the tensor needs to be a float tensor.</p>
<pre><code class="lang-python">a = torch.tensor([[[1, 2, 3, 4, 5]]]).float()
print(a)
# tensor([[[1., 2., 3., 4., 5.]]])

a_pad = F.pad(a, (2, 3), "replicate")
print(a_pad)
# tensor([[[1., 1., 1., 2., 3., 4., 5., 5., 5., 5.]]])
</code></pre>
GPU Machine Learning Mac OS

username: dspa
cooked: <p>PyTorch recently announced support for GPU processing on Mac using Apple’s Metal Performance Shaders (MPS) as a backend on the PyTorch v1.12 preview(nightly) build. Cuda implements this simply using</p>
<p>device = torch.device(“cuda:0”)</p>
<p>Is there an equilavent script for MacOS on this new build?</p>

username: dspa
cooked: <p>I found an equivalent script:<br>
<code>torch.device("mps")</code></p>
Cuda out of memory when tensorflow and used with torch

username: sourav
cooked: <p>am using Yolov3 by Ultralytics (PyTorch) to detect the behavior of cows in a video. The Yolov3 was trained to detect each individual cow in the video. Each image of the cow is cropped using the X and Y coordinates of the bounding box. Each image then goes through another model to determine whether they are sitting or standing. The second model was also trained with our own dataset. The second model uses Tensorflow and it’s a very simple InceptionV3 model.</p>
<p>However, whenever I try to load both models, I am getting the following errors</p>
<pre><code>RuntimeError: CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 16.00 GiB total capacity; 427.42 MiB already allocated; 7.50 MiB free; 448.00 MiB reserved in total by PyTorch) 
If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>
<p>If the second model is not loaded then the yolov3 (PyTorch) runs without any issue and it does not even use the whole 16GB of VRAM.  Any solution?</p>

username: ptrblck
cooked: <pre><code class="lang-python">7.50 MiB free; 448.00 MiB reserved in total by PyTorch
</code></pre>
<p>This indicates that the other application (TensorFlow in this case) is grabbing the entire device memory and doesn’t allow PyTorch to allocate it anymore.<br>
If I’m not mistaken, this is the default behavior in TensorFlow, so you might want to change it via <code>tf.config.experimental.set_memory_growth</code> (I don’t know if this is the proper API and I just found it via a quick search).</p>
Automatically Select Arithmetic Operations Between Tensors

username: calusbr
cooked: <p>Hello, I’m developing a model that makes the union between two tensors through arithmetic operations. (Example: addition, subtraction, maximum, minimum, multiplication).</p>
<p>But I need to do all the operations individually to check which one has the lowest loss rate. Is there any way to dynamically compare these operations at runtime and select the best one?</p>
<pre><code class="lang-auto">class Model(nn.Module):

		def __init__(self, model_one: nn.Module,
                                            model_two: nn.Module,
                                            in_features_one: int,
                                            in_features_two: int, 
		                            classes: int):
			
			super(Model, self).__init__()

			self.classes = classes
			self.model_one = model_one
			self.model_two = model_two
			
			# Normalize
			self.bn_model_one = nn.BatchNorm1d(in_features_one)
			self.bn_model_two = nn.BatchNorm1d(in_features_two)
			self.bn_fused     = nn.BatchNorm1d(768)
			
			# Layers and FC
			self.layer_1 = torch.nn.Linear(768, 768)
			self.fc      = torch.nn.Linear(768,self.classes)
			
			# Dropout / ReLU / Tanh / Sigmoid
			self.dropout = nn.Dropout(0.1)
			self.relu    = nn.ReLU()
			
		#######################################
		# Forward
		#######################################
		def forward(self, data_model_one, data_model_two, op):

			model_one = self.model_one(data_model_one)
			model_two = self.model_two(data_model_two)

			##############################################################
			# Operations
			##############################################################
			#['cat', 'add', 'sub', '-sub', 'div', 'max', 'min', 'mul', 'pow', '-pow']

			# Concatenate
			if op == 1:
				combined = torch.cat([model_one, model_two], dim=1)
			# Sum
			elif op == 2:
				combined = torch.add(model_one, model_two)
			# Sub
			elif op == 3:
				combined = torch.sub(model_one, model_two)
			# -Sub
			elif op == 4:
				combined = torch.sub(model_two, model_one)
			# Max
			elif op == 5:
				combined = torch.max(model_two, model_one)
			# Min
			elif op == 6:
				combined = torch.min(model_two, model_one)
			# Multi
			elif op == 7:
				combined = torch.mul(model_two, model_one)

			fused = self.dropout(self.bn_fused(self.fusion(combined)))
			fused = self.layer_1(fused)
			fused = self.fc(fused)

			return fused
</code></pre>

username: marksaroufim
cooked: <p>PyTorch is an eager mode framework which means you can have branches and for loops to take specific runtime behavior. Didn’t try running your code but it should in principle work just fine</p>
Layer fusion not fusing BatchNorm

username: Harshil_Dadlani
cooked: <pre><code class="lang-auto">fused_model = copy.deepcopy(model)
model.eval()
fused_model.eval()
fused_model = torch.quantization.fuse_modules(fused_model, [["conv1", "bn1", "relu"]], inplace=False)
</code></pre>
<p><strong>Original model:</strong></p>
<pre><code class="lang-auto">ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
</code></pre>
<p><strong>Fused model:</strong></p>
<pre><code class="lang-auto">ResNet(
  (conv1): ConvReLU2d(
    (0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3))
    (1): ReLU(inplace=True)
  )
  (bn1): Identity()
  (relu): Identity()
</code></pre>
<p>Shouldn’t <code>ConvBnReLU2d</code> be the block name?</p>

username: Megh_Bhalerao
cooked: <p>What do you mean by <strong>block</strong>?</p>

username: Harshil_Dadlani
cooked: <p>A block can also be thought of as a layer.</p>

username: jerryzh168
cooked: <p>ConvReLU2d is the type of the fused module, from fusing (conv - bn - relu) modules i nthe model. and after fusion we assign the result to <code>conv</code> module. What is the question?</p>
Specifying gpu or compute capability when compiling with cmake

username: adamce
cooked: <p><strong>Problem:</strong><br>
I have two GPUs in my system, one Geforce 960 for display and another 3080 for compute. I don’t need to compile my cmake + pytorch program for the 960, because it’ll never be executed on it. I also can’t compile it for the 960, because it has an low compute capabilities and doesn’t support all the features that i’m using.</p>
<p>The problem is, that <code>find_package(Torch REQUIRED)</code> automatically selects compiler options to support all available GPUs. specifying <code>set(CMAKE_CUDA_ARCHITECTURES 86)</code> in cmake is overriden by the find package torch script.</p>
<p>cmake logs</p>
<pre><code class="lang-auto">-- Autodetected CUDA architecture(s):  8.6 5.2
-- Added CUDA NVCC flags for: -gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_52,code=sm_52
</code></pre>
<p>compilation then fails because sm_52 is too low for the code.</p>
<p><strong>Solution:</strong><br>
I started writing the question, but while doing so I found the solution. I’m writing it here, maybe it’ll help others.</p>
<p>Same as with other CUDA programs, you can <code>export CUDA_VISIBLE_DEVICES=0</code> <strong>before</strong> calling cmake. In my case the order of devices did not agree with nvidia_smi, that’s why it didn’t work at first (it shows the 960 first and 3080 second, with indices 0 and 1, respectively).</p>

username: ptrblck
cooked: <p>Instead of masking the device with <code>CUDA_VISIBLE_DEVICES</code> you could also specify the architectures via <code>TORCH_CUDA_ARCH_LIST=8.6 python setup.py install</code>.</p>
Unfolding tensor based on binary map

username: anyfin22
cooked: <p>I have a 4d tensor that I need to perform some operations along the fourth dimension.</p>
<p>I have a 2d binary map for the first two dimensions, where I need to perform the operation only if it’s true.</p>
<p>I found that I can do this to get the sub tensor that I need to perform the operations on:</p>
<pre><code class="lang-auto">sub_patches=patches[binary_map, :, :]
</code></pre>
<p>which gives the desired time speed up.</p>
<p>but this combines the first two dimension, I need to put back where the values were.</p>
<p>Does anybody know how to do that?</p>
<p>Thanks</p>

username: Andrei_Cristea
cooked: <p>Can you just assign back the sub tensor to the original at the desired index?</p>
<pre><code class="lang-auto">t = torch.arange(0, 36).view(2, 2, 3, 3)
binary_map = (torch.arange(0, 4) % 2 == 0).view(2, 2)
sub_t = t[binary_map, :, :]
sub_t = sub_t * 0
t[binary_map, :, :] = sub_t  # just assigning it back
print(t)

Output:
tensor([[[[ 0,  0,  0],
          [ 0,  0,  0],
          [ 0,  0,  0]],

         [[ 9, 10, 11],
          [12, 13, 14],
          [15, 16, 17]]],


        [[[ 0,  0,  0],
          [ 0,  0,  0],
          [ 0,  0,  0]],

         [[27, 28, 29],
          [30, 31, 32],
          [33, 34, 35]]]])
</code></pre>

username: anyfin22
cooked: <p>thanks. I tried this and thought things were put back in different order. It turns out it’s because I was calculating median and the result I was comparing against took the smaller number when there’s even number, while pytorch took the average.</p>
Training a weight matrix

username: S_I_R_U_S
cooked: <p>Hi,</p>
<p>I am trying to train a weight matrix but I am having a hard time when working with a matrix that has a larger size (let’s say 500x500 or more). I have the following equation :</p>
<p>x(t+dt) = x(t) * (1-dt)+sigma(<strong>W</strong> * x(t) - u)*dt</p>
<p>Here, dt and u are some constants. <strong>W</strong> is a matrix nxn that I want to train and x(t) is a vector nx1. sigma is the sigmoid function that is apply to every element of the matrix. To put you into context, I want to train a matrix <strong>W</strong> (initially random) to converge to a specific matrix. To find the value at a time x(t+dt), I need the value of x at a time x(t). My code is able to converge to a close matrix. However, when working with a bigger matrix, I am unable to obtain the correct matrix. Here’s my code:</p>
<p>I begin by creating a random matrix of size nxn:</p>
<pre><code class="lang-auto">    W = torch.randn(n, n, dtype=torch.float32, requires_grad=True)
    W = nn.Parameter(W) 
</code></pre>
<p>Then I set up my model :</p>
<pre><code class="lang-auto">    model = RNNModel(W=W, dt=dt, mu=mu)
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
    criterion= nn.MSELoss()
</code></pre>
<p>RNNModel is a class that I have created :</p>
<pre><code class="lang-auto">class RNNModel(nn.Module):

    def __init__(self, W, dt, mu):
        super(RNNModel, self).__init__()
        self.W = W
        self.dt = dt
        self.mu = mu
    
    def forward(self, x):
        return x * (1-self.dt) + (torch.sigmoid(torch.matmul(self.W, x) - self.mu))*self.dt
</code></pre>
<p>Then I do the training :</p>
<pre><code class="lang-auto">    #Training
    for epoch in range(n_iters):
        #Forward pass
        #I call my model to find x_pred
        #Loss
        loss = criterion(x, x_pred)

        #Gradient
        loss.backward()

       #update weight
        optimizer.step()
        optimizer.zero_grad()
</code></pre>
<p>As I mention up top, I train found my value x_pred with x. Then I modify my matrix <strong>W</strong>. Do you find a better way or do you have any tips to help me?</p>
<p>Thanks in advance!</p>
How to flatten arrays via column-major ordering like numpy's "C" vs "F" arguments?

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I was wondering if it were possible to specify how flattening or reshaping is done via PyTorch in a similar way to how Numpy (<a href="https://numpy.org/doc/stable/reference/generated/numpy.matrix.flatten.html" rel="noopener nofollow ugc">here</a>) allows you to pass a “C” or “F” argument which specifies row or column ordering respectively.</p>
<pre><code class="lang-auto">a = torch.Tensor([[1,2], 
                  [3,4]])

a.detach().numpy().flatten('C') #returns array([1., 2., 3., 4.], dtype=float32) (row)
a.detach().numpy().flatten('F') #returns array([1., 3., 2., 4.], dtype=float32) (column)
a.flatten() #returns tensor([1., 2., 3., 4.]) (row)
</code></pre>
<p>As you can see <code>.reshape</code> defaults to row ordering, is it possible to specify column majoring in some way? Potentially this might be something to open as an issue?</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>I don’t think PyTorch supports the Fortran- style flattening, but you should be able to transpose the tensor before flattening it if this order is needed.</p>
How to drop top layer /head of a finetuned transformers model

username: slayer
cooked: <p>how to drop top layer or head from a pretrained/fine tuned transformers model.<br>
for example i have fine tuned a deberta-v3-large for NER, and i want to use same model for multi class classification, as data is same so i have decided that i will use same finetuned model for classification task and add a new head in place of NER head and will train for one or 2 epochs, and is there any way to freeze previous layers and train only head on this task like we do in CNN models(like ResNet50).<br>
I am new to pytorch, i am familiar how can we freeze initial layers of CNN in tensorflow, can we do same in pytorch.<br>
thanks!</p>

username: ptrblck
cooked: <p>You can iterate the parameters you want to freeze and set their <code>.requires_grad</code> attribute to <code>False</code>.<br>
Depending on the model architecture, something like this could work:</p>
<pre><code class="lang-python">for param in model.feature_extractor.parameters():
    param.requires_grad = False
</code></pre>
Interleaving FFT real & complex parts in a tensor

username: grid_world
cooked: <p>I have a use-case where I have to do FFT for a given tensor as. Here, FFT is applied to each of the 10 rows, in a column-wise manner which gives the dimension (10, 11) post FFT.</p>
<pre><code class="lang-auto"># Random data-
x = torch.rand((10, 20))

# Compute RFFT of 'x'-
x_fft = torch.fft.rfft(x)

# Sanity check-
x.shape, x_fft.shape
# (torch.Size([10, 20]), torch.Size([10, 11]))

# FFT for the first 2 rows are-
x_fft[:2, :]
'''
tensor([[12.2561+0.0000j,  0.7551-1.2075j,  1.1119-0.0458j, -0.2814-1.5266j,
          1.4083-0.7302j,  0.6648+0.3311j,  0.3969+0.0632j, -0.8031-0.1904j,
         -0.4206+0.9066j, -0.2149+0.9160j,  0.4800+0.0000j],
        [ 9.8967+0.0000j, -0.5100-0.2377j, -0.6344+2.2406j,  0.4584-1.0705j,
          0.2235+0.4788j, -0.3923+0.8205j, -1.0372-0.0292j, -1.6368+0.5517j,
          1.5093+0.0419j,  0.5755-1.2133j,  2.9269+0.0000j]])
'''

# The goal is to have for each row, 1-D vector (of size 11) as follows:
# So, for first row, the desired 1-D vector (size = 11) is-
[12.2561, 0.0000, 0.7551, -1.2075, 1.1119, -0.0458, -0.2814, -1.5266,
1.4083, -0.7302, 0.6648, 0.3311, 0.3969, 0.0632, -0.8031, -0.1904,
-0.4206, 0.9066, -0.2149, 0.9160, 0.4800, 0.0000]
'''

</code></pre>
<p>Here, you are taking the real and imaginary components and placing them adjacent to each other.</p>
<p>Adjacent means:</p>
<pre><code>[a_1_real, a_1_imag, a_2_real, a_2_imag, a_3_real, a_3_imag, ....., a_n_real, a_n_imag]
</code></pre>
<p>Since for each row, you get 11 FFT complex numbers, a_n = a_11.</p>
<p>How to go about it?</p>

username: Andrei_Cristea
cooked: <p>Could you clarify your desired output, considering that <code>x_fft[:-2,:].real</code> and <code>.imag</code> are 2d Tensors with shape (8, 11). What exactly is <code>an</code> from your message? That suggests a 1d Tensor and is ambiguous in the 2d case. Perhaps I misunderstood!</p>
<p>Also in your example above, the first 4 numbers are a bit confusing:</p>
<pre><code class="lang-auto">[1.1547e+01, 0.0000, -0.8197, 0.1383, ...
...
[a1_real, a1_imag, a2_real, a2_imag, ...
</code></pre>
<p>The 3rd number -0.8197 should be a2_real, but -0.8197 looks like it’s in the imaginary part when looking at the tensors above.</p>

username: grid_world
cooked: <p>I have edited the question. Hope it helps?</p>

username: Andrei_Cristea
cooked: <p>Got it! Try this (stacking along the 3rd dimension):</p>
<pre><code class="lang-auto">torch.dstack([x_fft.real, x_fft.imag]).view(10, 11 * 2)
</code></pre>