akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Not able to append multiple batches to single output whilst tokenizing

seanfarr788: <p>I am trying to batch iterate over my dataset. So say my dataset is 5000 lines long, I am dividing this into 1000 blocks, and then passing this blocks through a loop, where I am tokenizing the block and then appending the result to the original ‘inputs’.</p>
<p>The loop then goes back to the top, and gets the next batch however in doing so it is overwriting the previous block, so the resultant product is a tensor with length of 1000 (containing the last 1000 block)</p>
<pre><code class="lang-auto">def batchReader(Dataset, block_size=1000):
    block = []
    for line in Dataset:
        block.append(line)
        if len(block) == block_size:
            yield block
            block = []
    if block:
        yield block

input_ids = []
attention_masks = []
token_type_ids = []
count=0
with open('5000.txt') as Dataset:
    blocks = batchReader(Dataset)
    for block in blocks:
        inputs = tokenizer(
                block,     
                truncation=True,
                max_length=512,
                padding='max_length',
                return_tensors='pt', 
            )
        input_ids.append(inputs['input_ids'])
        attention_masks.append(inputs['attention_mask'])
        token_type_ids.append(inputs['token_type_ids'])
        inputs['labels'] = inputs.input_ids.detach().clone()
</code></pre>
<p>I am looking for the output of this code to yield a tensor with 5000 length, any help much appreciated!</p>
How to find Action in an Actor Critic Method with infinite possible actions

Khabbab_Zakaria: <p>Hi,<br>
I am working on an Actor Critic Method where there are infinite numbers of possible actions (let’s say movement of a robot, where the possible movement is not just left-right-up-down but every possible angle).<br>
My obvious idea was to have output mu and sigma^2 in the policy:</p>
<pre><code class="lang-auto">class Policy(nn.Module):
    """
    implements both actor and critic in one model
    """
    def __init__(self):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(state_size+1, 128)

        self.fc2 = nn.Linear(128, 64)

        # actor's layer
        self.action_head = nn.Linear(64, 1)
        self.mu = nn.Sigmoid()
        self.var = nn.Softplus()

        # critic's layer
        self.value_head = nn.Linear(64, 1)


    def forward(self, x):
        """
        forward of both actor and critic
        """
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))

        action_prob = self.action_head(x)
        mu = self.mu(action_prob)
        var = self.var(action_prob)

        state_values = self.value_head(x)

        return mu, var, state_values
</code></pre>
<p>Now, I also need to calculate the action from this (mu, sigma**2=var) pair. I also need to calculate the probability of happening of that action from that Normal Distribution. I am doing these for them:</p>
<pre><code class="lang-auto">sigma = torch.sqrt(var)
action = torch.normal(mu, sigma)
action = torch.clip(action, 0, 1)
pdf_probability = stats.norm.pdf(action.cpu().detach().numpy(), loc=mu.cpu().detach().numpy(), scale=sigma.cpu().detach().numpy())
</code></pre>
<p>Now I have some questions. What I am doing, is it ok? It does not feel ok, as I suppose the pdf_probability should be backpropagated as well, but during conversion Tensor-&gt;Numpy-&gt;Tensor, we lose it.<br>
I have gone through <a href="https://discuss.pytorch.org/t/resolved-actor-critic-with-a-large-amount-of-possible-actions/3933/5">[resolved] Actor Critic with a large amount of possible actions - reinforcement-learning - PyTorch Forums</a>, where they discussed a similar issue. They are also calculating mu, sigma^2 from the policy, but they never talked about how to calculate Action from these (mu,sigma^2).<br>
I also read the A3C paper <a href="https://arxiv.org/pdf/1602.01783v2.pdf" rel="noopener nofollow ugc">Asynchronous Methods for Deep Reinforcement Learning (arxiv.org)</a>, where they stated that the mu should be calculated by a Linear Layer. Is it mandatory? In my case, I want to only consider positive angles, so, mu should be followed by Sigmoid or ReLU right?</p>
Addinf GPU-based LU decomposition of sparse matrix

Hao_Peng: <p>Hi guys,</p>
<p>Thanks for your excellent job!</p>
<p>I had some troubles using PyTorch, especially when I needed to solve the sparse linear equations Ax=b (usually by LU decomposition).   Unfortunately, PyTorch doesn’t seem to have this feature.<br>
I wonder if you have any plans to add this feature?<br>
I found a great library to do LU decomposition based on GPU, but it’s only in c++.<br>
<a href="https://github.com/sheldonucr/GLU_public" rel="noopener nofollow ugc">sheldonucr/GLU_public: GLU - GLU-accelerated Sparse Parellel LU factorization solver V3.0 (github.com)</a></p>
<p>Would you like to add a Pytorch wrapper for it?</p>
<p>Many thanks!</p>
Plot ellipse with pytorch

Vijish_Madhavan: <p>I want to plot ellipse with pytorch, Convert the below code to pytorch.</p>
<pre><code class="lang-auto">def ellipse(n, R=1):  
theta = random.uniform(0, 360, n)
X1 = R * np.cos(theta)
X2 = R * np.sin(theta)

X1 = X1.reshape(n, 1)
X2 = X2.reshape(n, 1)
X = hstack((X1, X2))
y = ones((n, 1))
return X, y
</code></pre>

anantguptadbl: <p><a class="mention" href="/u/vijish_madhavan">@Vijish_Madhavan</a> plotting ellipse in pytorch does not make sense. Any numpy array can be converted to a Tensor</p>
<pre><code class="lang-auto">a=np.array([1,2,3])
torch.Tensor(a)
</code></pre>
<p>Is there more to this?</p>
Is there any research about value distribution of Conv's weight

Forceless: <p>I find that most CONV’s weights are distributed in a small range, a subset of [-1,1].<br>
So I did some works based on this, and I want to know if there was any research paper or explanation about this</p>

thecho7: <p>I couldn’t find the original paper which proposes the importance of weight normalization.<br>
However, the reason is simple. The network can be converged with the values in range of [-1, 1]</p>
<p>Here’s a sequence of multiplication of 3 numbers</p>
<ol>
<li>10 x 10 x 10</li>
<li>0.1 x 0.1 x 0.1</li>
</ol>
<p>Convergence/divergence is important. We can easily handle the value of 0 but not 1000000000000…</p>

Forceless: <p>Thank you, your words inspired me.<br>
I searched weight normalization and I don’t know is that what you said.<br>
Seems like Weight Normalization is helpful to training, and was introduced in this <a href="https://arxiv.org/pdf/1602.07868v3.pdf" rel="noopener nofollow ugc">paper</a>.<br>
Is there any evidence that torch implemented this?<br>
And looks like weight normalization is an optional operation in models.<br>
Is there any evidence showing that a network’s weight will be normalized</p>

anantguptadbl: <p><a class="mention" href="/u/forceless">@Forceless</a></p>
<p>As far as i could find, in the torch documentation for conv2d, the weights of conv2d<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d</a></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92e49618913c0a0d51e742ac9d29bd45f63d40fa.png" alt="image" data-base62-sha1="kXtsbub6QTQAGtqcs2IxX8U5adk" width="606" height="103"></p>
<pre><code class="lang-auto">import math
import torch
from torch.autograd import grad
import torch.nn as nn

m = nn.Conv2d(16, 33, 3, stride=2)
input = torch.randn(20, 16, 50, 100)
output = m(input)

# According to the equation
k = math.sqrt(1.0/(16 * 3 *3))
print([-k, k])

# UNIT TEST
print([torch.min(m.weight), torch.max(m.weight)])
</code></pre>
<pre><code class="lang-auto">[-0.08333333333333333, 0.08333333333333333]
[tensor(-0.0833, grad_fn=&lt;MinBackward1&gt;), tensor(0.0833, grad_fn=&lt;MaxBackward1&gt;)]
</code></pre>

Forceless: <p>I have a question.<br>
It looks like the weight was just initialized like that.<br>
After several epochs of training, did weight value still distributed in this range</p>

thecho7: <p>Nope, the weights are just trained to minimize the loss.<br>
If there is no constraint like <code>clip</code> or something, the weights can be varied in free.</p>

Forceless: <p>Yes, that’s what I mean.</p>
How to run autoencoder on single image/sample for inference

sparshgarg23: <p>I have trained an autoencoder and the training results seem to be okay.<br>
But when i run the model on a single image,the generated results are incosistent.<br>
Any ideas on how I can run the autoencoder on a single example.<br>
The autoencoder model in my case accepts an input of dimension (256x256+3,1)<br>
My evaluation code is as follows</p>
<pre><code class="lang-auto">img_dir='blender_files/Model_0/image_0/model_0_0.jpg'
img=cv2.resize(cv2.imread(img_dir,0),(256,256))#the image of dimension 256x256
img=torch.from_numpy(img)
#print(img)
coord=np.array([0,0,0]) #3x1 vector which i need to predict using the autoencoder
coord=torch.from_numpy(coord)
img=torch.unsqueeze(torch.flatten(img),1)
#print(img)
coord=torch.unsqueeze(torch.flatten(coord),1)
X=torch.cat((img,coord),dim=0)#the input feature which i am feeding to the model ,whose size is ```torch.Size([65539])```



#Feed feature vector to model
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
features=X.to(device)
features=torch.flatten(features)
print(features.shape)
model=AutoEncoder(num_features=features.shape[0],num_hidden_1=num_hidden_1,num_hidden_2=num_hidden_2,num_hidden_3=num_hidden_3)
model=model.double()
model.load_state_dict(torch.load('blender_files/Model_0/image_0/saved_model_img.pth'))
model.eval()
torch.manual_seed(123)
with torch.no_grad():
    decoded_rep=model(features[None,...])
print(decoded_rep)
</code></pre>
<p>The output i am getting is as follows</p>
<pre><code class="lang-auto">tensor([[3.2402e+16, 3.4111e+16, 3.2839e+16,  ..., 4.4640e+16, 4.4089e+16,
         7.7656e+15]], dtype=torch.float64)
</code></pre>
<p>But the decoded output obtained during training is</p>
<pre><code class="lang-auto">[ 67.5205,  67.6745,  67.6265,  ..., 124.9578, 124.8637,   4.7602]
</code></pre>
<p>As you can see both outputs are not even close to one another.<br>
I was looking at vanilla autoencoders and it seems for generation purposes,they are not really a good choice,as such what other models can I use .<br>
Since in my case I am interested in predicting the last 3 values of the feature ,would an autoregressive model suit my case better.</p>
<p>The following also adds more weight to my point<br>
One common application done with autoregressive models is auto-completing an image. As autoregressive models predict pixels one by one, we can set the first N pixels to predefined values and check how the model completes the image</p>
<p><a href="https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial12/Autoregressive_Image_Modeling.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial12/Autoregressive_Image_Modeling.html</a></p>
Torch.distributed.barrier stuck with cuda 11.4

Hyeonuk_Woo: <p>Hi, I have a problem with using torch.distributed with CUDA 11.4 environment(Nvidia rtx A6000).</p>
<p>It seems that torch.distributed  and DDP don’t work properly. I test with simple code below.</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel
import os
def get_local_rank() -&gt; int:
    return int(os.environ.get('LOCAL_RANK', 0))
def init_distributed() -&gt; bool:
    world_size = int(os.environ.get('WORLD_SIZE', 1))
    distributed = world_size &gt; 1
    if distributed:
        backend = 'nccl' if torch.cuda.is_available() else 'gloo'
        dist.init_process_group(backend=backend, init_method='env://')
        if backend == 'nccl':
            print ('set nccl correctly!')
            torch.cuda.set_device(get_local_rank())
        else:
            logging.warning('Running on CPU only!')
        assert torch.distributed.is_initialized()
    return distributed
dist_stat=init_distributed()
local_rank = get_local_rank()
print ('hi',local_rank)
dist.barrier(device_ids=[get_local_rank()]) 
print ('bye',local_rank)
</code></pre>
<p>Running  above code with command as below,<br>
<code>export CUDA_VISIBLE_DEVICES=0,1,2,3</code><br>
<code>python -u -m torch.distributed.run --nnodes=1 --nproc_per_node=gpu test_dist.py </code></p>
<p>then I got message:<br>
``<br>
set nccl correctly!<br>
set nccl correctly!set nccl correctly!</p>
<p>set nccl correctly!<br>
hi 0<br>
hi 1<br>
hi 2<br>
hi 3<br>
``<br>
And stuck at dist.barrier() forever. However, when I use only two gpus, everything is fine. What should I look for in order to solve the problem?</p>
<p>The version of Python is 3.9.7 and  torch is 1.10.1+cu113</p>

GeislerSimon: <p>I experience the same issue with a very similar setup.</p>
Most Efficient Way to Get Ordered Slice of Array as Tensor

J_Johnson: <p>Suppose we have an array:</p>
<p><code>A = torch.arange(start = 1, end = 101)</code></p>
<p>And we want to take a rolling window over the array to produce matrix B such that:</p>
<pre><code class="lang-auto">B=([1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
      [2, 3, 4, ...],
      [3, 4, 5, ...],
      [...],
      [91, 92, 93, 94, ..., 99, 100])
</code></pre>
<p>I know we could run a loop through to make it, but is there a faster, parallelized way in Pytorch to render this?</p>

thecho7: <pre><code class="lang-auto">A = torch.arange(start = 1, end = 101)
B = (A.reshape(-1, 10))
</code></pre>
<p>Any idea?</p>

J_Johnson: <p>Hi <a class="mention" href="/u/thecho7">@thecho7</a> ,</p>
<p>That is not quite what I am looking for. Here is an example via loop:</p>
<pre><code class="lang-auto">A=torch.arange(start=1, end=101)

def stack_sequence(sequence, window_size):
    thread_range=sequence.size(-1)-window_size+1
    out=torch.empty(0,window_size)
    for i in range(thread_range):
        out=torch.cat([out,sequence[i:i+window_size].unsqueeze(0)])

    return out

B = stack_sequence(A,10)
print(B)
</code></pre>
<p>What I am looking for is a more efficient solution, if one exists.</p>

anantguptadbl: <pre><code class="lang-auto">torch.stack([A[x:x+10] for x in range(91)])
</code></pre>

J_Johnson: <p>Thanks <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> . That works much faster than the loop.</p>
<pre><code class="lang-auto">import torch
import time


def stack_sequence(sequence, window_size):
    thread_range=sequence.size(-1)-window_size+1
    out=torch.empty(0,window_size)
    for i in range(thread_range):
        out=torch.cat([out,sequence[i:i+window_size].unsqueeze(0)])

    return out

def stack_sequence2(sequence, window_size):
    return torch.stack([sequence[x:x+window_size] for x in range(sequence.size()[-1]-window_size+1)])



A=torch.arange(start=1, end=100001)
now=time.time()
stack_sequence(A, 10)
print(time.time()-now)

now=time.time()
stack_sequence2(A,10)
print(time.time()-now)
</code></pre>
<p>Results:</p>
<pre><code class="lang-auto">32.05299949645996
0.2589995861053467
</code></pre>
Learning rate using "ExponentialLR"

cbd: <p>Is it possible to call function to set the value of “gamma” using “ExponentialLR”?</p>
<p>Below code call the function “lambda_rule”</p>
<pre><code class="lang-auto">lr.LambdaLR(optimizer, lr_lambda=lambda_rule)
</code></pre>
<p>but gives error “TypeError: unsupported operand type(s) for *: ‘float’ and ‘function’” as it is not calling “lambda_rule” function for below code.</p>
<pre><code class="lang-auto">lr.ExponentialLR(optimizer, gamma=lambda_rule)

</code></pre>
Bazel link to torch as external dependency

jackshi: <p>I use the following WORKSPACE rule:</p>
<pre><code class="lang-auto">git_repository(
    name = "pytorch",
    init_submodules = True,
    remote = "https://github.com/pytorch/pytorch.git",
    tag = "v1.7.1",
)
</code></pre>
<p>Everytime I try to invoke <code>@pytorch//:torch</code> I get <code>Unable to find package for @pybind11_bazel//:build_defs.bzl: The repository '@pybind11_bazel' could not be resolved.</code></p>
<p>If I download the repo and link it with <code>local_repository</code>, the same problem happens.<br>
If I download the repo and just <code>bazel build :torch</code> everything is fine. It seems that there’s something preventing the linking of pytorch as an external dependency.</p>
<p>Has anyone ran into this issue before? Any thoughts on what could have gone wrong?</p>

jackshi: <p>If I add <code>pybind11_bazel</code> to my <code>WORKSPACE</code>, the error becomes:<br>
<code>no such package 'tools/config': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package: /home/usr_name/projects/tools/config</code><br>
It seems that bazel is trying to resolve dependency from my local workspace instead of pytorch’s remote workspace.</p>

aikez: <p><a class="mention" href="/u/jackshi">@jackshi</a> Hi Jack, do you solve the problem?</p>
Resizing does not work

ElQadasi: <p>I have this code and I spent a lot of time figuring out why the size is still the same [28,28]:</p>
<pre><code class="lang-auto">class FashionDataset(Dataset):
    def __init__(self,dataset_dir,train_option,transform=None):
        self.dataset = datasets.FashionMNIST(root=dataset_dir,
                                           train=train_option,
                                           download=True)
        self.transform = transform
        self.labels = self.dataset.targets
        
    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, index):
        data, target = self.dataset[index]
        data = self.transform(data)
        return index, data, target

transform = transforms.Compose([transforms.ToTensor(),transforms.Resize((224, 224))])

train_dataset = FashionDataset(dataset_dir = dataset_dir, train_option = True, transform = transform)

test_dataset = FashionDataset(dataset_dir = dataset_dir, train_option = False, transform = transform)

train_loader = DataLoader(train_dataset, 
                          batch_size=batch_size,
                          sampler=SubsetRandomSampler(range(len(train_dataset))))

test_loader = DataLoader(test_dataset, 
                         batch_size=batch_size, 
                         shuffle=False)
</code></pre>
<p>Thank you for your help in advance.</p>

ptrblck: <p>Your code works as expected:</p>
<pre><code class="lang-python">idx, x, y = next(iter(train_loader))
print(x.shape)
# &gt; torch.Size([10, 1, 224, 224])

idx, x, y = next(iter(test_loader))
print(x.shape)
# &gt; torch.Size([10, 1, 224, 224])
</code></pre>

ElQadasi: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> , my mistake was I was trying to print the shape by printing train_loader.dataset.data.shape which is different when you iterate through the for loop or next() because it calls <em>getitem</em> which involves the transforms.</p>
How to avoid pickling errors when using PyTorch DDP with S3 data?

Olivier-CR: <p>Hi,<br>
I’m using Pytorch DDP to data parellize this code <a href="https://github.com/aws-samples/sagemaker-a2d2-segmentation-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - aws-samples/sagemaker-a2d2-segmentation-pytorch</a></p>
<p>I’m launching the training with</p>
<pre><code class="lang-auto">def main():
    
    # launches 1 data parallel replica per GPU
    mp.spawn(
        fn=train_replica,
        nprocs=world_size,
        join=True)


if __name__ == "__main__":
    
    main()
</code></pre>
<p>My dataset class uses boto3 to move records from S3 to memory.<br>
To avoid boto3 serialization issue, the dataset and dataloaders are instantiated from within the train_replica function, so that they should all be created on their own and not boto3 things should need to be serialized.</p>
<p>However, my training fails for</p>
<pre><code class="lang-auto">
_pickle.PicklingError: Can't pickle &lt;class 'boto3.resources.factory.s3.ServiceResource'&gt;: attribute lookup s3.ServiceResource on boto3.resources.factory failed
</code></pre>
<p>How to avoid that? Given how popular S3 is, I guess it is possible to use DDP with a dataset <strong>getitem</strong> that uses S3 right? how to avoid the serialization problem?</p>

rvarm1: <p>It looks like some multiprocessing code is attempting to pickle some data to send across processes, but <code>boto3</code> resource can’t be pickled.</p>
<p>Similar to this comment: <a href="https://github.com/pytorch/pytorch/issues/67681#issuecomment-959303197" class="inline-onebox" rel="noopener nofollow ugc">PicklingError encountered when using multiple GPUs · Issue #67681 · pytorch/pytorch · GitHub</a>, you might want to try:</p>
<ol>
<li>Passing arguments and construct boto3 client in the subprocess</li>
<li>Use/cache/store results of using boto3 APIs in some format that can be pickled, and pass around the result.</li>
</ol>
Auto Grad for External Tensor?

hgdev: <p>Hi there PyTorch beginner here.</p>
<p>So I’m trying to use make an autoencoder in PyTorch in combination of with an external custom graphics library. I know that generally, the loss function of the antoencoder takes two tensors as inputs, an input image tensor A, and its generated counterpart image tensor B.</p>
<p>But what I’m trying to do is slightly different. Instead of using the generated image tensor B directly in the loss function, I need to do some external graphic processing on it, turn the resulting array of numbers into another tensor C (which has the same shape as B), and then calculate the loss between A and C. But since C is processed externally, it’s connection to the recorded graph is broken. I have very limited knowledge about autograd (and PyTorch in general), so I’m a bit lost about what can be done properly to achieve this.</p>
<p>I’ve made a few attempts:</p>
<ol>
<li>
<p>Plug C directly in the loss function. It didn’t work (which is expected). I got the following error:<br>
“RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”. My understanding is that since C is a new tensor without any prior operation history, autograd doesn’t know what to do backward for it.</p>
</li>
<li>
<p>I also tried to “fake” the history for C by doing something like C = B.mul(0.001).add(C), in an attempt to “borrow” the autograd history from B while keeping the influence of B as minimum as I could. This time the loss function didn’t complain, but the loss seemed to plateau after a while, and the model didn’t converge. I tried other remixing formulas, like C = B.mul(0.003).add(C.mul(3)), etc., but none of them worked.</p>
</li>
<li>
<p>And if I plug B into the loss function, and model converged with no issue. This is just to double check the model &amp; hyperparameters are correct. And I can’t re-implement the processes that turns B to be C using PyTorch because B has to go through some simulation in the external graphics library to become C.</p>
</li>
</ol>
<p>My questions are:</p>
<ol>
<li>Is what I was doing in attempt <span class="hashtag">#2</span> the right direction to do it? Meaning it’s just the number in the formula are incorrect? Or a more complicated / different formula is needed?</li>
<li>If not, what method should be used for this kind of situation?</li>
<li>Or, using external tensor in autograd is not possible at all?</li>
</ol>
<p>Thanks for your time reading this. Any help is greatly appreciated!</p>

ptrblck: <p>To use operations from other libraries, you could write a custom <code>autograd.Function</code> and implement the <code>forward</code> as well as <code>backward</code> methods manually as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>
Re-initialization of torch.jit.script module

nitya05: <p>Hi,</p>
<p>I have a torch.jit.script module saved in Python and I load it in C++ for training. There are certain parameters which are passed to the <strong>init</strong>() of this module class, which I understand, are a part of the torch.jit.script module. Is there any way of changing these module class parameters after loading the module in CPP?</p>
<p>Note: If I create an api for modifying a class parameter, and if that parameter is being used by the forward api, then I get a runtime error while creating a torch.jit.script module of that class.</p>
Model does not learn after ternarization of weights contrary to the paper mentioned below

puranjay_mishra: <p>Hello. I apologize if this not the right place to ask this but I don’t now any place better!</p>
<p>I’m implementing the ‘Ternary Weights Network’ paper by Fengfu Li , Bo Zhang and Bin Liu(<a href="https://arxiv.org/abs/1605.04711" class="inline-onebox" rel="noopener nofollow ugc">[1605.04711] Ternary Weight Networks</a>).</p>
<p>I’m training a simple Covnet with linear layers on the MNIST dataset. Without ternarization, the exact same model converges with high accuracy, but after ternarization of the linear layers, the model predicts all the classes with equal probability(0.1 for all the classes).</p>
<p>What could be the reason for this?</p>
<p>Code which ternarizes the linear layers -</p>
<p>class TernarizeOp():<br>
def <strong>init</strong>(self, model):<br>
count_targets = 0<br>
self.model = model<br>
for m in model.modules():<br>
if isinstance(m, nn.Linear):<br>
count_targets += 1<br>
self.ternarize_range = np.linspace(0, count_targets - 1, count_targets).astype(‘int’).tolist()<br>
self.num_of_params = len(self.ternarize_range)<br>
self.saved_params = []<br>
self.target_modules = []</p>
<pre><code>    for m in model.modules():
        if isinstance(m, nn.Linear):
            tmp = m.weight.data.clone()
            self.saved_params.append(tmp)  # tensor
            self.target_modules.append(m.weight)  # Parameter

def SaveWeights(self):
    for index in range(self.num_of_params):
        self.saved_params[index].copy_(self.target_modules[index].data)

def TernarizeWeights(self):
    alpha = []
    for index in range(self.num_of_params):
        output,alpha_tmp = self.Ternarize(self.target_modules[index].data)
        self.target_modules[index].data = output
        alpha.append(alpha_tmp)
    return alpha

def Ternarize(self, tensor):
    tensor = tensor.cuda()
    # print(tensor[0])

    output = torch.zeros(tensor.size()).type(torch.cuda.FloatTensor)

    new_tensor = tensor.abs()
    delta = torch.mul(0.75, torch.mean(new_tensor, dim=1))
    # print(delta[0])
    new_tensor = torch.t(new_tensor)

    t = torch.greater_equal(new_tensor,delta).type(torch.cuda.FloatTensor)
    # print(t[0])
    x = torch.greater(tensor,0).type(torch.cuda.FloatTensor)
    y = torch.less(tensor,0).type(torch.cuda.FloatTensor)
    y = torch.mul(y,-1)
    z = torch.add(x,y)
    t = torch.t(t)
    final = torch.mul(t,z)

    new_tensor = torch.t(new_tensor)

    final.cuda()
    alpha = torch.mean(torch.mul(final,new_tensor),dim=1)

    output = torch.add(output,final)

    # print(output[0])

    return (output,alpha)
</code></pre>
<h5><a name="h-1" class="anchor" href="#h-1"></a></h5>
<p>The code which calls the Ternarizeop class in the training loop is as follows -</p>
<p>alpha = ternarize_op.TernarizeWeights()<br>
l = []<br>
l.append(imgs)<br>
l.append(alpha)<br>
output = model(l)</p>
How is im2col()'s backward being called?

zfzhang: <p>Hi,<br>
I know that <code>im2col()</code>'s backward is <code>col2im()</code>, and I’m studying how <code>F.unfold()</code> triggers the backward pass of <code> im2col()</code>.</p>
<p>I noticed there are functions like <code>im2col_backward_cuda()/im2col_backward_out_cuda()/im2col_backward_out_cuda_template()</code> in <a href="https://github.com/pytorch/pytorch/blob/46b3fc032aad4ff71209aea287e9258d5c1256e4/aten/src/ATen/native/cuda/Im2Col.cu" rel="noopener nofollow ugc">Pytorch repository</a>. However, when searching for these names, the only hit is the file itself.</p>
<p>My question is:</p>
<ol>
<li>Are the above <em>backward</em> functions ever called at all?</li>
<li>How is <code>im2col()</code>'s backward triggered?</li>
</ol>
<p>Thank you!</p>
How to get the scores from torchvision fasterRCNN based on an image and bounding box coordinates

Shantanu_Ghosh: <p>Hi,</p>
<p>I am using  torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) for object detection. Now this method gives me the scores as one of the outputs. Now i have the co-ordinates of the bounding box, so i want to get the scores based on that image and the bounding box.<br>
For example, i want to do something like this =&gt;<br>
num_classes = 2</p>
<pre><code>in_features = model.roi_heads.box_predictor.cls_score.in_features

model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
chkpt_path = "/chkpt_path"
model.load_state_dict(torch.load(chkpt_path))
model.to(device)
images, target = iter(val_data_loaders).next()
images = list(image.to(device, dtype=torch.float) for image in images)
[outputs = model(images)] # replace this with the following
outputs = model(images, precomputed_bbox)
</code></pre>
<p>Please suggest me how to do this? I want the score from this</p>

thecho7: <p>I’m not sure that I clearly understand your problem.</p>
<p>You want to get the scores of your custom bboxes from the image, right?<br>
I think it is hard to get because inside the model, there are several modules that generate anchors and cutout the boxes which has a score under threshold and NMS…</p>

Shantanu_Ghosh: <p>So, i have an image and a bbox, all i want a score of the image inside the bbox from the model taking the input and bbox</p>
How to debug "Expected all tensors to be on the same device"

mamunm: <p>When running the <code>optim.step()</code> the code is throwing this error that the tensors are not on the same device. I explicitly transferred the model and data <code>.to(device)</code> multiple times to make sure they are all on GPU. When I do</p>
<pre><code class="lang-auto">for param in model.parameters():
    print(param.device)
</code></pre>
<p>It prints <code>"cuda:0"</code> for all the params. I checked all the tensors going in and out of the model but still can’t figure out the problem. Is there any hook or debug tool that can help me pinpoint the tensor causing this problem? Thanks in advance!</p>
<p>PS: if I change the device to <code>"cpu"</code> it runs fine without any complain.</p>

thecho7: <p>Check all tensors outside of the model.<br>
Sorry for not giving you an useful advise</p>
Pytorch Version 1.6.0 Broke LBFGS Optimizer

jonahclarsen: <p>Hello!</p>
<p>I am building a Libtorch implementation of Neural Style Transfer based on <a href="https://github.com/andrew98450/Style-Transfer" class="inline-onebox" rel="noopener nofollow ugc">GitHub - andrew98450/Style-Transfer: Style Transfer using libtorch C++ Frontend</a></p>
<p>This was written before Pytorch 1.6.0, and it works fine with Pytorch 1.4.0, 1.5.0, or 1.5.1. But if I try to use it with version 1.6.0 and above, the program throws an exception right after the lambda function returns the loss (and hence, right as the optimizer takes a step). At line 13 in WrapFunctionIntoFunctor.h: “c10::Error at memory location 0x00000029AECFCF10.”</p>
<p>For some reason, the error is also times at WrapFunctionIntoRuntimeFunctor.h memory error on line 18 and in lbfgs.cpp, even in (seemingly) the same conditions.</p>
<p>It works fine with the Adam optimizer in all versions.</p>
<p>Any hints would be appreciated. I couldn’t find any commits on the 1.6.0 release that seemed relevant, but it must be there.</p>
<p>I am on Windows and have tried this with both CUDA and CPU (I modified the code to use CPU). I am happy to provide any other potentially helpful information.</p>

ptrblck: <p>Would you mind creating an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> with this description and a reproducible code snippet, if possible, please?</p>

jonahclarsen: <p>Done. <a href="https://github.com/pytorch/pytorch/issues/71391" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/71391</a></p>
Grid_sample for sparse matrices

fraazor: <p>Hello everybody,</p>
<p>I want to translate and rotate a map matrix using the affine_grid and grid_sample method but want to avoid using a full 2d tensor because of very large RAM demand. My data is quite sparse, therefore I reduce the size by only keeping coordinates where values actually exist and reconstruct the map when needed. However I am struggeling to see whether there is any way to use/configure the grid_sample function to transform sparse matrices of this format:<br>
x       y         value<br>
[[4],   [5]],     [0.12],<br>
[[4],   [10]],   [0.35],<br>
[[24], [12]],   [0.84],<br>
[[14], [595]], [0.55],<br>
…</p>
<p>Does anybody know whether this won’t be possible using sample_grid? The values don’t need to be included at all costs either. Thank you!</p>

ThibaultGROUEIX: <p>Hi,</p>
<p>I have exactly the same problem. Did you find a solution?</p>
<p>Thanks,<br>
Thibault</p>

Richard_Hakim: <p>I’m also having this problem. Any solutions? Is this worth a feature request?</p>
<p>Rich</p>
Adding layers after defining the model

wassimseif: <p>Hello,<br>
I am trying to find a way to add layers to a model after without changing the definition of the model</p>
<p>For example let’s say i have this layer that I want to add after every <code>ReLU</code> layer i have in a <code>ResNet</code></p>
<pre><code class="lang-python">class SimpleCustomLayer(nn.Module):
    def __init__(self):
        pass

    def forward(self,x):
        log_tensor(x)
        return x

</code></pre>
<pre><code class="lang-python">
def add_module_after_layer(module:nn.Module, layer:nn.Module,model:nn.Module):
    for i in model.modules():
        if type(i) == type(layer):
            # Here i should add the module after the layer

    

model = ResNet()

add_module_after_layer(module= SimpleCustomLayer,
                       layer= torch.nn.ReLU,
                       model=model)
</code></pre>
<p>Any idea how to add this layer after layer <code>i</code> ?</p>

ptrblck: <p>If your original model uses <code>nn.ReLU</code> modules for each of these activation functions (i.e. not the functional API via <code>F.relu</code> not reusing the same <code>nn.ReLU</code> module), you could replace each <code>nn.ReLU</code> with an <code>nn.Sequential</code> container containing the <code>nn.ReLU</code> and your <code>SimpleCustomLayer</code>.</p>

wassimseif: <pre><code class="lang-python">model = torchvision.models.resnet18(pretrained=False)

class SimpleCustomLayer(nn.Module):
    def __init__(self):
        super(SimpleCustomLayer).__init__()

    def forward(self, x):
        return x


block = nn.Sequential(SimpleCustomLayer(), nn.ReLU())
for i, l in enumerate(model.modules()):
    if isinstance(l, nn.ReLU):
        print(f"Replacing at {i}")
        model[i] = block

</code></pre>
<p>This gives the following error which is very understandable.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/path/", line 23, in &lt;module&gt;
    model[i] = block
TypeError: 'ResNet' object does not support item assignment
</code></pre>
<p>Does PyTorch provide any API for switching such layers ?</p>

ptrblck: <p>You would have to assign the new module to the attribute name e.g. via <code>setattr</code>. However, note that the resnet implementation reuses the <code>nn.ReLU</code> modules as seen in its <a href="https://github.com/pytorch/vision/blob/49468279d9070a5631b6e0198ee562c00ecedb10/torchvision/models/resnet.py">definition</a> so your approach won’t work.</p>

wassimseif: <p>Why would it not work if it’s reusing? in both <code>nn.ReLU</code> and my custom layer there’s no parameters to learn</p>

ptrblck: <p>I’ve assumed that your custom layer would have parameters, but you are right that it would work if that’s not the case.</p>

wassimseif: <p>Understood!<br>
Thank you for your help!</p>
Regarding transforms.MelSpectrogram output length

kevin_zhang: <p>Hi,</p>
<p>So I initialize my melspectrogram as follows:</p>
<p>transform = torchaudio.transforms.MelSpectrogram(sample_rate=8000, n_mels=80, win_length=200, hop_length=80, center=False)</p>
<p>Then here’s how I use it:<br>
x_in.shape == [1,5360]<br>
x_out = transform(x_in)</p>
<p>x_out.shape == [1, 80, 63]</p>
<p>However, based on my (introductory) understanding of Fourier Transform, I thought the output length is supposed to be (input_length - win_length) / hop_length + 1 = (5360 - 200)./ 80 + 1 = 65.5</p>
<p>So shouldn’t be either 65 or 66, instead of 63? Or do I just not understand FT?</p>
<p>Thanks,<br>
Kevin</p>
Draw_bounding_boxes not available in torchvision.utils package

Shantanu_Ghosh: <p>Is draw_bounding_boxes not available in torchvision.utils package? i am getting error while doing the import as<br>
from torchvision.utils import draw_bounding_boxes</p>
<p>The error message:<br>
ImportError                               Traceback (most recent call last)<br>
/tmp/ipykernel_9914/1881906353.py in <br>
----&gt; 1 from torchvision.utils import draw_bounding_boxes</p>
<p>ImportError: cannot import name ‘draw_bounding_boxes’ from ‘torchvision.utils’ (/ocean/projects/asc170022p/shg121/anaconda3/envs/python_3_7/lib/python3.7/site-packages/torchvision/utils.py)</p>
<p>My torch versions:<br>
import torch</p>
<p>print(torch.<strong>version</strong>) # 1.7.0<br>
print(torch.version.cuda) # 10.2</p>

ptrblck: <p>You haven’t posted the <code>torchvision</code> version, so I guess it might be too old and you would have to update it.</p>

Shantanu_Ghosh: <p>torchvision version is 0.6</p>

ptrblck: <p><code>0.6</code> was released in April 2020, while <code>draw_bounding_boxes</code> was added in <a href="https://github.com/pytorch/vision/pull/2785">this PR</a> in Nov. 2020 so update <code>torchvision</code>.</p>
How can I permute the value of a Torch tensor?

Rami_Ismael: <p>How can I permute the value of a Torch tensor?</p>

ptrblck: <p>I don’t know what “permute” means in this context, but you could check if calling <code>permute</code> works (will change the order of the dimensions) or e.g. random indexing (would shuffle the tensor).</p>
Best way to store feature vectors from pretrained model to hard disk?

Gkv: <p>Hi all,<br>
Suppose I extracted the image features from a pretrained CNN and wants to save these features to disk for later use. Which of the following will be a good method in terms of memory and speed.</p>
<ol>
<li>Store the feature dictionary in a pickle file</li>
<li>Convert the feature tensor to numpy arrays and save using np.save()</li>
<li>Save it in a json file ( TypeError: Object of type ‘FloatTensor’ is not JSON serializable is coming)</li>
</ol>
<p>Thanks!!</p>

ptrblck: <p>I would probably save them as <code>torch.tensors</code> with <code>torch.save</code>.<br>
This should be fast and as long as you load them again in PyTorch there shouldn’t be a problem.</p>

Neda: <p>would be possible to extract features from the saved model and save them as a .np file?<br>
I have saved the model using</p>
<pre><code class="lang-auto">     if epoch % 10 == 9:
             
            torch.save({
                    'epoch': epoch + 1,
                    'state_dict': model.state_dict(),
                    'optimizer': optimizer.state_dict(),
            }, 'NAS_exp_epoch{}.pth'.format(epoch+1))
             
</code></pre>
<p>I wonder how can I extract features from the model for each image patch?</p>

ptrblck: <p>I assume “features” refers to an intermediate forward activation. If so, you could use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a> to grab them and that you could store them as numpy arrays afterwards.</p>
Grad values update in optimizer.step()

ffan1980: <p>Hello everyone,</p>
<p>I am learningg SGD optimizer to train LeNet and AlexNet. I found out that during optimizer.step(), the param’s are updated by -1 * lr * grad’s, instead of -1 * lr * grad’s/batch_size ( in <em>functional.py:  param.add</em>(d_p, alpha=-lr) ). This way, when I increase the batch_size from e.g. 100 to 1000,  I need to decrease the learning_rate by the same magnitude.</p>
<p>To try to keep the learning_rate less impacted by the change of batch_size, I manually add the following codes before optimizer.step():</p>
<pre><code>    for p in optimizer.param_groups[0]['params']:
        p.grad = p.grad / batch_size
    optimizer.step()
</code></pre>
<p>Does anyone experience the same confusion? Or is this the correct way to train the network with the optimizer?</p>
<p>Thanks,<br>
Fan</p>

googlebot: <p>normally, loss magnitude doesn’t increase with batch size because of reduction=‘mean’, hence this adjustment is not needed/correct</p>

anantguptadbl: <p><a class="mention" href="/u/ffan1980">@ffan1980</a> as mentioned by <a class="mention" href="/u/googlebot">@googlebot</a> imagine it this way. You are diving the grads by a static number. The only thing you are doing is effectively using a lower learning rate.</p>
<p>If you want to add value by doing this operation, you should use some dynamic value extracted from the data/grads</p>

ffan1980: <p><a class="mention" href="/u/googlebot">@googlebot</a> <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> Thank you for the replies.</p>
EOFError: Ran out of input (Resnet model)

mlamrini: <p>I’m attempting to train the model myself, I took the code from (<a href="https://github.com/AndreyGuzhov/ESResNet" rel="noopener nofollow ugc">GitHub</a>), but when I tried to train the model I got an error.</p>
<p>I run this in a Anaconda Promp: <code>python main.py --config protocols/us8k/esresnet-us8k-stereo-cv1.json --Dataset.args.root D:/Data_Urb8k/Urb8K/Descr_Data/UrbanSound8K/</code></p>
<p>However, it returns:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68e06691204920a63af15ec7d95f4d58276a3962.png" data-download-href="https://discuss.pytorch.org/uploads/default/68e06691204920a63af15ec7d95f4d58276a3962" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68e06691204920a63af15ec7d95f4d58276a3962.png" alt="Capture" data-base62-sha1="eXMqQRkwAfFhDG6QFAcXu5SoxEK" width="690" height="167" data-dominant-color="181818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1309×317 14.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m sort of stranded on this, unfortunately. Does anyone recognize what could be happening here? I’m assuming some setting on my end doesn’t allow the model to run as I haven’t changed anything in his original code.</p>
<p><strong><a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a></strong></p>

ptrblck: <p>The screenshot shows a <code>MemoryError</code> and the <code>EOFError</code> might just be the following error message triggered by the original one. Check if you are running our of memory and if so, reduce the batch size or use lazy loading if not already applied.<br>
PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

mlamrini: <p>when I am training the model, firstly I got  loading name_of_data(train=true) after becoming loading name_of_data(train=false), As indicated in the picture<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e85b968eff31587e5ae5c3d0a82877c6ca00e97.png" data-download-href="https://discuss.pytorch.org/uploads/default/8e85b968eff31587e5ae5c3d0a82877c6ca00e97" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e85b968eff31587e5ae5c3d0a82877c6ca00e97.png" alt="1" data-base62-sha1="kkOiPrb51W2QF2gaqCWDn8AKhSL" width="690" height="357" data-dominant-color="232424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1389×719 26.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Recording local derivative

hamedB: <p>Hi</p>
<p>The autograd module computes the gradient of the output respect to the each internal component such as activation maps and we can record the values using hook module. My question is that, given an input image, how I can access to the local derivative, for example the gradient of each activation maps in layer L (e.x. a_l) w.r.t each activation maps in layer L-1 (e.x. a_(l-1))? Is there a way to record these values in one backpropagation procedure or should I apply backward operation each time for each pair activation maps (a_l,a_(l-1))?</p>
<p>Thank you.</p>
Can we resume training with only torchscript model?

hshwang: <p>I saw some transfer learning Libtorch codes using pretrained torchscript models and all of them is only training additional layers added in Libtorch codes.<br>
(e.g) <a href="https://github.com/krshrimali/Transfer-Learning-Dogs-Cats-Libtorch/blob/master/main.cpp" class="inline-onebox" rel="noopener nofollow ugc">Transfer-Learning-Dogs-Cats-Libtorch/main.cpp at master · krshrimali/Transfer-Learning-Dogs-Cats-Libtorch · GitHub</a>)</p>
<p>My question is, can we train the torchscript model without defining network model in Libtorch?</p>
<p>I think torchscript model has already model’s weights and structure is similar to torch::nn::Module(actually it is instance of torch::jit::Module object) so I assume it’s not impossible…but still can’t find the way.</p>
<p>Best Regards</p>

hshwang: <p>Solved from this issue.</p>
<p><a href="https://github.com/pytorch/pytorch/issues/28478#issuecomment-610897844" rel="noopener nofollow ugc">How to train a torch::jit::script::Module? · Issue #28478 · pytorch/pytorch (github.com)</a></p>
Cudnn_cnn_train64_8.dll existed but could not be found when using a library

hhh: <p>Hi, when I used spacy (a library) to download something, it showed that</p>
<p><code>OSError: [WinError 127] The specified procedure could not be found. Error loading "C:\Users\haoli\anaconda3\envs\ifcmapping\lib\site-packages\torch\lib\cudnn_cnn_train64_8.dll" or one of its dependencies.</code></p>
<p>But <code>cudnn_cnn_train64_8.dll</code> actually existed there, as the picture below shows. Then what is the problem? Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81b1a7703a99082e020ad65f8b04a4204f38a220.png" data-download-href="https://discuss.pytorch.org/uploads/default/81b1a7703a99082e020ad65f8b04a4204f38a220" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81b1a7703a99082e020ad65f8b04a4204f38a220_2_690x268.png" alt="image" data-base62-sha1="ivkdnkMHMJv3uJ72SXaGsGERw4g" width="690" height="268" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81b1a7703a99082e020ad65f8b04a4204f38a220_2_690x268.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81b1a7703a99082e020ad65f8b04a4204f38a220_2_1035x402.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81b1a7703a99082e020ad65f8b04a4204f38a220_2_1380x536.png 2x" data-dominant-color="858B8E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2043×794 232 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

hhh: <p>It seems that it is because I used conda to install PyTorch and used pip to install the spacy library. When I used conda to install both of them it worked.</p>