akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
SubsetRandomSampler

username: samanmo
cooked: <p>I am using SubsetRandomSampler() class in pytorch to train on a subset of my data set,<br>
I have got these problems:</p>
<ol>
<li>as dataset I use CocoDetection and my data loader seems  not returning labels for all objects.</li>
<li>how could I get the indices randomly chosen by SubsetRandomSampler from the list of indices that I provided at each batch run?</li>
</ol>
Torchvision normalize - how it operates on tuple of means/sds?

username: jchaykow
cooked: <p>I don’t understand how this transform works from torchvision. Ultimately I want to build a custom normalize class so I need to figure out how this works first.</p>
<p>Here in the docs it describes the init like this:</p>
<pre><code class="lang-auto">def __init__(self, mean, std, inplace=False):
        self.mean = mean
        self.std = std
        self.inplace = inplace
</code></pre>
<p>And when I pass these parameters usually (not custom class) I pass them as a list or tuple for each channel:</p>
<p><code>transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])</code></p>
<p>But if I look at the call:</p>
<p><code>return F.normalize(tensor, self.mean, self.std, self.inplace)</code></p>
<p>All this passes the tuple to is F.normalize() which only accepts a single value for the p parameter.</p>
<p>The class must iterate through the channels somehow to allow this to be implemented but how does it do this and how can I implement it in custom class?</p>
<p>Based on <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">this tutorial</a>, I would describe it like this:</p>
<pre><code class="lang-auto">class Normalize(object):
    """Convert ndarrays in sample to Tensors."""
    
    def __init__(self, mean, std, inplace=False):
        self.mean = mean
        self.std = std
        self.inplace = inplace

    def __call__(self, sample):
        image, landmarks = sample['image'], sample['landmarks']
        return {'image': F.normalize(image, self.mean, self.std, self.inplace),
                'landmarks': landmarks}
</code></pre>
<p>But this does not work because it does not go through each channel.</p>
Can bool slices operation be placed inside a back propagation loop

username: 111142
cooked: <p>Hi,</p>
<p>I just encountered an error, when I perform a bool slices operation like:</p>
<p>average = torch.mean(tensor[tensor[:] != 0]),</p>
<p>the system returns me an error: Function ‘AddmmBackward’ returned nan values in its 2th output.<br>
I realized that this equation may not be a differentiable function, so I was wondering if there is a differentiable way can let me get the same result as the equation stated above.</p>
<p>Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This operation is fine. Note that you don’t need the <code>[:]</code>.<br>
This error occurs when you run in AnomalyMode? You should check which forward function it correspond to and print the output value to make sure they are in acceptable ranges.</p>

username: 111142
cooked: <p>Hey,</p>
<p>Thanks for your suggestions and this operation really worked after I fixed a little error somewhere else.</p>
How dose work class of Network

username: Arvin1366
cooked: <p>I have this class.<br>
class Net(torch.nn.Module):<br>
def <strong>init</strong>(self, D_in, H, D_out):<br>
super(Net, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(D_in, H)<br>
self.fc2 = nn.Linear(H, H)<br>
self.fc3 = nn.Linear(H,H)<br>
self.fc4 = nn.Linear(H, D_out)<br>
self.active_1 = torch.nn.Tanh()<br>
self.active_2 = torch.nn.Tanh()<br>
self.active_3 = torch.nn.Tanh()<br>
self.fc11 = nn.Linear(D_in, H)<br>
self.fc22 = nn.Linear(H, H)<br>
self.fc33 = nn.Linear(H, H)<br>
self.fc_tot = nn.Linear(D_out, D_out)<br>
##################################</p>
<pre><code>def forward(self, x):
    """
    In the forward function we accept a Tensor of input data and we must return
    a Tensor of output data. We can use Modules defined in the constructor as
    well as arbitrary operators on Tensors.
    """
    y = self.active_1(self.fc1(x))
    y = self.active_2(self.fc2(y))
    y = self.active_3(self.fc3(y))
    y = self.fc4(y)
    y = y*y########################
    y = self.fc_tot(y)
    return y
</code></pre>
<p>As you see I have not to use these three functions( self.fc11 = nn.Linear(D_in, H),self.fc22 = nn.Linear(H, H),self.fc33 = nn.Linear(H, H)) in forward part, but when I delete from the class, the result gets completely change? why?</p>

username: spanev
cooked: <p>This looks quite odd. Is the complete model you are running?</p>

username: Arvin1366
cooked: <p>yes, and the result is good but I do not know, how can I explain it.<br>
the only parameter that, get change in class is the " _cdata".</p>
Run tensorboard

username: slavavs
cooked: <p>I installed <code>pip install tb-nightly</code><br>
I’m trying to visualize the code</p>
<pre><code class="lang-auto">from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter(log_dir='./logs')
x = range(100)
for i in x:
    writer.add_scalar('y=2x', i * 2, i)
writer.close()
</code></pre>
<p>The documentation says the command <code>tensorboard --logdir=logs</code></p>
<p>But getting an error <code>tensorboard --logdir=logs ^ SyntaxError: can't assign to operator</code></p>
<p>How it works?</p>

username: spanev
cooked: <p>It might be related to this <a href="https://stackoverflow.com/a/49084133" rel="nofollow noopener">thread</a>. Are you running the command inside a bash terminal?</p>
Can a kernel get involve and updated in the gradient map by this way?

username: SupremePROC
cooked: <p>This question might be naive but qusetioned me for a long while. If we wrap a convolution function by a custom function, and pass the parameters, like this</p>
<pre><code class="lang-python">my_w = nn.Parameter(torch.randn(1,1,3,3))
my_b = nn.Parameter(torch.randn(1,1,1))

def conv1(x, my_w, my_b):
    x = F.conv2d(x, weight=my_w, bias=my_b, stride, padding)
    return x
</code></pre>
<p>Can my_w and my_b get updated? as they are global parameters, they do not really live in this function</p>
<p>If can, why? If cant, how to fix it(without ‘global’ tag)?</p>
<p>Thank you guys!</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/supremeproc">@SupremePROC</a></p>
<p>The root node <code>x</code> returned by the function point to the recorded backward graph, which has <code>my_w</code> and <code>my_b</code> as leaf nodes. They are recorded in <code>x</code>'s graph, independently of their scope (global or local).<br>
So their <code>grad</code> will get updated (as they are leaf nodes).</p>
<p>Find more details about <a href="https://pytorch.org/docs/stable/autograd.html" rel="nofollow noopener">autograd</a> here.</p>
PyWarm: A cleaner way to build neural networks for PyTorch

username: blue-season
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/blue-season/pywarm" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/54463427?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/blue-season/pywarm" target="_blank" rel="nofollow noopener">blue-season/pywarm</a></h3>

<p>A cleaner way to build neural networks for PyTorch. - blue-season/pywarm</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: blue-season
cooked: <p><a href="https://github.com/blue-season/pywarm" rel="nofollow noopener">PyWarm</a> is a high-level neural network construction API for PyTorch.</p>
<p>Using PyWarm, you can put  <em>all</em>  network data flow logic in the  <code>forward()</code>  method of your model, without the need to define children modules in the  <code>__init__()</code>  method then call it again in the  <code>forward()</code>. This result in a much more readable model definition in fewer lines of code. Check the GitHub repository for code examples and more details!</p>

username: blue-season
cooked: <p>PyWarm just hit v0.2!</p>
<p>Featuring the warm functional interface to Transformer, and a new example for MobileNet.</p>

username: blue-season
cooked: <p>See an example of the <a href="https://blue-season.github.io/pywarm/docs/example/#transformer" rel="nofollow noopener">transformer model</a> (<em>Attention is all you need</em>) in less than 50 lines of code.</p>
Retrieve neural network name inside registered hook

username: vainaijr
cooked: <p>how do I get the neural network name inside hook function, so that I see output of different neural network under a different name in tensorboard</p>
<pre><code class="lang-auto">writer = SummaryWriter()
def hook(module, input, output):
    # print(input[0].shape, output.shape)
    grid = torchvision.utils.make_grid(output[0][0])
    # print(grid.shape)
    
    # how to change conv_features to neural network name?
    # so it is model.modelone.encoder for a, model.modeltwo.encoder for b
    writer.add_image('conv_features', grid, 0) 

a = model.modelone.encoder.register_forward_hook(hook)
b = model.modeltwo.encoder.register_forward_hook(hook)
</code></pre>
<p><code>module.__class__, module.__class__.__name__, module._get_name</code> do not give neural network name.</p>

username: ptrblck
cooked: <p>Have a look at <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">this example</a> where you can pass additional names to the <code>get_activation</code> function.</p>

username: vainaijr
cooked: <p>I did something like this to get output of each encoder of each neural network</p>
<pre><code class="lang-auto">writer = SummaryWriter()
activations = {}
def get_activation(name):
  def hook(module, input, output):
      activations[name] = output.detach()
      grid = torchvision.utils.make_grid(output)
      writer.add_image(name, grid, 0)
  return hook

for name, layer in model.named_modules():
  if 'encoder' in name and 'encoder.' not in name:
    print(name, layer)
    layer.register_forward_hook(get_activation(name))
</code></pre>

username: vainaijr
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, this way I am able to visualize what all patterns are learnt by base neural networks, but their output is passed in the form of 10 scores, which is then concatenated in higher level neural network, how do I visualize what  are the higher level neural networks learning, currently I do something like this, I add image output from base networks, and plot them</p>
<pre><code class="lang-auto">def forward(self, x):
    img = self.modelone(x)[0] + self.modeltwo(x)[0]
    grid = torchvision.utils.make_grid(img)
    writer.add_image('img_combine', grid)
    img_2d = self.encoder(img)
    img = self.lin_two(self.lin_one(img_2d.view(img_2d.size(0), -1)))
    out = torch.cat((self.modelone(x)[1], self.modeltwo(x)[1], img), dim=-1)
    out = (self.linear((out)))
    return img_2d, out
</code></pre>
<p>where ModelOne forward looks like</p>
<pre><code class="lang-auto">def forward(self, x):
    x = self.encoder(x)
    return (x, self.lin_two(self.lin_one(x.view(x.size(0), -1))))
</code></pre>
<p>encoder is conv, batchnorm, relu</p>
Dataloader to load video with neighboring windows (without nvvl)?

username: Pritesh_Gohil
cooked: <p>Hi again,</p>
<p>I am looking for help in writing pytorch dataloader for my dataset, which can load small video clips ( loading sequence of frames using neighbouring window) from 1 big video.</p>
<p>I have some dataset which contains lots of videos and each video can have multiple labels.</p>
<p>Now I would like to train my network with 16 frames with neighbouring video. Also, each video is of different length. I strictly don’t want to use nvvl because of some package dependancy issue which I have already tried.</p>
Switchable Normalization

username: arjung
cooked: <p>I recently came across switchable normalization in Andrew Ng’s ‘The Batch.’ Is this a layer that can be plugged into an existing model for normalization purposes? If so, is there a pytorch implementation of it?</p>

username: tshadley
cooked: <p>A version here: <a href="https://github.com/switchablenorms/Switchable-Normalization/blob/master/devkit/ops/switchable_norm.py" rel="nofollow noopener">https://github.com/switchablenorms/Switchable-Normalization/blob/master/devkit/ops/switchable_norm.py</a></p>
Judge the model is train or validation

username: linyu
cooked: <p>How can I judge the model is train or validation in the ‘forward’ function?<br>
In fact, I have construct a network, and write the ‘forward’ function. I want to judge whether this forward process happened in the training process or in the validation process.</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/linyu">@linyu</a>,</p>
<p>You can know by checking the value of the <code>nn.Module</code> attribute <code>self.training</code> in <code>forward</code> (set to <code>True</code> in <code>net.train()</code> and to <code>False</code> in <code>net.eval()</code>) .</p>
Default Weight Initialization vs my kaiming_uniform init

username: 07hyx06
cooked: <p>I read some topics, and know that most of the layers are initialized by <code>kaiming_uniform_initialization</code>, including <code>Conv2D</code>, <code>Linear layers</code>…And when i use these default initialization, my VGG runs well.</p>
<p>I try to reset these layers’ initialization with my own <code>initialize_weights</code> function, in my function, i use nn.init.kaiming_uniform_ as well. But when i train my VGG with same hyperparameters, loss fly to NAN.</p>
<p>Is there any wrong in my codes?Thanks for your help!!</p>
<pre><code class="lang-auto">def initialize_weights(layer):
    if isinstance(layer, nn.Conv2d):
        nn.init.kaiming_uniform_(layer.weight, mode='fan_in', nonlinearity='relu')
        if layer.bias is not None:
            nn.init.constant_(layer.bias, 0)

model.apply(initialize_weights)
</code></pre>

username: 07hyx06
cooked: <p>I find that the gain value in the default initialization is sqrt(5).<br>
(means <code>a=sqrt(5)</code> in the code below)</p>
<pre><code class="lang-auto">torch.nn.init.kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')
</code></pre>
<p>but based on the paper, if my activation function are all ReLU, i should set gain value equal to 0.<br>
So i set a=0 in my code, but got NAN loss. What’s the problem?</p>
Measuring cosine similarity reconstructed vector

username: lorenzo_fabbri
cooked: <p>I’m trying to use a convolutional auto-encoder to reconstruct some vectors (not images).<br>
In order to assess whether it’s working I’d like to plot the cosine similarity between the input data and the reconstructed vector. What I’m doing right now is this:</p>
<pre><code class="lang-auto">model.eval()
    with torch.no_grad():
        for data in loader_val: # Validation dataset
            # CAE
            data, noise = data # Use data w/ noise as input model
            data = data.unsqueeze(1).float()
            noise = noise.unsqueeze(1).float()
            
            out = model(noise)
            loss = criterion(out, data)
            
            # Cosine similarity
            cos = CosineSimilarity()
            cosine_sims.append(cos(data, out))
</code></pre>
<p>I wanted to visually see how it changes after each epoch and so I used the following commands to plot it:</p>
<pre><code class="lang-auto">plt.plot([np.mean(np.abs(el.numpy())) for el in cosine_sims])
</code></pre>
<p>The <em>problem</em> is that the values start at 0.9 and they just fluctuate (not decreasing or increasing). Is there something wrong in the way I use <code>CosineSimilarity</code>?</p>

username: ptrblck
cooked: <p>What kind of criterion are you currently using to train your model?</p>

username: lorenzo_fabbri
cooked: <p>I’m currently using an MSELoss.</p>

username: ptrblck
cooked: <p>The Euclidean distance does not necessarily correlate with the angle (cosine similarity).<br>
Imagine a point cloud near the origin of a coordinate system. While the <code>MSELoss</code> could be low, the cosine similarity might be high between certain points.</p>
<p>Would it be possible to use the <code>CosineSimilarity</code> as the criterion to train the model?</p>

username: lorenzo_fabbri
cooked: <p>I’ve being thinking about this for a while now. I guess it’s possible but I don’t think I understand how to actually use it. The <code>CosineSimilarity</code> returns a number for each pair of vectors, and not a <em>real</em> loss. Should I take the mean or should I add it to <code>MSELoss</code>? Thanks.</p>

username: Skalwar
cooked: <p>Can a loss function be like:-<br>
Total_loss = MSEloss()+CosineSimilarity()<br>
I think this will be better measure.<br>
Build your custom CosineSimilarityLoss if possible.</p>

username: InnovArul
cooked: <p>I guess,  the sign for cosine similarity shall be - (minus). Also,  if the vectors are L2 normalized,  MSELoss ~ -CosineSimilarity.</p>

username: Skalwar
cooked: <p>Yeah that’s right MSE ~ -(CosineSimilarity) if vectors are l2 normalized.</p>

username: lorenzo_fabbri
cooked: <p>But I cannot simply add (or subtract) the cosine similarity to the loss function, since the former is not just a real number. Do you recommend taking the mean?</p>

username: Skalwar
cooked: <p>I think MSE should work as MSE ~Approx of (-(CosineSimilarity)).</p>

username: Skalwar
cooked: <p>Given that vectors are l2 normalized.</p>

username: lorenzo_fabbri
cooked: <p>Why are they L2-normalized?</p>
Video classification) I have a question about 64f in the dataloader of the video

username: 439327bcc62cd8e45e96
cooked: <p>Hi . I am a university student studying in South Korea. I’m confused about the videos dataloader, so I leave a question.</p>
<p>I want to make a 3 x 3 x 3 cube by stacking 64 frames in the video, but if the entire frame of the video is 200 frames, how is it common and best to select 64 frames?</p>
<p>If the total frame of the entire video is 40, how is it best to increase it to 64 frames?</p>
<p>If there’s any problem, I’ll delete it. I am sorry for my short English.</p>
Sentiment analysis

username: slavavs
cooked: <p>Guys, help solve the problem. I took the nn.transformer module. At the entrance is a sub-offer, at the output I need to get an analysis, positive or negative. I want to use softmax (0,1). How can I do it? nn.transformer for decoder is a required parameter (tgt - the sequence to the decoder (required)). What should I apply to this input?</p>
100% GPU Util at the beginning of training, but become extremely unstable later

username: fanl
cooked: <p>Hi,<br>
I transformed the ImageNet dataset into LMDB database. During training, I find that the GPU Util is around 99% stably at the beginning of training,  but it jumps between 99%  and 0% after hundreds of iterations, as a result, the speed of training slows down significantly.<br>
In addition, I have run the code several times before, but such phenomenon has never happened. The code has not been changed. Dataloader is shown below.</p>
<pre><code class="lang-auto">class train_dataset(Dataset):
    def __init__(self):
        super(train_dataset,self).__init__()
        self.root = r'/data/imagenet/train/'
        self.env = lmdb.open(self.root)
        self.txn = self.env.begin(write = False)
        self.transforms = transforms.Compose([
            transforms.RandomResizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            normalize
            ])
        #        print(self.__len__())
    def __getitem__(self,i):
        image_bin =  self.txn.get((str(i)+'_img').encode())
# another way to open binary data, but got the same phenomenon
#        image = Image.open(BytesIO(image_bin))
#        if image.mode != 'RGB':
#            image = image.convert('RGB')
        image_buf = np.frombuffer(image_bin,dtype = np.uint8)
        image = cv2.imdecode(image_buf,cv2.IMREAD_COLOR)
        image = Image.fromarray(cv2.cvtColor(image,cv2.COLOR_BGR2RGB))
        image = self.transforms(image)
        label =  self.txn.get((str(i)+'_label').encode()).decode()
        label = int(label)
        return {'image':image,'label':label}
    def __len__(self):
        return self.txn.stat()['entries']//2

loader_train = torch.utils.data.DataLoader(
	train_dataset(), batch_size=bs_train, shuffle=True,
	num_workers=n_worker, pin_memory=True)
</code></pre>
<p>What’s the possible causes?<br>
Sincerely</p>

username: ptrblck
cooked: <p>It the decrease in GPU utilization observable after an approx. fixed number of iterations or does your code slow down consistently?<br>
Are you storing some variables somewhere, e.g. <code>losses.append(loss)</code>?<br>
Do you see an increased usage in GPU or CPU memory?</p>

username: fanl
cooked: <p>Thanks.<br>
It become unstable after an approx. fixed number of iterations.<br>
Yes, I used <code>losses.append(loss)</code>, but memory usage is constant. How does storage of variables affect GPU Util?</p>

username: fanl
cooked: <p>I forgot to mention that I store <code>losses</code> in CPU memory.</p>
atomicAdd does not work

username: Kevin96
cooked: <p>Hi,</p>
<p>I am relatively new to PyTorch C++ and cuda implementation. I wrote the following code to compute the dot product between two vectors.</p>
<blockquote>
<p><span class="hashtag-raw">#include</span> &lt;torch/extension.h&gt;<br>
<span class="hashtag-raw">#include</span> <br>
using namespace std;</p>
<p>__global__<br>
void dot_cuda_kernel(<br>
torch::PackedTensorAccessor&lt;float, 1&gt; &amp;x_a,<br>
torch::PackedTensorAccessor&lt;float, 1&gt; &amp;y_a,<br>
int n,<br>
float *c)<br>
{<br>
int index = threadIdx.x;<br>
int stride = blockDim.x;</p>
<pre><code>// for(int i = index; i &lt; n; i += stride)
for(int i = 0; i &lt; n; i++)
   atomicAdd(c, x_a[i] * y_a[i]);
</code></pre>
<p>}</p>
<p>float dot_cuda(torch::Tensor &amp;x, torch::Tensor &amp;y)<br>
{<br>
auto x_a = x.packed_accessor&lt;float, 1&gt;();<br>
auto y_a = y.packed_accessor&lt;float, 1&gt;();</p>
<pre><code>float c = 0;

dot_cuda_kernel&lt;&lt;&lt;1, 1&gt;&gt;&gt;(x_a, y_a, x.size(0), &amp;c);

cout&lt;&lt;c&lt;&lt;endl;

return c;
</code></pre>
<p>}</p>
</blockquote>
<p>But it does not work. The function dot_cuda always return 0. What could possibly go wrong? Thanks in advance!</p>

username: SimonW
cooked: <p><code>c</code> is in cpu memory. you can’t add to it in a cuda kernel.</p>
How to model element wise trainable scalars for a matrix input?

username: WeiHao97
cooked: <p>Say that I want a model looks like y=A*x where X is a matrix and * is element wise product. How can I construct a tranable A?<br>
Thank you very much</p>

username: WeiHao97
cooked: <p>Revise:I want y=(A*B) * x where x is a vector as one input and A is a matrix as another input and B is the trainable elementwise product matrix I desire. So far, I did :<br>
<a href="http://self.net" rel="nofollow noopener">self.net</a> = nn.Sequential(A, )<br>
And I am confused how to step forward.</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/weihao97">@WeiHao97</a>,</p>
<p>If I understand correctly you want to train your model w.r.t. <code>B</code>, the learning parameter matrix.</p>
<pre><code class="lang-auto">class Network(nn.Module):
   def __init__(self, shape):
       super().__init__()
       self.B = nn.Parameter(torch.zeros(shape))
       torch.nn.init.xavier_uniform_(self.B) #or any other init method
   def forward(self, x, A):
       M = A*self.B
       return x @ M.t()
</code></pre>
<p><code>shape</code> has to be <code>(output_dim, input_dim)</code>.  <code>input_dim</code> being the dim of <code>x</code>.<br>
Hope this is what you wanted to do.</p>

username: WeiHao97
cooked: <p><a class="mention" href="/u/spanev">@spanev</a> Thank you very much! This is exactly what I want to do.</p>
Lib and dll name of debug vs release have same name

username: jshanab
cooked: <p>I applaud you for providing builds, but…<br>
I think it is a bad practice to make the lib and dlls the same name when doing debug and release. it  makes using the lib difficult. You seem to like CMake and it has and expects to create both release and debug from same generated instance. (you could provide one zip for download if you just packaged them in debug and release subfolders but that depends on the FindPackage…</p>
Sparse operation in libtorch

username: sanketgujar
cooked: <p>Hi,<br>
I just want to do the below operation in libtorch but wouldn’t find any documentation or examples,<br>
<code>torch.sparse.FloatTensor(index, torch.ones(30), torch.Size([400,200,36])).to_dense()</code></p>
<p>Would you please direct me to the documentation or any example.<br>
Thanks</p>

username: tom
cooked: <p>I think you want the <a href="https://pytorch.org/docs/stable/torch.html#torch.sparse_coo_tensor" rel="nofollow noopener"><code>sparse_coo_tensor</code></a> function. In C++:</p>
<pre><code class="lang-auto">Tensor sparse_coo_tensor(const Tensor &amp; indices, const Tensor &amp; values, IntArrayRef size, const TensorOptions &amp; options={});
</code></pre>
<p>Best regards</p>
<p>Thomas</p>
Include C++ header files after building pytorch from source

username: johannesgensheimer
cooked: <p>Hello,</p>
<p>I’m working on a Jetson TX2 and as it’s a aarch64 architectur this doens’t work: <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_frontend.html</a><br>
I don’t know how to include the torch header files to my c++ code.</p>
<p>So I successfully built Pytorch from source with Ubuntu 16.04 und Cuda 9.0.<br>
Now I want to copy the header files for the C++ API to the standard include directory: /usr/local/include.<br>
Is this possible? Which directories do I have to copy?</p>
<p>And if I don’t copy it what would be the -DCMAKE_PREFIX_PATH=/path/to/libtorch path?<br>
Because just the pytorch directory doesn’t work.</p>
<p>I’m new to cmake and also to pytorch, so if you need any further information please let me know.<br>
Thanks in advance for all your help.</p>
<p>Best regards,<br>
Johannes</p>

username: tom
cooked: <p>After the Python build, the headers will be in <code>torch/include</code>. For libtorch builds, I needed to make sure to call cmake with <code>--target install</code> to get the headers.</p>
<p>Best regards</p>
<p>Thomas</p>
Remove dataparallel from submodules

username: Ke_MA
cooked: <p><strong>Context:</strong></p>
<p>I have a main model <code>netG</code>, which includes a bunch of modules (e.g. <code>nn.conv2D</code>, <code>nn.Sequential</code> and some other modules defined by myself).</p>
<p>Due to some issues on the code, some submodules of <code>netG</code> don’t support data parallel.</p>
<p>So, instead of wrapping the whole <code>netG</code> like this</p>
<blockquote>
<p>nn.DataParallel(netG)</p>
</blockquote>
<p>I do</p>
<blockquote>
<p>nn.DataParallel(nn.conv2D)<br>
nn.DataParallel(nn.Sequential)<br>
…</p>
</blockquote>
<p>for modules that support nn.DataParallel, and leaves other modules as it is (no data parallel)</p>
<p><strong>Problem:</strong></p>
<p>This works fine, but when I save <code>netG</code>, I want to remove the DataParallel model wrapper for the whole model. Clearly I can’t do <code>netG.module.state_dict()</code> because <code>netG</code> is not wrapped so it doesn’t have the attribute module.</p>
<p><strong>Candidate solutions:</strong></p>
<p>I am thinking of recursively removing the DataParallel wrapper for <code>netG</code>’s children, but I am not sure if I can do that in-palce, or I have to construct a new <code>netG</code> object from scratch.</p>
<p>What it the solution?</p>
<p>Thanks!</p>

username: nooblyh
cooked: <p>Maybe it’s not a good practice to separate your modules. But have you try this way to save your separate modules?</p>
<pre><code class="lang-auto">torch.save({
            'nn.conv2D_state_dict': nn.conv2D.module.state_dict(),
            'nn.Sequential_state_dict': nn.Sequential.module.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            ...
            }, PATH)
</code></pre>

username: Ke_MA
cooked: <p>Thank you! I haven’t try that, but that makes perfect sense.</p>
<p>Also, if some sub-modules can’t be parallelized (as in my case), what would be a good practice? Should we not parallelize the whole module at all?</p>

username: nooblyh
cooked: <p>I think DataParallel would work well, if you can make the batch size of your sub-modules at the same position for DataParallel to scatter. <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
Do i need a filter for the Input for LSTMs?

username: N3m0
cooked: <p>Hi everyone,<br>
im working on an LSTM which gets vibration signal of a mechanical part to predict the remaining useful lifetime. Since the signal is very noisy,  i’m not sure if the LSTM is able to learn the data properly. Is it necessary to filter noisy inputs for LSTMs? How is the approach in other fields, where the signal is noisy as well,  like for voice signal processing?<br>
I wanted to use the scipy.savgol_filter for the problem.!<br>
Thanks in advance.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/88ca635485c2e027270d90267ab3c0781fa65ffd.png" alt="Filter" data-base62-sha1="jw6y7jmWtD186MO3EdhAQhE8evz" width="640" height="480"><br>
blue is the original signal, orange is the filtered signal<br>
The signal is normalized for the training.</p>
Hidden state lstm

username: slavavs
cooked: <p>Can I use def __init__hidden (self, batch) instead:</p>
<pre><code class="lang-auto">hidden = torch.randn (1, batch, 512) .to (device)
       return hidden
</code></pre>
<p>hidden - which was obtained from the previous sequence. that is, do not make the new hidden, but use the previous one each time? The fact is that I am exploring a continuous sequence and it seems to me that hidden should be continuous.</p>

username: slavavs
cooked: <p>And another question, in my version, do I have the right to do randn initialization or do I need to do it with zeros?</p>

username: ptrblck
cooked: <p>I’m not sure to understand the question clearly, but it seems you would like to initialize the hidden state once and then just use if for the whole training?</p>

username: slavavs
cooked: <p>Yes, initialize once per epocha, and do not show once per batch</p>

username: ptrblck
cooked: <p>Would it be similar to <a href="https://github.com/pytorch/examples/blob/632d385444ae16afe3e4003c94864f9f97dc8541/word_language_model/main.py#L163" rel="nofollow noopener">this example</a>?</p>
<p>What do you mean by “do not show once per batch”?<br>
What hidden state should be passed instead for the next data batch?</p>

username: slavavs
cooked: <p>Take a look:</p>
<pre><code class="lang-auto">def forward(self, out):
        out = self.fc1(out)
        out = torch.transpose(out,0,1)
        hidden = self.__init__hidden(batch)
        out,hidden = self.gru(out, hidden)
        out = self.fc3(hidden)
        out = out.reshape(batch,3)
        return out
    
    def __init__hidden(self, batch):
        hidden = torch.randn(1, batch, 512).to(device)
        return hidden
</code></pre>
<p>When I call <code>outputs = net (wn)</code>, <code>def forward</code> is triggered.<br>
This happens for every sequence in batch.</p>
<p>And each time we initialize the hidden variable, which calls <code>def __init__hidden</code>. I want <code>def __init__hidden</code> to work the first time, and for the second and next passes hidden should be remembered from the previous sequence (out,<strong>hidden</strong> = self.gru(out, hidden)).</p>

username: DoubtWang
cooked: <p>u can use a constant to remember the hidden for the next sequence.<br>
this constant will not update.</p>
How to save and restore entire model without class definition

username: kevin_sandy
cooked: <p>Hi, everyone!<br>
I want to deploy my pytorch model and I don’t want other people to know the detail of my model’s parameter setting.<br>
I noticed that when using save&amp;load, you should always include your model’s definition code, so I wonder if there exists a way that I can save the model weights and parameters simultaneously, and restore without the model class defined in somewhere.</p>
How to save grads of parameters in checkpoint for following model training?

username: wade
cooked: <p>a = m.weight.grad.data.clone()<br>
AttributeError: ‘NoneType’ object has no attribute ‘data’</p>

username: ptrblck
cooked: <p>Your gradients were cleared or never calculated.<br>
What is your use case and could you provide a bit more information?</p>

username: wade
cooked: <p>I use checkpoint to save the information of trained model, but when I load model, the grad of parameter is none. How to maintain grad in checkpoint ?</p>
<p>print("Best accuracy: ", best_prec1)<br>
save_checkpoint({<br>
‘epoch’: epoch + 1,<br>
‘state_dict’: model.state_dict(keep_vars=True),<br>
‘best_prec1’: best_prec1,<br>
‘optimizer’: optimizer.state_dict(),<br>
}, is_best, filepath=args.save)</p>
<p>if args.model:<br>
if os.path.isfile(args.model):<br>
print("=&gt; loading checkpoint ‘{}’".format(args.model))<br>
checkpoint = torch.load(args.model)<br>
args.start_epoch = checkpoint[‘epoch’]<br>
best_prec1 = checkpoint[‘best_prec1’]<br>
model.load_state_dict(checkpoint[‘state_dict’])<br>
print("=&gt; loaded checkpoint ‘{}’ (epoch {}) Prec1: {:f}"<br>
.format(args.model, checkpoint[‘epoch’], best_prec1))<br>
else:<br>
print("=&gt; no checkpoint found at ‘{}’".format(args.resume))</p>

username: ptrblck
cooked: <p>You could probably store the gradients manually, as they won’t be saved using the <code>state_dict</code> method.</p>

username: wade
cooked: <p>but I want to use parameter at same time, how to store the gradients in checkpoint manually for following deal?</p>
How to get a batch with equal number of images of each class?

username: Zhang_Chi
cooked: <p>i have a dataset that there is a huge difference in the number of images in different classes. So i want to get a batch with equal number of images of each class. anyone knows how to do that?</p>

username: ptrblck
cooked: <p>You could use the <a href="http://pytorch.org/docs/master/data.html#torch.utils.data.sampler.WeightedRandomSampler" rel="nofollow noopener">WeightedRandomSampler</a>.</p>
<p>The issue was also discussed in <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2?u=ptrblck">this thread</a>.</p>

username: aliutkus
cooked: <p>hi,</p>
<p>thanks for that answer, I have the same question, but the thing is: can you guarantee that you’ll get <em>exactly</em> the same number of samples from each class within a batch this way ?</p>

username: aliutkus
cooked: <p>answering to myself, here’s the solution I came up with. It amounts to creating a loader for each class, and using all those loaders for each batch:</p>
<pre><code class="lang-auto">    dataset = datasets.ImageFolder(
        my_rootfolder,
        transform=my_transforms)
    # now create the loaders for the different classes
    loaders = []
    for class_name in dataset.classes:
        # get the indices in the dataset that are relative to that class
        idx = [
            pos for pos, item in enumerate(dataset.samples)
            if item[1] == dataset.class_to_idx[class_name]]
        # construct the corresponding dataloader thanks to a SubsetRandomSampler
        loaders += [torchdata.DataLoader(
            dataset, batch_size=args.batch_size,
            sampler = torchdata.sampler.SubsetRandomSampler(idx),
            **kwargs)]


    # now using the thing:
    for items in zip(*loaders):
        # items[idx] is the batch for class idx,
        # ...
</code></pre>

username: pheonix511
cooked: <p>I have a dataset already splited manually into train, validation and test with two classes each 0 and 1<br>
i want my network to fetch equal number of images in every batch<br>
constrain i am not having equal number of images in each class<br>
Train —&gt; class 0 —&gt; 1000, class 1 —&gt;100<br>
say if my batch size is 30 , i need to have 15 from class 0 and 15 from class 1 for every iteration.<br>
how to achieve this? also i am using Vgg16 architecture in keras</p>
A "Nonetype" problem about grad

username: GROX
cooked: <p>Hey,guys.I’m a beginner of machine learning and pytorch.Now i’m follow a chinese guide video but meet a problem i can’t solve.I follow the teacher and create a sample nn in a stupid way but when i train it ,my grad become “Nonetype” and i don’t know how to do.Here is my code :</p>
<p>n,input,h,output=64,1000,100,10</p>
<p>x=torch.randn(n,input)<br>
y=torch.randn(n,output)</p>
<p>w1=torch.randn(input,h,requires_grad=True)<br>
w2=torch.randn(h,output,requires_grad=True)</p>
<p>learningrate=0.000001</p>
<p>for t in range(500):<br>
<span class="hashtag">#forward</span><br>
h=x.mm(w1)<br>
h_relu=h.clamp(min=0)<br>
y_pred=h_relu.mm(w2)</p>
<pre><code>#loss 


loss=(y_pred-y).pow(2).sum()
print(t,loss.item())


#backward 
loss.backward()




with torch.no_grad():
    w1=w1-learningrate*w1.grad.data
    w2=w2-learningrate*w2.grad.data
    w1.grad.zero_()
    w2.grad.zero_()
</code></pre>
<hr>
<p>AttributeError                            Traceback (most recent call last)<br>
 in <br>
35         w1=w1-0.00001<em>w1.grad.data<br>
36         w2=w2-0.00001</em>w2.grad.data<br>
—&gt; 37         w1.grad.zero_()<br>
38         w2.grad.zero_()<br>
39</p>
<p>AttributeError: ‘NoneType’ object has no attribute ‘zero_’</p>
<p>I will be very grateful if some master give me a hand and show me the way to dell with the problem.</p>

username: ptrblck
cooked: <p>You are overwriting your parameters in the update block.<br>
To avoid this. you could use inplace operations:</p>
<pre><code class="lang-python">with torch.no_grad():
    w1.sub_(learningrate*w1.grad.data)
    w2.sub_(learningrate*w2.grad.data)
    w1.grad.zero_()
    w2.grad.zero_()
</code></pre>

username: GROX
cooked: <p>Thanks,<br>
It really helps a lot.Finally,i can finish my home work!<img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
How to skip .grad computation but still run backward hooks?

username: Yaroslav_Bulatov
cooked: <p>For my application it’s more efficient to keep gradients in factored form, ie, without applying the backprops @ activations matmul. Is there a recommended way to disable this computation, but run the rest of the autograd as expected?</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/yaroslav_bulatov">@Yaroslav_Bulatov</a>,</p>
<p>Setting <code>requires_grad=False</code> should work in your use case. It won’t stop autograd to run the backprop on the rest of the model, and only compute intermediate buffers needed by the rest of the computations.</p>

username: Yaroslav_Bulatov
cooked: <p>Thanks for the tip, I think that works for me. It seems some care needs to be taken not to set this on leaf parameters</p>
<pre><code class="lang-auto">layer1 = nn.Linear(1,1, bias=False)
layer2 = nn.Linear(1,1, bias=False)
net = nn.Sequential(layer1, layer2)
x = torch.ones((1))
layer2.weight.requires_grad=False
net(x*x).backward()
assert layer2.weight.grad is None
layer1.weight.requires_grad=False
net(x*x).backward()   # nothing to propagate
</code></pre>

username: spanev
cooked: <p>Great!<br>
That’s correct, at least one leaf node has to <code>require_grad</code> so <code>autograd</code> starts recording the graph.</p>
Error message when training torch.nn.Transformer: "ERROR:root:Creating dummy sequence since config seq length is greater than sample sequence"

username: changebo
cooked: <p>Hi,</p>
<p>I got the following error message when training a transformer model using <code>torch.nn.Transformer</code>.</p>
<blockquote>
<p>ERROR:root:Creating dummy sequence since config seq length is greater than sample sequence</p>
</blockquote>
<p>Could anyone shed light on potential issues? Thanks!</p>

username: zhangguanheng66
cooked: <p>You have to paste your code for debugging. Feel free to open an issue on Github and at zhangguanheng66 for the questions related to nn.Transformer.</p>

username: changebo
cooked: <p>Thanks, I will create a minimal working example!</p>
Same output size of Conv2d

username: razvanc92
cooked: <p>I’m working with dilated temporal 2d convolutions and in order for the output to have the same shape as the input, I need to add left padding (same as the dilatation). I looked at nn.Conv2d and it only accepts symmetric padding, which is not idea in my case. Also I’m trying to avoid the manual padding, since the code became hard to read. Are there any other solutions that I am not aware of?</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/razvanc92">@razvanc92</a>,</p>
<p>I dont know a better way than manual padding. This wrapper should leave the code readable:</p>
<pre><code class="lang-auto">class LeftPaddedConv2D(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, stride, left_padding, dilation, bias):
        super().__init__()
        self.pad = nn.ZeroPad2d((left_padding, 0, 0, 0))
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, stride, 0, dilation, bias=bias)
    def forward(x):
        x = self.pad(x)
        return self.conv(x)
</code></pre>

username: razvanc92
cooked: <p>Thanks for the answer. I ended up doing the same thing.</p>
Torch version automatic update

username: airwater
cooked: <p>Today, I found suddenly my torch version is 1.3.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/890d2714e9198e14f61a73e7291cc3fb49977411.png" alt="image" data-base62-sha1="jypAL73YPir2eGX7bvWSTBaUVd7" width="347" height="56"></p>
<p>I don’t know this is why.</p>
<p>My pytorch build from source, It can  do automatic update ?</p>
<p>Hope get your reply.</p>

username: ptrblck
cooked: <p>The code after a release gets bumped to the next version counter.<br>
I.e. if you build PyTorch from source after the <code>1.2.0</code> release, you’ll see a <code>1.3.0a0+commit</code> version.<br>
The same applies to the nightly builds, which will be named <code>1.3.0.devBUILDDATE</code>.</p>

username: airwater
cooked: <p>Thank you to reply me very much!<br>
From your answer, I know it, and first know what is nightly builds.<br>
Finally, confirm one thing with you:<br>
because I build torch from source(it isn’t stable), so torch can do nightly builds?<br>
If I conda install stable  through <a href="https://pytorch.org/" rel="nofollow noopener">https://pytorch.org/</a>  , torch will do not update.<br>
Is this what I understand?<br>
Thank you again!</p>
Why TorchVision needs to install coco-api?

username: shihyung
cooked: <p>Why don’t you release torch version coco-api?<br>
It should be quite simple for pytorch to implement an embedded similar function calls or classes, like RLE, iou, mAp, mean average precision, etc.<br>
Frankly, it’s so difficult to install coco-api on Mac…<br>
Could you please add them into pytorch? (no need to install coco-api)</p>
How do I interpolate directly on tensor?

username: 12_DHH
cooked: <p>Is there any easy way?<br>
I want to do interpolation(BICUBIC) directly on the variable<br>
(torch.cuda.FloatTensor of size 1x3x256x256 (GPU 0))  without other any transform</p>

username: InnovArul
cooked: <p>I am not sure if Bicubic interpolation is available. I can see ‘nearest’ | ‘linear’ | ‘bilinear’ | ‘trilinear’.</p>
<p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.interpolate" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.functional.interpolate</a></p>

username: 12_DHH
cooked: <p>Wow, that’ s cool !<br>
but, I used the 0.3.1 version, this version does not have such a function…</p>

username: nareshr8
cooked: <p>When I tried <code>interpolate(x,scale_factor=3)</code>, I am getting output of shape <code>torch.Size([3, 21, 63])</code> but I expected <code>torch.Size([3, 63, 63])</code>. What should I do. I tried to make tuple of shapes like (3,3), it didnt help as well. Kindly note that x is of shape <code>torch.Size([3, 21, 21])</code></p>

username: ptrblck
cooked: <p>Your input tensor is treated as a temporal signal, thus only the sequence length (dim2) will be interpolated, while the batch size and channels stay the same (dim0, dim1).</p>
<p>If your input is an image tensor, unsqueeze the batch dimension before passing it to <code>F.interpolate</code>:</p>
<pre><code class="lang-python">x = x.unsqueeze(0)
out = F.interpolate(x, scale_factor=3)
</code></pre>
[HELP] Could I process sequence of 2D images with nn.Conv3d?

username: MariosOreo
cooked: <p>Hello,</p>
<p>I have sequences of 2d images, each image is a single frame from a video.</p>
<p>With the length of each sequence given, the shape of the input is <code>[batch_size, seq_length, c, h, w]</code>, so I want to process them with <code>nn.Conv3d()</code>. But I met some unexpected problem that I had to transpose the input data into the shape of <code>[batch_size, c, seq_length, h, w]</code>, and finally the performance is poor (i.e. segmentation task of sequences of medical images).</p>
<p><strong>So</strong>, Could I consider the sequences of images as 3D task and process them with conv3d?</p>
<p>Thanks in advance!</p>

username: JuanFMontesinos
cooked: <p>You are doing it right. Proper order is BxCxSXHxW.<br>
3Dconv are really data greedy and they tend to overfit. You should have a really huge dataset to obtain good results. You can have a look at this paper.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.2.5/images/icons/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://arxiv.org/abs/1711.11248" target="_blank" rel="nofollow noopener">arXiv.org</a>
  </header>
  <article class="onebox-body">
    

<h3><a href="https://arxiv.org/abs/1711.11248" target="_blank" rel="nofollow noopener">A Closer Look at Spatiotemporal Convolutions for Action Recognition</a></h3>

<p>In this paper we discuss several forms of spatiotemporal convolutions for
video analysis and study their effects on action recognition. Our motivation
stems from the observation that 2D CNNs applied to individual frames of the
video have remained...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
They create networks modeling motion with 3dconvs at the beginning and 2d conv in the end.</p>
<p>BTW image2image translation is performed with encoder-decoder architectures, what are you exactly using?</p>
pytorch collate_fn reject sample and yield another

username: Aniloid2
cooked: <p>I have built a Dataset, where I’m doing various checks on the images I’m loading. I’m then passing this DataSet to a DataLoader.</p>
<p>In my DataSet class I’m returning the sample as None if a picture fails my checks and I have a custom collate_fn function which removes all Nones from the retrieved batch and returns the remaining valid samples.</p>
<p>However at this point the returned batch can be of varying size. Is there a way to tell the collate_fn to keep sourcing data until the batch size meets a certain length?</p>
<pre><code class="lang-auto">class DataSet():
     def __init__(self, example):
          # initialise dataset
          # load csv file and image directory
          self.example = example
     def __getitem__(self,idx):
          # load one sample
          # if image is too dark return None
          # else 
          # return one image and its equivalent label

dataset = Dataset(csv_file='../', image_dir='../../')

dataloader = DataLoader(dataset , batch_size=4,
                        shuffle=True, num_workers=1, collate_fn = my_collate )

def my_collate(batch): # batch size 4 [{tensor image, tensor label},{},{},{}] could return something like G = [None, {},{},{}]
    batch = list(filter (lambda x:x is not None, batch)) # this gets rid of nones in batch. For example above it would result to G = [{},{},{}]
    # I want len(G) = 4
    # so how to sample another dataset entry?
    return torch.utils.data.dataloader.default_collate(batch) 
</code></pre>