akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to speedup element wise update?

username: odats
cooked: <p>In my custom dataset, the data is loaded from Pandas. There is augmentation by creating a new item based on a simple rule:</p>
<pre><code class="lang-auto">def augmentation_by_color(x, colors):
    x = np.array(x)
    return np.array([colors[xi] for xi in x.flatten()]).reshape(x.shape)
colors = [0,1,2,3,4,5,6,7,8,9]
colors = np.random.permutation(colors)
new_data = (dataitem_from_pandas, colors)
# new_data to tensor and return for training on CUDA
</code></pre>
<p>Everything works on CPU and then converted to tensor that will be processed on CUDA. How to speed up this process? Should I do all operations on tensors (convert Pandas item to tensor and the do permutations there)? If so can you suggest a code?</p>
<p>If loading data takes most of time then should I increase the num_workers?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes I would say that doing this as a single indexing operation would make it much faster.<br>
Maybe something as simple as <code>colors[x]</code>? Or <code>colors[x.flatten()].reshape(x.shape)</code> ?</p>

username: odats
cooked: <p>Thank you it made it much faster. Do you think I should keep it on CPU or switch to GPU? In general case how to speedup dataloaders?</p>

username: albanD
cooked: <p>If this is done in the preprocessing step of the dataloader, you can keep it on CPU.<br>
In general, large indexing ops will be better on CPU unless the data is already on the GPU.</p>

username: odats
cooked: <p>How can increasing num_workers help?</p>

username: albanD
cooked: <p>I will help if your CPU is underutilized. But it can be detrimental if it is already fully used.<br>
The only way to know here is to test different values and see which one gives you the best result as it will depend on many things (like disk, preprocessing, caching, etc)</p>
Pytorch DistributedDataParallel Help

username: kleingeo
cooked: <p>I am currently using the <code>nn.DataParellel</code> to parallelize my model, but I am planning to switch to a framework that utilizes the <code>nn.DistributedDataParallel</code> with multiple GPU from a single system. Is there a tutorial out there or some information to help with this? I am a little confused with how to set up <code>torch.distributed.init_process_group</code> with only a single job on a single machine.</p>

username: mrshenli
cooked: <p><a class="mention" href="/u/kleingeo">@kleingeo</a></p>
<p>Please checkout this <a href="https://pytorch.org/docs/master/notes/ddp.html" rel="nofollow noopener">this note</a> and <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">this tutorial</a></p>
<p>BTW, for questions related to distributed training, please add a “distributed” tag, so that the team monitoring this tag can get back to you promptly.</p>

username: kleingeo
cooked: <p>Thanks, I’ll look into those and try that.</p>
Model.to(device) does't work in dynamic graph?

username: Baron_Tsai
cooked: <p>I create a dynamic conv-net by <code>for in range</code>, and data.to(device) as well as model.to(device).<br>
However it raise ‘RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same’.<br>
Is there something wrong when i create dynamic net and use cuda?<br>
Here is the key code, thanks very much!</p>
<pre><code class="lang-auto"># create dynamic net
class AutoNet(nn.Module):
    def __init__(self, channels=[32, 32, 32], conv_kernel_size=3, conv_padding=1, pool_kernel_size=2, pool_stride=2):
        super(AutoNet, self).__init__()
        self.pool = nn.MaxPool2d(kernel_size=pool_kernel_size, stride=pool_stride)

        self.convs = []
        channel_in = 1
        layer_out = args.image_size
        for i in range(len(channels)):
            channel = channels[i]
            layer_out = (layer_out - conv_kernel_size + 2 * conv_padding) / 1 + 1
            layer_out = (layer_out - pool_kernel_size) / pool_stride + 1
            self.convs.append(
                nn.Conv2d(in_channels=channel_in, out_channels=channel, kernel_size=conv_kernel_size, padding=conv_padding))
            channel_in = channel
        self.fc = nn.Linear(layer_out * layer_out * channels[-1], args.num_class)

    def forward(self, x):
        for i in range(len(self.convs)):
            x = self.pool(self.convs[i](x))
        x = x.view(args.batch_size, -1)
        x = self.fc(x)
        return x

# model
model = AutoNet(channels=args.channels,
                    conv_kernel_size=args.conv_kernel_size,
                    conv_padding=args.conv_padding,
                    pool_kernel_size=args.pool_kernel_size,
                    pool_stride=args.pool_stride)
device = torch.device('cuda:' + str(args.gpu) if args.gpu != -1 else 'cpu')
model.to(device)

# train
inputs, labels = data[0].to(device), data[1].to(device)
outs = model(inputs)

</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>The problem is that you store you convolutions in a plain python list. This means that <code>.to()</code> cannot find these convs to move them to cuda.<br>
You should do <code>self.convs = nn.ModuleList()</code> instead of <code>self.convs = []</code> and that will work fine.</p>
MultivariateNormal, Categorical, etc. do not inherit from ExponentialFamily

username: richardk
cooked: <p>I noticed that several exponential family distributions such as MultivariateNormal and Categorical do not inherit from the ExponentialFamily base class. Is there a reason for this?</p>
<p>On a side note, it would be very useful to make the natural_params property of the ExponentialFamily public. Several operations, e.g. the product of N densities (probabilistic sensor fusion), can then be trivially implemented for any/all distributions in this family.</p>
CNN that only looks at local area

username: Fibbi
cooked: <p>I’m trying to look at local orientation in images. I have a set of images which have the size <em>150x150x150</em> and for each voxel I would like to produce a unit vector i.e. an output of size <em>3x150x150x150</em> for each image. Each of these unit vectors should reflect the dominant orientation in its neighborhood, in my case the size of this neighborhood is <em>17x17x17</em> (<code>kernel_size = 17</code>), which means my model shouldn’t care about other values which are outside of this neighborhood.</p>
<p>I tried naively setting up a series of 3D convolutions and trained on a small test-set. But I’m pretty sure that I’m already reaching beyond my neighborhood in my second convolution.</p>
<pre><code class="lang-python">class CNN3(torch.nn.Module):
    def __init__(self,kernel_size):
        super(CNN3, self).__init__()
        self.k = kernel_size
        self.pad = kernel_size//2
        self.conv1 = torch.nn.Conv3d(1, 3, kernel_size=self.k, stride=1, padding=self.pad)
        self.conv2 = torch.nn.Conv3d(3, 3, kernel_size=self.k, stride=1, padding=self.pad)
        self.conv3 = torch.nn.Conv3d(3, 3, kernel_size=self.k, stride=1, padding=self.pad)
        self.conv4 = torch.nn.Conv3d(3, 3, kernel_size=self.k, stride=1, padding=self.pad)
    def forward(self,x):
        out = F.relu(self.conv1(x))
        out = F.relu(self.conv2(out))
        out = F.relu(self.conv3(out))
        out = self.conv4(out)
        return out
</code></pre>
<p>Is there a way to set up a CNN to only look at a particular neighborhood or is it simply not the right tool for the job?</p>
OneCycleLR Why specfiy number of steps

username: JanoschMenke
cooked: <p>Hi,</p>
<p>I do not understand why for the OneCycle LR scheduler I need to specify the number of epochs or the total number of steps. I was under the impression that the OneCycle scheduler cycles through the learning rate and momentum, per batch within a epoch but is reset for each new epoch.</p>
<p>How would I set up the scheduler if I do not know the number of epochs because I am using Early Stopping.</p>
Index lookup without for loop

username: chris-clem
cooked: <p>Hi,</p>
<p>I have been struggling with the following problem for a while now and would appreciate your help.</p>
<p>I have a for loop in my forward pass that slows down the training immensely, but I do not know how to get rid of it.<br>
What I want to do is to resample features of an image based on optical flow. To do that, I have the following code:</p>
<pre><code class="lang-auto"># Get the indices where the flow is not zero
flow_files_non_zero_indices = torch.nonzero(flow_files_shifted, as_tuple=True)
face_images_features_resampled = face_images_features

# Iterate over these indices
for index in zip(
    flow_files_non_zero_indices[0],
    flow_files_non_zero_indices[1],
    flow_files_non_zero_indices[2],
    flow_files_non_zero_indices[3],
):
    # Get index
    n, b, u, v = index

    # Get delta_u and delta_v from flow
    delta_u = flow_files_shifted[n, b, u, v, 0].type(torch.int64)
    delta_v = flow_files_shifted[n, b, u, v, 1].type(torch.int64)

    # Index lookup
    try:
        face_images_features_resampled[n, b, u, v] = face_images_features[
            n, b, u + delta_u, v + delta_v
        ]
    except IndexError:
        face_images_features_resampled[n, b, u, v] = face_images_features[
            n, b, u, v
        ]
</code></pre>
<p>I know of <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.index_put_" rel="nofollow noopener">index_put_</a>, but I do not see how I can use it here as I need to look up delta_u and delta_v based on the current index.</p>
<p>Is there a way to get rid of the for loop?</p>
<p>Your help is very much appreciated!</p>
Easy way for mutil-type data input?

username: moluchase
cooked: <p>I’m trying joint model training,so my model’s input data is various.<br>
example :</p>
<pre><code class="lang-auto">uid_data: [12,3,23,4,54,23,12,43,32...]  
feature_data:[1,0,2.3,23,3,0.4...]
text_data:[[1,2,5,3,23,43..],[4,34,54,34,5,...],...]
and many...
</code></pre>
<p>for normal way, i need to write these variables one by one to forward function ,like:</p>
<pre><code class="lang-python">class MyModel():
    def __init__():
        pass
    def forward(self,uid_data,uid_mask,feature_data,feature_mask,text_data,text_mask):
        pass
#
uid_data=torch.LongTensor(uid)
feature_data=torch.LongTensor(feature)
text_data=torch.LongTensor(text)
...
</code></pre>
<p>I want to know if there is an easy way to achieve these like:</p>
<pre><code class="lang-python">class MyModel():
    def __init__():
        pass
    def forward(self,x):
        pass

#
X_data=torch.LongTensor(train_X) # element of train_X like: [[12,3,23,4,54,23,12,43,32...] , [1,0,2.3,23,3,0.4...],[[1,2,5,3,23,43..],[4,34,54,34,5,...],...],...]
</code></pre>
<p>thanks a lot !</p>

username: emad
cooked: <p>I used to concatenate them at the data loader into one X, then split them back at the beginning of “forward”</p>
Change one value in a tensor?

username: Jesper.Lindberg
cooked: <p>I want to change a value in a tensor to a specific value. The current way I try to do it: <code>next_Q[replay_memory.at(trainingIDX).action].item() = reward;</code></p>
<p>My entire training loop and some output from it can be seen below.</p>
<pre><code>  void AiModel::train(Model &amp;model_to_train,
	  torch::optim::Optimizer &amp;optimizer,
	  std::vector&lt;AiModel::Experience&gt; &amp;replay_memory) {
	  model_to_train.train();
	  //traning loop
	  for (size_t trainingIDX = 0; trainingIDX &lt; replay_memory.size(); ++trainingIDX) {

		  float reward;
		  torch::Tensor next_state;
		  torch::Tensor next_Q;
		  torch::Tensor current_state = torch::from_blob(replay_memory.at(trainingIDX).current_state.data(), { k_input_size_ }).to(*device);

		  if (std::isnan(replay_memory.at(trainingIDX).next_state.at(0))){
			  //If next state is terminal:
			  next_state = torch::from_blob(replay_memory.at(trainingIDX).current_state.data(), { k_input_size_ }).to(*device);
			  next_Q = model_to_train.forward(next_state).detach();
			  reward = replay_memory.at(trainingIDX).reward;
			  next_Q[replay_memory.at(trainingIDX).action].item() = reward;
		  } else {
			  next_state = torch::from_blob(replay_memory.at(trainingIDX).next_state.data(), { k_input_size_ }).to(*device);
			  next_Q = model_to_train.forward(next_state).detach();
			  std::cout &lt;&lt; "next_Q: \n" &lt;&lt; next_Q  &lt;&lt; std::endl;
 			  reward = replay_memory.at(trainingIDX).reward + k_gamma_ * next_Q.max().item().toFloat();
			  reward = 1;
			  std::cout &lt;&lt; "reward: " &lt;&lt; reward &lt;&lt; std::endl;
			  next_Q[replay_memory.at(trainingIDX).action].item() = reward;
			  std::cout &lt;&lt; "next_Q after: \n" &lt;&lt; next_Q &lt;&lt; std::endl;
		  }

		  optimizer.zero_grad();
		  auto current_Q = model_to_train.forward(current_state);
		  std::cout &lt;&lt; "current_Q: \n" &lt;&lt; current_Q &lt;&lt; std::endl;
		  auto loss = torch::mse_loss(current_Q, next_Q);
		  std::cout &lt;&lt; "loss: " &lt;&lt; loss &lt;&lt; std::endl;
		  loss.backward();
		  optimizer.step();
	 }//For traning loop
  } // train-function
</code></pre>
<p>Which gives me the output:</p>
<pre><code>next_Q:
-0.0558
 0.0243
 0.1972
-0.1091
 0.0342
[ CPUFloatType{5} ]
reward: 1
next_Q after:
-0.0558
 0.0243
 0.1972
-0.1091
 0.0342
[ CPUFloatType{5} ]
current_Q:
-0.0543
 0.0240
 0.1965
-0.1083
 0.0341
[ CPUFloatType{5} ]
loss: 6.94254e-07
[ CPUFloatType{} ]</code></pre>

username: Jesper.Lindberg
cooked: <p>Solve by removing <code>.item()</code>…</p>
Using Conv1d with four dimensional input

username: pgp
cooked: <p>I’m using Conv1d to process input which is in the format [batch_size, Nc, Nh, Nw].</p>
<pre><code class="lang-auto">m = nn.Conv1d(16, 2, 3, stride=2) 
input = torch.randn(20, 16, 50, 50) 
output = m(input)
</code></pre>
<p>It gives the following error:<br>
RuntimeError: Expected 3-dimensional input for 3-dimensional weight 33 16 3, but got 4-dimensional input of size [20, 16, 50, 50] instead</p>
<p>I want to reduce the 50x50x16(height, width, channel) to 50x50x2(height, width, channel). How can this be done? Thanks in advance!!!</p>

username: ptrblck
cooked: <p>You are passing a 4-dimensional input, so you should use <code>nn.Conv2d</code> for it.<br>
Also, if you don’t want to reduce the spatial size, you could use a <code>1x1</code> kernel or add padding, as currently your setup will yield an output of <code>torch.Size([20, 2, 24, 24])</code>.</p>
Cannot set output_size in ConvTranspose2d

username: pgp
cooked: <p>I’ve been trying to use ConvTranspose2d with the following code.</p>
<pre><code class="lang-auto">upsample = nn.ConvTranspose2d(1024, 512, kernel_size=2)                                                                                                                              
e4=torch.randn(2, 512, 56, 56)                                                                                                                                                       
b=torch.randn(2, 1024, 28, 28)                                                                                                                                                       
b_1=upsample(b, output_size=e4.size())
</code></pre>
<p>It is giving the error<br>
ValueError: requested an output size of torch.Size([56, 56]), but valid sizes range from [29, 29] to [29, 29] (for an input of torch.Size([28, 28]))</p>
<p>I am confused with this and not able to solve. Can anyone help me on this. Thanks in advance!!!</p>

username: Tony-Y
cooked: <pre><code class="lang-auto">upsample = nn.ConvTranspose2d(1024, 512, 3, stride=2, padding=1) 
</code></pre>
<p>Why not use this?</p>

username: pgp
cooked: <p>Thank you. This works. I get the intuition behind adding padding but why is stride needed for this?</p>

username: Tony-Y
cooked: <p><a href="https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md" rel="nofollow noopener">This site</a> provides a nice visualization about stride and padding.</p>
[Tutorial] Self-Supervision with FastAI

username: Epoching
cooked: <p>Hey guys!</p>
<p>I recently got started with <a href="https://fastpages.fast.ai/" rel="nofollow noopener">fastpages</a> and started my own blog! Feel free to check it out: <a href="https://amarsaini.github.io/Epoching-Blog/" rel="nofollow noopener">Epoching’s Blog</a></p>
<p>My first blog post is on self-supervised learning, with FastAI2 <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Check it out here: <a href="https://amarsaini.github.io/Epoching-Blog/jupyter/2020/03/23/Self-Supervision-with-FastAI.html" rel="nofollow noopener">Self-Supervision with FastAI </a></p>
<p>It’s a friendly coder introduction to rotation-based self-supervised pretraining. Let me know what you guys think! Feedback is appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Epoching (Amar Saini)</p>

username: ptrblck
cooked: <p>Thanks for sharing! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
What loss does Detectron2 uses?

username: alx
cooked: <p>Hey guys I was wondering if anyone knew what loss detectron2 was using? <a href="https://github.com/facebookresearch/detectron2/blob/5e2a1ecccd228227c5a605c0a98d58e1b2db3640/detectron2/modeling/roi_heads/roi_heads.py#L648-L661" rel="nofollow noopener">I found L1 here but I’m not too sure.</a></p>

username: ptrblck
cooked: <p>Based on <a href="https://github.com/facebookresearch/detectron2/blob/6c0ff7e2bb9ba50947ea3dcfd756b9c5a411f1e3/detectron2/modeling/roi_heads/fast_rcnn.py#L307-L318">this code</a>, it seems cross entropy and a smooth L1 loss are used.</p>
Using WeightedRandomSampler in PyTorch

username: SurajSubramanian
cooked: <p>I need to implement a multi-label image classification model in PyTorch. However my data is not balanced, so I used the <code>WeightedRandomSampler</code> in PyTorch to create a custom dataloader. But when I iterate through the custom dataloader, I get the error : <code>IndexError: list index out of range</code></p>
<p><a href="https://discuss.pytorch.org/t/balanced-sampling-between-classes-with-torchvision-dataloader/2703/3" class="inline-onebox">Balanced Sampling between classes with torchvision DataLoader</a> (Implemented the following code using this link)</p>
<pre><code class="lang-auto">def make_weights_for_balanced_classes(images, nclasses):                        
    count = [0] * nclasses                                                      
    for item in images:                                                         
        count[item[1]] += 1                                                     
    weight_per_class = [0.] * nclasses                                      
    N = float(sum(count))                                                   
    for i in range(nclasses):                                                   
        weight_per_class[i] = N/float(count[i])                                 
    weight = [0] * len(images)                                              
    for idx, val in enumerate(images):                                          
        weight[idx] = weight_per_class[val[1]]                                  
    return weight 
</code></pre>
<pre><code class="lang-auto">weights = make_weights_for_balanced_classes(train_dataset.imgs, len(full_dataset.classes))
weights = torch.DoubleTensor(weights)
sampler = WeightedRandomSampler(weights, len(weights))

train_loader = DataLoader(train_dataset, batch_size=4,sampler = sampler, pin_memory=True)   
</code></pre>
<p>Thanks a lot in advance !</p>

username: ptrblck
cooked: <p>Which line of code is throwing this error?<br>
The code looks generally alright and I cannot spot an obvious mistake.</p>
Pywick - High-Level Training framework for Pytorch

username: achaiah
cooked: <p>Hi Pytorch community, please allow me to introduce you to <a href="https://github.com/achaiah/pywick" rel="nofollow noopener">Pywick</a> - a high-level Pytorch training framework that aims to get you up and running quickly with state of the art neural networks for <a href="https://pywick.readthedocs.io/en/latest/api/pywick.models.html" rel="nofollow noopener">image classification and segmentation</a>, <a href="https://pywick.readthedocs.io/en/latest/api/pywick.optimizers.html" rel="nofollow noopener">optimizers</a> (like SWA, AdamW), activation functions (swish/aria) etc. Does the world really need another Pytorch framework? Probably not. But I began working on this project about three years ago when no good frameworks were available and it just kept growing. So it probably doesn’t hurt to have another open-source framework out there.</p>
<p>The goal is to try to keep up with the state of the art in terms of <a href="https://pywick.readthedocs.io/en/latest/api/pywick.models.html" rel="nofollow noopener">models</a> and various training enhancements so that you have something to play around with. Does SWA or Swish really guarantee you best results? Probably not… but it’s there in case you want to check it out.</p>
<p>Lastly, I just wanted to mention that I really can’t take credit for any of the cool stuff. I mostly assembled and cleaned up what other awesome people have already done. But that way you don’t need to do the dirty work and can just use it.</p>

username: braindotai
cooked: <p>I really like its api consistency and completeness, and it really is a great framework. But its highly unmaintained, I can only find a single good tutorial example and while running it showed many errors!<br>
Updating documentation is a must, otherwise its a great framework- its perfectly like keras + pytorch.</p>

username: achaiah
cooked: <p>Thanks for the feedback! Providing updated examples is a big goal of mine. Keep in mind that the models and the framework itself are pretty well maintained – it’s just lacking good templates to get people started. Stay tuned - I’ll try to push a few out in the near future.</p>

username: braindotai
cooked: <aside class="quote no-group" data-username="achaiah" data-post="3" data-topic="45723">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/achaiah/48/3366_2.png" class="avatar"> achaiah:</div>
<blockquote>
<p>Keep in mind that the models and the framework itself are pretty well maintained</p>
</blockquote>
</aside>
<p>That’s true indeed.</p>
<p>Thanks</p>
Network parameter sync in forward pass

username: DzReal
cooked: <p>Hi, all</p>
<p>I’m trying to use the distributed package for multi-gpu training. Because of the way the code is written, the master process does all the initialisations (creating model replicas, optimisers etc.). From pytorch source code, it seems like during forward pass, all model replicas will be synced with the one running in subprocess with rank 0. Does that mean I could just initialise one optimiser for subprocess 0 and only update the parameters of the first model replica?</p>
<p>Thanks,</p>

username: mrshenli
cooked: <p>Hi <a class="mention" href="/u/dzreal">@DzReal</a></p>
<blockquote>
<p>From pytorch source code, it seems like during forward pass, all model replicas will be synced with the one running in subprocess with rank 0.</p>
</blockquote>
<p>If you are using DistributedDataParallel, above is actually not true. The distributed sync occurs during the backward pass, and it averages all gradients instead of parameters (check <code>torch/csrc/distributed/c10d/reducer.cpp</code>). So that when the optimizer consumes those gradients, they are already global gradients.</p>
<p>The sync you mentioned in the forward pass might be <a href="https://github.com/pytorch/pytorch/blob/46539eee0363e25ce5eb408c85cefd808cd6f878/torch/nn/parallel/distributed.py#L442" rel="noopener nofollow ugc">this</a>. This only does intra-node sync when you use one DDP process to work on multiple GPUs.</p>
<blockquote>
<p>Does that mean I could just initialise one optimiser for subprocess 0 and only update the parameters of the first model replica?</p>
</blockquote>
<p>No. Each DDP process should have its own local optimizer.</p>

username: DzReal
cooked: <p>So when running on a single node with multiple GPUs, could I only use one optimiser?</p>

username: mrshenli
cooked: <blockquote>
<p>So when running on a single node with multiple GPUs, could I only use one optimiser?</p>
</blockquote>
<p>You will need one optimizer per DDP process, regardless of where those DDP processes are. I hope the following note could help explain it:  <a href="https://pytorch.org/docs/master/notes/ddp.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch master documentation</a></p>
GridSearchCV with pytorch tensor in cuda

username: nushib
cooked: <p>I am using GridSearchCV for cross validation as indicated here (<a href="https://github.com/skorch-dev/skorch" rel="nofollow noopener">https://github.com/skorch-dev/skorch</a>). However it seems like I am not able to pass X and y as pytorch tensors loaded in cuda. This is my unsuccessful attempt:</p>
<p>device = “cuda”<br>
X = torch.from_numpy(X).to(device)<br>
y = torch.from_numpy(y).to(device)<br>
gs.fit(X, y)</p>
<p>The error I get is “can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.”.  This appears during gs.fit(X, y). Of course if I convert this to cpu() it works but that defeats the purpose because I’d like to avoid data transfers from memory to the gpu. Anything I am missing?</p>

username: acarlier
cooked: <p>This means X and y already are PyTorch tensors, so you don’t need to import them from NumPy. Just remove the first 3 lines <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: nushib
cooked: <p>Thanks! I also just read in the skorch documentation that fit() converts X and y to pytorch tensors. It however doesn’t say whether it transfers the data to the gpu. To make this more concrete, below is the code I am running. The problem appears on net.fit() as well. Here X, y are just numpy.ndarray-s. If used as such, they will be located in main memory and not in the gpu which is why I am importing, or am I wrong? Because of this, the following code runs 1.5 faster on cpu.</p>
<pre><code class="lang-auto">import numpy as np
from sklearn.datasets import make_classification
from torch import nn
import torch.nn.functional as F
import torch
from skorch import NeuralNetClassifier
from datetime import datetime, time as datetime_time, timedelta
from dateutil.relativedelta import relativedelta
from sklearn.model_selection import GridSearchCV


def diff(t_a, t_b):
    t_diff = relativedelta(t_b, t_a)  # later/end time comes first!
    return '{h}h {m}m {s}s'.format(h=t_diff.hours, m=t_diff.minutes, s=t_diff.seconds)

class MyModule(nn.Module):
    def __init__(self, num_units=10, nonlin=F.relu):
        super(MyModule, self).__init__()

        self.dense0 = nn.Linear(20, num_units)
        self.nonlin = nonlin
        self.dropout = nn.Dropout(0.5)
        self.dense1 = nn.Linear(num_units, 10)
        self.output = nn.Linear(10, 2)

    def forward(self, X, **kwargs):
        X = self.nonlin(self.dense0(X))
        X = self.dropout(X)
        X = F.relu(self.dense1(X))
        X = F.softmax(self.output(X), dim=-1)
        return X

device = "cuda"
net = NeuralNetClassifier(
    MyModule,
    max_epochs=100,
    lr=0.1, 
    # Shuffle training data on each epoch
    iterator_train__shuffle=True,device=device
)

X, y = make_classification(1000, 20, n_informative=10, random_state=0)
X = X.astype(np.float32)
y = y.astype(np.int64)

X =  torch.from_numpy(X).to(device)
y = torch.from_numpy(y).to(device)

t_a = datetime.now()
net.fit(X, y)
y_proba = net.predict_proba(X)
t_b = datetime.now()
print(diff(t_a, t_b))

</code></pre>
Pytorch multi gpu training

username: Sangwon_Jake
cooked: <p>I’m trying to run 2 gpu with torch tranning. <br></p>
<pre><code>from apex.parallel import DistributedDataParallel as DDP

   #multi gpu
    os.environ["CUDA_VISIBLE_DEVICES"] = '0, 1'
    
    torch.distributed.init_process_group(backend='nccl',
                                             init_method='env://')
    #model = nn.DataParallel(model, output_device=1)
    model = DDP(model, delay_allreduce=True)
</code></pre>
<p>I added this init_process_group and DDP part <br></p>
<ol>
<li>this is not running without error <br>
</li>
<li>and I dont know what backend and init_method mean</li>
</ol>

username: mrshenli
cooked: <p><a class="mention" href="/u/sangwon_jake">@Sangwon_Jake</a></p>
<p>Checkout <a href="https://pytorch.org/docs/master/notes/ddp.html" rel="nofollow noopener">this doc</a> and <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">this tutorial</a>.</p>
<p>The <code>init_method</code> arg helps process groups to perform rendezvous. You can use either of the following two options:</p>
<ol>
<li>set with a string. E.g., for local training, you can specify sth like <code>tcp://localhost:23456</code> to tell all processes to rendezvous on that port.</li>
<li>set <code>MASTER_ADDR</code> and <code>MASTER_PORT</code> env vars.</li>
</ol>
<p>BTW, for questions regarding distributed training, please add a “distributed” tag, so that the corresponding team can closely monitor that.</p>
Save the output of the network in .ark format

username: Mohamed_Nabih
cooked: <p>I trained a PyTorch model for speech recognition and I want to save the output of the model as .ark file<br>
Can anyone give me a help?</p>
<p>Thanks</p>

username: George3d6
cooked: <p>Not really pytroch related it seems ?</p>
<p>Are you referring to this ? <a href="https://fileinfo.com/extension/ark" rel="nofollow noopener">https://fileinfo.com/extension/ark</a><br>
If so… it’s a pretty arcane format, I’m pretty sure people switched to .zip and .gzip a few decade ago, if you need it for some reason you can use arc (<a href="https://sourceforge.net/projects/arc/" rel="nofollow noopener">https://sourceforge.net/projects/arc/</a>), at least on linux (but should work on OSX and windows as well I’d assume)</p>
<p>Just output whatever you have to a text file and then use the arc tool to compress it. There’s also a python library for it seemingly: <a href="https://pypi.org/project/arc/" rel="nofollow noopener">https://pypi.org/project/arc/</a>, but I’m not sure if it actually works.</p>

username: Mohamed_Nabih
cooked: <p>So I need to save my network posteriors in any format can you give me a help on this</p>
how to assign value to a tensor using index correctly?

username: chenxiahan
cooked: <p>I defined four tensors that represent index_x,index_y,index_z,and value,respectively and assigned value to a new tensor using these three index. Why were the results of the two assignments different?</p>
<pre><code class="lang-auto">import torch
import numpy as np
import random
import os

def seed_torch(seed=0):
    random.seed(seed)
    np.random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

seed_torch(1)
a_list, b_list, c_list = [], [], []
for i in range(0, 512*512):
    a_ = random.randint(0, 399)
    b_ = random.randint(0, 399)
    c_ = random.randint(0, 199)
    a_list.append(a_)
    b_list.append(b_)
    c_list.append(c_)
a = torch.tensor(a_list)
b = torch.tensor(b_list)
c = torch.tensor(c_list)
v = torch.rand(512*512)
matrix1 = torch.zeros(400,400,200)
matrix2 = torch.zeros(400,400,200)
index=[a,b,c]
matrix1[index]=v
matrix2[index]=v
m = matrix1 - matrix2
print(m.sum())
</code></pre>
<p>print(m.sum()) is not zero</p>

username: ptrblck
cooked: <p>You are using repetitive indices, which will write to the same index in an undefined order.<br>
You can check it by comparing the shape of the indices and their unique length:</p>
<pre><code class="lang-python">index = torch.stack(index)
u = index.unique(dim=1)
print(index.shape)
&gt; torch.Size([3, 262144])
print(u.shape)
&gt; torch.Size([3, 261021])
</code></pre>

username: chenxiahan
cooked: <p>Thank you very much! If I want to calculate the mean of repetitive indices as the value of this index, how can I achieve it fast? If I use circular judgement, it will be very slow.</p>

username: ptrblck
cooked: <p>You could use <code>torch.unique</code> with <code>return_counts=True</code>, check for each index with <code>counts &gt; 1</code>, and grab the index. Once you have all repeated indices, you could use it to calculate the position of these indices and use this to calculate the mean of <code>v</code>.</p>
Is there a way to set a custom scatter function for DataParallel?

username: lalord
cooked: <p>Hey, I’ve been trying to use the DataParallel class for training on several local GPUs. The issue I have is that my input data doesn’t fit the type of data that DataParallel, and in particular its scatter function, expects.</p>
<p>The scatter function in DataParallel recursively looks for Tensor types in the input data and distributes it between your GPUs along the axis you tell it to. This is all well and fine but it doesn’t consider the use case of people using lists of Tensors as minibatches. Is this a use case that will be supported, or is there a way to customize the scatter function myself?</p>
<p>Cheers!</p>

username: 1234
cooked: <p>Hi, were you able to find a solution for this? I have a similar issue.</p>

username: George3d6
cooked: <p>As per <a href="https://discuss.pytorch.org/u/SimonW">SimonW</a>’s answer here: <a href="https://discuss.pytorch.org/t/dataparallel-chunking-for-a-list-of-3d-tensors/15962" class="inline-onebox">Dataparallel chunking for a list of 3d tensors?</a>. You might find luck overriding the scatter function: <a href="https://github.com/pytorch/pytorch/blob/v0.3.1/torch/nn/parallel/data_parallel.py#L76-L80" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/v0.3.1/torch/nn/parallel/data_parallel.py#L76-L80</a> to make it work.</p>
<p>Other than that, this guy seems to have it working with a list with no issues: <a href="https://discuss.pytorch.org/t/nn-dataparallel-with-input-as-a-list-not-a-tensor/39485" class="inline-onebox">nn.DataParallel with input as a list not a tensor</a></p>
<p>Not sure as to the specific customizations that should be done to the function, or if there is a workaround.</p>
<p>Could you post an exact mock example of the code you are trying to run ? That might help.</p>
Quantization support for 1D convolutions?

username: snakers41
cooked: <p>Hi!</p>
<p>Are you planning to add support for 1D convolutions for <a href="https://pytorch.org/docs/stable/quantization.html" rel="nofollow noopener">quantization</a>?</p>

username: jerryzh168
cooked: <p><a class="mention" href="/u/dskhudia">@dskhudia</a> does fbgemm support 1D convolution?</p>

username: dskhudia
cooked: <p>1D can go though 2D path but we need some work here to make it work.</p>

username: dskhudia
cooked: <p><a href="https://github.com/pytorch/pytorch/pull/35093" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/35093</a> adds 1d qconvs.</p>
Add cmake in two different libs, cmake error

username: Jesper.Lindberg
cooked: <p>I want to add libtorch to two existing projects. So I add:</p>
<blockquote>
<p>find_package(torch REQUIRED)</p>
</blockquote>
<p>in both cmake files, which gives me the following error:</p>
<blockquote>
<p>add_library cannot create target “torch_library” because another target<br>
with the same name already exists.  The existing target is an interface<br>
library created in source directory</p>
</blockquote>
<p>How do I correctly add it to both projects?</p>

username: tom
cooked: <p>So the above hints at CMake seeing both <code>find_package</code> statements in one invocation.<br>
If it’s in the same CMake project, I think you can use <code>find_package</code> once and then use the torch “environment” set up by CMake for both.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Jesper.Lindberg
cooked: <p>Tried this, however then torch/torch.h is not found in the other project <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>So I can have it at any one of them at each time without any problems. However, I get issues when I try to have it in both projects.</p>

username: tom
cooked: <p>So you’re trying to make them work separately as well as “in combination”?<br>
One thing you could try is to make the <code>find_package</code> conditional on <code>if (NOT TARGET torch_library)</code> or somesuch to have it executed exactly once.<br>
If that doesn’t solve it, maybe it would help if you showed a bit about your CMake setup.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Jesper.Lindberg
cooked: <p>Thanks you, this can be marked as sloved.<br>
Tried your suggestion but it did not work, this is a huge cmake-project with about 800 cpp different projects. However, I managed to get it working in the end.</p>

username: sinafz
cooked: <p><a class="mention" href="/u/jesper.lindberg">@Jesper.Lindberg</a> we’re running into the same issue. How did you get it t work?</p>

username: Jesper.Lindberg
cooked: <p>Sorry for late reply but I won’t be able to help you with this issue as I did not solve it myself but rather got help from a colleague who specializes in cmake builds.</p>

username: sinafz
cooked: <p>Does your colleague have a pytorch discuss account? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> Or maybe a github ID?</p>

username: Jesper.Lindberg
cooked: <p>Sorry, I am afraid not. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Share multiprocessing Manager.list among many instances?

username: apytorch
cooked: <p>Running on 4+ gpus on the same machine, and I’m trying to save time and memory by just caching all of the Dataset items once and sharing that list among all of the Dataloader workers for all of the currently training models on the same machine.</p>
<p>I’ve been able to get a Manager.list that is shared among all of the Dataloader workers for one model just fine. However, I’m not sure how to share a pointer to this list with the other models that are started later.</p>
How reduce_range affects model performance?

username: Reuven_Peretz
cooked: <p>Hi,<br>
I’m trying to do a post-training static quantization on mobilenetV2, as demonstrated in <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="nofollow noopener">this</a> tutorial.<br>
I wanted to check how different observers affects the preformance of the model, and I got this:</p>
<p>activation=MinMaxObserver, weight=MinMaxObserve(dtype=torch.qint8) -&gt; 67.27<br>
activation=MinMaxObserver, weight=MovingAverageMinMaxObserver(dtype=torch.qint8) -&gt; 67.21<br>
activation=MinMaxObserver, weight=HistogramObserver(dtype=torch.qint8) -&gt; 65.98<br>
activation=MinMaxObserver, weight=PerChannelMinMaxObserver(dtype=torch.qint8) -&gt; 42.87<br>
activation=MinMaxObserver, weight=MovingAveragePerChannelMinMaxObserver(dtype=torch.qint8) -&gt; 43.92</p>
<p>I noticed that the per-channel observers gives significantly worse accuracies, so I tried it using the reduce_range flag and got those results:<br>
activation=MinMaxObserver, weight=PerChannelMinMaxObserver(dtype=torch.qint8, , reduce_range=True) -&gt; 69.19<br>
activation=MinMaxObserver, weight=MovingAveragePerChannelMinMaxObserver(dtype=torch.qint8, reduce_range=True) -&gt; 68.93</p>
<p>I saw in the comments that “reduce_range reduces the range of the quantized data type by 1 bit. This is sometimes required to avoid instruction overflow”.<br>
How can I tell if this is the case I’m issuing?<br>
Is there some “thumb rule” for when reducing the range by 1 bit might help?</p>
<p>Thanks!</p>