akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Log_prob() not returning any gradients, even though it should

username: Lugi
cooked: <p>So I have the following code snippet:</p>
<pre><code class="lang-auto">import torch

loc = torch.tensor(0., requires_grad=True)
scale = torch.tensor(1., requires_grad=True)
gaussian_test = torch.distributions.Normal(loc, scale)
gaussian_y.log_prob(torch.tensor(0.)).backward()
print(loc.grad, scale.grad) # None, None
</code></pre>
<p>Since calculating a logPDF at a particular point consists of exclusively differentiable operations, then I would expect to be able to get a gradient of a <code>log_prob</code> operation with respect to the distribution parameters. However this is not the case. Why would that happen?</p>

username: KFrank
cooked: <p>Hi Lugi!</p>
<aside class="quote no-group" data-username="Lugi" data-post="1" data-topic="158359" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lugi/48/15014_2.png" class="avatar"> Lugi:</div>
<blockquote>
<pre><code class="lang-auto">gaussian_test = torch.distributions.Normal(loc, scale)
gaussian_y.log_prob(torch.tensor(0.)).backward()
</code></pre>
</blockquote>
</aside>
<p><code>gaussian_y</code> is not defined within the code you posted.  Did you mean:</p>
<pre data-code-wrap="python"><code class="lang-python">gaussian_test.log_prob(torch.tensor(0.)).backward()
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>I would expect to be able to get a gradient of a <code>log_prob</code> operation with respect to the distribution parameters.</p>
</blockquote>
</aside>
<p>Your reasoning is correct and I believe that you will get a gradient if you<br>
correct the line noted above that appears to be a typo.</p>
<p>Best.</p>
<p>K. Frank</p>
Applying strict=False for using a modified pretrained model

username: Tm4
cooked: <p>I want to use a pretrained model as the encoder part in my model. You can find a version of my model:</p>
<pre><code class="lang-auto">class MyClass(nn.Module):
    def __init__(self, pretrained=False):
        super(MyClass, self).__init__()
        self.encoder=S3D_featureExtractor_multi_output()
        if pretrained:

            weight_dict=torch.load(os.path.join('models','weights.pt'))

            model_dict=self.encoder.state_dict()
            
            list_weight_dict=list(weight_dict.items())
            list_model_dict=list(model_dict.items())
            
            for i in range(len(list_model_dict)):
                assert list_model_dict[i][1].shape==list_weight_dict[i][1].shape
                model_dict[list_model_dict[i][0]].copy_(weight_dict[list_weight_dict[i][0]])
            
            for i in range(len(list_model_dict)):
                assert torch.all(torch.eq(model_dict[list_model_dict[i][0]],weight_dict[list_weight_dict[i][0]].to('cpu')))
            print('Loading finished!')
                    
    def forward(self, x):
        
        a, b = self.encoder(x)
        return a, b
</code></pre>
<p>Because I modified some parts of the code of this pretrained model, based on <a href="https://discuss.pytorch.org/t/how-to-delete-some-layers-in-pretrained-model/122607/2">this post</a> I need to apply <code>strict=False</code> but based on the scenario that I load the pretrained weights, I cannot find a place in the code to apply <code>strict=False</code>. How can I apply that or how can I change the scenario of loading the pretrained model taht makes it possible to apply <code>strict=False</code>?</p>

username: JuanFMontesinos
cooked: <p>it seems whoever that coder your snippet is doing a manual weight loading.</p>
<p>You should be able to do:</p>
<pre><code class="lang-auto">model_dict=self.encoder.load_state_dict(weight_dict)
</code></pre>
<p>as shown in <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-state-dict-recommended" class="inline-onebox" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 1.12.0+cu102 documentation</a><br>
and that function accepts the strict=False</p>
Splitting datapipe into train/eval

username: Lugi
cooked: <p>Hi, what is the most canonical way to perform a train/test split on a datapipe? I start off with a <code>FileLister</code>, and then would like to split it into 2 subsequent datapipes - one for train data and one for eval data. I don’t see any documentation related to that, and would like to do this in the most straightforward way possible.</p>

username: nivek
cooked: <p>You can use the <a href="https://pytorch.org/data/beta/generated/torchdata.datapipes.iter.Demultiplexer.html#torchdata.datapipes.iter.Demultiplexer" rel="noopener nofollow ugc">Demultiplexer</a> by doing something like this:</p>
<pre><code class="lang-python">train_dp, eval_dp = dp.demux(num_instances=2, classifier_fn=rand_fn)
</code></pre>
<p>where <code>rand_fn</code> assigns a value 0 or 1 to each sample, which would determines the sample belongs to train or eval.</p>
<p>Let me know if this works for you.</p>

username: Lugi
cooked: <p>I have seen this proposition already, however the trouble I have with it is following: since I’m using the iterable datapipeline I do not know up front which sample should go to which set. The solution I guess would be to preload all the filepaths, and then split them to train/val, and then let the classifier function assign them to one or the other based on that.</p>
<p>I guess this would be the way to do this right now, since there is no straightforward implementation of a datapipeline component that does just that (a stateful splitter which would remember which samples were in which set, so the split is the same in every epoch).</p>

username: nivek
cooked: <p>If you pass in a classifier function that is random but provided with a specific seed you should get what you are describing:</p>
<blockquote>
<p>a stateful splitter which would remember which samples were in which set, so the split is the same in every epoch</p>
</blockquote>
<p>Does that make sense?</p>

username: Lugi
cooked: <p>But wouldn’t the “inside randomness” of the function change between epochs? Would I have to reinitialize the pipeline with the same seed inbetween epochs?</p>

username: nivek
cooked: <aside class="quote no-group" data-username="Lugi" data-post="5" data-topic="158160">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lugi/48/15014_2.png" class="avatar"> Lugi:</div>
<blockquote>
<p>inbetween</p>
</blockquote>
</aside>
<p>Yep, your classifier function will use a RNG and you will have to reset the seed of the RNG after each epoch (or at the beginning).</p>
<p>I can see how this can be a bad user experience and we should build a DataPipe/function to handle this. I will track this within a GitHub issue.</p>
<p>Thanks for bringing this issue up! We appreciate your feedback!</p>
Loading an HDF5 dataset with DataLoader

username: aizyuval
cooked: <p>Hi,<br>
I have two HDF5 datasets that has cat images and non cat images (64x64x3 [x209 train, x50 test]) for training and testing.<br>
Each with a list of classes (0 for non cat, 1 for cat), a train_set_x → the images, and a train_set_y → the labels for the images.</p>
<p>I know I need to make a custom dataset with <strong>init</strong>, <strong>getitem</strong>, <strong>len</strong>, but what should be the value of those? and what should be the corresponding code to be able to load it?</p>
<p>I just want to load the data and be able to enumerate over it, shuffle it, transform it and such, just like we do with the MNIST dataset from pytorch datasets.</p>
<p>Im really lost! Thank you in advance!<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1391b8d330b312e53eeec7d07e560057371a665.png" alt="bbba" data-base62-sha1="tQS8UVGmsZvGYpLVzmDywCaMHyt" width="575" height="457"></p>

username: nivek
cooked: <aside class="quote no-group" data-username="aizyuval" data-post="1" data-topic="158271">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aizyuval/48/53728_2.png" class="avatar"> aizyuval:</div>
<blockquote>
<p>HDF5 dataset</p>
</blockquote>
</aside>
<p>One possible options is to have <code>__init__</code> load both <code>train_set_x</code> and <code>train_set_y</code> into memory, <code>__getitem__</code> can accept an index and return a tuple of image and label, then <code>__len__</code> would be the size of the training data set.</p>
<p>You can consider other implementation (e.g. lazy loading or iter-style) if the dataset doesn’t fit into memory.</p>
<p>You may find our <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html" rel="noopener nofollow ugc">tutorial</a> helpful as well.</p>
Add target attribute to torchvision.datasets.inaturalist

username: Torcione
cooked: <p>Hi everyone,<br>
I have a code implemented for CIFAR10/CIFAR100 built-in datasets.<br>
I want now to extend it for INaturalist dataset, which is also a pytorch built-in dataset.<br>
The latter however misses the attribute <code>dataset.targets</code><br>
As I want to adapt the rest of the code to the new dataset option, is it possible to add the <code>targets</code> attribute to the INaturalist dataset before defining the dataloader?</p>
About size of gradient update

username: Haeju_Lee
cooked: <p>Hi!<br>
I’m going to build a model like the picture below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a6cc326a869b8a37c89a040c1aeeb1255728774.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9a6cc326a869b8a37c89a040c1aeeb1255728774" title="File (2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a6cc326a869b8a37c89a040c1aeeb1255728774_2_690x227.jpeg" alt="File (2)" data-base62-sha1="m26yggZ09wv3V1nZd33miDmLx5O" width="690" height="227" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a6cc326a869b8a37c89a040c1aeeb1255728774_2_690x227.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a6cc326a869b8a37c89a040c1aeeb1255728774_2_1035x340.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a6cc326a869b8a37c89a040c1aeeb1255728774_2_1380x454.jpeg 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">File (2)</span><span class="informations">1563×516 38.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
where a vision transformer(ViT) model’s outputs for different pictures are fed into GPT model.</p>
<p>I’m worried about the size of gradient update, and also the training time for this model.</p>
<p>Let’s say that 10 outputs of the ViT are fed into the GPT,<br>
then does this mean that it requires 10 x (size of gradient update of ViT) size GPU memory for training? Or gradient updates are sequentially aggregated on-the-fly, so maybe it requires only 2 x (size of gradient update of ViT)?</p>
<p>Also I guess training this model would require significant amount of time, being very slow…<br>
but I’m not sure whether I’m right or not.</p>
<p>Any guess would be appreciated.<br>
Thanks!</p>
RuntimeError: DataLoader worker is killed by signal: Segmentation fault

username: Sfitsos
cooked: <p>I am going through my dataset using the data loader and I get the following error:</p>
<p>ERROR: Unexpected segmentation fault encountered in worker.<br>
Traceback (most recent call last):<br>
File “/home/kvougiou/miniconda3/envs/dev/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 986, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/home/kvougiou/miniconda3/envs/dev/lib/python3.8/multiprocessing/queues.py”, line 107, in get<br>
if not self._poll(timeout):<br>
File “/home/kvougiou/miniconda3/envs/dev/lib/python3.8/multiprocessing/connection.py”, line 257, in poll<br>
return self._poll(timeout)<br>
File “/home/kvougiou/miniconda3/envs/dev/lib/python3.8/multiprocessing/connection.py”, line 424, in _poll<br>
r = wait([self], timeout)<br>
File “/home/kvougiou/miniconda3/envs/dev/lib/python3.8/multiprocessing/connection.py”, line 931, in wait<br>
ready = selector.select(timeout)<br>
File “/home/kvougiou/miniconda3/envs/dev/lib/python3.8/selectors.py”, line 415, in select<br>
fd_event_list = self._selector.poll(timeout)<br>
File “/home/kvougiou/miniconda3/envs/dev/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 89199) is killed by signal: Segmentation fault.</p>
<p>The dataset that I am using is massive 1.6M videos but I can never seem to do even a single pass of my data before I see this crash. In the dataset, I am reading videos (using decord), reading audio (using torchaudio which shouldn’t be the problem since I have used this before on similar data and it works) and loading numpy arrays.</p>
<p>Specs of my machine:<br>
128 GB Ram<br>
CPU: AMD Ryzen Threadripper 1950X 16 Core CPU</p>
<p>The versions of the libraries I am using are:<br>
torch                  1.8.1<br>
torchaudio             0.8.0a0+e4e171a<br>
torchmetrics           0.3.2<br>
torchvision            0.9.1</p>
<p>My shared memory is:</p>
<p>kernel.shm_rmid_forced = 0<br>
kernel.shmall = 18446744073692774399<br>
kernel.shmmax = 18446744073692774399<br>
kernel.shmmni = 4096</p>
<p>Other info:<br>
I am using 10 workers in the dataloader</p>
<p>I have now looked at similar issues and I have no idea why this happens. It has happened even when setting workers=0 (this was before some changes I tried but I expect it to happen again). Also when I run the exact same code on a server with 256 RAM and way more cores it works so this is specific to this one machine. Does anyone have any idea on how I can debug this further because I am now stomped?</p>
<p><strong>Update:</strong><br>
I noticed that if I increase the number of workers then the issue happens faster. If I increase to 32 (same as the cores) then it sometimes happens instantly. Then my computer goes to a state that it will keep segfaulting instantly on the code until I reduce the workers.</p>

username: czh
cooked: <p>Did you solve the issue? I think I am experiencing the same thing or something very similar. Any help would be appreciated.</p>
Multi threading slow down

username: Amelia
cooked: <p>Hi, when I tried to run 2 shallow Neural Network in 2 threads within one process, both get the 2x slow down(10s) than running standalone(5s).  The memory is enough, CPU utilization is not saturated when running 2 jobs in parallel and the data they used are dummy data initialized in their own thread. Under same environment, I run them in 2 process, same behavior What was blocking those jobs? The pseudocode is as follows. Thanks</p>
<p>def train():<br>
//x = …some random data<br>
model = nn.Sequential(OrderedDict([<br>
(“hidden_layer_1”, nn.Linear(inFeatures, hiddenDim)),<br>
(“activation_1”, nn.ReLU()),<br>
(“hidden_layer_2”, nn.Linear(hiddenDim, hiddenDim)),<br>
(“activation_2”, nn.ReLU()),<br>
(“output_layer”, nn.Linear(hiddenDim, nbClasses))<br>
]))<br>
optimizer = torch.optim.RMSprop(model.parameters(), lr=0.001)<br>
criterion = nn.MSELoss()</p>
<pre><code>start_time = time.time()
for epoch in range(epoch_size):
    predicted = model(normed_train_data)
    loss = criterion(predicted, train_target)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
end_time = time.time()
print(end_time-start_time)
</code></pre>
<p>threadA = Thread(target=train, args=())<br>
threadB = Thread(target=train, args=())<br>
threadA.start()<br>
threadB.start()<br>
threadA.join()<br>
threadB.join()`</p>

username: sudomaze
cooked: <p>Could you provide more code?</p>

username: Amelia
cooked: <p>updated…That’s basically the most of code. There are some preprocessing of data (x) using pandas DataFrame but I don’t think that’s a problem since the timing starts after. Thanks!</p>
TypeError: can't convert cuda:0 device type tensor to numpy.,am getting this error how to solve ? Use Tensor.cpu() to copy the tensor to host memory first

username: uzumaki_N
cooked: <p>frame_str = base64.b64decode(input_data[“frame1”])<br>
frame = torch.Tensor(resize(jpeg.decode(frame_str),(640,640))).permute(2,0,1)<br>
frame = frame<br>
predictions = model.predict(frame)<br>
boxes = numpy.asarray(predictions[0][‘boxes’])<br>
print(“Converted Predictions to numpy format - %s” %(type(boxes)))<br>
print(“Predictions %s” %(predictions))</p>
<pre><code>    if isinstance(boxes, list):
        boxes = np.array(boxes)
        boxes = boxes.astype("int")
</code></pre>
<p>TypeError: can’t convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.<br>
i’m getting this error ,what am i doing wrong?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/uzumaki_N">@uzumaki_N</a>, can you share the exact line where this error is shown? It’ll be the last line with the stack trace.</p>
<p>The error is because you have a Tensor which is currently stored on your GPU but you’re trying to use it with numpy which is a CPU only library. Therefore, before sending this Tensor to numpy you need to first send it to your CPU, then call whatever numpy operation you want.</p>
<p>You can send the Tensor to the CPU via the <code>.cpu()</code> method,</p>
<pre><code class="lang-auto">x = torch.randn(4,4,device='cuda') #currently on GPU
x = x.cpu() #moved back to CPU
</code></pre>
How do I choose the learning rate for SGD?

username: BobKim
cooked: <p>It usually uses 0.001 when learning a network using Adam, but the appropriate learning rate for SGD is all different (even set the learning rate to 30 !! when learning LSTM for PTB dataset, Language modeling task)</p>
<p>If so, is there a specific criterion or tendency to set the learning rate of SGD?<br>
(For example, it depends on the type of networks or tasks.)<br>
Or should I rely entirely on the user’s experience?</p>
<p>Im very confused…</p>

username: JuanFMontesinos
cooked: <p>It’s user experience and depends on fields and problems to solve. There are some methods that suggest a lr. In my experience SGD’s is higher than adam’s</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1506.01186">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.4/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/1506.01186" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/1506.01186" target="_blank" rel="noopener nofollow ugc">Cyclical Learning Rates for Training Neural Networks</a></h3>

  <p>It is known that the learning rate is the most important hyper-parameter to
tune for training deep neural networks. This paper describes a new method for
setting the learning rate, named cyclical learning rates, which practically
eliminates the need...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: BobKim
cooked: <p>Thank you so much<br>
I’ll read this paper<br>
It will probably help me a lot<br>
Have a nice day!</p>
[Error] Pytorch Multiprocessing

username: arya_singh1
cooked: <pre><code class="lang-auto">assert group is none, 'group argument must be none'
</code></pre>
<p>Error occurs when using multiprocessing pool for training multiple client models in parallel. I don’t know much about this error and couldn’t find any useful information on the same anywhere. Any sort of help is highly appreciated in this regard.</p>

username: sudomaze
cooked: <p>Can you give code and more context?</p>
How to initialize weights with one line

username: robinho
cooked: <p>With mynet =  nn.Sequential(nn.Linear(1,10), nn.Sigmoid()), how to initialize weights <strong>with one line</strong> as ones or zeros instead of as the default random weights?</p>

username: sudomaze
cooked: <pre><code class="lang-python">def init_weights(m):
    if isinstance(m, nn.Linear):
        m.weight.data.fill_(1)
        m.bias.data.fill_(0)

net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))
net.apply(init_weights)
</code></pre>
<p>For more information, <a href="https://stackoverflow.com/a/49433937/5862635" rel="noopener nofollow ugc">check this Stackoverflow answer</a></p>
Optimizer problem when copying a model

username: kharrat_salma
cooked: <p>Hello, I have to train <em>n</em> clients where each client has its own model</p>
<p>so I instantiated a model <code>model=Net()</code> and then created <em>n</em> copies:</p>
<p><code>models.append(copy.deepcopy(model))</code></p>
<p>after every batch I aggregate all weights and do the following:<br>
<code>models[i].load_state_dict(average_weights)</code></p>
<p>I noticed that there is no learning, and the loss gets stuck.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f56a91f97ddab75e6f01105ea6167fe713660085.png" data-download-href="https://discuss.pytorch.org/uploads/default/f56a91f97ddab75e6f01105ea6167fe713660085" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f56a91f97ddab75e6f01105ea6167fe713660085.png" alt="image" data-base62-sha1="z13iXPQXIvAXfuVYPGygkzauOCV" width="690" height="252" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">854×312 6.79 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any suggestion for this problem?</p>

username: sudomaze
cooked: <p>Why do you have multiple models and then average their weights?</p>
<p>Each model might learn something different, and taking the average of the weights, especially during training, won’t result in a better estimation to solve the problem at hand.</p>
Loading weights from pretrained model with different module names

username: Nicholas_Wickman
cooked: <p>I would like to initialize my model with the first 14 layers of VGG16. A nice solution suggested at <a href="https://discuss.pytorch.org/t/how-to-load-part-of-pre-trained-model/1113/3?u=nicholas_wickman">https://discuss.pytorch.org/t/how-to-load-part-of-pre-trained-model/1113/3?u=nicholas_wickman</a> does not work for me, because my parameters are not named the same as the model I would like to load from.</p>
<p>My parameters are named like <code>conv1.weight</code> and <code>conv1.bias</code>. VGG’s layers are named like <code>features.0.weight</code> and <code>features.0.bias</code>.</p>
<p>Does anyone have any suggestions?</p>
<pre><code>pretrained_dict = torch.load('VGG_dict.pth')
model_dict = model.state_dict()

# Modification to the dictionary will go here?

model_dict.update(pretrained_dict)
model.load_state_dict(model_dict)</code></pre>

username: smth
cooked: <p>you could change the key names of the state dict to match your layer’s key names…</p>

username: vsag
cooked: <p><a class="mention" href="/u/smth">@smth</a> shouldn’t this work?</p>
<pre><code class="lang-auto">pre_trained_model=torch.load("Path to the .pth file")
 new=list(pre_trained.items())

my_model_kvpair=mymodel.state_dict()
count=0
for key,value in my_model_kvpair.item():
  layer_name,weights=new[count]      
mymodel_kvpair[key]=weights
count+=1
</code></pre>

username: lijunzh
cooked: <p>Thanks a lot. This is exactly what I am looking for.</p>
<p>In addition, do you know a good way to transfer the optimizer state to the new model with different names? The reason I am asking here is that the pre-trained model I have is constructed via <code>nn.Sequential</code>, but I want to insert some new blocks in between and retrain the new network. It will be easier for me if I can start with what they have.</p>

username: vsag
cooked: <p>It’s been a While since I have used pytorch, I have no idea how to do that. But you can find it easily as it was asked before iirc.</p>

username: lijunzh
cooked: <p>Thanks for the prompt reply. I will search around again.</p>

username: Shijie_Wu
cooked: <p>For me on a nested nn, this code piece did not work. Though the dictionary is updated as expected, the weights and bias actually didn’t change. Do you have other recommendations?</p>

username: Ayush_Gupta
cooked: <p>Pretty old thread, but I was having the same issue and have solved it. Here is what you need to do:</p>
<pre><code class="lang-auto"> new=list(pre_trained.items())

my_model_kvpair=my_model.state_dict()
count=0
for key,value in my_model_kvpair.item():
  layer_name,weights=new[count]      
  mymodel_kvpair[key]=weights
  count+=1

my_model.load_state_dict(my_model_kv_pair)
</code></pre>
<p>After modifying the key value pairs, you need to actually load the updated state_dict back into your model so that the weights and biases change</p>

username: naveenmarthala
cooked: <p>I would modify it to:</p>
<pre><code class="lang-auto">    mymodel_kvpair[key] = weights.detach().clone()
</code></pre>
How can I apply a single transform?

username: David1
cooked: <p>I’m new to PyTorch, and I am just trying to use the pretrained Mask R-CNN model to segment a single image. To do this, I’m reading the image using <code>img=PIL.Image.open("apple.jpg")</code>. Now I simply need to transform the image to a tensor to pass to the model, but how can I do this in the simplest way possible? I can use <code>torchvision.transforms.transforms.Compose([transforms.ToTensor()])(img)</code>, but is it not possible to apply the <code>ToTensor</code> transform without using <code>Compose</code>?</p>

username: JuanFMontesinos
cooked: <p>You don’t even need to use pil.<br>
You can use any image reader you want imageio, opencv, scikit-image (all those returns numpy arrays)<br>
and just</p>
<pre><code class="lang-auto">img = load(...)
img = torch.from_numpy(img)/255.
</code></pre>
<p>and there you go<br>
or alternatevely you can do <code>transforms.ToTensor()(img)</code></p>
Illegal instruction (core dumped) because avx

username: Jesse_Stone
cooked: <p>both version happen</p>
<p>Illegal instruction (core dumped)</p>
<p>pip3 install torch torchvision torchaudio --extra-index-url <a href="https://download.pytorch.org/whl/cu116" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu116</a></p>
<pre><code class="lang-auto">pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113
</code></pre>
<p>we need make a version auto check avx cpu</p>
Save and further calculate the test accuracy result

username: shangguan91
cooked: <p>Dear all,<br>
After running the model, how could i save the result and then calculate the best accuracy on certain epoch and average accuracy in python?  is there some way to load the result into excel as well?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/031b0998ddafbf99e8cb9fd1fbf7fbefcf67eb75.png" data-download-href="https://discuss.pytorch.org/uploads/default/031b0998ddafbf99e8cb9fd1fbf7fbefcf67eb75" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/031b0998ddafbf99e8cb9fd1fbf7fbefcf67eb75.png" alt="image" data-base62-sha1="rtmjHu8JcsMxibnGoZHuoYXYvr" width="618" height="500" data-dominant-color="DBDBDB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">802×648 15 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Depending how the results should be stored you could:</p>
<ul>
<li>store the print output in a text file using plain <code>open()</code> calls and append the strings,</li>
<li>store the tensors via <code>torch.save()</code>
</li>
<li>transform the tensors to numpy arrays via <code>tensor.numpy()</code> and either store it using a numpy method or create a <code>pd.DataFrame</code> using <code>pandas</code>, which would also allow you to create Excel sheets and other formats.</li>
</ul>

username: Berny
cooked: <p>Hello. I also need to save some parameters like accuracy. I used this method:</p>
<pre><code>torch.save(accuracy, 'accuracy.pt')
</code></pre>
<p>but it seems to require tools such as Panther with license to open .pt files. Is there another way to open this kind of file? Thanks for your help.</p>

username: ptrblck
cooked: <p>I’m not familiar with the Python package “Panther”, but you can use PyTorch to open the file and you can also use any file extension you like.<br>
The <code>.pt</code> extension is just a commonly used one, but not a requirement:</p>
<pre><code class="lang-python">x = torch.randn(10)
torch.save(x, "x.ptrblck")
y = torch.load("x.ptrblck")
print((x == y).all())
# tensor(True)
</code></pre>

username: Berny
cooked: <p>I added this lines:</p>
<pre><code>torch.save(accuracy, "accuracy.txt")
                    y = torch.load("accuracy.txt")
                    print((accuracy == y))
</code></pre>
<p>but when I open the txt file, the content is not a number but something incomprehensible.</p>
Adam: how to output 1st and 2nd moments m and v

username: robinho
cooked: <p>adam: how to output such intermediate values involved as 1st and 2nd moments m and v during training?</p>

username: ptrblck
cooked: <p>You can check the <code>optmizer.param_goups</code> or <code>optimizer.state_dict()</code>.</p>

username: robinho
cooked: <p>thank you but all I can see are the trained parameters etc, not the moments</p>

username: ptrblck
cooked: <p>I can see the entire state including the running averages:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

out = model(torch.randn(1, 1))
out.backward()
optimizer.step()

print(optimizer.state_dict())
# {'state': {0: {'step': tensor(1.), 'exp_avg': tensor([[-0.0219]]), 'exp_avg_sq': tensor([[4.7941e-05]])}, 1: {'step': tensor(1.), 'exp_avg': tensor([0.1000]), 'exp_avg_sq': tensor([0.0010])}}, 'param_groups': [{'lr': 0.001, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'maximize': False, 'foreach': None, 'capturable': False, 'params': [0, 1]}]}
</code></pre>

username: robinho
cooked: <p>thank you I saw the same output, but what is exp_avg and which are the moments?</p>

username: robinho
cooked: <p>could anyone help with my last question pls thanks</p>

username: IneedMrmeeseeks
cooked: <p>When I read <a href="https://arxiv.org/pdf/1412.6980.pdf" rel="noopener nofollow ugc">this paper</a> and <a href="https://pytorch.org/docs/stable/_modules/torch/optim/adam.html#Adam" rel="noopener nofollow ugc">source code</a>,<br>
exp_avg is the exponential moving average of gradient value and same as 1st moment vector<br>
exp_avg_sq is the exponential moving average of the sqared gradient and same as 2nd moment vector.<br>
betas are exponential decay rates.</p>
<p>Have a nice day.</p>
How to catch exception for single elements of a minibatch in forward function of a model

username: DanielMa
cooked: <p>I’m trying to train a deep model using PyTorch. One of the layers is a differentiable optimization layer which I implemented using cvxpylayers (<a href="https://github.com/cvxgrp/cvxpylayers" class="inline-onebox" rel="noopener nofollow ugc">GitHub - cvxgrp/cvxpylayers: Differentiable convex optimization layers</a>). Depending on the input to the optimization layer, the optimization might be infeasible (this is the wanted behavior for some inputs). In case of infeasibility the optimization layer will raise an exception. In that case I would like to circumvent the optimization layer using a different layer that will not return an exception. I have implemented this using a try/except block in the forward function of the model.</p>
<p>My problem is, that when training with minibatches, an exception in the optimization layer for any one training example will lead to the optimization layer being circumvented for all training examples in the minibatch.</p>
<p>Is there any way to basically do an element-wise try/except inside the forward function of a PyTorch model that allows for batch-mode training in my situation?</p>
<p>Thanks a lot for your help!</p>
Based on the exact same code, random seed and data, ran on different servers but got different results

username: Tnak
cooked: <p>Hello, community<br>
I am testing the reproducibility of the results of the code on different server devices. Both servers are trained with a single card, one card is 2080Ti and one card is 3080, and the operating systems are Ubuntu 18.04 and Ubuntu 20.04 respectively. Both servers use the same conda virtual environment, i.e. the same pytorch, numpy, python versions. Also using the same code, the same random seed, etc. But strangely, I got different results, the estimated accuracy of the models trained on the two servers differed by almost 2%. This is confusing me, hope someone can help me.</p>

username: ptrblck
cooked: <p>Deterministic results via seeding etc. are not guaranteed between different hardware/software setups.<br>
Depending on the variance seen in the final accuracy using different seeds, the difference of 2% might be expected.</p>

username: Tnak
cooked: <p>Sincerely thank you for your reply, it has benefited me a lot.</p>
How do i reimplement the following logic in torch 1.7+?

username: Kenny_Mccormic
cooked: <pre><code class="lang-auto">        # loss1, loss2, loss3 are depend on both modelA and modelB
        # first we want to use loss1, loss2 to update modelA
        # then use loss2, loss3 to update modelB
        modelB.zero_grad()
        modelA.zero_grad()
        lossA = loss1 + loss2
        lossA.backward(retain_graph=True)
        modelA_optimizer.step()

        modelB.zero_grad()
        lossB = loss2 + loss3
        lossB.backward()
        modelB_optimizer.step()
</code></pre>
<p>Apparently the code above works for torch1.0.0; but fails in torch1.7.0.</p>

username: ptrblck
cooked: <p>The logic is flawed and PyTorch now properly raises an error, since you are using stale forward activations in the second <code>backward</code> call as described <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">here</a>.</p>

username: Kenny_Mccormic
cooked: <p>Sorry for the late reply, i wonder if something like this works:</p>
<pre><code class="lang-auto">gradA, gradB = {}, {}

lossA.backward(retain_graph=True)

for name, param in modelA.named_parameters():
    gradA[name] = param.grad.clone().detach()
for name, param in modelB.named_parameters():
    gradB[name] = param.grad.clone().detach()

lossB.backward()
# restore modelA's grad
for name, param in modelA.named_parameters():
    param.grad = gradA[name]
for name, param in modelB.named_parameters():
    param.grad -= gradB[name]
# update model afterwards
    
</code></pre>

username: ptrblck
cooked: <p>I don’t think it would help, as the intermediate forward activations are stale compared to the already updated parameters as explained in the linked post.</p>
How to add gradient noise?

username: BobKim
cooked: <p>Hello! everyone!</p>
<p>my pytorch version is 1.10.2.</p>
<p>I want to add noise to the gradient of Adam optimizer using torch.normal.</p>
<p>But grad tensor is variable. so how can i add the normal distribution noise to gradient?</p>

username: ptrblck
cooked: <p>You can directly add the noise to the gradient via:</p>
<pre><code class="lang-python">loss.backward()
model.layer.weight.grad = model.layer.weight.grad + torch.randn_like(model.layer.weight.grad)
optimizer.step()
</code></pre>

username: BobKim
cooked: <p>Thanks!</p>
<p>then, couldn’t add the normal distribution noise with mean and standard deviation?</p>

username: ptrblck
cooked: <p>You can add the <code>mean</code> and scale with the <code>stddev</code> as:</p>
<pre><code class="lang-python">... + (torch.randn_like(model.layer.weight.grad) * stddev + mean)
</code></pre>

username: BobKim
cooked: <p>Thanks!! i’ll try this!</p>
Random split alternative

username: Anonymous3
cooked: <p>I do not want to split data into train, test and validation randomly but split them sequentially. What can I do?</p>
<p>Please help</p>

username: ptrblck
cooked: <p>I’m not sure I understand the issue correctly, but a “sequential” split could just be done via indexing/slicing:</p>
<pre><code class="lang-python">data = torch.arange(100)
train, val, test = data[:80], data[80:90], data[90:]
</code></pre>
A bug of cross entropy

username: Limingxing00
cooked: <p>Hi,</p>
<p>This is very strange, I calculate the cross-entropy loss using the perfectly correct prediction results and GT. as follows:</p>
<p>pred = tensor([[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.],<br>
[1., 0.]], device=‘cuda:0’, grad_fn=)</p>
<p>label =tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 , 0,<br>
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,<br>
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,<br>
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])</p>
<p>then use<br>
F.cross_entropy(pred, finial_label.to(parent_embedding.device))</p>
<p>The result of the output is<br>
tensor(0.3133, device=‘cuda:0’, grad_fn=NllLossBackward)</p>
<p>Why does it appear?</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects logits as the model output, not probabilities, so your <code>pred</code> tensor is not representing a “perfect” prediction.<br>
Using high/low logits would work:</p>
<pre><code class="lang-python">pred = torch.tensor([10000., -10000.])
pred = pred.unsqueeze(0).expand(10, -1)

label = torch.zeros(10).long()


criterion = nn.CrossEntropyLoss()
loss = criterion(pred, label)
print(loss)
# tensor(0.)
</code></pre>

username: Limingxing00
cooked: <p>Sorry, this is my mistake, not a pytorch bug. thank you very much!</p>
Get Authorization to Analyze Discussion Forum Data

username: Mohammad_Abdul_Hadi
cooked: <p>We are performing some exploratory analysis on the crowd-sourced Software Engineering related Q&amp;A Platforms, such as Stack Overflow (SO) questions and answers or GitHub (GH) issues. PyTorch is another Q&amp;A platform that has been gaining popularity in the research arena almost exponentially over the past few years.<br>
Having said that, can we collect the public posts available on the PyTorch Discussion Forum and perform topic modeling analysis? May we make that dataset and analysis results publicly available? There is a lot of analysis like this available for GitHub and SO, but not for PyTorch.<br>
I am reaching out to the admins (<a class="mention" href="/u/ptrblck">@ptrblck</a>, <a class="mention" href="/u/smth">@smth</a> ) as in the <em>Terms of Service</em> of the forum there was a mention about “<em>no unauthorized automated means to compile information</em>”.</p>
<p>What is the process for obtaining this authorization?<br>
If we do not publish the dataset, are we authorized to publish the analysis?</p>
<p>Thanks in advance.</p>
Index out of range

username: StrafeNDestroy
cooked: <p>Hello im confused as why I am getting and index error even though when I check my output of my dataset class before i put it into a dataloader and it is of size [N]. I know that i must account for the background which should be index 0 so perhaps that may be the issue. But that leaves me with more questions because I have another model with the exact code that does not have a for loop to gather multply labels and it works. Soon as I add a for loop to the dataset class I get the error even though they are the exact same datasets with the same outputs.</p>
<p>MODEL Documentation<br>
During training, the model expects both the input tensors, as well as a targets (list of dictionary), containing:</p>
<blockquote>
<ul>
<li>boxes (<code>FloatTensor[N, 4]</code>): the ground-truth boxes in <code>[x1, y1, x2, y2]</code> format, with <code>0 &lt;= x1 &lt; x2 &lt;= W</code> and <code>0 &lt;= y1 &lt; y2 &lt;= H</code>.</li>
<li>labels (<code>Int64Tensor[N]</code>): the class label for each ground-truth box<br>
ERROR CODE</li>
</ul>
</blockquote>
<pre data-code-wrap="python"><code class="lang-python">IndexError                                Traceback (most recent call last)
&lt;ipython-input-31-fffd8dd57861&gt; in &lt;module&gt;()
     12         targets = [{k: v.to(device) for k, v in t.items()} for t in targets] # sending targets to the GPU
     13         bs = BATCH_SIZE
---&gt; 14         loss_dict = model(images, targets) # passing our model a single batch of images with repective targets
     15         totalLoss = sum(loss for loss in loss_dict.values()) # adds up all the losses from the models output
     16         lossValue = totalLoss.item() # Converts tensor loss to interger Loss

5 frames
/usr/local/lib/python3.7/dist-packages/torchvision/models/detection/roi_heads.py in assign_targets_to_proposals(self, proposals, gt_boxes, gt_labels)
    586                 clamped_matched_idxs_in_image = matched_idxs_in_image.clamp(min=0)
    587 
--&gt; 588                 labels_in_image = gt_labels_in_image[clamped_matched_idxs_in_image]
    589                 labels_in_image = labels_in_image.to(dtype=torch.int64)
    590 
</code></pre>
<p>GET LABEL in DATASET CLASS</p>
<pre data-code-wrap="python"><code class="lang-python">    label_list =["raccoon"]
    annotation_path = self.annotation_names[index]
    annotation_tree = ET.parse(annotation_path)
    label_name = annotation_tree.find("object").find("name").text
    
    if label_name in label_list:
      label = (label_list.index(label_name)+1) 
      label = torch.tensor([label],dtype=torch.int64)
</code></pre>
<p>Dataset Class output at index 9</p>
<pre data-code-wrap="python"><code class="lang-python">{'area': tensor(1006000),
  'boxes': tensor([[ 52,   7, 948, 999]]),
  'image_id': tensor(9),
  'iscrowd': tensor([0]),
  'labels': tensor([1])})
</code></pre>
Push a GPU tensor (output from network) into CPU but on preallocated memory

username: soulslicer
cooked: <p>Assuming i have a tensor output <code>a</code> from a network of some float [100, 100]. I then have some preallocated <code>a_cpu</code> float*. I could do a.cpu(), then memcpy the data to that pointer which would be double work. Is there a way i can do .cpu(), but let it target some preallocated memory?</p>
Roberta-base takes too long

username: Rafi
cooked: <p>Hello,<br>
I am fine-tuning roberta-base using <a href="https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification" rel="noopener nofollow ugc">this script</a> on ~8M example for classification. I am using NVIDIA RTX A6000 with 6 gpus. Max length is 64, batch_size=32 gradient_accumulation_steps=4. 1 epochs takes ~8 hours. Is it in the normal? I see that the original version was trained 1024 gpus</p>
Are the batch norm parameters updated every batch or every epoch?

username: jpcbertoldo
cooked: <p>Are the batch norm parameters updated every batch or every epoch?</p>

username: eqy
cooked: <p>They should be updated every batch.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; b = torch.nn.BatchNorm2d(3)
&gt;&gt;&gt; b.running_mean
tensor([0., 0., 0.])
&gt;&gt;&gt; x = b(torch.randn(8, 3, 224, 224))
&gt;&gt;&gt; b.running_mean
tensor([ 0.0002, -0.0002, -0.0002])
&gt;&gt;&gt; x2 = b(torch.randn(8, 3, 224, 224))
&gt;&gt;&gt; b.running_mean
tensor([-5.2687e-05, -4.1823e-04, -1.2951e-04])
&gt;&gt;&gt; 
</code></pre>
Most efficient way to use Column major when reshape a 1D array in PyTorch

username: miraboreasu
cooked: <pre><code class="lang-auto">import torch

    p = torch.arange(0, 12, requires_grad=False, dtype=torch.int32)
    pr = torch.reshape(p, (4, 3))

</code></pre>
<p>what I want is</p>
<pre><code class="lang-auto">pr = [0 4 8 
      1 5 9
      2 6 10
      3 7 11]
</code></pre>
<p>but it actually becomes</p>
<pre><code class="lang-auto">pr = [0 1 2 
      3 4 5
      6 7 8
      9 10 11]
</code></pre>
<p>I search online it said permute can do it, but it will make a copy of your array, what is the most efficient way to reshape it in PyTorch?</p>

username: eqy
cooked: <p>I am not sure permute will necessarily make a copy of the array:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; p = torch.arange(0, 12, requires_grad=False, dtype=torch.int32)
&gt;&gt;&gt; p.data_ptr()
94744753781760
&gt;&gt;&gt; p.is_contiguous()
True
&gt;&gt;&gt; prt = torch.reshape(p, (3, 4)).t()
&gt;&gt;&gt; prt
tensor([[ 0,  4,  8],
        [ 1,  5,  9],
        [ 2,  6, 10],
        [ 3,  7, 11]], dtype=torch.int32)
&gt;&gt;&gt; prt.data_ptr()
94744753781760
&gt;&gt;&gt; prt.is_contiguous()
False
&gt;&gt;&gt; 
</code></pre>
Is permute() lazy?

username: borisb
cooked: <p>I am using <code>permute()</code> and <code>reshape()</code> to pass data to a library/module which by default operates on the last two dimensions. For performance reasons I am wondering whether this will involve data movement?</p>
<p>Or does PyTorch rather keep track of how to index into the data and just change that?</p>

username: eqy
cooked: <p>I believe that by permute is lazy in that sense as it returns a <code>view</code> rather than a copied tensor. However, this also means that the result isn’t guaranteed to be contiguous:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.randn(2, 3, 224, 224)
&gt;&gt;&gt; a.data_ptr()
140712810848320
&gt;&gt;&gt; a.is_contiguous()
True
&gt;&gt;&gt; a = a.permute(0, 2, 3, 1)
&gt;&gt;&gt; a.data_ptr()
140712810848320
&gt;&gt;&gt; a.is_contiguous()
False
&gt;&gt;&gt; 

</code></pre>