akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to normalize uint16 depth image for training?

username: qiminchen
cooked: <p>The depth image rendered from the ScanNet dataset is in <code>uint16</code>, after dividing the depth by the shift (1000), the depth values are in the range of <code>[0, 1 &lt; some positive float &lt; 10]</code>, how to normalize the depth to <code>[0, 1]</code> (per dataset) for training?</p>

username: tom
cooked: <p>So the question is whether you want per-dataset or per-image normalization - I would imagine that you want global because depth is a physical quantity (but in medical imaging it is not uncommon to use per-image, in particular when working with images taken by different scanners). If you need global, you’d need to iterate over the dataset once during preprocessing and find the maximum.</p>
<p>Best regards</p>
<p>Thomas</p>

username: qiminchen
cooked: <p>I need the per-dataset normalization. By saying iterate over the dataset and find the maximum, do you mean over the training set? or including the test set?</p>

username: tom
cooked: <p>The rules are that you should use the training set.<br>
The next question then is what to do when the test set a larger depth somewhere. I’d probably just clamp it down to 1. In a good training set, I would expect the training set to max out the depth reported by the camera somewhere.</p>
<p>Best regards</p>
<p>Thomas</p>

username: qiminchen
cooked: <p>Thanks, Tom. So then I can normalize the depth by using <code>(depth - min_depth) / (max_depth - min_depth)</code> where <code>max_depth</code> is the maximum depth value over the whole training set, right?</p>

username: tom
cooked: <p>That’s what I’d do, though maybe you might just keep min_depth 0 or so.</p>

username: Lakshman
cooked: <p>I have a question regarding normalising depth images. How did you find the maximum depth value over the whole training set?</p>
Index select from tensors

username: Jaideep_Valani
cooked: <pre><code class="lang-auto">selection_randn=torch.randn(4,4,3,4)
curr_idx=torch.tensor([2,3,1,0])

selection_randn[:,curr_idx,:,:].shape
</code></pre>
<p>I want to select   2,3,1,0 value  from each row  on dim 1 .<br>
let me know how to do that.</p>

username: Jaideep_Valani
cooked: <p>Expect size is  4,3,4 but i get same size as input</p>

username: Jaideep_Valani
cooked: <p>I think this will work</p>
<pre><code class="lang-auto">
selection_randn[range(len(curr_idx)),curr_idx]
shape =4,3,4
</code></pre>
How does ADAPTIVEAVGPOOL2D increase spatial dimension

username: Pramesh_Gautam1
cooked: <p>I need to use ADAPTIVEAVGPOOL2D so that the spatial dimension across all batches be same. I understand the case where it is used to decrease the dimensions. The question I have is regarding the increase in dimension.</p>
<pre><code class="lang-auto">layer = nn.AdaptiveAvgPool2d((1,64))

input = torch.randn(1, 3, 64, 16) # B, C, H, W

output = layer(input)

print(output.shape)
torch.Size([1, 3, 1, 64])
</code></pre>
<p>As seen in the code, height is reduced to 1 but width is increased to 64. Can someone explain how pooling achieves this? Is padding used internally?</p>
Converting list to torch tensor

username: Aigerim_Bazarkhanova
cooked: <p>I am trying to convert list to torch tensor. However, it shows an error len() of unsized object.</p>
<p>torch.tensor(a, dtype=torch.float64)</p>
<p>len(a)  is 11.</p>
<p>Thank you!</p>

username: Matias_Vasquez
cooked: <p>Could you post how <code>a</code> is defined?</p>

username: Aigerim_Bazarkhanova
cooked: <p>Hello! I defined a  = cvqnn_layers_all(L, M), where cvqnn_layers_all(n_layers, n_wires, seed=None) Creates a list of all eleven parameter arrays for CVNeuralNetLayers().<br>
Actually my aim was to convert 11 parameters of cvqnn_layers_all to tensors</p>

username: rahulvigneswaran
cooked: <p>Can you print <code>a</code> and show?</p>

username: Aigerim_Bazarkhanova
cooked: <p>[tensor([[2.66190161, 4.0582724 , 2.74944154]], requires_grad=True),<br>
tensor([[5.60317502, 6.0548717 , 2.40923412]], requires_grad=True),<br>
tensor([[4.97455513, 3.32314479, 3.56912924]], requires_grad=True),<br>
tensor([[0.22408932, 0.04843122, 0.05791405]], requires_grad=True),<br>
tensor([[5.46644755, 6.14884039, 5.02126135]], requires_grad=True),<br>
tensor([[2.89956035, 4.90420945, 0.74314013]], requires_grad=True),<br>
tensor([[4.02074236, 0.90071527, 5.93552986]], requires_grad=True),<br>
tensor([[3.27886971, 2.60539781, 1.66225193]], requires_grad=True),<br>
tensor([[-0.02421495,  0.15143913, -0.03330574]], requires_grad=True),<br>
tensor([[3.88071828, 3.84591085, 3.87631062]], requires_grad=True),<br>
tensor([[0.00473648, 0.01549474, 0.03781625]], requires_grad=True)]</p>

username: Matias_Vasquez
cooked: <p>You can use <code>cat</code> to concatenate them.</p>
<pre><code class="lang-python">b = torch.cat(a, 0)
print(b)
</code></pre>
<pre><code class="lang-python"># Output
tensor([[ 2.6619e+00,  4.0583e+00,  2.7494e+00],
        [ 5.6032e+00,  6.0549e+00,  2.4092e+00],
        [ 4.9746e+00,  3.3231e+00,  3.5691e+00],
        [ 2.2409e-01,  4.8431e-02,  5.7914e-02],
        [ 5.4664e+00,  6.1488e+00,  5.0213e+00],
        [ 2.8996e+00,  4.9042e+00,  7.4314e-01],
        [ 4.0207e+00,  9.0072e-01,  5.9355e+00],
        [ 3.2789e+00,  2.6054e+00,  1.6623e+00],
        [-2.4215e-02,  1.5144e-01, -3.3306e-02],
        [ 3.8807e+00,  3.8459e+00,  3.8763e+00],
        [ 4.7365e-03,  1.5495e-02,  3.7816e-02]], grad_fn=&lt;CatBackward0&gt;)
</code></pre>
Build the architecture of an ensemble model for tabular classification

username: Onur_Ozbek
cooked: <p>I have data that looks like this:</p>
<pre><code class="lang-auto">	X000	X001	X002	X003	X004	X005	X006	X007	X008	X009	...	X384	X385	X386	X387	X388	X389	X390	X391	X392	Y
0	0.435294	0.568627	0.470588	0.239216	0.062745	0.003922	0.000000	0.000000	0.133333	0.321569	...	0.694118	0.000000	0.000000	0.000000	0.000000	0.000000	0.054902	0.243137	0.537255	5
1	0.290196	0.192157	0.098039	0.039216	0.011765	0.000000	0.000000	0.000000	0.482353	0.627451	...	0.278431	0.007843	0.019608	0.035294	0.054902	0.082353	0.133333	0.141176	0.098039	0
2	0.000000	0.000000	0.000000	0.000000	0.003922	0.027451	0.082353	0.184314	0.000000	0.000000	...	0.380392	0.007843	0.050980	0.164706	0.301961	0.400000	0.443137	0.415686	0.360784	4
3	0.745098	0.952941	0.870588	0.329412	0.035294	0.000000	0.000000	0.000000	0.000000	0.058824	...	0.000000	0.015686	0.211765	0.341176	0.800000	0.101961	0.000000	0.000000	0.000000	1
4	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	...	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	9
</code></pre>
<p><code>X000</code> to <code>X392</code> are the data columns and <code>Y</code> is the column I’m trying to classify for.<br>
I want to build an ensemble of 5 models with specific constraints, such as:</p>
<ul>
<li>The base classifier needs to be a fully connected neural network</li>
<li>For the ensemble, I need to use neural networks of different hyper-parameters</li>
<li>Better to create a NN factory that receives a list of neurons per layer</li>
<li>The ensemble can be simple bagging or majority vote</li>
<li>The ensemble needs to be a class and not inherited from Pytorch/Keras</li>
<li>5 models</li>
<li>Accuracy to measure model performance</li>
<li>During training, training and validation accuracies need to be plotted</li>
</ul>
<p>I have never architected a model like this before. May I get some help on how I should go about it?</p>
<p>Previously, I have experience doing this with pure Machine Learning:</p>
<pre><code class="lang-auto">import pandas as pd
from sklearn import model_selection, svm
from sklearn.ensemble import AdaBoostClassifier
from xgboost import XGBClassifier

dataframe = pd.read_csv('pima-indians-diabetes.csv')
array = dataframe.values

X = array[:,0:8]
Y = array[:,8]

seed = 7
num_trees = 30

kfold1 = model_selection.KFold(n_splits=10)
model1 = AdaBoostClassifier(n_estimators=num_trees, random_state=seed)
results1 = model_selection.cross_val_score(model1, X, Y, cv = kfold1)
print('AdaBoost: ', results1.mean())

kfold2 = model_selection.KFold(n_splits=10)
model2 = XGBClassifier(n_estimators=num_trees, random_state=seed)
results2 = model_selection.cross_val_score(model2, X, Y, cv=kfold2)
print('XGBoost: ', results2.mean())
</code></pre>
How does the training behave when half the model is wrapped with dataparallel?

username: YASJAY
cooked: <p>What is the effect on training if 2nd half of the model is wrapped with data parallel?</p>

username: InnovArul
cooked: <p>In my understanding,  the tricky part would be due to the batch norms (if any) used in the second part of the model, as the batch statistics will be calculated for each device separately.</p>
Pytorch lightning with GPU

username: ricky1
cooked: <p>I am trying to follow the official doc <a href="https://pytorch-lightning.readthedocs.io/en/latest/accelerators/gpu.html" class="inline-onebox" rel="noopener nofollow ugc">Accelerator: GPU training — PyTorch Lightning 1.7.0dev documentation</a> to use gpu to train. There is basic, intermediate and advanced level tutorial in the doc. I am only following the basic one.</p>
<p>there is only two changes to be made in the tutorial:<br>
<strong>1st change from</strong></p>
<pre><code class="lang-auto">trainer = pl.Trainer(max_epochs=20)

</code></pre>
<p>to</p>
<pre><code class="lang-auto">trainer = pl.Trainer(max_epochs=20, accelerator='gpu', devices=1)
</code></pre>
<p><strong>2nd change use .type_as behind every newly created tensor e.g. from</strong></p>
<pre><code class="lang-auto">class GetContextVector(pl.LightningModule):
    def __init__(self):
        super(GetContextVector, self).__init__()

    def forward(self, memory, attentions):
        attentions = F.softmax(attentions, dim=-1)
        output = torch.tensor([])
        .......
</code></pre>
<p>to</p>
<pre><code class="lang-auto">class GetContextVector(pl.LightningModule):
    def __init__(self):
        super(GetContextVector, self).__init__()

    def forward(self, memory, attentions):
        attentions = F.softmax(attentions, dim=-1)
        output = torch.tensor([]).type_as(attentions)
        .......
</code></pre>
<p>These two are the only changes I made.<br>
Then I decided to log the time taken between it taken to perform one training step and the time taken to perform one full epoch. I also did it with each validation step.</p>
<pre><code class="lang-auto">    def training_step(self, batch, batch_idx):  # batch should contain both x, y label
        tic = time.perf_counter()


        if batch_idx == 0:
            if self.epochStartTime is None:
                self.epochStartTime = time.perf_counter()
            else:
                a = time.perf_counter()
                wandb.log({"epochTime": a - self.epochStartTime})
                self.epochStartTime = time.perf_counter()

        ..... compute loss, backpropagation, etc. 


        toc = time.perf_counter()
        wandb.log({"oneStepTime": toc - tic})
        return {'idx': batch_idx}
</code></pre>
<p>The results are<br>
Not using gpu:<br>
one training step: 0.14 seconds<br>
one validation step: 0.05<br>
one epoch: 60 seconds</p>
<p>using gpu:<br>
one training step: 0.2 seconds<br>
one validation step: 0.15 seconds<br>
one epoch: 120 seconds</p>
<p>So using gpu actually makes things a lot slower in this case.<br>
For each epoch there is 100 training step and 100 validation step:<br>
(0.2 - 0.14) x 100 + (0.15 - 0.05) x 100 = 16 seconds.<br>
It seems that the validation step and training steps are not the only computation that is slowed down because the epoch step slowed down by 60 seconds instead of 16.</p>
<p>What have I done wrong?</p>
DataLoader with cache and shuffle implementation

username: Matteo_Ciotola
cooked: <p>I would like to implement something equivalent to the <code>tf.data.Dataset</code> joint features: cache and shuffle.</p>
<p>More precisely, I would like to create a code, to speed up the training, equivalent to:</p>
<pre><code class="lang-auto">inp, gt = train_generator[0]
train_dataset = tf.data.Dataset.from_generator(train_generator,
                                               output_shapes=(inp.shape, gt.shape),
                                               output_types=(inp.dtype, gt.dtype)).batch(batch_size)
train_dataset = train_dataset.prefetch(4)
train_dataset = train_dataset.cache()
train_dataset = train_dataset.shuffle(len(train_generator) + 1, reshuffle_each_iteration=True)
</code></pre>
<p>I have already found some partial answers such as <a href="https://discuss.pytorch.org/t/best-practice-to-cache-the-entire-dataset-during-first-epoch/19608" class="inline-onebox">Best practice to cache the entire dataset during first epoch</a> but I think none of them supports shuffle.</p>
<p>My starting point is my dataset class:</p>
<pre><code class="lang-auto">class MatDataset (Dataset):

    def __init__(self, img_path_list):
        self.img_path_list = img_path_list

    def __len__(self):
        return len(self.img_path_list)

    def __getitem__(self, index):
        temp = io.loadmat(self.img_path_list[index])
        I_PAN = temp['I_PAN']
        I_MS = temp['I_MS']
        # I_MS = interp23tap(I_MS, 4)
        I_MS = np.moveaxis(I_MS, -1, 0)
        I_PAN = np.expand_dims(I_PAN, 0)

        return torch.from_numpy(I_MS.astype(np.float32)), torch.from_numpy(I_PAN.astype(np.float32))
</code></pre>
<p>How may I do?</p>
<p>Thanks a lot!</p>

username: nivek
cooked: <p>You can check out <a href="https://github.com/pytorch/data" rel="noopener nofollow ugc"><code>torchdata</code></a> which introduces <code>DataPipe</code> as replacement for <code>torch</code>'s <code>Dataset</code>. The built-in DataPipes have some of the <a href="https://pytorch.org/data/main/torchdata.datapipes.iter.html" rel="noopener nofollow ugc">functionalities</a> you need (<a href="https://pytorch.org/data/main/generated/torchdata.datapipes.iter.InMemoryCacheHolder.html#torchdata.datapipes.iter.InMemoryCacheHolder" rel="noopener nofollow ugc">in-memory cache</a>, <a href="https://pytorch.org/data/main/generated/torchdata.datapipes.iter.OnDiskCacheHolder.html#torchdata.datapipes.iter.OnDiskCacheHolder" rel="noopener nofollow ugc">on-disk cache</a>, <a href="https://pytorch.org/data/main/generated/torchdata.datapipes.iter.Shuffler.html#torchdata.datapipes.iter.Shuffler" rel="noopener nofollow ugc">shuffle</a>).</p>
<p>We do not have a DataPipe for prefetching yet, you can create your own or try using the built-in prefetching feature of <code>DataLoader</code>.</p>
AlexNet Input Size: 224 or 227?

username: zhanwenchen
cooked: <p>When I looked at the <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py" rel="nofollow noopener">source code for vision/AlexNet</a>, I’m not sure whether the implementation assumes a 227 or 224 input size, because if you did use 227, the output of the first conv layer should be (227-11+2x2)/4 + 1 = 56 even. This would not have made much sense for the first pooling layer, which would produce an output size of (56 - 3) / 2 + 1 = 27.5.</p>
<p>One possible understanding is that the pool1 output is then floored, but this account differs from <a href="https://medium.com/@smallfishbigsea/a-walk-through-of-alexnet-6cbd137a5637" rel="nofollow noopener">this Medium article</a>, which suggests that in fact, conv1’s input may in fact be 224, and that the conv1 output is thus floored: (224 - 11 + 2x2) / 4 + 1 = 55.25, so that the pool1 output produces an integer without flooring: (55 - 3) / 2 + 1 = 27. Can someone please comment assertively whether the input should actually be 224 or 227? I believe this may be important for users who need this information to resize their images correctly.</p>

username: ptrblck
cooked: <p>As described in <a href="https://pytorch.org/docs/stable/torchvision/models.html">the docs</a>, the input size should be at least <code>224x224</code> and as far as I know this size was used for training as shown <a href="https://github.com/pytorch/examples/blob/b142c4bde84b7d6e2d61fd063f902621d014291d/imagenet/main.py#L204">here</a>.</p>
<blockquote>
<p>All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224.</p>
</blockquote>

username: mattgoh
cooked: <p>There are several sources that point out the same discrepancy. The video transcript <a href="https://www.coursera.org/lecture/convolutional-neural-networks/classic-networks-MmYe2" rel="noopener nofollow ugc">here</a> (I believe from Andrew Ng) states:</p>
<blockquote>
<p>So, AlexNet input starts with 227 by 227 by 3 images. And if you read the paper, the paper refers to 224 by 224 by 3 images. But if you look at the numbers, I think that the numbers make sense only of actually 227 by 227.</p>
</blockquote>
<p>In the Caffe file <code>deploy.prototxt</code>, the input dim is indicated to be 227x227:</p>
<pre><code class="lang-auto">layer {
  name: "data"
  type: "Input"
  top: "data"
  input_param { shape: { dim: 10 dim: 3 dim: 227 dim: 227 } }
}
</code></pre>
<p>There are other sources (from searching the web) that reference this discrepancy, and work through the calculations to support the claim.</p>
<p>As a side note, the AlexNet paper states that random cropping was used, whereas the PyTorch docs <a href="https://pytorch.org/hub/pytorch_vision_alexnet/" rel="noopener nofollow ugc">here</a> use <code>CenterCrop</code> as an example</p>
Synchronize CUDA calls in Libtorch

username: Dan_Sagher
cooked: <p>Hi,<br>
I’m trying to improve performance and in order to do so I want to measure the accurate running time of different functions calls.<br>
Do anybody knows how to synchronize CUDA calls in Libtorch?<br>
In python you can do:<br>
torch.cuda.synchronize()</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">CUDAStream stream = getCurrentCUDAStream();
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>

username: Ja_Klar
cooked: <p>I’m new to libtorch myself. But isn’t it possible to simply use</p>
<pre><code>#include &lt;torch/cuda.h&gt;
torch::cuda::synchronize();
</code></pre>
<p>See <a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1_1cuda_1a576cf6ec2b223bcabee2f80dfdf8cdc8.html" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/api/function_namespacetorch_1_1cuda_1a576cf6ec2b223bcabee2f80dfdf8cdc8.html</a>.</p>
Torch.jit.trace throwing an error for quantized model

username: Harshavardhana
cooked: <p>Hi,<br>
I am trying to quantize the CRAFT pre-trained model and optimize it for Android mobile. I am getting below error while calling torch.jit.trace<br>
<strong>“NotImplementedError: Could not run ‘quantized::conv2d.new’ with arguments from the ‘CPU’ backend.”</strong><br>
[Pretrained model is present <a href="https://github.com/clovaai/CRAFT-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - clovaai/CRAFT-pytorch: Official implementation of Character Region Awareness for Text Detection (CRAFT)</a>]</p>
<p>The code I used is below. Please let me know where I am going wrong?</p>
<p><em>model = craft.CRAFT()</em><br>
<em>model.load_state_dict(copyStateDict(torch.load(‘craft_mlt_25k.pth’, map_location=‘cpu’)))</em><br>
<em>model.eval()</em></p>
<p><em>backend = ‘qnnpack’</em><br>
<em>qconfig = torch.quantization.get_default_qconfig(backend)</em><br>
<em>model.qconfig = qconfig</em></p>
<p><em>torch.quantization.prepare(model, inplace=True)</em><br>
<em>torch.quantization.convert(model, inplace=True)</em></p>
<p><em>example = torch.randn(1, 3, 1024, 1024)</em><br>
<em>traced_script_module = torch.jit.trace(model, example)</em><br>
<em>traced_script_module_optimized = optimize_for_mobile(traced_script_module)</em><br>
<em>traced_script_module_optimized._save_for_lite_interpreter(‘craft_mlt_25k.ptl’)</em></p>
<p>I have also tried <strong>torch.jit.script</strong> instead of <strong>torch.jit.trace</strong> and ended up in error mentioned in <a href="https://discuss.pytorch.org/t/convert-pytorch-model-to-ptl/150310" class="inline-onebox">Convert pytorch model to ptl</a></p>
Weighted mean state_dict

username: eban
cooked: <p>Hello I am working with federated learning. I am following <a href="https://towardsdatascience.com/preserving-data-privacy-in-deep-learning-part-3-ae2103c40c22?gi=3ecc7ab79c8c" rel="noopener nofollow ugc">this tutorial</a>.</p>
<p>Everything is understandable, however, the author is trying to use a weighted average</p>
<pre><code class="lang-auto">def server_aggregate(global_model, client_models,client_lens):
    """
    This function has aggregation method 'wmean'
    wmean takes the weighted mean of the weights of models
    """
    total = sum(client_lens)
    n = len(client_models)
    global_dict = global_model.state_dict()
    for k in global_dict.keys():
        global_dict[k] = torch.stack([client_models[i].state_dict()[k].float()*(n*client_lens[i]/total) for i in range(len(client_models))], 0).mean(0)
    global_model.load_state_dict(global_dict)
    for model in client_models:
        model.load_state_dict(global_model.state_dict())
</code></pre>
<p>Total: number of images used<br>
n : number of deep learning models obtained<br>
client_lens[i]: number of data samples used to train model i</p>
<p>As you can see, there is a for loop where the weighted average of weights is performed:</p>
<pre><code class="lang-auto">for k in global_dict.keys():
        global_dict[k] = torch.stack([client_models[i].state_dict()[k].float()*(n*client_lens[i]/total) for i in range(len(client_models))], 0).mean(0)
</code></pre>
<p>my question is:</p>
<p>why is the author multiplying by n when taking the mean?:</p>
<pre><code class="lang-auto">n*client_lens[i]/total
</code></pre>
Torchaudio.functional.lfilter VS scipy.signal.lfilter

username: a7brusco
cooked: <p>I’m testing some filters features using <code>torchaudio</code>.<br>
I wanted to filter a sinus sweep using an IIR A-Weighting filter (cf. <a href="https://en.wikipedia.org/wiki/A-weighting" class="inline-onebox" rel="noopener nofollow ugc">A-weighting - Wikipedia</a>).<br>
I also use an old version of torchaudio (0.7.2) but I’ve done my tests using the latest one with the same results.<br>
There is the code that I used with Python 3.8.3: <a href="https://github.com/a7brusco/torchaudio_vs_scipy" class="inline-onebox" rel="noopener nofollow ugc">GitHub - a7brusco/torchaudio_vs_scipy: Testing torchaudio feature and scipy.signal (specially filters)</a><br>
Is there an meaning why the torchaudio lfilter and the scipy lfilter functions are so different ?</p>

username: a7brusco
cooked: <p>After many tries with my code, I found that the difference between the 2 filters are caused by precision error on my coeffs. Scipy used my coeffs as an float64  <code>np.ndarray</code> while pytorch converted the coeffs as <code>FloatTensor</code> (which have float32 precision). IIR filters are pretty sensible about precision error (especially high order filters). Moreover, it seems that the <code>torchaudio.functional.lfilter</code> doesn’t support <code>DoubleTensor</code> (float64 dtype). I have no choice to use the scipy function for that kind of filter…</p>

username: yoyololicon
cooked: <p>Yes, the precision error can be magnified when dealing with IIR filters, especially when the order is high. This is also stated in the <a href="https://pytorch.org/audio/stable/functional.html#lfilter" rel="noopener nofollow ugc">online document</a>. I recommend you either use double-precision or break down the filter coefficients into cascaded second-order filters using some root-finding algorithms (e.g. <a href="https://numpy.org/doc/stable/reference/generated/numpy.roots.html" rel="noopener nofollow ugc">np.roots</a>).</p>
<blockquote>
<p>Moreover, it seems that the <code>torchaudio.functional.lfilter</code> doesn’t support <code>DoubleTensor</code> (float64 dtype).</p>
</blockquote>
<p><code>torchaudio.functional.lfilter</code> does support double-precision, which is included in our <a href="https://github.com/pytorch/audio/blob/69467ea59003e950152e2abb9e447807c45cad79/test/torchaudio_unittest/functional/functional_cpu_test.py#L20" rel="noopener nofollow ugc">unit tests</a>. If you believe it’s a bug please submit an issue to our repo so we can look into it.</p>

username: a7brusco
cooked: <p>Thank you for the response. I’m using an old version of torchaudio (0.7.2) which explains why <code>torchaudio.functional.lfilter</code> doesn’t support double-precision for me. I retry using the last version and double precision and it works prefectly !</p>
How to write a customized autograd Function which accepts lists/tuples of tensors as its inputs?

username: Jiabao
cooked: <p>I recently encountered a problem when I try to make a customized autograd Function on my own. In my case, it is quite different from conventional requirements, because the inputs to my Function need to be some lists/tuples of tensors.</p>
<p>To be more specific, my problem can be simplified as follows.<br>
The input to my Function is a list of tensor (in the presented example, we may only consider one input list; but in fact, the Function may need to take more than one input lists).</p>
<blockquote>
<p>x = [torch.randn(4,5), torch.randn(8,9)]</p>
</blockquote>
<p>And let’s consider the simplest case, a customized ReLU Function.<br>
I would suppose the following codes would run, but it won’t actually.</p>
<blockquote>
<p>class MyReLU(torch.autograd.Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(ctx, x):<br>
# <code>x</code> need to be a list of tensors<br>
out = [torch.relu(t) for t in x]<br>
ctx.save_for_backward(out)<br>
return out<br>
<span class="mention">@staticmethod</span><br>
def backward(ctx, y):<br>
act, = ctx.saved_tensors<br>
out = [(t1 &gt; 0.0) * t2 for t1, t2 in zip(act, y)]<br>
return out</p>
</blockquote>
<p>I know the actual implementation might be different, but you may get the basic idea from the above codes. The above codes didn’t work, it seems that the Function only takes <code>torch.Tensor</code> as inputs.</p>
<p>I need to implement this because these tensors have distinct sizes and cannot be stacked together. Besides, this shown codes involves some <code>for</code> loop to iterate over all the tensors, but actually it won’t be necessary, since my real operation is written in CUDA C++ and this can be efficiently computed. And the length of the tensor list could be very large, iterating over all the tensors should be time-consuming and inefficient. So that’s why the input to the Function need to be a list of tensors.</p>
<p>Could you give any suggestions or workarounds about this? Thank you very much!!</p>
Custom structured kernel

username: Phthonos
cooked: <p>Greetings<br>
Is there a standard way to train a convolutional kernel where the kernel has a particular structure, for my specific example, I’d like the kernel to be rotationally symmetric:</p>
<pre><code class="lang-auto">c b c
b a b
c b c
</code></pre>
<p>And more generally, I’d like to be able to create a function F parametrized by (a_0, a_1,…) such that the kernel is just F applied to the elements of a matrix of the distances (squared) from the center of the kernel. Then I’d like the parameters (a_0, a_1,…) to be trained directly.<br>
I’ve tried guessing so far but without success.</p>

username: Michal_Bogacz
cooked: <p>Hello,<br>
Although I haven’t done this myself, I found several threads on the forum.<br>
<a href="https://discuss.pytorch.org/t/how-to-implement-a-custom-convolutional-layer-and-call-it-from-your-own-network/120196">How to implement a custom convolutional layer and call it from your own network? - PyTorch Forums</a><br>
<a href="https://discuss.pytorch.org/t/custom-convolution-layer/45979">Custom convolution layer - PyTorch Forums</a><br>
I believe there is no better way than writing your own class, this class should be based on nn.Module.</p>
<p>Hope I have helped.</p>
ValueError: expected sequence of length 1300000 at dim 1 (got 650000)

username: insung3511
cooked: <p>Hi, I troubling with train tensors. here is the code…</p>
<pre><code class="lang-python">X_train, X_test, y_train, y_test = train_test_split(
    (db1_sig + db2_sig), 
    (db1_label + db2_label) * 2,
    test_size=0.33,
    shuffle=True
)

print()

train_dataloader = DataLoader(X_train,
                              batch_size=BATCH_SIZE,
                              num_workers=0, collate_fn=lambda x: x,
                              shuffle=True)

test_dataloader = DataLoader(X_test,
                             batch_size=BATCH_SIZE,
                             shuffle=True)
...

loss_ = []
for epoch in range(EPOCH):
    '''First bbrbm'''
    for _, (data) in enumerate(train_dataloader):
        print(len(data[0]))
        data = torch.tensor(data, dtype=torch.float32)
</code></pre>
<p>and error contents are like this.</p>
<pre><code class="lang-auto">ValueError: expected sequence of length 1300000 at dim 1 (got 650000)
</code></pre>
<p>I think <code>data</code> variable is a problem because <code>data</code> is list that made of 650000 length numpy arrays. I still figuring out what is wrong with this code and plz help anyone. thanks</p>

username: Michal_Bogacz
cooked: <p>Hello,<br>
Can you specify at which line of the presented code the error occurs?</p>

username: insung3511
cooked: <p>oh sorry error on this line</p>
<pre><code class="lang-python">        data = torch.tensor(data, dtype=torch.float32)
</code></pre>
Clarification on LSTM

username: thistlillo
cooked: <p>I am looking at the example at the bottom of this page:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html</a></p>
<pre><code class="lang-auto">rnn = nn.LSTM(10, 20, 2)
input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
c0 = torch.randn(2, 3, 20)
output, (hn, cn) = rnn(input, (h0, c0))
</code></pre>
<p>If I understand correctly:</p>
<ol>
<li>the LSTM has an input dimension of 10, a hidden dimension of 20 and 2 layers;</li>
<li>the input is a batch of size 5, each item being a sequence of three elements of size 10 (i.e. the input dimension of the LSMT);<br>
Am I correct?<br>
If I am, why do I need three hidden initial tensors (h0)?</li>
</ol>

username: Andrei_Cristea
cooked: <p>Hello!</p>
<p>Per <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">the docs</a> unless you specify batch_first=True, the input to the LSTM is shaped as (sequence length, batch size, number of input features), so in this case the sequence length is 5, the batch size is 3, and the number of input features is 10.</p>
<p>h0 represents the initial hidden state used when processing each input, so you need one for each element in the input sequence. It’s shaped as (number of lstm layers, batch size, number of output features)  which in this case is precisely (2, 3, 20).</p>
<p>Hope this helps!</p>
Load model from *.pt with different model definition path

username: jame
cooked: <p>Language: Python<br>
Version: PyTorch 1-6-0</p>
<p>Suppose, a neural network is saved using <code>torch.save(model, "model.pt")</code>.</p>
<p>Now, the model definition (source code) is placed somewhere else, so that it cannot find this definition when using <code>torch.load("model.pt")</code>.</p>
<ol>
<li>Is it possible to load <code>model.pt</code> with a different model definition path?</li>
<li>Is it possible to extract the <code>state_dict</code> from <code>model.pt</code>?</li>
</ol>
In pytorch, can two tensors with different dtype multiply？

username: 111102
cooked: <pre><code class="lang-auto">import torch
import numpy as np
num_input = 2
num_example = 1000
true_w = np.array([2, -3.4])
true_b = 4.2

features = torch.normal(torch.zeros(num_example, num_input), 1).float()
w = torch.tensor([true_w]).t()

Xw = features.mm(w)


</code></pre>
<p>I run the code in Pycharm, get the error:<br>
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument <span class="hashtag">#2</span> ‘mat2’<br>
So I want to know whether two tensors( features and w )with different dtype can multiply？</p>

username: albanD
cooked: <p>Hi,</p>
<p>No this is not supported.<br>
Note that the problem here is that <code>np.array</code> created a float64 number while <code>.float()</code> returns a float32. So either changing the dtype in np.array or changing .float() to .double() will solve your issue.</p>

username: 111102
cooked: <p>Thank you for your replying！</p>

username: Danfoa
cooked: <p>Any reason for not supporting these type of operations? In my case I neet a multiplication between a quasy binary matrix and a float32 matrix. Converting the binary/int matrix to float seems like a huge waste of memory.</p>
Is QUADRO A2000 4GB gpu compatible with pytorch?

username: Alon_Yariv
cooked: <p>Is QUADRO A2000 4GB gpu compatible with pytorch?</p>
<p>Considering now to but the Thinkpad Gen 4 P1 workstation and I am not sure if I could do my research on it.</p>
<p>What do you say?</p>

username: Michal_Bogacz
cooked: <p>Hello,<br>
In my understanding it is not about pytorch compatibility, but about CUDA drivers, and if your card can cooperate with such drivers. Here is a list of CUDA compatible Nvidia devices, you should look of yours <a href="https://developer.nvidia.com/cuda-gpus" rel="noopener nofollow ugc">CUDA GPUs | NVIDIA Developer</a>.<br>
Unfortunately I can’t see your card in the compatible ones.<br>
You can also try in installing the drivers and using <code>nvidia-smi</code> command, and there looking if your card has any cuda version.</p>
Save PyTorch Model for Evaluation

username: Andre_Amaral_IST
cooked: <p>Hey,<br>
In the documentation is this part of the code:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6ce006e9f45ead3c8d5a59bcdd70592cadc459b4.png" alt="image" data-base62-sha1="fx9xDcXT7Bcg3xl4HwPht9jBcVe" width="481" height="405"></p>
<p>However, how does the script knows the “TheModelClass” since here it does not say that we have to import the model class?</p>

username: Michal_Bogacz
cooked: <p>Hello,<br>
Note that in documentation there are two methods of saving the model. If you have access to created model class, you can save only the states/weights in form of dictionary. But you then are required to have the code of the class → next you create “empty” model, and apply the weights.<br>
Second method saves everything, including the model structure. This can be found in documentation here: <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#:~:text=Save/Load%20Entire%20Model" class="inline-onebox" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 1.11.0+cu102 documentation</a></p>
<p>I hope, I have helped and understood you question correctly.</p>
Use libtorch 1.10.2+cu102 vs libtorch 1.9.0+cu102

username: Tumb1eweed
cooked: <p>I try to build a conan package for libtorch in ubuntu 18.04, Environment is:<br>
NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2  GeForce RTX 2080</p>
<p>test_package.cpp:</p>
<pre><code class="lang-auto">#include &lt;torch/cuda.h&gt;
#include &lt;iostream&gt;

int main(void) {
  std::cout &lt;&lt; "CUDA DEVICE COUNT: " &lt;&lt; torch::cuda::device_count() &lt;&lt; std::endl;
  if (torch::cuda::is_available()) {
    std::cout &lt;&lt; "CUDA is available! Training on GPU." &lt;&lt; std::endl;
  }
  return 0;
}
</code></pre>
<p>When I test libtorch 1.10.2+cu102， it returns</p>
<pre><code class="lang-auto">libtorch/1.10.2+cu102 (test package): Running test()
CUDA DEVICE COUNT: 1
CUDA is available! Training on GPU.
</code></pre>
<p>But when I test libtorch 1.9.0+cu102(or libtorch 1.9.1+cu102), it returns:</p>
<pre><code class="lang-auto">libtorch/1.9.1+cu102 (test package): Running test()
CUDA DEVICE COUNT: 0
</code></pre>
<p>how does this happen? I can’t figure out why.</p>

username: Tumb1eweed
cooked: <p>BTW, Python version is 3.6.9</p>
Compute the weighted average in PyTorch

username: Li-Yun
cooked: <p>Assuming we have a tensor A, with shape</p>
<pre><code>CxH
</code></pre>
<p>And tensor W, representing the weights, with shape <code>H</code></p>
<p>Tensor A represents some features with dim <code>C</code> for each spatial location in <code>H</code> (flattened), and weights <code>W</code> represents the weight for each spatial location. Instead of computing the mean via:</p>
<pre><code>torch.mul(A, W).mean(1) 
</code></pre>
<p>How can we compute the weighted average ?<br>
The output dim should be of size <code>C</code>.</p>
<p>Would it be:</p>
<pre><code>Z = torch.mul(A, W)
Weighted_average = torch.sum(Z, dim=1) / torch.sum(W)</code></pre>

username: sadra-barikbin
cooked: <p>Yes, that’s correct. To write it shorter:</p>
<pre><code class="lang-python">weighted_average = (A@W)/W.sum()
</code></pre>
Self written SGEMM similar to cuBLAS on .cu, but compiled to python library, much slower than torch.matmul !?

username: Ziyu_Huang
cooked: <p>Hi! I am writing cuda kernel for matmul, for research reason, (I have some ideas…so really need to write kernel by myself) I compared this kernel with cublas, the cublas is about 2 times faster than my kernel.<br>
Later I packaged my kernel into python package(follow pytorch’s official method!!<a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#integrating-a-c-cuda-operation-with-pytorch" class="inline-onebox" rel="noopener nofollow ugc">Custom C++ and CUDA Extensions — PyTorch Tutorials 1.11.0+cu102 documentation</a>), and compared it with torch.matmul, now, about 400 times faster than my kernel!! Why it changes???</p>
<p>(I tested this on A100 and GTX1650 , similar result!! much slower!!!)</p>
<p><strong>My guess reason: my way to package is slower than pytorch’s method???</strong></p>
<p>I have uploaded all the files in my github. With detailed steps, you will feel comfortable to compile .cu files and package them into python library.</p>
<p>Some parameters and experiment results are:<br>
matrix size: 64 * 5000 ~ 5000 * 5000 =&gt; 64 * 5000<br>
on .cu result: my kernel:0.540128 ms    cublas: 1.16879 ms   <strong>2 times faster!</strong><br>
on python result:  my kernel: 197.69s    pytorch: 59s first iteration, then 0.5s and stable.   <strong>400 times faster!!!</strong></p>
<p>This is my code, and you can see all details!!!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Ziyu_Huang" data-post="1" data-topic="151350">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziyu_huang/48/14158_2.png" class="avatar"> Ziyu_Huang:</div>
<blockquote>
<p>pytorch: 59s first iteration, then 0.5s and stable</p>
</blockquote>
</aside>
<p>Note that CUDA operations are executed asynchronously so you would have to synchronize the code before starting and stopping the timers. Alternatively, use <code>torch.utils.benchmark</code> to profile the workloads which would add warmup iterations as well as synchronizations.</p>

username: Ziyu_Huang
cooked: <p>Your suggestion is correct…Even pytorch need torch.cuda.synchronize…Previously I did not realize that. Thank you!!!</p>
Is Pytorch still unable to have some smaller version of it for inference or JavaScript solution?

username: JonathanSum
cooked: <p>I have been deploying Pytorch model on Web and mobile. However, I often need some other framework, such as ONNX to make it runs in the browser.<br>
I know we have jit or torchserve. However, do we have other suggestion beside them? And what is the file size requirement when installing the dependency? Let’s say I want to deploy a Web application. I can not install all kinds of Pytorch library in the server.<br>
I say torchserve does not work. For example, you can’t use a torchserve if you want to build a NLP transformer model NPC installed at client’s computer because it requires a HTTP backend serve.</p>

username: dbish
cooked: <p>There isn’t a PyTorch for the browser or javascript version at the moment. What kind of issues do you run into with onnx.js? Is it the transformation from pytorch to onnx that causes issues or something else? It would also be great to hear what you use ML in the browser for if you can share.</p>
How to access sliding window data with only one index in the __getitem__ method?

username: xiaohao-lin1
cooked: <p>Hi I am working on a continuous classification problem. My time series data is of the following shape<br>
x: (n_trials, n_features, n_time)<br>
y: (n_trials, ).</p>
<p>The trials are of different length, so the n_windows is different for each trial.</p>
<p>Then, I applied sliding window. Now, the data is of this 4D shape:<br>
x: (n_trials, n_windows, n_features, window_length).<br>
y: (n_trials, n_windows)</p>
<p>Not all trials will be used, and I will be passing a list eligible of trial number.</p>
<p>There are three things I would like to ask regarding the Pytorch Dataset class:</p>
<ol>
<li>
<p>The <strong>getitem</strong> method only takes in one index, and my objective is to return one window of 2D data shaped (n_features, window_length), and one integer for y. I hope that the index would refer to a specific <strong>window number</strong>. However, to access the window, I must get into the corresponding trial first.<br>
Also, the input trial number may not always start from 1.  <strong>With only one index, how can I get to the specific window I want?</strong></p>
</li>
<li>
<p>Is the logic in (1) correct? Is it a must to return one window at a time? Can I put in an entire trial instead of a specific window? My eventual goal is to make a prediction every 40ms, with a window size of 500ms.</p>
</li>
</ol>
<p>Thank you!</p>

username: tom
cooked: <p>If I understand you correctly, you want to have a single (sometimes “linear” in other context) index into (n_trials, n_windows). The easiest likely is to just loop over the eligible trials/windows at the beginning and build a list of <code>(idx_trial, id_window)</code>. Then you can use the index passed into <code>__getitem__</code> to get the corresponding <code>(idx_trial, id_window)</code> and then the data from that.<br>
If the loop takes too much time to do at the start of the training, an alternative is to do this once in preprocessing and store the result (I would probably use Pandas and a CSV file, but you could also use Python’s <code>pickle</code> or even <code>torch.save</code>).</p>
<p>Best regards</p>
<p>Thomas</p>

username: xiaohao-lin1
cooked: <p>Hi Tom</p>
<p>Thanks so much for your help! Really appreciate it.</p>
SSIM: matching two images

username: Anastacia_Rombolakis
cooked: <p>How do i find an API for generating similarity indexing and matching between two images ?</p>
<p>So how can I get percentage index matching between two images</p>

username: ksmdanl
cooked: <p><a class="mention" href="/u/anastacia_rombolakis">@Anastacia_Rombolakis</a> there are few implementations for SSIM/MS-SSIM. One that I used personally is <a href="https://github.com/VainF/pytorch-msssim.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - VainF/pytorch-msssim: Fast and differentiable MS-SSIM and SSIM for pytorch.</a></p>
Specify GPU for training but other GPU is allocated

username: I_H_Yoo
cooked: <p>Hi.</p>
<p>I specify a second GPU for training my model via <code>DEVICE = torch.device("cuda:1" if torch.cuda.is_available() is True else "cpu")</code> but the memory of the first GPU, which is cuda:0 is allocated. Can you please let me know what am I missing?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3bd59bb8ebc9025298fae7769ca22dedcc63c04e.png" data-download-href="https://discuss.pytorch.org/uploads/default/3bd59bb8ebc9025298fae7769ca22dedcc63c04e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3bd59bb8ebc9025298fae7769ca22dedcc63c04e.png" alt="image" data-base62-sha1="8xjP71whfZT3c8P6FZRhmtbOQFg" width="690" height="455" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">955×630 18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You are most likely creating the CUDA context on the default device and/or accidentally allocating additional memory on it.<br>
Mask the device you would like to use via <code>CUDA_VISIBLE_DEVICES=1 python script.py args</code> and use <code>cuda:0</code> in the script to select the masked device.</p>

username: I_H_Yoo
cooked: <p>Thank you for the immediate answer and for providing me the solution. However, I didn’t allocate it on the default device. If we assume that I didn’t make a mistake at all, what would be a possible option?</p>

username: Hui_Yu
cooked: <p>Did you solve the problem? If so, how  to solve it?</p>