akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Optimizing Inference Time for Chat Conversations on Falcon

Humza_Sami: <p>How can I optimize the inference time of my 4-bit quantized Falcon 7B, which was trained on a chat dataset using Qlora+PEFT. During inference, I loaded the model in 4 bits using the bits and bytes library. While the model performs well in inference, I’ve observed that the inference time increases significantly as the length of the chat grows. To give you an idea, here’s an example of how I’m using the model:</p>
<p>First Message prompt:<br>
&lt; user&gt;: Hi …<br>
&lt; bot&gt;:</p>
<p>2nd Message prompt:<br>
&lt; user&gt;: Hi …<br>
&lt; bot&gt;: How are you ?<br>
&lt; user&gt;: I am good thanks. I need your help !<br>
&lt; bot&gt;:</p>
<p>As the length of the chat increases, the inference time sometimes doubles and can take up to 2-3 minutes per prompt. I’m using an NVIDIA RTX 3090 Ti for inference. Below is the code snippet I’m using for prediction:</p>
<pre><code class="lang-auto">MODEL_NAME = "tiiuae/falcon-7b"

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_use_double_quant=True,
                                bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
tokenizer.pad_token = tokenizer.eos_token

SAVED_MODEL = "Saved_models/model_path"

saved_model_config = PeftConfig.from_pretrained(SAVED_MODEL)
saved_model = AutoModelForCausalLM.from_pretrained(saved_model_config.base_model_name_or_path,
                                             return_dict=True,
                                             quantization_config=bnb_config,
                                             device_map="auto",
                                             trust_remote_code=True
                                            )

tokenizer = AutoTokenizer.from_pretrained(saved_model_config.base_model_name_or_path)
tokenizer.pad_token = tokenizer.eos_token

saved_model = PeftModel.from_pretrained(saved_model, SAVED_MODEL)

pipeline = transformers.pipeline(
    "text-generation",
    model=saved_model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)

prompt = "&lt; user&gt;: Hi ..
&lt; bot&gt;: How are you ?
&lt; user&gt;: I am good thanks. I need your help ! 
&lt; bot&gt;: "

response = pipeline(
        prompt,
        bos_token_id=11,
        max_length=2000,
        temperature=0.7,
        top_p=0.7,
        do_sample=True,
        num_return_sequences=1,
        eos_token_id=[15564]
        )[0]['generated_text']
</code></pre>
<p>I would greatly appreciate any insights or suggestions on how to improve the inference time of model while dealing with longer chat interactions. Thank you in advance for your assistance!</p>
How do I get pytorch 1.9.0 through anaconda?

George_Stepaniants: <p>I really would like to get the last release of pytorch 1.9.0 (not sure if this is a nightly version). I need this in order to use the parametrizations library: <a href="https://pytorch.org/tutorials/intermediate/parametrizations.html" class="inline-onebox" rel="noopener nofollow ugc">Parametrizations Tutorial — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>
<p>For some reason, I can not find a single command to do this. I have run<br>
conda install -c pytorch pytorch-nightly -c conda-forge<br>
but it fails.</p>
<p>Any help would be much appreciated!</p>

ptrblck: <p>The install command if given <a href="https://pytorch.org/get-started/locally/">here</a> as:</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch-nightly
# or
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch-nightly -c nvidia
</code></pre>
<p>so you would need to change the <code>pytorch</code> and <code>pytorch-nightly</code> channel and package names.</p>

fermat97: <p>Hi is there any specific reason version 1.9.0 is not available for installation in the <code>previous version</code> section? If I need to access that specific version is there a straight forward way to go?</p>

ptrblck: <p>Would specifying the version via <code>conda install pytorch==1.9.0...</code> work?</p>

fermat97: <p>Sorry for my very late response. Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> Yes, it works:</p>
<pre><code class="lang-auto">conda install -n test pytorch==1.9.0 torchvision torchaudio cudatoolkit=10.2 -c pytorch
</code></pre>
<p>However, I cannot understand why it is not listed in the <code>previous versions</code> section. Is there any specific reason or something?</p>

Muhammad_Adeel_Altaf: <p>Will this command also work?  conda install -n test pytorch==1.9.0 torchvision torchaudio cudatoolkit=11.1 -c pytorch  Actually, i need to install cuda11.1 with pytorch 1.9.0.</p>
The generated triton MaxPool2d kernel has poor performance on some platforms

lzxdn: <p>I conducted maxpool2d forward kernel testing on some platforms and found that the new triton kernel improved performance on some platforms, mainly due to the efficient use of L1 Cache. However, on some platforms with different L1 cache architectures, the performance decreased significantly compared to the native kernel. MaxPool2d is regarded as a pointwise op in torch.Because the size of the parameter stride changes, the data loading performance will change greatly. Is this a question to consider?</p>

ptrblck: <p>Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> describing your profiling in detail and sharing the code you’ve used to profile these kernels?</p>

lzxdn: <p>Ok， thank you. here is the issue. <a href="https://github.com/pytorch/pytorch/issues/107441" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/107441</a></p>
Error occurred while compiling CUDA program using PyTorch

BearBiscuit05: <p>I encountered this issue while compiling a CUDA file using PyTorch, <strong>along with a series of errors related to CUB</strong>. However, if I directly compile the corresponding file using nvcc, no issues arise. I’m eager to know how to resolve this problem.<br>
correct command without error：*<em>nvcc -o main <em>.cu</em></em></p>
<p><strong>setup.py</strong></p>
<pre><code class="lang-auto">from setuptools import setup, find_packages
from torch.utils.cpp_extension import BuildExtension, CUDAExtension, CppExtension

setup(
    name = "xxx",
    include_dirs = ["."],
    ext_modules = [
        CUDAExtension(
        "xxx", 
        sources = [
            "xxx.cu","xxx1.cu","xxx2.cu","xxx3.cpp",
        ],
        extra_compile_args={
                'cxx': ['-std=c++14', '-g',
                        '-fPIC',
                        '-Ofast',
                        '-DSXN_REVISED',
                        '-Wall', '-fopenmp', '-march=native'],
                'nvcc': ['-std=c++14',
                         '-g',
                         '-DSXN_REVISED',
                         '--compiler-options', "'-fPIC'",
                         ]
            }
        )
    ],
    cmdclass={
        "build_ext": BuildExtension 
    },
)
</code></pre>
<p>Related file</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;curand_kernel.h&gt;
#include &lt;vector&gt;
#include &lt;chrono&gt;
#include &lt;numeric&gt;
#include &lt;fstream&gt;
#include &lt;algorithm&gt;
#include &lt;map&gt;
#include &lt;sstream&gt;
#include &lt;cassert&gt;
#include &lt;cuda_runtime.h&gt;
#include &lt;stdint.h&gt;
#include &lt;cub/cub.cuh&gt;
#include &lt;torch/extension.h&gt;

inline __device__ int64_t
AtomicCAS(int64_t* const address, const int64_t compare, const int64_t val) {
  using Type = unsigned long long int;  // NOLINT

  static_assert(sizeof(Type) == sizeof(*address), "Type width must match");

  return atomicCAS(
      reinterpret_cast&lt;Type*&gt;(address), static_cast&lt;Type&gt;(compare),
      static_cast&lt;Type&gt;(val));
}
</code></pre>
<p>The brief error message is <code> error: ‘atomicCAS’ was not declared in this scope; did you mean ‘AtomicCAS’?</code></p>
The c++ source code for torchvision.csrc.ops.nms.nms, where is it?

freebsdx: <p>Greetings, smart people.</p>
<ol>
<li>as the title mentioned, I want to check the source code(c++) for the torchvision.csrc.ops.nms.nms.</li>
<li>you may point out it is here:<br>
<a href="https://github.com/pytorch/vision/blob/ba7ce61e1d1804771bd25343cc7bd4bfb6cb5b76/torchvision/csrc/ops/nms.cpp#L10" rel="noopener nofollow ugc">vision/torchvision/csrc/ops/nms.cpp at ba7ce61e1d1804771bd25343cc7bd4bfb6cb5b76 · pytorch/vision (github.com)</a></li>
<li>yes, I have seen it. However, so far I have not found out the detailed implementation for this line of code:<br>
"static auto op = c10::Dispatcher::singleton()<br>
.findSchemaOrThrow(“torchvision::nms”, “”)<br>
.typed&lt;decltype(nms)&gt;();<br>
return op.call(dets, scores, iou_threshold);<br>
"</li>
<li>what I actually want is the complete implementation of this “torchvision::nms”.<br>
Could you please give me any hint/clue? Thank you &lt;3</li>
</ol>
Jit trace issues with EagerTensor

yangkl96: <p>I am trying to convert a model I downloaded to TorchScript.</p>
<pre><code class="lang-auto">loaded_model = keras.models.load_model("model.hdf5")
loaded_model.predict(X) #this works, where X is a numpy array
torch.jit.trace(loaded_model.predict, tf.convert_to_tensor(X)) #returns the error RuntimeError: Tracer cannot infer type of (&lt;tf.Tensor: shape=(60, 6), dtype=float16, numpy=... dtype=float16)&gt;)
:Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type EagerTensor.
</code></pre>
<p>I then disable eager execution with <code>tf.compat.v1.disable_eager_execution()</code>,  but get the following error:</p>
<pre><code class="lang-auto">OperatorNotAllowedInGraphError: Iterating over a symbolic `tf.Tensor` is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.
</code></pre>
<p>I do not have access to decorate this function myself, since it’s just a model I downloaded but do not have access to the code. Is there a way to make trace work here?</p>

ptrblck: <p>It seems you are trying to mix TF with PyTorch, which I doubt is supported. Are you following a tutorial, which claims this should work?</p>

yangkl96: <p>Sorry I think I got confused between TF and PyTorch</p>
I want to apply one common transformation to image, followed by three different independent transformation

Kapil_Rana: <p>as per shown here</p>
<p>transform = T.Compose([<br>
T.RandomHorizontalFlip(),<br>
T.RandomVerticalFlip(),<br>
T.RandomApply(torch.nn.ModuleList([T.ColorJitter()]), p=0.25)<br>
])</p>
<p>transform1 = T.Compose([<br>
T.Resize((224,224)),<br>
T.ToTensor(),<br>
])</p>
<p>transform2 = T.Compose([<br>
T.Resize((384, 384)),<br>
T.ToTensor(),<br>
])</p>
<p>transform3 = T.Compose([<br>
T.Resize((512, 512)),<br>
T.ToTensor(),<br>
])</p>
<p>My code in dataloader</p>
<pre><code>   if(self.mode1=="train"):
               img1 = self.img_transform(img1) 
               print("OK") 
    
   if self.img_transform1 is not None:
       img1 = self.img_transform1(img1)
       img2 = self.img_transform2(img1)
       img3 = self.img_transform3(img1) 
</code></pre>
<p>however I am gettng error<br>
TypeError: pic should be PIL Image or ndarray. Got &lt;class ‘torch.Tensor’&gt;</p>
<p>I know the other transformation want input as PIL image, thats why it showing error. How to tackle this situation. transform is common transformation I want to apply.</p>

Mah_Neh: <p>What happens if you remove the “ToTensor()” conversion ?</p>

Kapil_Rana: <p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘PIL.Image.Image’&gt;</p>
<p>It will give this error. Also, i do want to apply the normalise the values in 0 and 1</p>

Mah_Neh: <p>are you using <code>read_image</code> to load the image?<br>
Now paying more attention it seems you use PIL, but no info included in the post about it.</p>
<p>Can you include a more complete code ?</p>

Kapil_Rana: <p>I am using<br>
def default_loader(path):<br>
return Image.open(path).convert(‘RGB’)</p>

Mah_Neh: <p>I could take a look if the full relevant code snippet is included, so I can reproduce it locally, not by pieces like this</p>
Torch.compile() before or after .cuda()

zhihao_zhu: <p>Hi, should I call torch.compile() before or after moving the model to the GPU (i.e. call cuda()). Or it doesn’t matter? Because for some model I noticed run torch.compile() before .cuda() actually slows down the model inference speed. Another similar questions: should I call torch.compile() before or after model = torch.nn.parallel.DistributedDataParallel(model) for DDP?</p>

marksaroufim: <p>Call .cuda() before torch.compile and compile before passing in to DDP or FSDP</p>

zhihao_zhu: <p>Thanks for the answer! That’s aligned with my experiment results. Btw, may I ask what’s the reason for this ordering, just trying to understand the Pytorch 2 fundamentals a bit more. Thanks in advance!</p>

marksaroufim: <p>It’s best to call <code>.cuda()</code> first since then Inductor doesn’t need to reason about Device Copies, just makes the compilers job simpler</p>
<p>It’s best to call before DDP because the communication collectives aren’t traced - they will be though in the future <a href="https://github.com/pytorch/pytorch/issues/93173" class="inline-onebox" rel="noopener nofollow ugc">[RFC] PT2-Friendly Traceable, Functional Collective Communication APIs · Issue #93173 · pytorch/pytorch · GitHub</a></p>

marksaroufim: <p>For people coming to this issue please keep in mind that the instructions I gave for DDP are out of date</p>
<p>Please follow instructions here instead <a href="https://github.com/pytorch/pytorch/issues/107362#issuecomment-1682658771" class="inline-onebox">[Inductor] Run compiled model failed on 2023_08_17 nightly · Issue #107362 · pytorch/pytorch · GitHub</a></p>
Attributes to be considered while using model interpretability methods

kountaydwivedi: <p>Hello all,<br>
I am using model interpretability methods such as IntegratedGradients, GradSHAP, DeepLift, etc. After executing the attributions method, I get a matrix of attributions corresponding to every feature in the dataset. My question is:<br>
Which attributes should I consider as important attributes? The ones with high positive correlation, or the ones with negative correlation, or both? Maybe I want to ask how to assign the attribution values to important or not-important set?</p>
<p>Thank you.</p>

googlebot: <p>Both. Check plots in captum tutorials to get an intuition. Not sure if there is an universal recipe for importance thresholds…</p>

Jugal_Patel: <p>New model-agnostic interpretability methods, both local and global, are being released by Leap Labs in a few weeks (<a href="https://www.leap-labs.com/" rel="noopener nofollow ugc">https://www.leap-labs.com/</a>). There is a waiting list on their site. Their global methods extract what a model has learned directly from the model, independent of the training data. Their local explanations use hierarchical perturbation.</p>
The 1th argument dim does not have a specified value or default value

areebbashir: <p>0</p>
<p>I’m trying to deploy an image to image model on Android. Model consists of encoder and decoder. The encoder works fine. But this issue comes in decoder. My model takes one tensor as input and it should return a tensor of size [1,3,1024,1024] as output. But there comes a Fatal in the app.</p>
<pre><code class="lang-auto">final long[] styleshape = new long[]{3};
final FloatBuffer styledBuffer = Tensor.allocateFloatBuffer(3);
styledBuffer.put(-0.4951f);
styledBuffer.put(-1.0065f);
styledBuffer.put(-1.0413f);

styledBuffer.get(0);
Tensor styledTensor = Tensor.fromBlob(styledBuffer, styleshape);
IValue style = IValue.from(styledTensor);
final Tensor styledOutputTensor = decoderModule.forward(style).toTensor();
</code></pre>
<p>I tried passing a random tensor of the similar shape and then I didn’t get the error.</p>
<pre><code class="lang-auto">def forward(self,  style_tensor):
        spade_input=None 
        pure_generation=False
        content_list=[]
        content_list.append(torch.rand([1, 192, 256, 256]))
        content_list.append(torch.rand([1, 5, 512, 512]))
        content_list.append(torch.rand([1, 5, 1024, 1024]))
        style_tensor = torch.randn(1,1,3,1)
        style_tensor=style_tensor.view(1,1,3,1)
        if self.pred_adain_params:
            adain_params = self.adain_net(style_tensor)
            self.assign_adain_params(adain_params, self)

        if self.pred_conv_kernel:
            assert style_tensor.size(0) == 1, 'prediction of convilution does not work with batch size &gt; 1'
            self.assign_style(style_tensor.view(1, -1), self)

        tensor = module_list_forward(self.body, content_list[0], spade_input)
        for skip_content, up_layer, up_postprocess_layer, skip_preprocess_layer in     zip(content_list[1:], self.upsample_head, self.upsample_postprocess,self.skip_preprocess):                            
                                                                                       
                                                                                       
                                                                                       
            tensor = up_layer(tensor)
            skip_tensor = skip_preprocess_layer(skip_content)
            tensor = torch.cat([tensor, skip_tensor], 1)
            tensor = up_postprocess_layer(tensor)
        for layer in self.model_postprocess:
            tensor = layer(tensor)    

        tensor = torch.tanh(tensor[:, : 3])

                            
        #ret = torch.randn([1, 3, 1024, 1024])
        
        return tensor

</code></pre>
<p>Expecting a Tensor but getting the above error.</p>
Fine-tuning only new weights

sl5035: <p>I am trying to train a transformer model after extending its vocabulary. The problem is I want to keep the original weights frozen and train only the weights associated with the new vocabulary. I was thinking of doing something like this:</p>
<pre><code class="lang-auto">processor = processor() # Loading processor
model = model() # Loading model
for param in model.parameters():
    param.requires_grad = False

processor.tokenizer.add_tokens(new_tokens)
model.resize_token_embeddings(len(processor.tokenizer))
</code></pre>
<p>Is this a valid option? If not, what other options do I have?</p>

ptrblck: <p>It’s unclear what exactly happens in your code as neither object definitions are posted nor did you explain what the custom methods perform.<br>
If you want to freeze a <code>weight</code> parameter partially, you could zero out its gradients before calling <code>optimizer.step()</code> (which should work for stateless optimizers) or you could restore the original weight values after the update (which would also work for optimizers using running stats).</p>
Transform an image to tensor using libtorch

Jesus: <p>Hello everyone. I’m am using the c++ version of Pytorch called libtorch. Do you know a standard way to convert an image into a tensor with libtorch?</p>

ptrblck: <p>You could load and process an image with <code>OpenCV</code> and transform it to a tensor via <code>torch::from_blob</code>.</p>

Jesus: <p>OK. I see. Thank you Sir</p>
Torchaudio VAD reverse effect

aNR0: <p>from Torchaudio documentation, regarding the Voice Activity Detector transform (<a href="https://pytorch.org/audio/stable/transforms.html#vad" class="inline-onebox" rel="noopener nofollow ugc">torchaudio.transforms — Torchaudio 0.11.0 documentation</a>), it says:</p>
<p>“The effect can trim only from the front of the audio, so in order to trim from the back, the reverse effect must also be used.”</p>
<p>Well, I have a simple question: how can I apply the reverse effect? <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

nateanl: <p>Hi <a class="mention" href="/u/anr0">@aNR0</a>, you can try <a href="https://pytorch.org/audio/stable/sox_effects.html#torchaudio.sox_effects.apply_effects_tensor" rel="noopener nofollow ugc">torchaudio.sox_effects.apply_effects_tensor</a> with <code>reverse</code> as the effects.<br>
For example,</p>
<pre><code class="lang-python">effects = [
    ['reverse'],  # reverse
]
sample_rate = 16000
waveform = 2 * torch.rand([2, sample_rate * 1]) - 1
waveform, sample_rate = apply_effects_tensor(waveform, sample_rate, effects, channels_first=True)
</code></pre>

Shahin_Konadath: <p>Try this!</p>
<pre><code class="lang-auto">import torchaudio
import torchaudio.functional as taf

def trim_silence(source: str, target: str = None, sample_rate: int = 22050, resample: bool = True):
    if target is None:
        target = source.replace(".wav", "_trim.wav")
    try:
        aud, sr = torchaudio.load(source)
        if aud.shape[0] &gt; 1:
            aud = aud.mean(aud, dim=0, keepdim=True)

        if resample:
            aud = taf.resample(aud, sr, sample_rate)

        aud = taf.vad(aud, sample_rate)
        aud, sr = torchaudio.sox_effects.apply_effects_tensor(aud, sample_rate, [['reverse']])
        aud = taf.vad(aud, sample_rate)
        aud, sr = torchaudio.sox_effects.apply_effects_tensor(aud, sample_rate, [['reverse']])
        torchaudio.save(target, aud, sample_rate, bits_per_sample=32)
        return target
    except Exception as et:
        raise et
</code></pre>
Finetuning pretrained Transformers using pipeline parallelism

molokanov: <p>Hello. I’m looking for a solution analogous to<br>
<a href="https://pytorch.org/tutorials/intermediate/pipeline_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/pipeline_tutorial.html</a><br>
with the only difference. The tutorial says how to pipeline-parallelize while training from scratch, and I need to pipeline-parallelize while finetuning an already pretrained Transformer.<br>
Can anyone tell how to do such a finetuning?</p>
Geometric temporal GConvLSTM

Davide_Michelon: <p>Hello everyone,</p>
<p>I’ve been working on a project involving graph convolutional LSTM models in PyTorch, and I’ve encountered a shape mismatch error that I’m having trouble resolving.</p>
<p>I have implemented a <code>GConvLSTMModel</code> using the source code for torch_geometric_temporal.nn.recurrent.gconv_lstm available at <a href="https://pytorch-geometric-temporal.readthedocs.io/en/latest/_modules/torch_geometric_temporal/nn/recurrent/gconv_lstm.html#GConvLSTM" class="inline-onebox" rel="noopener nofollow ugc">torch_geometric_temporal.nn.recurrent.gconv_lstm — PyTorch Geometric Temporal documentation</a>. This is my model:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2ea54981a47be7e2ef5288d3ecdd553dd54a7ca0.png" data-download-href="https://discuss.pytorch.org/uploads/default/2ea54981a47be7e2ef5288d3ecdd553dd54a7ca0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2ea54981a47be7e2ef5288d3ecdd553dd54a7ca0.png" alt="image" data-base62-sha1="6EE5yMlzzrougzg5KxCl0OaVAJi" width="323" height="500" data-dominant-color="F0F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">484×748 22.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<blockquote>
<p>model_GconvLSTM= GConvLSTMModel(in_channels=6, out_channels=22, num_layers=6, K=2)</p>
</blockquote>
<p>During training, I have the error:<br>
…</p>
<blockquote>
<p><code> /usr/local/lib/python3.10/dist-packages/torch_geometric/nn/dense/linear.py in forward(self, x)     130             x (torch.Tensor): The input features.     131         """ --&gt; 132         return F.linear(x, self.weight, self.bias)     133      134     @torch.no_grad() RuntimeError: mat1 and mat2 shapes cannot be multiplied (22x6 and 22x22)</code></p>
</blockquote>
<p>In the <code>forward</code> method of the model, the <code>data</code> tensor has dimensions <code>(batch_size=1, num_nodes=22, num_features=6, num_time_steps=5)</code>. Here, <code>num_nodes</code> is the number of graph nodes, <code>num_features</code> represents the number of features for each node, and <code>num_time_steps</code> indicates the number of graph representations over time.</p>
<p>I’ve double-checked the shapes of my tensors and input data, and I’m fairly confident that the issue arises from a shape mismatch somewhere in the model architecture. I suspect that I might be passing incorrect dimensions or not handling the temporal dimension properly.</p>
<p>Could someone kindly guide me on how to properly configure the input data shape and architecture to resolve this error? I’d greatly appreciate any insights or suggestions to help me move forward</p>
Return torch.stack(batch, 0, out=out) RuntimeError: stack expects each tensor to be equal size, but got [21] at entry 0 and [23] at entry 1

Muhammad12: <p>why this error is occuring?</p>

whichone: <p>your batch consists of tensors of different shapes. i guess, maybe <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pad_sequence.html" rel="noopener nofollow ugc">pad_sequence</a> or <a href="https://pytorch-forecasting.readthedocs.io/en/stable/api/pytorch_forecasting.utils.padded_stack.html" rel="noopener nofollow ugc">padded_stack</a> from external library would be useful in your case.</p>
Dealing with Flooded Batteries in PyTorch Applications

Annijohn: <p>Hello PyTorch community members!</p>
<p>I hope you’re all doing well. I’m reaching out here because I’ve encountered a challenge related to working with flooded batteries in my PyTorch project, and I could really use some guidance and advice.</p>
<p>In my current project, I’m developing a deep learning model using PyTorch that involves processing data from sensors attached to flooded batteries. These batteries are used in a variety of applications, and accurate analysis of their behavior is crucial.</p>
<p>I’ve run into some difficulties in effectively preprocessing the sensor data from these <strong><a href="https://www.trojanbattery.com/products/technology/flooded" rel="noopener nofollow ugc">Flooded Batteries</a></strong> before feeding it into my PyTorch model. The data is quite complex, with various parameters affected by the battery’s state and the environment. I’m not entirely sure about the best practices for data preprocessing in such a scenario.</p>
<p>I’m turning to this community for advice. Have any of you worked with similar data or faced challenges with preprocessing complex sensor data in PyTorch? I’m particularly interested in techniques or libraries that can assist in handling this kind of data effectively within the PyTorch framework.</p>
<p>Has anyone encountered similar challenges when working with complex sensor data?<br>
What preprocessing steps would you recommend for dealing with flooded battery data specifically?<br>
Are there any PyTorch modules or best practices that could be useful in this context?<br>
I truly appreciate any insights, suggestions, or guidance you can provide. This community has been a tremendous source of knowledge, and I’m looking forward to learning from your experiences.</p>
<p>Thank you in advance for your time and help!</p>
<p>Best regards,<br>
Annijohn</p>
Shuffle batch retrieval but not values within batch

gursi261: <p>I am working on a machine translation dataset and the input entries are sorted from the shortest sequence to the longest sequence. I pad them to the max length in each batch using collate_fn.</p>
<p>Is there a way to make batch creation unshuffled so that similar length entries end up in the same batch (since the data is sorted by length), but still shuffle the retrieval of batches themselves?</p>

ptrblck: <p>Take a look at e.g. <a href="https://discuss.pytorch.org/t/gru-and-padded-sequences-tipps-and-tricks/90729/3">this post</a> from <a class="mention" href="/u/vdw">@vdw</a> where he shared a similar approach.</p>

gursi261: <p>I see, thanks for your reply!</p>

vdw: <p>This <a href="https://discuss.pytorch.org/t/using-batches-for-seq2seq-models/55205">post</a> is more specifically for Seq2Seq models like machine translation.</p>

gursi261: <p>Got it, thanks for the answer!</p>
Iter.device(arg).is_cuda() INTERNAL ASSERT FAILED

Pranavk: <p>Hi I am using PPO algorithm</p>
<p>during the back-propagation there is this step<br>
" return torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))"</p>
<p>and the error is<br>
RuntimeError: iter.device(arg).is_cuda() INTERNAL ASSERT FAILED at “/opt/conda/conda-bld/pytorch_1603728993639/work/aten/src/ATen/native/cuda/Loops.cuh”:94, please report a bug to PyTorch.</p>
<p>Can anyone help me with this ? thank you …</p>

ptrblck: <p>Could you rerun your code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check, if you get any error message?<br>
If so, could you post it here please?<br>
Also, make sure to use the latest PyTorch version.</p>

Pranavk: <p>Thank you for the replay …I am using PyTorch 1.7.0 and I am running it in a Notebook so I do not know how can I pass this argument there…</p>

ptrblck: <p>You could try to use <code>os.environ</code> to set this env var at the beginning of the notebook.<br>
Note, that you would have to set it before importing PyTorch, as it won’t have any effect otherwise.<br>
Alternatively you could export the notebook as a Python script and run it in the terminal, which should work as well.</p>

Pranavk: <p>Hii I did that but still it is the same error.</p>
<p>Error :  “RuntimeError: iter.device(arg).is_cuda() INTERNAL ASSERT FAILED at “/opt/conda/conda-bld/pytorch_1603728993639/work/aten/src/ATen/native/cuda/Loops.cuh”:94, please report a bug to PyTorch.”</p>

ptrblck: <p>OK, I think the blocking launch didn’t work or maybe Jupyter is not forwarding the stack trace.<br>
Could you post an executable code snippet to reproduce this issue and give us information about your setup, i.e. PyTorch CUDA, cublas version, how did you install it, which GPU are you using?</p>

Pranavk: <p>Hi thank you it was solved I had a different issue I guess</p>

berlin: <p>how did you solve the problem?</p>
If prediction impossible continue

pounnaullaugeivi: <p>Hello, I trained an LSTM model. I wish I could continue to predict even after the user is asking for an impossible prediction for the model. Can you tell me how to do it? (If prediction possible give the prediction, if prediction impossible continue). Thank you in advance.</p>

pounnaullaugeivi: <p>Sorry my question was stupid, I found the answer (try / except)…</p>
RuntimeError: using pytorch model with fastai deployment

berlin: <p>I am trying to train deep learning with my dna sequence. However I am keep getting an error message which I can not locate. Can anyone help me with this?</p>
<p>Model:</p>
<pre><code class="lang-auto"># import the necessary packages
from torch.nn import Module,Dropout,Conv1d,Linear,MaxPool1d,ReLU,Conv2d,MaxPool2d
import pytorch_lightning as pl
from torch import flatten
from typing import List
class model_test(nn.Module): # deepcre model  
    def __init__(self,
                 seq_len: int =1000,
                 kernel_size: int = 8,
                 p = 0.25): # drop out value 
        super().__init__()
        self.seq_len = seq_len
        # adjusting window size corresponding to sequence length 
        window_size = int(seq_len*(8/3000)) # 1000* = 2 
        
        # CNN module
        self.conv11 = Conv1d(4,64,kernel_size=(kernel_size),padding= "same",stride = 1  )
        self.relu11 = ReLU()
        self.maxpool1 = MaxPool1d(kernel_size=window_size)
        self.Dropout1 = Dropout(p)
        self.fc1 =Linear( batch_size_init*(seq_len//window_size) , 1)
    
    def forward(self, x):
       # x = xb.permute(0,2,1).unsqueeze(1)
        """Forward pass."""
        # reshape view to batch_size x 4channel x seq_len
        # permute to put channel in correct order
        # rearranges the original tensor according to the desired ordering and returns a new multidimensional rotated tensor. 
        # The size of the returned tensor remains the same as that of the original. (0,2,1) means (batch size, 4 channel - OHE(DNA), Seq.length  )
        x = x.permute(0,2,1)
        x = self.conv11(x)
        x = self.relu11(x)
        print("conv1 1st : ",x.size())
        x = self.maxpool1(x)
        print("maxpool : ",x.size())
        x = self.Dropout1(x)
        x = x.view(batch_size_init, 64*500)
        print("view1 : ",x.size())
        print("view2 : ",x.size())
        print("checking processed dimension",x.dim()
        x = self.fc1(x)
        print("linear : ",x.size())
        return x
</code></pre>
<p>Learner :</p>
<pre><code class="lang-auto">
dls= DataLoaders(train_dl,val_dl)
net = model_test()
learn= Learner(dls, net, loss_func= nn.MSELoss(), opt_func = SGD, lr= 0.001) 

learn.fit(2)
</code></pre>
<p>Error :</p>
<pre><code class="lang-auto">RuntimeError: Exception occured in `Recorder` when calling event `after_batch`:
        iter.device(arg).is_cuda() INTERNAL ASSERT FAILED at "../aten/src/ATen/native/cuda/Loops.cuh":89, please 
report a bug to PyTorch. argument 1: expected a CUDA device but found cpu
</code></pre>
Machine translation transformer predicts repeating output without generating <EOS>

gursi261: <p>I created and tried to train my own machine translation transformer model with the regular encoder decoder architecture, attempting to keep it as close to the original “Attention is all you need” paper.</p>
<p>The problem is that my model seems to output the same thing repeated without generating an  token until many timesteps later. The outputs look something like this for an english to spanish translation.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; inference("i like to swim", model, dataset, DEV, 50)
'&lt;SOS&gt; me gusta nadar a nadar te gusta nadar a nadar a nadar les gusta nadar a me gusta nadar como me gusta nadar &lt;EOS&gt;'
</code></pre>
<p>The correct output should just be ‘me gusta nadar’, which the model has generated and then gone past, repeating previous outputs again and again.</p>
<p>What could be the reason for behaviour like this? I trained the model for 20 epochs on a dataset with 130,000 translations. Do i keep training for longer?</p>
<p>For context, the code can be found <a href="https://github.com/gursi26/seq2seq-transformer" rel="noopener nofollow ugc">here</a> (with <a href="https://github.com/gursi26/seq2seq-transformer/blob/7cd586b15eea00f0bbd5b08cccfc1e238f6f2165/utils.py#L41" rel="noopener nofollow ugc">this</a> being the definition of the inference function). I didn’t include it in this post because it would end up being very long.</p>
Difference between using 1 LSTM with 2 features and 2 LSTMs with 1 feature

math_art: <p>If there’s a sequential dataset with with 2 features (eg. a Pytorch tensor of shape <code>(N,2)</code> ) and I want to use an LSTM to generate more elements in the sequence what would be the difference between using a single LSTM <code>torch.nn.LSTM(input_size=2)</code> for both features, or 2 LSTMs <code>torch.nn.LSTM(input_size=1)</code> one for each feature?</p>
Asynchronous copy in c++ when input has been destructed

Nanlalalalalal: <p>I want to copy an at::Tensor input with pinned memory with<br>
<code>input.to(at::TensorOptions().device(at::kCUDA, -1), true, false, input.suggest_memory_format());</code><br>
As it is an asynchronous operation, how to ensure that the storage area of the original input is not released before completion if input will be destructed?</p>
<p>Is the following function safe if input is in pinned memory, and on a different GPU device?.</p>
<pre><code class="lang-auto">at::Tensor to_current_device(at::Tensor input) {
  if (input.device() == at::kCPU) return input.cuda();
  if (input.device().index() == at::cuda::current_device()) return input;
  at::TensorOptions options;
  // input.is_pinned()
  return input.to(at::TensorOptions().device(at::kCUDA, -1), input.is_pinned(), 
                  false, // &lt;==
                  input.suggest_memory_format());  
}

input = to_current_device(input);
</code></pre>
<p>And also about when copy is true:</p>
<pre><code class="lang-auto">at::Tensor to_current_device(at::Tensor input) {
  if (input.device() == at::kCPU) return input.cuda();
  if (input.device().index() == at::cuda::current_device()) return input;
  at::TensorOptions options;
  // input.is_pinned()
  return input.to(at::TensorOptions().device(at::kCUDA, -1), input.is_pinned(), 
                  true,  // &lt;==
                  input.suggest_memory_format());  
                                                   
}

input = to_current_device(input);
</code></pre>
<p>Thanks so much if anyone can help.</p>
My Simplest Neural Network Object Detector Not Working Well

takbess: <p>I made one simplest object detection task and a model to study object detection.</p>
<ul>
<li>TASK</li>
</ul>
<p>The task is to predict the bounding box of a person. Each image contains only one person. An image example is the following.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6eb41b43754267d0a368519a022bce2665c5148.png" alt="Image.1" data-base62-sha1="zeluyi2Rd5rJjGJ4Uzv0J4FtHtK" width="384" height="398"><br>
I made the dataset from MSCOCO dataset. I chose the images containing only one person, and then resized them into (32,32). I got 1k images.</p>
<ul>
<li>MODEL</li>
</ul>
<p>The SimpleObjectDetector class is made as follows:</p>
<pre><code class="lang-auto">class ConvLayer(nn.Module):
    def __init__(self,in_channel,out_channel):
        super().__init__()
        self.conv = nn.Conv2d(in_channel,out_channel,kernel_size=3,stride=1,padding=1)
        self.act = nn.ReLU()
        self.pool = nn.MaxPool2d(2,2)
    def forward(self,x):
        x = self.pool(self.act(self.conv(x)))
        return x

class SimpleObjectDetector(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = ConvLayer(3,8)
        self.conv2 = ConvLayer(8,16)
        self.l1 = nn.Linear(8*8*16,64)
        self.l2 = nn.Linear(64,4)
        self.act = nn.ReLU()

    def forward(self,x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size()[0],-1)
        x = self.act(self.l1(x))
        x = self.l2(x) * 32 + 16
        return x    # (x,y,w,h) is expected. 
</code></pre>
<p>x = self.l2(x) * 32 + 16 is to make the output distribution fit the image size (32,32).</p>
<ul>
<li>TRAINING</li>
</ul>
<p>I tried the loss function nn.L1Loss() and IoU loss. At 1~3 epoch, the loss stops decreasing.</p>
<ul>
<li>RESULT</li>
</ul>
<p>I found the predicted bounding boxes become almost the same for all images.</p>
<ul>
<li>ADDITIONAL STUDY</li>
</ul>
<p>I further studied the problem, and found the bounding box prediction becomes the same for all images even before the training! It seems the final output has very low degrees of freedom.</p>
<ul>
<li>QUESTION</li>
</ul>
<p>Is there any way of making this model correctly predict bounding boxes? I want to try to predict bounding boxes without using larger models like YOLO.</p>
Layer-by-layer profiler in `torch.profiler`

dsgl: <p>How to set up parameters to show layer-by-layer profile by <code>torch.profiler</code>?</p>
<p>The layer-by-layer profile is the same functionality in the package of <code>torchprof</code>, which shows the running time of each layers in NN. However, the current <code>torch.profiler</code> only displays low-level operations.</p>
<p>See <a href="https://github.com/awwong1/torchprof/blob/master/README.md" rel="noopener nofollow ugc">Readme of torchprof</a> for the difference of layer profiles and low-level operations, where the first table is layer-by-layer profile and the second table is low-level operations.</p>

Yeshwanth_V: <p>Same question. OP please share if you found a solution for this. Thank you!</p>
How to efficiently calculate pairwise differences between values in a list (tensor)?

Davi_P: <p><a href="https://discuss.pytorch.org/t/how-to-calculate-pair-wise-differences-between-two-tensors-in-a-vectorized-way/37451/9">This post</a> helped me to reach a solution, however it is still 7x slower than pdist (which is a similar and a little more complex operation, but not what I need):</p>
<p>I have put <a href="https://stackoverflow.com/a/76916911/9681577" rel="noopener nofollow ugc">more details here</a>, in case anyone is interested reading or answering there:</p>
<p>Thanks</p>
Multiple forward passes

smani: <p>Is it possible that uncommenting the line</p>
<pre><code class="lang-auto">out_u, mid_u = self.net(Iun)
</code></pre>
<p>may affect the performance as the loss does not depend on this line?<br>
In my understanding commenting / uncommenting this line should not affect the test accuracy based on the net. However, it does. What could be the reason?</p>
<pre><code class="lang-auto">self.net.train()
for batch_idx in range(n):
	Il, lbls, Iun = self.getNextBatch()

	# labeled
	outputs, mid = self.net(Il)
	loss = self.criterion(outputs, lbls)
	self.optimizer.zero_grad()
	loss.backward()
	self.optimizer.step()
	
	# unlabeled
	out_u, mid_u = self.net(Iun)

acc = testModel()
</code></pre>
<p>However, the following two codes (CODE 1 and CODE 2) give me the same test classification performances (i.e., acc1 = acc2).</p>
<pre><code class="lang-auto"># CODE 1
self.net.train()
for batch_idx in range(n):
	Il, lbls, Iun = self.getNextBatch()

	# labeled
	outputs, mid = self.net(Il)
	loss = self.criterion(outputs, lbls)
	self.optimizer.zero_grad()
	loss.backward()
	self.optimizer.step()

    # unlabeled
    # out_u, mid_u = self.net(Iun)

acc1 = testModel()
</code></pre>
<pre><code class="lang-auto"># CODE 2

for batch_idx in range(n):
	Il, lbls, Iun = self.getNextBatch()

	# labeled
    self.net.train()
	outputs, mid = self.net(Il)
	loss = self.criterion(outputs, lbls)
	self.optimizer.zero_grad()
	loss.backward()
	self.optimizer.step()

    self.net.eval()
    # unlabeled
    out_u, mid_u = self.net(Iun)

acc2 = testModel()
</code></pre>

whichone: <p>of course any extra calculations result in decreasing performance.</p>
<p>may be i misunderstood your question. what do you mean by “test performance”? also, what do you mean by “Am I doing something wrong somewhere else?”. your code without “# unlabaled” line is a proper abstract code for any typical training procedure. perhaps, what is the purpose of this “# unlabeled” line?</p>

smani: <p>I have refined my question. Test performance = classification accuracy on the test set based on <a href="http://self.net" rel="noopener nofollow ugc">self.net</a>.<br>
Am I doing something wrong somewhere else? = May be some bugs in the other part of my code?<br>
I need to calculate the loss based on the <span class="hashtag-raw">#unlabeled</span>,  however, I havent done that so far.</p>

smani: <p>Sorry I found it. It is due to the batch norm, and it seems my Il and Iun are normalized differently.</p>

whichone: <p>good to hear you’ve found an answer<br>
in addition, I want to mention, that computations on GPU are non-detirministic, unless you set some torch.cudnn options manually. thus, two runs of the similar code fragments may produce different results. Also, randomness affects the result, which occurs from BatchNorm in your case, which you’ve said you use.</p>
Are older pre-compiled wheels available? Where do older torch_nightly.html whl files go?

agostini01: <p>Hi,<br>
I am aware that some newer dev python wheels are available here: <a href="https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html</a></p>
<p>However I need a dev version that is more than 2-3 months old (for example: <code>torch==2.0.0.dev20221231, torchvision==0.15.0.dev20221231</code>). Do these older nightly wheels get moved to another server? Are they still publicly available?</p>

ptrblck: <p>These wheels are still available and you can download them by explicitly specifying the URL. The index is just cleared and keeps a few months of wheels. However, wheels are not deleted themselves, just from the index.</p>

agostini01: <p>Thank you for the reply! It solved my issue.</p>
<p>For future reference, I was able to download the <code>whl</code> files with:</p>
<pre><code class="lang-auto"># Download torch-2.0.0.dev...
wget https://download.pytorch.org/whl/nightly/cpu/torch-2.0.0.dev20221231%2Bcpu-cp38-cp38-linux_x86_64.whl
</code></pre>
<p>or directly install with:</p>
<pre><code class="lang-bash"># Download and install torchvision-0.15.0.dev...
pip install https://download.pytorch.org/whl/nightly/cpu/torchvision-0.15.0.dev20221231%2Bcpu-cp38-cp38-linux_x86_64.whl
</code></pre>
<hr>
<p>Do you know if it is possible to omit the final part <code>%2Bcpu-cp38-cp38-linux_x86_64.whl</code>?<br>
Similar to how it is done when the whl files are in a visible index?</p>
<pre><code class="lang-bash"># Example that currently does not work 
# as the .html file does not contain these version anymore:
python3 -m pip --no-cache-dir install \
    -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html \
    --pre \
    torch==2.0.0.dev20221231 \
    torchvision==0.15.0.dev20221231
</code></pre>
<p>Thanks again!</p>