akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
sparse.FloatTensor: size is inconsistent with indices

mmp: <p>With PyTorch 1.13, I want to create a sparse FloatTensor:</p>
<pre><code class="lang-auto">num_nodes = 29
num_edges = 75
n2n_idxes = torch.LongTensor(2, num_edges * 2)
n2n_vals = torch.FloatTensor(num_edges * 2)
n2n_sp = torch.sparse.FloatTensor(n2n_idxes, n2n_vals, torch.Size([num_nodes, num_nodes]))
</code></pre>
<p>Run script 1:<br>
<strong>RuntimeError: found negative index -4745092176981434354 for dim 0</strong><br>
Run script 2:<br>
<strong>RuntimeError: size is inconsistent with indices: for dim 0, size is 29 but found index 140147970303920</strong></p>
<p>I found a similar question here: <a href="https://discuss.pytorch.org/t/runtimeerror-sizes-is-inconsistent-with-indices-pytorch-0-4-1/94181" class="inline-onebox">RuntimeError: sizes is inconsistent with indices. Pytorch 0.4.1</a><br>
But the problem seems to be version related.<br>
Any recommendations or solutions are highly appreciated!</p>

ptrblck: <p>You are using uninitialized memory in these lines of code:</p>
<pre><code class="lang-python">num_nodes = 29
num_edges = 75
n2n_idxes = torch.LongTensor(2, num_edges * 2)
n2n_vals = torch.FloatTensor(num_edges * 2)
</code></pre>
<p>which means <code>n2n_idxes</code> and <code>n2n_vals</code> can contain any value including Infs, NaNs etc.<br>
The error is thus expected, so make sure you are properly initializing the tensors before trying to create the sparse tensor.<br>
I would generally recommend to use the factory method, such as <code>torch.randn</code>, <code>torch.randint</code> etc. instead of the explicit constructors for a dtype.</p>

akuysal: <p>I think, you may try the code below</p>
<p>n2n_idxes = torch.LongTensor(torch.randint(1, num_nodes, [2, num_edges * 2]))</p>
<p>instead of</p>
<p>n2n_idxes = torch.LongTensor(2, num_edges * 2)</p>

mmp: <p>Many thanks, this solved the problem!</p>
Is there any function in pytorch like plot_model function in the tensor flow to print the network architecture?

Kamal_Acharya: <p>I want to print the network architecture like the one shown in the figure. Is there any function in pytorch which can do so?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ee42a6add2296316eae172baeda68dca382e892.png" data-download-href="https://discuss.pytorch.org/uploads/default/6ee42a6add2296316eae172baeda68dca382e892" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ee42a6add2296316eae172baeda68dca382e892.png" alt="Capture" data-base62-sha1="fOZmB6JJyGla0vuS7dc3gMoSIvg" width="518" height="500" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">561×541 7.66 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

wi-sh: <p>You can use <a href="https://github.com/szagoruyko/pytorchviz" rel="noopener nofollow ugc">torchviz</a> library for the same!</p>

Kamal_Acharya: <p>This is my code which throwing errors. Can you provide me some more reference about the torchviz?</p>
<p>!pip install git+https://github.com/szagoruyko/pytorchviz<br>
from torchviz import make_dot<br>
make_dot(mlp,mlp.parameters())</p>

wi-sh: <p>Take a look at this issue:<a href="https://github.com/szagoruyko/pytorchviz/issues/74" rel="noopener nofollow ugc">https://github.com/szagoruyko/pytorchviz/issues/74</a></p>

ConvolutionalAtom: <p>Another library is <code>torchview</code>, which is very close to <code>plot_model</code> of keras as it can capture module hierarchy. The similarity to <code>plot_model</code> API was a big factor in the design of the library<br>
For instance, output for mlp model is the following<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1aca7df9481765ea98b5713c724b8c09c6c65e6.png" alt="mlp" data-base62-sha1="wcplkIQBg8ghMtID5SGMGvPFk0e" width="221" height="441"></p>
<p>Disclaimer: I am the author of library</p>
'mps' and manual_seed_all()

David_Laxer: <p>How do I do manual_seed_all() for an ‘mps’ device<br>
E.g. - for cuda:</p>
<pre><code class="lang-auto">torch.cuda.manual_seed_al
</code></pre>

junwucs: <p>Hi David: have you solved this problem?</p>
How to use RNN with images

Ioannis_D: <p>Hello, I found this paper here: <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6377428/" rel="noopener nofollow ugc">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6377428/</a><br>
It seems like in order to deal with different ecg lengths, it deals with it by splitting each to 5s segments.<br>
I have two questions regarding this:</p>
<pre><code>During training, does each batch contain the segments of a single ecg?
What is the input (following the documentation notation (L,N,Hin​)) in the RNN in such cases?
</code></pre>
<p>For question 2, here is how I interpret it.<br>
The output of the FC layer is 1024 thus the input size is 1024(corresponding to 5s of ecg), batch size = 1 if we are using a single ecg per batch and the sequence length is the number of 5s segments per ecg.</p>
<p>Thank you in advance.</p>
Is there any way to generate compile_commands.json for the c++ extensions?

umiswing: <p>When using <code>python setup.py develop</code>, PyTorch will generate a <code>compile_commands.json</code> file that can be used by many editors to provide command completion and error highlighting for PyTorch’s C++ code. But it seems that this file is not for custom c++ extension. Is there any way to generate a <code>compile_commands.json</code> for the custom extension?</p>

umiswing: <p>I find just use <code>bear</code> when run <code>python setup.py install</code> to compile the extensions will generate the <code>compile_commands.json</code>.  The root <code>setup.py</code> of <code>pytorch</code> write some functions to generate <code>compile_commands.json</code>, but just use <code>bear</code> will work.</p>
Exporting the operator ::flatten_dense_tensors to ONNX opset version 12 is not supported

Weiwei_Shi: <p>I am trying to export my model to ONNX, but get the Error,</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/code/2022/SR/BigViT1x/basicsr/train.py", line 223, in &lt;module&gt;
    train_pipeline(root_path)
  File "/home/code/2022/SR/BigViT1x/basicsr/train.py", line 195, in train_pipeline
    model.save_to_onnx()
  File "/home/code/2022/SR/BigViT1x/basicsr/models/sr_model.py", line 248, in save_to_onnx
    torch.onnx.export(self.net_g,  # model being run
  File "/home/anaconda3.9/lib/python3.9/site-packages/torch/onnx/__init__.py", line 350, in export
    return utils.export(
  File "/home/anaconda3.9/lib/python3.9/site-packages/torch/onnx/utils.py", line 163, in export
    _export(
  File "/home/anaconda3.9/lib/python3.9/site-packages/torch/onnx/utils.py", line 1074, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/home/anaconda3.9/lib/python3.9/site-packages/torch/onnx/utils.py", line 731, in _model_to_graph
    graph = _optimize_graph(
  File "/home/anaconda3.9/lib/python3.9/site-packages/torch/onnx/utils.py", line 308, in _optimize_graph
    graph = _C._jit_pass_onnx(graph, operator_export_type)
  File "/home/anaconda3.9/lib/python3.9/site-packages/torch/onnx/__init__.py", line 416, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/anaconda3.9/lib/python3.9/site-packages/torch/onnx/utils.py", line 1421, in _run_symbolic_function
    raise symbolic_registry.UnsupportedOperatorError(
torch.onnx.symbolic_registry.UnsupportedOperatorError: Exporting the operator ::flatten_dense_tensors to ONNX opset version 12 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.
</code></pre>
<p>I have no idea how to solve the issue.</p>
<p>Thank you.</p>
Multiprocessing dataloader with num_workers = 1

amirhf: <p>Hello, I have custom map-style dataset created and I iterate with the native PyTorch Dataloader. I use num_workers = 1 for debugging purposes and understanding the bottlenecks with pytorch profiler. Here is what I see in PyTorch Profiler’s trace tab:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a584d021813e0dc7520131df4aac5a9292673b25.png" data-download-href="https://discuss.pytorch.org/uploads/default/a584d021813e0dc7520131df4aac5a9292673b25" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a584d021813e0dc7520131df4aac5a9292673b25_2_690x153.png" alt="image" data-base62-sha1="nCfkEM70sVnXxtdZF3bLTsKPnfv" width="690" height="153" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a584d021813e0dc7520131df4aac5a9292673b25_2_690x153.png, https://discuss.pytorch.org/uploads/default/original/3X/a/5/a584d021813e0dc7520131df4aac5a9292673b25.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/5/a584d021813e0dc7520131df4aac5a9292673b25.png 2x" data-dominant-color="B2A1C9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">897×199 36.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m confused why there are 7 different threads still created that wait for other thread to finish in each iteration of dataloader. Can someone help me understand and guide me how to definitely get a single process dataloader?</p>

ejguan: <p>To get a single-process dataloader, you should specify <code>num_wokers=0</code>.</p>

amirhf: <p>I wonder why. It doesn’t sound intuitive!</p>

ejguan: <p><code>num_worker</code> represents the number of subprocesses used by <code>DataLoader</code>. Using <code>num_worker=0</code> means make everything running in the main process.</p>
Calling backward after multiple forwards

enterthevoidf22: <p>for some reason (related to triplet loss training in parallel to cross-entropy loss) i want to create a forward pass like this:</p>
<pre><code class="lang-auto">        def forward(self,triplet):
            total_outputs = torch.empty((0,self.num_categories))
            output_embedding = {}
            for name, x in triplet.items():
                output = self.main_model(x)
                output_embedding[name] = self.get_current_embedding()
                total_outputs = torch.cat([total_outputs,output],dim=0)
            return total_outputs, output_embedding
</code></pre>
<p>what i did is create a triplet model that inherits from nn.Module and accepts a main_model which is a normal CNN. the triplet model takes a dict as input, passes each value (a triplet component) )through the main_model and collects the triplet embedding, but also aggregates the regular binary output of the three triplet components and pass them out as total_outputs</p>
<p>my question:<br>
is there anything problematic i’m not seeing about the gradients being accumulated on main_model weights through 3 consecutive forward passes? excluding memory issues that are under control, is there anything inherently wrong with the assumption that passing the anchor, positive and negative one after the other and then concatenating the binary outputs as  one and calculating the loss and gradients on the total is identical to if i didn’t do it in three passes?</p>
Native pytorch activation checkpointing implementation gives different performance than without

Idriss_Mghabbar: <p>Hello,<br>
I am using activation checkpointing in order to trade compute for memory. The idea is to run the forward in a torch.no_grad() manner and to run it again before performing the backpropagation to get the necessary activations.</p>
<p>I was therefore expecting to get the same loss and performances values compared to a training done without it given that my script is deterministic.</p>
<p>I also tried the activation checkpointing wrapper of fairscale but I could not get the same values.</p>
AttributeError: module 'syft' has no attribute 'FederatedDataLoader'

jarvico: <p>Hi,</p>
<p>I am trying to use syft for federated learning.<br>
I used below commands to import syft and define the virtual machines.</p>
<pre><code class="lang-auto">import syft as sy

alice = sy.VirtualMachine(name="alice")
bob = sy.VirtualMachine(name= "bob")
ken= sy.VirtualMachine(name="ken")
ryu = sy.VirtualMachine(name= "ryu")
tim= sy.VirtualMachine(name="tim")
</code></pre>
<p>I have a dateset named “train”.</p>
<p>When I try to define my trainloader, I got an error as below:</p>
<pre><code class="lang-auto">train_loader_1 = sy.FederatedDataLoader(train.federate((bob, alice, ken, ryu , tim)),  batch_size=120, shuffle=False)

AttributeError: module 'syft' has no attribute 'FederatedDataLoader

</code></pre>
<p>Would you please help me on how to define my federateddataloader?</p>
<p>torch version: 1.10.0+cpu<br>
syft version 0.6.0</p>
<p>Best wishes,<br>
Ömer</p>

jarvico: <p>The question is still open.</p>
<p>I would appreciate if anyone can comment on this.</p>

ptrblck: <p>Based on <a href="https://github.com/OpenMined/PySyft/issues/5530#issuecomment-852033529">this comment</a> I guess this class is deprecated and removed.<br>
Also, tracked <a href="https://github.com/OpenMined/PySyft/issues/7188">here</a>.</p>
PyTorch memory leak reference cycle in for loop

David_Van_Dijcke: <p>I am facing a memory leak when iteratively updating tensors in PyTorch on my Mac M1 GPU using the PyTorch mps interface. The following is a minimal reproducible example that replicates the behavior:</p>
<pre><code class="lang-auto">import torch 

def leak_example(p1, device):
    
    t1 = torch.rand_like(p1, device = device) # torch.cat((torch.diff(ubar.detach(), dim=0).detach().clone(), torch.zeros_like(ubar.detach()[:1,:,:,:], dtype = torch.float32)), dim = 0)
    u1 = p1.detach() + 2 * (t1.detach())
    
    B = torch.rand_like(u1, device = device)
    mask = u1 &lt; B
    
    a1 = u1.detach().clone()
    a1[~mask] = torch.rand_like(a1)[~mask]
    return a1

if torch.cuda.is_available(): # cuda gpus
    device = torch.device("cuda")
elif torch.backends.mps.is_available(): # mac gpus
    device = torch.device("mps")
torch.set_grad_enabled(False)
        
p1 = torch.rand(5, 5, 224, 224, device = device)
for i in range(10000):
    p1 = leak_example(p1, device)    
</code></pre>
<p>My Mac’s GPU memory steadily grows when I execute this loop. I have tried running it on a CUDA GPU in Google Colab and it seems to be behaving similarly, with the GPU’s Active memory, Non-releasable memory, and Allocated memory increasing as the loop progresses.</p>
<p>I have tried detaching and cloning the tensors and using weakrefs, to no avail. Interestingly, if I don’t reassign the output of <code>leak_example</code> to <code>p1</code>, the behavior disappears, so it really seems related to the recursive assignment. Does anyone have any idea how I could resolve this?</p>

ptrblck: <p>I cannot reproduce the issue on a 3090 and after adding <code>print(torch.cuda.memory_allocated()/1024**2)</code> to the for loop I get:</p>
<pre><code class="lang-python">4.78515625
4.78515625
4.78515625
...
</code></pre>
<p>Is this post also related to <a href="https://discuss.pytorch.org/t/memory-grows-with-loop-with-recursively-assigned-attribute/168879">this one you’ve created yesterday</a>?<br>
If so, could you create a GitHub issue for the potential memory leak on MPS?</p>

David_Van_Dijcke: <p>Yes, it’s the same issue but this is a minimal reproducible example.<br>
I also get stable <code>memory_allocated()</code> on Colab but the output of <code>torch.cuda.memory_summary()</code> appears to indicate that some other background memory allocations are growing? I’m not sure though, I’m quite inexperienced with CUDA programming.</p>
<p>I’ll open a GitHub issue. Thank you!</p>

David_Van_Dijcke: <p>I think I found the cause, here is the GitHub issue with my own response <a href="https://github.com/pytorch/pytorch/issues/91368" class="inline-onebox" rel="noopener nofollow ugc">PyTorch memory leak reference cycle in for loop, Mac M1 · Issue #91368 · pytorch/pytorch · GitHub</a></p>
Why VAE Encoder outputs log variance and not standard deviation?

laro: <p>When talking about <code>VAE</code> (and viewing <code>VAE</code> implementations), the Encoder outputs:</p>
<p>μ, log(variance)</p>
<p>when we train the model (the part of the <code>decoder model)</code>, we convert the <code>log(variance)</code> to Standard deviation:</p>
<pre><code class="lang-auto">std = exp(0.5 * logvar)
</code></pre>
<p>(I took the example from here: <a href="https://github.com/AntixK/PyTorch-VAE/blob/master/models/vanilla_vae.py" class="inline-onebox" rel="noopener nofollow ugc">PyTorch-VAE/vanilla_vae.py at master · AntixK/PyTorch-VAE · GitHub</a>)</p>
<p>If we need to convert the <code>log(variance)</code> to <code>Standard deviation,</code> why won’t we output the Standard deviation from the encoder instead of making calculation to convert it to Standard deviation ?</p>
GPT2 and BERT embeddings (Hugging Face)

dreidizzle: <p>Hi, I have two questions related to the embeddings I am getting from a BERT model and a GPT2 model. I am inputting a sentence of 4 words. For GPT2 I get 4 tokens, for BERT I get 6 since I add SEP and CLS.</p>
<ol>
<li>If I want to “summarize” the sentence into one vector with BERT: should I use the CLS embedding or the mean of the tokens within the sentence (all embeddings for tokens other than CLS and SEP)? The CLS token was added for the NSP task so it some sense it captures if one sentence is related to the next sentence for that pre training task. However, what if there is no second sentence like in this case? Can CLS be viewed as a summary of the sentence?</li>
<li>The embedding I get from GPT2, what exactly are they? My understanding is you feed data to GPT2 in an autoregressive fashion and decode with it but, I get 4 vectors for the 4 words/tokens. Is this literally just putting 4 words through the decoder and not masking? To use GPT2, I’d put a word one at a time and see what it generates. But, I’m unsure what these 4 embeddings mean in this case and how you use them. I guess you’d use them the way ELMO (another language model) uses embeddings? I.e., for that, you basically use a weighted sum of all embeddings for each time step …</li>
</ol>
<pre><code class="lang-auto">text = "this is a sentence"
tokenizer_bert = BertTokenizer.from_pretrained('bert-base-uncased')
tokenizer_gpt2 = GPT2Tokenizer.from_pretrained("gpt2")

model_bert = BertModel.from_pretrained("bert-base-uncased")
model_gpt2 = GPT2Model.from_pretrained("gpt2")

# 6 tokens
encoded_input_bert = tokenizer_bert(text, return_tensors='pt')

# 4 tokens
encoded_input_gpt2 = tokenizer_gpt2(text, return_tensors='pt')

output_gpt2 = model_gpt2(**encoded_input_gpt2)
output_bert = model_bert(**encoded_input_bert)
</code></pre>
Why two PyTorch tensors with the same content are serialized to two files that differ?

josipK: <p>If I a question about saving (pickling) PyTorch tensors.</p>
<p>Let’s take a tensor and make a copy. We end with two exactly the same tensors, e.g. like here:</p>
<pre><code class="lang-auto">x = torch.rand(10)
y = torch.zeros_like(x)  
y.copy_(x)

# x and y are the same, you can check with: x.ne(y).sum() == 0
</code></pre>
<p>We then save them:</p>
<pre><code class="lang-auto">torch.save(x, 'x.pt')
torch.save(y, 'y.pt')
</code></pre>
<p>And check if the binary files differ:</p>
<pre><code class="lang-auto">diff x.pt y.pt
Binary files x.pt and y.pt differ
</code></pre>
<p>They do. Why?</p>
<p>NB: If I do exactly the same for NumPy arrays the pickle files don’t differ.</p>

codewithcheese: <p>Over 2 years late, but I tried to replicate this and <code>x.pt</code> and <code>y.pt</code> are identical using Python 3.8.8 and Torch 1.12.1. Using <code>cmp x.pt y.pt</code> and <code>diff x.pt y.pt</code></p>
Trainer.evaluate() >> TypeError: vars() argument must have __dict__ attribute

mystera: <p>When training my model, I get the following error during the evaluation step:</p>
<pre><code class="lang-auto">TypeError: vars() argument must have __dict__ attribute
</code></pre>
<p>I can recreate this error also by simply running trainer.evaluate().  The model appears functional as I can save then reload it and execute inference successfully.  So I’m unsure what would cause the trained.evaluate() to throw this error.</p>
<p>Any recommendations?</p>

mystera: <pre><code class="lang-auto">target_names=["false", "true"] # [0, 1]
employee_count_df['label_as_int'] = (employee_count_df['label'] == True).astype(int)
labels = employee_count_df.label_as_int.values.tolist()
labels = np.array(labels)
full_dataset = employee_count_df.text.values.tolist()

class TenKDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __getitem__(self, idx):
        item = {k: torch.tensor(v[idx]) for k, v in self.encodings.items()}
        item["labels"] = torch.tensor([self.labels[idx]])
        return item

    def __len__(self):
        return len(self.labels)
    
def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    # calculate accuracy using sklearn's function
    acc = accuracy_score(labels, preds)
    return {
      'accuracy': acc,
    }

(train_dataset, validation_dataset,train_labels,validation_labels)=train_test_split(full_dataset, labels, test_size=0.3)

train_encodings = tokenizer(train_dataset, truncation=True, padding=True, max_length=max_length)
train_dataset = TenKDataset(train_encodings, train_labels)

model_directory = "./output"
model=BertForSequenceClassification.from_pretrained(model_name, num_labels=len(target_names))
training_args = TrainingArguments(
    num_train_epochs=3,
    per_device_train_batch_size=8,  
    per_device_eval_batch_size=20,  
    weight_decay=0.01,              
    load_best_model_at_end=True,
    logging_steps=300,
    save_steps=300,
    evaluation_strategy="steps",
    output_dir=model_directory
)

trainer = Trainer(  model=model,
                    args=training_args,
                    train_dataset=train_dataset,
                    eval_dataset=validation_dataset,
                    compute_metrics=compute_metrics,)
trainer.train()
</code></pre>

ptrblck: <p>I assume you are using some higher-level API and the <code>Trainer</code> class comes from HuggingFace?<br>
If so, I guess <a href="https://discuss.huggingface.co/t/error-vars-argument-must-have-dict-attribute-when-trying-to-use-trainer-train/10708">this issue</a> might be related.</p>
Pytorch_lightning Trainer Predict execution time

thanhdat6716: <p>Hi,</p>
<p>I have a model and dataloader with 1 image.</p>
<p>The main function is updating the image by looping then predicting the image.</p>
<pre><code class="lang-auto">import time
# load model and weight
model = ...
if __name__ == "__main__":
    while(True):
        # load image
        start_time = time.time()
        dataloader = process_image_to_dataloader
        trainer.predict(model=model, dataloaders=[dataloader], return_predictions=True)
        print("--- %s seconds ---" % (time.time() - start_time))
</code></pre>
<p>As my experiment, the execution time is around 12s for 1 image prediction.<br>
Another experiment that predicts 10 images, the execution time is also around 12s.<br>
→ That means the number of images is not counting for execution time. (dataloader is not take long time ~0s)</p>
<p>Is there any efficient way to predict 1 image in loop that minimizes the execution time?</p>
<p>Thank you!</p>

eqy: <p>Are the predictions being done on CPU or GPU? If they are being done on GPU, you may want to add <code>torch.cuda.synchronize()</code> before you start and stop timing to ensure that the timing is done in a valid fashion.</p>
<p>In general, there are many ways to improve the performance of inference, you may want to check out <code>torch.jit.script</code> as well as <code>torch.compile</code>:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.jit.script.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.jit.script.html</a></p>
<p><a href="https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html</a></p>
<p>(the latter of which is new PyTorch 2.0 feature)</p>

thanhdat6716: <aside class="quote no-group" data-username="eqy" data-post="2" data-topic="169000">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>torch.cuda.synchronize()</p>
</blockquote>
</aside>
<p>I predict by GPU. I added <code>torch.cuda.synchronize()</code> but the execution time was still the same.<br>
I am using <code>torch==1.12.1+cu116</code>, so <code>torch.compile</code> might not be supported.<br>
Let’s me try to use <code>torch.jit.script</code>, it look quite difficult to me.~</p>