gopikrsmscs/torch-issues huggingface dataset:
torch.nn.functional.scaled_dot_product_attention() : support both attn_mask and is_causal
Labels: triaged, module: multi-headed-attention

### 🚀 The feature, motivation and pitch

It would still be great if `torch.nn.functional.scaled_dot_product_attention()` supported setting both `attn_mask` and `is_causal=True`. In which case it ignores the upper triangular part of `attn_mask` and implicitly assumes it's either set to False or -inf. This would allow us to do some sort of relative positional encoding and causal masking at the same time while saving compute and memory.

### Alternatives

At the moment you set `is_causal=False` and make sure `attn_mask` is inherently causal. That's fine, but we're missing an optimization.

### Additional context

Mentioned in #100709
DISABLED test_noncontiguous_samples_matmul_cuda_float32 (__main__.TestCommonCUDA)
Labels: module: cuda, triaged, module: flaky-tests, skipped

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_noncontiguous_samples_matmul_cuda_float32&suite=TestCommonCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/undefined).

Over the past 3 hours, it has been determined flaky in 13 workflow(s) with 13 failures and 13 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_noncontiguous_samples_matmul_cuda_float32`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_ops.py`

cc @ptrblck
Support pipeline parallelism with PyG
Labels: oncall: distributed, triaged, topic: new features

### 🚀 The feature, motivation and pitch

I would like to request support for pipeline parallelism for data constructed by PyG (PyTorch Geometric). Currently, PyG provides excellent capabilities for GNN modeling and processing, our teams use PyG to construct graph data for the drug field. However, there is a lack of native support for pipeline parallelism, which can significantly improve the performance and scalability of GNN training on large-scale graphs.

However, PyTorch does not support pipeline parallelism through the torch.distributed.pipeline.sync.Pipe modules, these modules are not directly compatible with PyG's graph-specific dataset. As a result, users currently need to implement custom solutions or workarounds to enable pipeline parallelism in PyG, which can be challenging and error-prone, our team has tried, but there have been no results.

Therefore, I kindly request that the PyTorch development team consider adding native support for pipeline parallelism. Such support would greatly benefit the PyG community, enabling faster and more efficient training of GNN models on large-scale graphs.

Thank you for considering this request. Please let me know if there are any questions or if I can provide further assistance in this matter.

### Alternatives

The main reason PyG data cannot be used with Pipeline is the inability to locate and correctly split the tensors within the data when dividing the minibatch into microbatches. PyG's tensors are wrapped within a dictionary-like data structure and are not directly exposed. Our team has attempted two approaches to address this issue.

The first approach involved manually splitting and recombining PyG data, but it was challenging to ensure the preservation of the graph topology within each data instance. The second approach attempted to load a certain number of minibatches (chunks) in dataloder first, combine them, and pass them into the Pipeline for internal microbatch splitting. However, this method encountered issues when invoking checkpoints during subsequent stages.

Regrettably, neither of these methods proved effective. It seems that microbatch requires specific handling of tensors.


### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Investigate random sequence number broadcast initially incorrect
Labels: oncall: distributed, triaged

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/pull/101422 removes an implementation of generating a random sequence number and communicating it via the store, instead opting to start off groups at a sequence number of 0. However, using a random initial sequence number is better for debugging UX as it allows users to easily tell the difference between different collectives corresponding to different subgroups. We should root cause and fix the store issue and re-enable.

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
onnx runtime error
Labels: module: onnx, triaged

### 🐛 Describe the bug

I have already converted my pytorch model to onnx, but I encountered new bugs when converting onnx to engine and using onnxruntime to infer onnx model. 
When I do onnx inference:
`onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Expand node. Name:'/encoder/node_encoder/Expand' Status Message: /encoder/node_encoder/Expand: left operand cannot broadcast on dim 1 LeftShape: {1,6,32}, RightShape: {1,33,32}`

Even if I fix the code concerning the shape error, and convert the model to onnx once more, the onnx inference would report new bugs concerning the shape in following calculations in the model:
`onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Add node. Name:'/aggregator/Add_80' Status Message: /aggregator/Add_80: right operand cannot broadcast on dim 0 LeftShape: {162,15}, RightShape: {152,15}
`
Seems the bug fixing has no end. Is there any efficient way to solve the problem, thank you.


### Versions

1st error related codes in the model:
```
feat_embedding_packed = pack_padded_sequence(feat_embedding_batched, seq_lens_batched, batch_first=True, enforce_sorted=False)

```
2ed error related codes in the model:
```
# Dummy encodings for goal nodes
dummy_enc = torch.zeros_like(node_encodings)
node_encodings = torch.cat((node_encodings, dummy_enc), dim=1)
# Gather node encodings along traversed paths
node_enc_selected = node_encodings[batch_idcs, traversals_batched]
```
[bazel] add inductor to bazel build
Labels: module: build, triaged, module: bazel

### 🚀 The feature, motivation and pitch

We should work towards allowing torch.compile() to be run from the bazel build.
After https://github.com/pytorch/pytorch/issues/101469
The next step would be enabling inductor backend in the bazel build.

### Alternatives

_No response_

### Additional context

_No response_

cc @malfet @seemethere
Regression in NCCL error handling
Labels: oncall: distributed

### 🐛 Describe the bug

There is a regression in NCCL error handling after https://github.com/pytorch/pytorch/pull/97066 on PyTorch master. If one rank of a training job is killed/dies, some ranks of a training job get stuck with this traceback of the workCleanupLoop:

```
Thread 9 (Thread 0x1554539fc700 (LWP 153338)):
#0  0x0000155520c47cdc in ?? () from lib/libcudart.so.11.0
#1  0x0000155520c14bde in ?? () from lib/libcudart.so.11.0
#2  0x0000155520c4a915 in cudaGetLastError () from lib/libcudart.so.11.0
#3  0x00001555097933b5 in c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const () from lib/python3.8/site-packages/torch/lib/libtorch_cuda.so
#4  0x0000155509794008 in c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() () from lib/python3.8/site-packages/torch/lib/libtorch_cuda.so
#5  0x0000155509793a25 in c10d::ProcessGroupNCCL::workCleanupLoop() () from lib/python3.8/site-packages/torch/lib/libtorch_cuda.so
#6  0x000015552081aa93 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#7  0x00001555554f5609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00001555552b4133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
```

Basically the `workCleanupLoop` is stuck on `cudaGetLastError` which is blocked since NCCL kernels are locking up the GPU. As a result, `workCleanupLoop` never gets a chance to `abort` the communicators and recover. This is essentially the potential issue reported in https://github.com/pytorch/pytorch/pull/97066#discussion_r1164692531. I was able to reliably reproduce the issue on a job with 24 ranks and manually killing one rank. After this 9 ranks did not exit and were stuck with the traceback above.

As mentioned in https://github.com/pytorch/pytorch/pull/97066#discussion_r1164692531, we should probably add a separate lightweight thread whose sole responsibility is to abort NCCL kernels that run into errors. 

### Versions

PyTorch: master
CUDA: 12.0
NCCL: 2.17.1

cc @mrshenli @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Do automatic replacement of scaled dot product with fast sdpa implementation in HF models. 
Labels: triaged, module: fsdp, oncall: pt2, module: inductor

#100609 adds sdpa patterns to pattern matcher that are seen in models from HF suite. The status of the benchmarked models is as follows:

 - [x] Albert 
 - [ ] AllenAILongformer broken on master
- [x] Bart 
- [x] Bert 
- [ ] BlenderBot - has a  different pattern
- [x] CamemBert 
- [ ] Deberta custom autograd functions
- [x] DistilBert - works, but will need a different pattern for const folding
- [ ] DistillGPT - has causal mask, hard to match
- [x] Electra - works
- [ ] GPT2 - has causal mask
- [x] LayoutLM 
- [ ] M2M100 has a different pattern
- [ ] MBart has a  different pattern
- [ ] MT5 - non-zero (maybe causal) bias
- [x] Megatron 
- [x] MobileBert 
- [ ] OPT - does additional clipping, different math from _sfdp_params_check
- [ ] PLBart - non-0 attn mask
- [ ] Pegasus - causal mask, can't pattern match
- [x] Roberta
- [ ] T5 - non-zero mask
- [ ] TrOCR - non-zero (possibly causal) mask
- [ ] XGLM - additional clipping, different math
- [ ] XLNet - different formulas
- [x] YituTechConvBert 

However, to make this actually show up in the benchmarks we need to 
1) do constant folding to get rid of all-zero mask that's getting added to attention scores and forces sdpa to go on a slow path cc @zhaojuanmao @mrshenli @rohan-varma @awgu @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire @eellison 
2) figure out either a way to pattern match with lowmem dropout, or run pattern-matching pass without lowmem dropout off, and if no match encountered, switch to  lowmem dropout cc @jancel
3) dynamic shapes present additional complications. During tracing, either memory efficient or flash or slow math implementation will be picked depending on the parameters, and, since (I think) we don't generate any guards for it, it will be used for other sizes that are later encountered, which will either lead to suboptimal performance or even runtime crashes, if fast implementation from trace cannot be used, cc @voznesenskym
RuntimeError: Triton Error [CUDA]: device-side assert triggered when trying torch.compile max-autotune on nanoGPT
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

Tried to run `torch.compile(model, mode='max-autotune')` on nanoGPT, got some cuda errors such as `Triton Error [CUDA]: device-side assert triggered` when doing inference.  I originally found this when experimenting with quantization, but it reproduces without quantization.

Repro script (requires nanoGPT repo): https://gist.github.com/vkuzo/7352cf7b18fd2602668fb3d43ad9467a
Stack trace: https://gist.github.com/vkuzo/57225adfacd7779309fd5444a5f80bcf

### Versions

https://gist.github.com/vkuzo/a5fbfc5905f9bd33732f4f73ec6ce0f4

cc @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305
Enhance FSDP debugability
Labels: oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

Some items we can add under torch_distributed_debug mode to improve debugability of FSDP:

- Shared parameter detection
- Logging when backward hooks are fired (per parameter or per individually wrapped FSDP unit), to debug convergence / parity with DDP issues
- These should also work with the composable API.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
not yet implemented the batching rule for torchaudio::_lfilter
Labels: triaged, module: batching, module: functorch

[W BatchedFallback.cpp:82] Warning: There is a performance drop because we have not yet implemented the batching rule for torchaudio::_lfilter. Please file us an issue on GitHub so that we can prioritize its implementation. (function warnFallback)

cc @zou3519 @Chillee @samdow @kshitij12345 @janeyx99
import functorch.dim monkeypatches torch
Labels: triaged, module: functorch

### 🐛 Describe the bug

Since it's in core now, we can just upstream the necessary changes into the main implementations themselves.

While we still monkeypatch, there is a risk the monkeypatches are not exactly semantics preserving. However, I took a quick look and they seem relatively benign (they only trigger alternate behavior when first class dims are involved.)

cc @zou3519 @Chillee @samdow @kshitij12345 @janeyx99 @zdevito 

### Versions

master
SymIntify first class dims
Labels: topic: not user facing

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #101413
* #101409

To get pybind translation for SymInt, I just use the stock
pybind11 from PyTorch proper.  This is kind of annoying but
since first class dims are now in tree I don't think there is
too much benefit in forcing us to stay this way.

Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Delete old vmap prototype
Labels: triaged, better-engineering, module: functorch

We have an old vmap prototype (basically, all of the `LegacyBatched*` or `LegacyVmap*` files (see image). We should delete it, because it continues to confuse people at a rate of 1-2 people per month.

![image](https://github.com/pytorch/pytorch/assets/5652049/91b2a2ea-7686-47c2-b0f1-90a0eab70773)

## Plan of action

torch.autograd.functional still uses the old vmap prototype. In general we're moving away from torch.autograd.functional, but it still needs vmap. We should attempt to replace torch.autograd.functional's usage of vmap with the new torch.vmap, verify that it passes all tests, and finally delete the old vmap prototype.



cc @Chillee @samdow @soumith @kshitij12345 @janeyx99
problem of compilation for torch2.0
Labels: needs reproduction, module: build, triaged, module: third_party

### 🐛 Describe the bug

Hello, I want to compile torch2.0 with -D_GLIBCXX_USE_CXX11_ABI=1, However, I met some problems.
<img width="755" alt="捕获" src="https://github.com/pytorch/pytorch/assets/45058324/b9ffb0ed-e627-4a8d-a15f-1f9a9ab51bd1">
 How can I solve this? 

### Versions

cuda:11.7
driver version: 470.103.01
gcc: 10.2

cc @malfet @seemethere @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @anijain2305
DataParallel for nested modules
Labels: oncall: distributed

### 🐛 Describe the bug

My module is not on the same devices
`
class ImageEncoder(nn.Module):
    def __init__(self):
        super(ImageEncoder, self).__init__()
        resnet_model = torchvision.models.resnet18()
        self.model = torch.nn.Sequential(*(list(resnet_model.children())[:-1]))

    def forward(self, x):
        print('x', x.device)
        x = x.float()
        print('x.float', x.device)
        print('model', self.model[4][0].conv1.weight.device)
        output = self.model(x).squeeze(-1).squeeze(-1)
        print('output', output.device)
        return output

class VideoEncoder(nn.Module):
    def __init__(self):
        super(VideoEncoder, self).__init__()
        self.model =  ImageEncoder()

    def forward(self, x):
        x = self.model(x.flat(0,1))
        return x

class FinalModel(nn.Module):
    def __init__(self):
        supe(FinalModel, self).__init__()
        self.model = VideoEncoder()
    
    def forward(self, x):
        return self.model(x)


  def data_to_cuda(data):
      for d in data:
          if(type(data[d]) == dict):
              for p in data[d]:
                  data[d][p] = data[d][p].cuda()
          else:
              data[d] = data[d].cuda()
      return data

model = FinalModel()
model= nn.DataParallel(model, device_ids=[0,1])
model = model.cuda()

loss = torch.tensor(0.0).cuda()
losses = []

optimizer = Adam(model.parameters(), lr = 0.0001)

for data in train_data:
    data = data_to_cuda(data)
    predicted_labels = model(data)
    loss_i = compute_loss(data['labels'], predicted_labels)
    loss = loss + loss_i
      
    if(self.steps % backprop_every == 0):
        loss = loss / self.backprop_every
        loss.backward()

        optimizer.step()
        optimizer.zero_grad()
        loss = torch.tensor(0.0).cuda()

        losses.append(loss.item())  
    steps += 1
`

For this I am getting following result:

`
x cuda:0
x cuda:1
x.float cuda:1
model cuda:0
x.float cuda:0
model cuda:0
output cuda:0
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

`

### Versions

pytorch                   1.10.1
gpu                         V100


cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
SparseAdam: working with dense parameters but sparse gradients - usecase 
Labels: module: optimizer, triaged, actionable

### 📚 The doc issue

I have a use case where some gradients are sparse while others are dense, and therefore I'm splitting them into two groups and optimizing them separately using Adam and AdamSparse. 

In the case of the parameter tensor that is dense while the gradient tensor is sparse (many of the values are set to zero as I do not want to update those parameters), which seems like a perfect use case for the SparseAdam optimizer, however, I get into the situation where if I pass my parameters associated with the sparse gradient matrix (when I say sparse here, I mean that many of the values are zero, not that the gradient tensor is converted to a sparse tensor using `torch.sparse_coo_tensor` or `torch.to_sparse`. 

According to the documentation of the SparseAdam optimizer the variant is applied for the following: 

> In this variant, only moments that appear in the gradient get updated, and only those portions of the gradient get applied to the parameters.

And according to the source code, in the `init()` method, sparse parameters are rejected after being checked with the `param.is_sparse`:
`if sparse_params:
      raise ValueError(
                f"Sparse params at indices {sparse_params}: SparseAdam requires dense parameter tensors"
            )
`
So the optimizer only accepts dense parameters. However, it does not accept dense gradients (which is the purpose of the optimizer). But the problem I am having here, is if I am passing my dense parameters with the sparse gradient tensor (many zeros) the optimizer fails at `optimizer.step` because of this check 

` if not p.grad.is_sparse:
       raise RuntimeError('SparseAdam does not support dense gradients, please consider Adam instead')`

A possible work around is using a hook to change the parameter gradient to a sparseTensor using `torch.sparse_coo_tensor`  but then i get the error 

`Exception has occurred: RuntimeError
hook 'hook' has changed the type of value (was torch.cuda.FloatTensor got torch.cuda.sparse.FloatTensor)`

It is unclear from the documentation how this optimizer is supposed to be used or if this use case can be covered by the sparseAdam optimizer. 

What I would think a possibility of this use-case being covered, which might be useful in many cases where parts of the layers/nodes would be updated, but not others, is passing the parameters and their zero gradients to the sparseAdam optimizer. The optimizer converts the gradient to a sparse tensor to enable this use case. 

If there's a workaround, I'd appreciate the help/input. 



### Suggest a potential alternative/fix

How the sparseAdam optimizer is used - which use cases does it cover, and which use cases it does not. The documentation mentions that it's used for sparse tensors, but not much else. It is specifically confusing because the source code has a check for if the parameter dense is dense (which is required) and the gradient tensor is sparse (which is required), but not on how to satisfy this use-case. 

cc @vincentqb @jbschlosser @albanD @janeyx99
Theme update
Labels: module: docs, triaged, module: doc infra

### 📚 The doc issue

To update `pytorch_sphinx_theme` to support new `sphinx>=7.0.0`, I made 2 PRs: 

- https://github.com/pytorch/pytorch_sphinx_theme/pull/180
- https://github.com/pytorch/pytorch_sphinx_theme/pull/181

And there was a very old PR that I submitted but sadly got no attention. It fixes multiple visual bugs and improve visual looking:
- https://github.com/pytorch/pytorch_sphinx_theme/pull/150

### Suggest a potential alternative/fix

_No response_

cc @svekars @carljparker @ezyang @zou3519 @holly1238
Functions for Calculating Skewness and Kurtosis 
Labels: feature, triaged, module: reductions

### 🚀 The feature, motivation and pitch

I was working on a ML project recently involving statistical descriptors. Usually, I try to stick to torch functions(e.g- torch.mean() etc) but I was unable to find anything for skewness and kurtosis. I was looking for something similar to scipy.stats skew and kurtosis functions.

### Alternatives

For now, I have used scipy.stats.skew() and scipy.stats.kurtosis()

### Additional context

_No response_
Terminate handler
Labels: triaged, open source, ciflow/trunk

Fixes #50051.
This PR is based on #50320 and I address the last feedback.
On Windows it is enabled by default. Can be enabled or disabled via USE_CUSTOM_TERMINATE env variable.

This PR adds support for overriding the terminate handler in order to log uncaught exceptions in the threads.
If an exception is thrown and not caught, it will print <Unhandled exception caught in c10/util/AbortHandler.h>
The point of doing this is that in issue #50051, exceptions were thrown but not logged. With this logging system it will be easier to debug it in the future.
Speed when installing from source is very low with CUDA 11
Labels: needs reproduction, module: performance, module: build, module: cuda, triaged

### 🐛 Describe the bug

Hello!

I have a performance issue as I am trying to build PyTorch 1.12 with CUDA 11.3 and CUDNN 8.2 from source.
Although the compilation is successful, running anything, especially with CUDNN is much slower compared to installing from source.

Here is a minimal example:
```

import torch
import time 

layer = torch.nn.BatchNorm2d(64).cuda()
input = torch.randn(4, 64, 112, 112).cuda()
for j in range(100):
   start = time.time()
   for i in range(100):
       output = layer(input)
   print(f"It took {time.time()-start} sec")
```

Option 1: 
* install with: `pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113 `
* time: 5.2 ms


Option 2:
* install from [source](https://github.com/pytorch/pytorch/tree/v1.12.1)
* time: 44 ms


I am wondering why there is so much difference in performance. Thank you!

### Versions

v.1.12.1

cc @malfet @seemethere @ngimel
Docs suggestion `FullyShardedDataParallel.summon_full_params` must be called on all ranks/processes
Labels: triaged, module: fsdp

### 📚 The doc issue

The docs for [summon_full_params](https://pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel.summon_full_params) don't state that it must be called on all processes. I think I was lead astray by the `rank0_only` argument, thinking that I'd get what I need only calling the context manager on the rank 0 process, but instead things hang (at least with `with_grads=True`).

I am guessing under the hood all_gather is being called, which would explain the hanging. But that's not explicit in the docs.

### Suggest a potential alternative/fix

Add a Note to the docs for summon_full_params that it should be called on all ranks, even if parameters are only being used on one process.

cc @zhaojuanmao @mrshenli @rohan-varma @awgu
Operations to shared tensors in the forked process could lead to silent crash
Labels: module: multiprocessing, triaged

### 🐛 Describe the bug

I am using `torch.multiprocessing` with the `fork` context, but the forked processes would mysteriously crash, with no exception thrown. I have minimize the scope of the problem into the following example:

```python
import torch
import torch.multiprocessing as torch_mp

def fn(q):
    tensor = q.get()
    print(tensor, tensor.is_shared())
    tensor += 1 # process silently crashes
    print("OK")

def test():
    ctx = torch_mp.get_context('fork')
    q = ctx.Queue()
    tensor = torch.zeros(1024*1024)
    q.put(tensor)
    proc = ctx.Process(target=fn, args=(q,))
    proc.start()
    proc.join()

if __name__ == "__main__":
    torch_mp.set_start_method('fork')
    test()
```

If, however, the tensor is created with `torch.randn` or `torch.empty`, the program wouldn't crash. `torch.ones, torch.arange` lead to crashes. Using the `spawn`/`forkserver` contexts also works.

Is this the expected behavior with `fork`?

### Versions

Tested with pytorch 2.0.1 under linux.

cc @VitalyFedyunin @ejguan
fused torch.optim.AdamW isn't faster than the unfused version
Labels: module: optimizer, triaged

### 🐛 Describe the bug

When I enabled the fused option of `torch.optim.AdamW` on PyTorch 2.0.1 for my vision model that has a low SM efficiency at the optimizer step, I didn't see any speedup. I profiled the fused version and unfused version, and I could see that although the fused version could improve SM efficiency by fusing multiple small CUDA kernels, the fused `aten::_fused_adamw_` was also launched much later, which actually cancelled out the performance benefit of fused optimizer.

See the attached screenshots of the two trace files:

**1. Original Unfused Version:**
![unfused version](https://github.com/pytorch/pytorch/assets/8884619/8e28e3d1-ff6a-43be-80bf-2b69d4a862c8)

**2. Fused Version in v2.0.1:**
![fused version](https://github.com/pytorch/pytorch/assets/8884619/aef8333a-b0ed-479e-be53-34e03ba5e239)


Will there be any further optimization to handle this issue and deliver more actual speedup?

### Versions

PyTorch 2.0.1

cc @vincentqb @jbschlosser @albanD @janeyx99
Support fake tensor real inputs in dynamo
Labels: triaged, module: dynamo

### 🐛 Describe the bug

Today, Dynamo assumes that if you have a fake tensor, it was was generated by fakeification in the compiler. But fake tensor is also a user visible concept, and you could for example call a torch.compile'd function with fake tensor inputs. We should be able to handle fake tensor inputs, distinguishing them from internal fake tensor inputs by checking which fake mode they're associated with (torch.compile will always allocate a fresh fake mode).

Some basic functionality works; for example, it works to `FakeTensorMode.from_tensor` a fake tensor (however, if you had a Parameter+fake tensor, the parameter-ness is lost; that needs to be fixed.)

Here are at least some of the spots where we make incorrect assumptions.

```
diff --git a/torch/_dynamo/variables/builder.py b/torch/_dynamo/variables/builder.py
index 1d7c33d50c2..64523f3e610 100644
--- a/torch/_dynamo/variables/builder.py
+++ b/torch/_dynamo/variables/builder.py
@@ -154,8 +154,6 @@ class GraphArg:
             assert isinstance(
                 self.fake_tensor, torch._subclasses.fake_tensor.FakeTensor
             )
-        if isinstance(self.example, torch._subclasses.fake_tensor.FakeTensor):
-            raise AssertionError("Fake Tensor observed in TorchDynamo Fx graph inputs")
 
     def load(self, tx):
         return self.source.reconstruct(tx)
@@ -796,7 +794,7 @@ class VariableBuilder:
             # a later point in time.
             ignore_subclass = True
         else:
-            assert type(value) in (torch.Tensor, torch.nn.Parameter), type(value)
+            assert type(value) in (torch.Tensor, torch.nn.Parameter, FakeTensor), type(value)
             ignore_subclass = False
 
         is_duplicate_tensor = source in self.tx.output.input_source_to_var
@@ -1024,10 +1022,6 @@ def wrap_fx_proxy_cls(
         if example_value is None:
             example_value = get_fake_value(proxy.node, tx)
 
-        # Handle recursive calls here
-        elif isinstance(example_value, FakeTensor):
-            pass
-
         elif isinstance(example_value, torch.Tensor):
             if tx.export:
                 # The legacy behavior for real value cache with subclasses was
@@ -1200,7 +1194,7 @@ class TrackedFake:
 def wrap_to_fake_tensor_and_record(
     e, tx, ignore_subclass=False, *, source: Optional[Source], is_tensor: bool
 ):
-    if type(e) in (torch.Tensor, torch.nn.Parameter) or (
+    if type(e) in (torch.Tensor, torch.nn.Parameter, FakeTensor) or (
         ignore_subclass and isinstance(e, torch.Tensor)
     ):
         assert source is not None
```

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire @thiagocrepaldi 

### Versions

master
Should be ok to call _dynamo.export and torch.compile under FakeTensorMode
Labels: triaged, module: dynamo

### 🐛 Describe the bug

This will require either: (1) ensuring that the eval frame handler disables fake tensor mode before doing anything serious, or (2) we refactor all of Dynamo to work if there is a fake tensor mode or not. Probably (1) is easiest.

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire @thiagocrepaldi 

### Versions

master
IPEX as TorchDynamo Backend Performance Dashboard
Labels: triaged, intel, oncall: pt2

Dashboard to track the performance of IPEX as TorchDynamo backend on CPU.

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @anijain2305
```[tasklist]
### Tasks
```
torch._dynamo.exc.UserError: Dynamic control flow is not supported at the moment.
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

 ERROR：torch._dynamo.exc.UserError: Dynamic control flow is not supported at the moment, Please use functorch.experimental.control_flow.cond to explicitly capture the control flow

i am trying to use a torch(2.1.0.dev20230510+cu118) dynamo export a function with control flow. The relevant code is as follows，interesting thing is： when the control condition is “if b.dim < 0:”， it works fine； if the control condition is “if b.sum() < 0:”， the error comes as  " torch._dynamo.exc.UserError: Dynamic control flow is not supported at the moment, Please use functorch.experimental.control_flow.cond to explicitly capture the control flow "。（The relevant situation is basically consistent with the description in the link ：https://pytorch.org/functorch/nightly/ux_limitations.html）

when i try to find how to use functorch.experimental.control_flow.cond,i found the document is empty。how can i get 
instructions for  this function？

~~~
from typing import List
import torch
import torch._dynamo as dynamo

def my_compiler(gm: torch.fx.GraphModule, example_inputs: List[torch.tensor]):
    print("my_compiler() called with FX graph:")
    gm.graph.print_tabular()
    return gm.forward

@dynamo.optimize(my_compiler)
def foo(a, b):
    x = a/(torch.abs(a)+1)
    if b.sum() < 0:
    # if b.dim < 0:
        b = b*-1
    else:
        b = b-1
    return x*b

graph, guards=torch._dynamo.export(foo,torch.rand(10),torch.rand(10))
print("the output is:---------------------------------------------")
print(guards) 
~~~
<img width="988" alt="image" src="https://github.com/pytorch/pytorch/assets/51430740/2cb9a50c-593a-4827-bc28-70307e83d912">


### Versions

torch 2.1.0 20230510;cuda 11.8

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @anijain2305
Tool for identifying where in eager model an operation is nondeterministic
Labels: triaged, module: determinism

### 🐛 Describe the bug

Let's say you have a model code and when you run it twice you get bitwise different results. Where did it diverge? We can use TorchFunctionMode/TorchDispatchMode to localize where the first divergence occurred.

### Versions

master

cc @mruberry @kurtamohler
GLCM implementation in pytorch C++ api and cuda
Labels: module: cpp, triaged

### 🚀 The feature, motivation and pitch

The Gray Level Co-occurrence Matrix GLCM https://en.wikipedia.org/wiki/Co-occurrence_matrix is heavily used to calculate a variety of texture features called haralick features, for many bio-medical imaging applications. Currently there is no GPU implementation available using pytorch. However, skimage and pyradiomics have CPU implementations available that are very slow. 

I have very elementary level understanding of Cuda, C++, and Torch C++ API. However, as you know, a proper documentation for Torch C++ is really lacking. I would your help to write these in Torch C++ API and Cuda. Thank you.

### Alternatives

Here are two CPU implementations available in Cpython or C.
calculate_glcm in:
https://github.com/AIM-Harvard/pyradiomics/blob/master/radiomics/src/cmatrices.c

GLCM Loop in:
https://github.com/scikit-image/scikit-image/blob/v0.20.0/skimage/feature/_texture.pyx

There is also the following paper, that has a simple comparison of implementation of it on CPU and GPU:
Computation of Gray Level Co-Occurrence Matrix Based on CUDA and Optimization for Medical Computer Vision Application

### Additional context

_No response_

cc @jbschlosser
Migrate windows runners to non-ephemeral instances
Labels: module: ci, triaged

### 🚀 The feature, motivation and pitch

Due to increased pressure over our windows runners, and the elevated cost of instantiating and bringing down those instances, we want to migrate instances from ephemeral to not ephemeral.

Possible impacts are related to breakages in or misbehaves on CI jobs that puts the runners in a bad state. Other possible impacts are related to exhaustion of resources, especially disk space, but memory might be a contender, as CI trash piles up on those instances.

As a somewhat middle of the road approach to this, currently nonephemeral instances are stochastically rotated as older instances get higher priority to be terminated when demand is lower.

Instances definition can be found here: https://github.com/pytorch/test-infra/pull/4072

* ✅ migrate `windows.4xlarge` to `windows.4xlarge.nonephemeral` instances under `pytorch/pytorch` (#100377)
* 📣 migrate `windows.8xlarge.nvidia.gpu` to `windows.8xlarge.nvidia.gpu.nonephemeral` instances under `pytorch/pytorch` (#104404)
* ⏳ submit PRs to all repositories under `pytorch/` organization to migrate `windows.4xlarge` to `windows.4xlarge.nonephemeral`
* ⏳ submit PRs to all repositories under `pytorch/` organization to migrate `windows.8xlarge.nvidia.gpu` to `windows.8xlarge.nvidia.gpu.nonephemeral`
* ⏳ terminate the existence of `windows.4xlarge` and `windows.8xlarge.nvidia.gpu`
* ⏳ evaluate and start the work related to the adoption of `windows.g5.4xlarge.nvidia.gpu` to replace `windows.8xlarge.nvidia.gpu.nonephemeral` in other repositories and use cases (proposed by @huydhn)

The reasoning for this phased approach is to reduce the scope of possible contenders to investigate in case of misbehave of particular CI jobs.

### Alternatives

_No response_

### Additional context

_No response_

cc @seemethere @malfet @pytorch/pytorch-dev-infra
AOTAutograd export path does not support training graphs with parameters that do not receive gradients.
Labels: triaged, module: aotdispatch

See the comment [here](https://github.com/pytorch/pytorch/pull/100587/files#diff-df954bbf954d2dcb81f687876053267ffa4ddb36ed86b7d2bd76319ff2b94416R3786): today, the AOTAutograd export path expects that every parameter in the module you export will receive some sort of gradient. It's possible to create a case where this doesn't happen. We can support it, but I'd like to gauge how much request there is for this first, since it's more of an edge case.

Example:

```
def f(p1, p2):
    # p1 will never receive any gradients. p2 will though
    return p1.detach() * p2
```
custom_op API follow-ups
Labels: triaged

@soulitzer, @bdhirsh, @ezyang if I missed any:

Tracking here so I don't need to keep all of them in my head
- [x] test symint works with Sequence[int]
- [x] FakeTensor notimplemented error message
- [x] Stop using python RAII (in general); prefer context-managers.
- [ ] Use the autograd notimplemented fallback
- [x] autograd API
- [ ] way to mark operator as non-differentiable
- [ ] ctx.needs_input_grad
- [ ] custom_op should not graph break on dynamo; dynamo safety rules for custom_op
- [ ] (maybe) CustomOp.impl to specify device-agnostic impl
- [ ] (maybe) torch.compile-like decorator
- [ ] (maybe) ban kwarg-only args
- [ ] (maybe) save_pytree_for_backward doesn't need to run if grad_mode is False
- [ ] Mechanism to wrap an existing custom operator with a CustomOp object to use the new APIs
- [ ] Triton kernels: see if we need to do any special handling. Ideally we would generate a cpp_wrapper (using AOTInductor) for it to lower overhead while running under torch.compile.
- [ ] Finish and ship the [usage documentation](https://docs.google.com/document/d/16YsyKY5uil_HF5mwa6UOtw0AxEHJkZp06NZcSJ3GyJs/edit#heading=h.deyt6pleeqxt)
- [ ] custom_op and determinism (likely needs some way to add the tag)
- [ ] custom_op and randomness (need to ban, likely through testing)
- [ ] custom_op and cuda graphs (Elias tells me not all custom ops work with cuda graphs)

Testing
- [ ] Test that under tracing, the operator has the same view+inplace semantics
- [ ] Test that under AOTAutograd, view+inplace edge cases work out.
`dense -> sparse compressed` to work with empty batches.
Labels: module: sparse, triaged

### 🚀 The feature, motivation and pitch

Currently, it is not possible to convert an empy-batched dense tensor to a sparse compressed tensor.
```python
In [1]: import torch

In [2]: torch.rand(0, 4, 4).to_sparse_bsr(2)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 torch.rand(0, 4, 4).to_sparse_bsr(2)

RuntimeError: to_sparse_bsr: Expected product of batch dimensions to be non-zero.

In [3]: torch.rand(0, 4, 4).to_sparse_csr()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 torch.rand(0, 4, 4).to_sparse_csr()

RuntimeError: to_sparse_csr: Expected product of batch dimensions to be non-zero.
```

### Alternatives

_No response_

### Additional context

_No response_

cc @alexsamardzic @pearu @cpuhrsch @amjames @bhosmer
Fork run CI from upstream remote (more than 10,000 emails) 
Labels: module: ci, triaged

### 🐛 Describe the bug

Fork run CI from upstream remote

### Versions

![image](https://github.com/pytorch/pytorch/assets/1328316/45e7f6af-434f-4e4b-b609-c2c5af3554de)


cc @seemethere @malfet @pytorch/pytorch-dev-infra
round float16 calculation error in mps backend
Labels: triaged, module: mps

### 🐛 Describe the bug

import torch
x = torch.tensor([8.5], dtype=torch.half).to("mps")
y = torch.round(x)

y is 9 but should be 8 (According to the rules of "round half to even" in the [doc](https://pytorch.org/docs/stable/generated/torch.round.html?highlight=torch+round#torch.round))

### Versions

This issue was found in pre-ci, and I don't have a mac to reproduce this bug. For details please see https://github.com/pytorch/pytorch/actions/runs/4944088105/jobs/8839307876

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
[Dynamo] TB hf_Reformer graph breaks
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Repro:
```
import torch
import logging
import sys
import torch._dynamo

# torch._logging.set_logs(dynamo=logging.DEBUG, bytecode=True)
torch._dynamo.config.print_graph_breaks = True

import torch.nn as nn
import torch.nn.functional as F

class MyModel(nn.Module):

    def __init__(self):
        super(MyModel, self).__init__()
        self.linear = torch.nn.Linear(5, 5)
        self.dropout = torch.nn.Dropout()

    def _init_attention_seed(self):
        """
        This function sets a new seed for the attention layer to make dropout deterministic for both forward calls: 1
        normal forward call and 1 forward call in backward to recalculate activations.
        """

        # randomize seeds
        # use cuda generator if available
        if hasattr(torch.cuda, "default_generators") and len(torch.cuda.default_generators) > 0:
            # GPU
            device_idx = torch.cuda.current_device()
            self.attention_seed = torch.cuda.default_generators[device_idx].seed()
        else:
            # CPU
            self.attention_seed = int(torch.seed() % sys.maxsize)

        torch.manual_seed(self.attention_seed)

    def forward(self, x):
        self._init_attention_seed()
        return self.dropout(self.linear(x))

x = torch.randn(5, 5)

m = MyModel()
print(m(x))

opt_m = torch.compile(backend="eager")(m)
print(opt_m(x))
```
There are several graph breaks:
```
[2023-05-11 04:12:58,513] torch._dynamo.symbolic_convert: [WARNING] Graph break: hasattr: TorchVariable(<module 'torch.cuda' from '/scratch/ybliang/work/repos/pytorch/torch/cuda/__init__.py'>) from user code at   File "/scratch/ybliang/work/repos/debug/debug3.py", line 39, in forward
    self._init_attention_seed()
  File "/scratch/ybliang/work/repos/debug/debug3.py", line 28, in _init_attention_seed
    if hasattr(torch.cuda, "default_generators") and len(torch.cuda.default_generators) > 0:

[2023-05-11 04:12:58,748] torch._dynamo.symbolic_convert: [WARNING] Graph break: inlining disallowed: <function current_device at 0x7f2ec26d8430> from user code at   File "/scratch/ybliang/work/repos/debug/debug3.py", line 30, in <resume in _init_attention_seed>
    device_idx = torch.cuda.current_device()

[2023-05-11 04:12:58,754] torch._dynamo.symbolic_convert: [WARNING] Graph break: call_method UserDefinedObjectVariable(seed) __call__ [] {} from user code at   File "/scratch/ybliang/work/repos/debug/debug3.py", line 31, in <resume in _init_attention_seed>
    self.attention_seed = torch.cuda.default_generators[device_idx].seed()
```

### Versions

N/A

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
[ONNX] OnnxFunction of aten_index_put_bool operation isn't consistent to aten::index_put inx FX exporter
Labels: module: onnx, triaged, onnx-triaged

### 🐛 Describe the bug

Need below in `test_fx_op_consistency.py`

```python
    xfail(
        "index_put",
        matcher=lambda sample: (sample.args[0][0].dtype == torch.bool) and (sample.kwargs.get("accumulate") == False),
        reason=onnx_test_common.reason_dynamo_does_not_support("Unknown reason!"),
    ),
```

However, the ops_test.py of this OnnxFunction is actually passed.



cc @justinchuby 

### Versions

Nightly
Pytorch compile failure on Windows with CUDA 12.1 because of lacking NVTX component
Labels: module: build, module: windows, triaged

### 🐛 Describe the bug

**Procedure to reproduce:**
1. For a new windows system
2. Install Miniconda and create a python=3.11 environment
3. Install cuda 12.1 with all components
4. git clone pytorch and run `python setup.py develop`
5. It fails.

**Reason:** There is no longer NVTX component in cuda 12.1, while pytorch assumes it installed at `C:\Program Files\NVIDIA Corporation\NvToolsExt`.

**Solution:** Open CUDA 11.8 installer and install NVTX component.

**Expected Behavior:** Pytorch shall be compiled with only CUDA 12.1. We don't need to install an extra NVTX component in CUDA 11.8.

**Error Log:**
```
Building wheel torch-2.1.0a0+git4052741
-- Building version 2.1.0a0+git4052741
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=D:\workspace\pytorch\torch -DCMAKE_PREFIX_PATH=C:\Users\ain-s\miniconda3\Lib\site-packages -DJAVA_HOME=C:\Program Files\Java\jdk1.8.0_301 -DNUMPY_INCLUDE_DIR=C:\Users\ain-s\miniconda3\lib\site-packages\numpy\core\include -DPYTHON_EXECUTABLE=C:\Users\ain-s\miniconda3\python.exe -DPYTHON_INCLUDE_DIR=C:\Users\ain-s\miniconda3\Include -DPYTHON_LIBRARY=C:\Users\ain-s\miniconda3/libs/python310.lib -DTORCH_BUILD_VERSION=2.1.0a0+git4052741 -DUSE_NUMPY=True D:\workspace\pytorch
CMake Warning at CMakeLists.txt:368 (message):
  TensorPipe cannot be used on Windows.  Set it to OFF


-- Current compiler supports avx2 extension. Will build perfkernels.
-- Current compiler supports avx512f extension. Will build fbgemm.
CMake Error at cmake/public/cuda.cmake:69 (message):
  Failed to find nvToolsExt
Call Stack (most recent call first):
  cmake/Dependencies.cmake:43 (include)
  CMakeLists.txt:710 (include)


-- Configuring incomplete, errors occurred!
```

### Versions

I've installed successfully. If you do need the script output in error case, I could uninstall NVTX and try it days later.

cc @malfet @seemethere @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht