gopikrsmscs/torch-issues huggingface dataset:
Clean up PyTorch's private operators
Labels: module: internals, triaged

### 🚀 The feature, motivation and pitch

There are some PyTorch operators that are private (i.e. prefaced with an underscore) that show up in the dispatcher. These operators are problematic, as 1. their semantics are completely undocumented, 2. it provides confusion to folks (including backends) about whether they should to be supported, and 3. they just look ugly/confusing.

Luckily, there are not that many private operators that show up in practice. Many of them are things like for_each operators, .In the OpInfo tests, we only run into 25 of them.

So, what's the goal here? Well, we don't want folks working with the PyTorch operator set (i.e. transforms or compiler backends) to need to worry about these private operators.

So, there's 3 main ways to handle these.

1. Kill them, and use other operators in their place. For example, @ngimel did this with `aten::_s_where` in https://github.com/pytorch/pytorch/pull/73468. Sometimes operators are split for not particularly great reasons.
2. Promote them to "proper" APIs. For example, `_fft_r2c`, `_fft_c2r`, and `_fft_c2c` are generic FFT APIs that actually cover all of PyTorch's FFT APIs! In this case, we should probably just expose these to users, perhaps as part of PrimTorch.
3. Decompose them. Some other private operators may need to exist for a reason - perhaps they map to an efficient kernel or need to be there for autograd reasons. In these cases, we can provide decompositions so that backends that don't want to deal with them no longer need to.

Removable:
- [ ] _log_softmax (https://github.com/pytorch/pytorch/issues/76433)
- [ ] _softmax (https://github.com/pytorch/pytorch/issues/76433)
- [ ] _log_softmax_backward_data (https://github.com/pytorch/pytorch/issues/76433)
- [ ] _softmax_backward_data (https://github.com/pytorch/pytorch/issues/76433)
- [ ] _unique2 (previous attempt: https://github.com/pytorch/pytorch/pull/18655)


Decomposable:
- [ ] _euclidean_dist
- [ ] _cdist_forward
- [ ] _trilinear

Promotable:
- [ ] _local_scalar_dense
- [ ] _fft_r2c
- [ ] _fft_c2r
- [ ] _fft_c2c
- [ ] _ctc_loss_backward
- [ ] _ctc_loss

Not sure: 
- [ ] _conj
- [ ] _conj_physical
- [ ] _embedding_bag
- [ ] _reshape_alias
- [ ] _unsafe_view
- [ ] _to_copy
- [ ] _linalg_check_errors
- [ ] _linalg_svd
- [ ] _embedding_bag_dense_backward
- [ ] _embedding_bag_per_sample_weights_backward
- [ ] _cdist_backward

cc: @ngimel, @ezyang 

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh
Make Torch FX function `_torchscript_type_to_python_type` public
Labels: oncall: jit, module: fx

### 🚀 The feature, motivation and pitch

Could we make the Torch FX function `from torch.fx.operator_schemas import _torchscript_type_to_python_type` public? 
This allows third party ML pipeline libraries to get arguments of `ScriptModule` objects.


### Alternatives

We would like to get the same Python level argument type that is available with

```
import inspect

arg_specs = inspect.getfullargspec(mod.forward)
```

Which returns a `FullArgSpec` object with an `annotations` field that maps to regular Python classes like `torch.Tensor`.


Unfortunately, for a given TorchScript module `script_module`,  the traditional outputs from the call `script_module.forward.schema` are not simple Python types but instead, return classes from the `torch._C.*` namespace.

This request is so that we can have a supported function that can be used to converted those objects to actual Python types.


### Additional context

The current usage is

```
from torch.fx.operator_schemas import _torchscript_type_to_python_type

_torchscript_type_to_python_type(script_module.forward.schema.arguments[1].type)
```
`Tensor.register_hook()` Source Link Broken
Labels: module: docs, triaged

### 📚 The doc issue

https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html
Clicking the "source" link redirects to https://pytorch.org/docs/stable/_modules/torch/_tensor.html#Tensor.register_hook, which does not exist.

This is true in the master docs as well: https://pytorch.org/docs/master/generated/torch.Tensor.register_hook.html

### Suggest a potential alternative/fix

Fix the link.

cc @brianjo @mruberry
[NVFuser] Automated generation of microbenchmarks
Labels: triaged, module: nvfuser

### 🚀 The feature, motivation and pitch

The NVFuser microbenchmarks in torchbench (https://github.com/pytorch/benchmark/pull/793) provide a good way to monitor nvfuser performance. They are generated by collecting the fusion groups from a torchbench run - which gives a closer look at nvfuser performance for each of the fusion groups. However, they are not updated when nvfuser or torchbench change.

Things that would change the microbenchmarks:
* changes to classify_graphs.py
* nvfuser changes that modify the fusion groups that get captured
* torchbench updates

Things that need to be automated:
* [ ] codify the torchbench command (& any modifications, if necessary) that is used to generate the microbenchmarks. The original run was done with lazy tensor (which should generate larger fusion groups than JIT) & nvfuser.
* [ ] write a tool that merges the existing microbenchmarks in torchbench with new IR graphs generated by a more recent run of torchbench (i.e. we don't want to lose old microbenchmarks or microbenchmarks that were given more descriptive names, especially if any of them were manually generated)
* [ ] codify the pipeline of (run torchbench with flags) -> (log_extract) -> (classify_graphs.py) -> (merge with existing graphs) -> (update torchbench IR)
* [ ] update classify_graphs.py to support more types of ops. It maps (op) -> (general type of op), e.g. pointwise, reduction, view. "would be trivial to expand this with unary ops at least, potential reduction via the classifications in shape analysis" -- @eellison 
* [ ] (optional) pipeline for automatically running the updates, so that a human doesn't need to remember to run the script in order to 

### Alternatives

_No response_

### Additional context

_No response_
`F.interpolate` uses incorrect size when `align_corners=True`
Labels: triaged, module: interpolation

### 🐛 Describe the bug

Currently for given `scale_factor`  `F.interpolate` uses the same formula to compute output size for both modes of `align_corners`.
For mode `align_corners=False` it's fine, but for `align_corners=True` it's incorrect. as it's makes actual scale factor (defined as scale of image features) dependant from input image size.

Current formula has [off-by-one error](https://en.wikipedia.org/wiki/Off-by-one_error).
For example, for image size of 4 there are 3 (4-1) sections in between pixels. With `scale_factor=2` each section will be populated with 1 new pixel. So correct new size will be 7.
Or, in general, `new_size = (size - 1) * scale_factor + 1`.

Given [this](http://entropymine.com/imageworsener/matching/) `align_corners=False` matches the "match edges" model, and `align_corners=True` - the "match centers" model, but currently scale factor controls scale of image dimensions, not of image features.

This is current behavior:
```
>>> x = torch.rand(1, 1, 4, 4)
>>> y = F.interpolate(x, scale_factor=2, mode='bilinear', align_corners=True)
>>> y.shape
(1, 1, 8, 8)  # incorrect
>>> torch.allclose(x, y[:, :, ::2, ::2])
False
```
Example of correct behavior:
```
>>> y = F.interpolate(x, (7, 7), mode='bilinear', align_corners=True)
>>> torch.allclose(x, y[:, :, ::2, ::2])
True
```

### Versions

Any currently available pytorch version.
Expand pow and float_pow sampling function for more coverage
Labels: module: tests, triaged

### 🐛 Describe the bug

This issue is linked to: https://github.com/pytorch/pytorch/issues/74279
with matching PR: #76825

After looking into gradient computations for the pow and float pow the nans arise when lhs or base tensor has values that are negative.  Since the exponent or rhs tensor has fractional values for the opinfo test the power function can and does map floats -> complex  values in the analytical case. An example of nans in the function.
```
import torch
def rand_lh(shape, low=0, high=1):
    rand_val = torch.rand(shape)
    result = high * rand_val + low * (1 - rand_val)
    return result

a = rand_lh((1,2,3,4), low = -1 )
power = torch.rand((1,2,3,4))

print(torch.pow(a, power))
```

<img width="430" alt="Screen Shot 2022-04-27 at 1 51 35 PM" src="https://user-images.githubusercontent.com/32754868/165628746-15e8aa4f-663b-4db2-ac2a-92f0e0c855ca.png">

In order to unskip TestGradients the lhs low value can be set to 0 so that this tensor doesn't include any negative values.   Follow up work would be maybe adding documentation that includes this note. Or updating pow to promote the output tensor to complex type. 

### Versions

Collecting environment information...
PyTorch version: 1.12.0a0+git2469525
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.3.1 (arm64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.3)
CMake version: version 3.22.1
Libc version: N/A

Python version: 3.10.4 (main, Mar 31 2022, 03:37:37) [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-12.3.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.942
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] torch==1.12.0a0+git2469525
[conda] mypy                      0.942                    pypi_0    pypi
[conda] mypy_extensions           0.4.3           py310hca03da5_1
[conda] nomkl                     3.0                           0
[conda] numpy                     1.21.2          py310hb38b75b_0
[conda] numpy-base                1.21.2          py310h6269429_0
[conda] torch                     1.12.0a0+git2469525           dev_0    <develop>

cc @mruberry
Pyre type checking fails 
Labels: module: typing, triaged

### 🐛 Describe the bug

Pyre type checker fails on PyTorch when running any of the following:

```
import torch

torch.device('cpu')
torch.distributed.all_gather(...)
# other distributed methods
```

Running pyre outputs something like the following:

```
utilities/device.py:39:17 Undefined attribute [16]: Module `torch` has no attribute `device`.
utilities/distributed.py:24:4 Undefined attribute [16]: Module `torch.distributed` has no attribute `all_gather`.
utilities/distributed.py:45:51 Undefined attribute [16]: Module `torch.distributed` has no attribute `is_initialized`.
utilities/distributed.py:50:16 Undefined attribute [16]: Module `torch.distributed` has no attribute `group`.
utilities/distributed.py:55:17 Undefined attribute [16]: Module `torch.distributed` has no attribute `get_world_size`.
utilities/distributed.py:56:4 Undefined attribute [16]: Module `torch.distributed` has no attribute `barrier`.
utilities/distributed.py:65:4 Undefined attribute [16]: Module `torch.distributed` has no attribute `all_gather`.
utilities/distributed.py:81:4 Undefined attribute [16]: Module `torch.distributed` has no attribute `all_gather`.
```


### Versions

Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.3.1 (arm64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.3)
CMake version: version 3.22.4
Libc version: N/A

Python version: 3.8.11 (default, Jul 29 2021, 14:57:32)  [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-12.3.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.812
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.3
[pip3] torch==1.11.0
[pip3] torchmetrics==0.8.0
[pip3] torchrec==0.1.0
[pip3] torchx-nightly==2022.4.26
[conda] mypy                      0.812                    pypi_0    pypi
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.22.3                   pypi_0    pypi
[conda] torch                     1.11.0                   pypi_0    pypi
[conda] torchmetrics              0.8.0                    pypi_0    pypi
[conda] torchrec                  0.1.0                    pypi_0    pypi
[conda] torchx-nightly            2022.4.26                pypi_0    pypi

cc @ezyang @malfet @rgommers @xuzhao9 @gramster
Add post-AccumulateGrad hook as a nice public API
Labels: feature, module: autograd, triaged, has workaround, actionable

This is used by DDP to trigger synchronization of the gradients and we also have other users that want the optimizer to run alongside the backward pass.

Today a workaround is doing `t.weight.view_as(t.weight).grad_fn.next_functions[0][0].register_hook(my_hook)` but that is pretty shady.



cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
[checkpoint] Extension hooks to support logging and telemetry
Labels: triaged, sharded_tensor

### 🚀 The feature, motivation and pitch

The current load/saving code, in particular the FileSystem one, doesn't have any hooks to enable telemetry or logging of its activity.

Given how fragile and time consuming checkpointing is, this is an important feature to support production usage of this API.

### Alternatives

_No response_

### Additional context

_No response_
Enhance _verify_param_shape_across_processes
Labels: high priority, triage review, oncall: distributed, better-engineering, pt_distributed_rampup

### 🚀 The feature, motivation and pitch

This check in reducer.cpp serves the purpose of checking the parameter shape (and # of params) across processes, although, when there is an issue, the issue is only reported on non-0 ranks and rank 0 must be taken down through either async error handling or nccl_blocking_wait.

We should enhance this to make rank 0 fail more quickly with an appropriate error message as well. This will enhance debugability for FSDP and DDP. 

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
[checkpoint] Switch away from pickle-base serialization
Labels: triaged, sharded_tensor

### 🚀 The feature, motivation and pitch

We currently use pickle for serializing everything.
Beyond efficiency issues, it makes backwards compatibility quite hard.

While have the default implementation pickle based is a reasonable default (it has worked for torch.save) we should make the interface plugable to other implementations.

### Alternatives

_No response_

### Additional context

_No response_
[NHWC] Extend the lowering function to support explicitly defining the output strides
Labels: NNC

Currently, we always convert the input and output tensor to contiguous regardless of channles-last. So the lowering function does not need the output stride information because it is always contiguous. But the situation will be changed as long as NNC supports channels-last without converting to contiguous. Hence, the lowering function needs to create the output `Buf` according to the stride information. That's why we need to extend the lowering function and pass the output stride information to it.

Before:
```c++
using NNCLoweringFunction = std::function<Tensor(
    const std::vector<ArgValue>& inputs,
    const std::vector<ExprHandle>& outputShape,
    const c10::optional<ScalarType>& outputType,
    at::Device device)>;
```
After:
```c++
using NNCLoweringFunction = std::function<Tensor(
    const std::vector<ArgValue>& inputs,
    const std::vector<ExprHandle>& outputShape,
    const std::vector<ExprHandle>& outputStrides,
    const c10::optional<ScalarType>& outputType,
    at::Device device)>;
```
Remove _log_softmax/_softmax in favor of log_softmax and softmax respectively.
Labels: module: internals, triaged

### 🚀 The feature, motivation and pitch

Similar rationale to https://github.com/pytorch/pytorch/issues/73135

The only difference between `_softmax` and `softmax` is that `softmax` takes an optional dtype argument, while `_softmax` takes a `half_to_float` argument. However, this `half_to_float` arg is not needed in the backwards pass, and even if it were, we could simply reconstruct it from the argument dtypes.

https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/SoftMax.cpp#L427

So, to remove these, we can simply turn the current `_softmax` and `_log_softmax` into C++ functions (not operators), and mostly preserve the same logic as today. The backwards pass would need to be modified slightly to take account of the cast.

cc: @albanD  @zou3519 @ngimel @ezy

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh
TorchFunction handling and overload resolution very slow in `torch.ops`
Labels: oncall: jit

### 🐛 Describe the bug

This was noticed while investigating a perf hit after using `torchbind` custom classes in the profiler api, see https://github.com/pytorch/pytorch/pull/72302#issuecomment-1107887355

We get a 30x slowdown on `_record_function_exit` after switching to the second of two overloads, changing from `Tensor` to `_RecordFunction` respectively. Tensor is much faster both because it's the first overload is the set, and fast-tracks `__torch_function__` checks by virtue of the `THPVariable_CheckTypeExact` special case in `check_has_torch_function`.

```python
import torch
record = torch.ops.profiler._record_function_enter_new('test')

# First call normally
%timeit torch.ops.profiler._record_function_exit(record)
# 30.3 µs ± 760 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

with torch._C.DisableTorchFunction():
    # Now remove calls to `has_torch_function_attr`
    %timeit torch.ops.profiler._record_function_exit(record)
    # 12.1 µs ± 115 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

    # Now also avoid overload resolution
    %timeit torch.ops.profiler._record_function_exit._RecordFunction(record)
    # 2.68 µs ± 5.87 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```

### Versions

2291960d3f0ea0f7c4e99e955bc9a23783ef70a3
[checkpoint] SPMD distributed checkpoint coordination
Labels: oncall: distributed, sharded_tensor

### 🚀 The feature, motivation and pitch

The current implementation provides no coordination around completion or failure.

A call to `load_state_dict` or `save_state_dict` should complete either when: 

- the global checkpoint has successfully being written and sync'd to storage.
- one or more ranks ran into an error

It should be possible to know if a checkpoint ran to completion by only looking at the stored objects. As an initial approach, I suggest we follow the following sequence:

- all ranks write their data and fsync it (or equivalent)
- all_gather_object write results (stop here in case of failure)
- rank 0 write metadata and fsync it (or equivalent)
- rank 0 scatter result of metadata write results

The last collective should be enough to provide the required coordination among ranks.



### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
Error in DistributedDataParallel with 'CPU' device
Labels: oncall: distributed

I'm using DistributedDataParallel with 'CPU' device.

Although I'm setting device_ids=None, output_device=None in model = DistributedDataParallel(model, device_ids=None, output_device=None) 

Below error:
File "/opt/mnist/src/distributeddataparallel.py", line 142, in main
model = DistributedDataParallel(model, device_ids=None, output_device=None)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 189, in __init__
output_device = device_ids[0]
IndexError: list index out of range

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
[checkpoint] Make prepare_sharded_tensor_read and prepare_sharded_tensor_write public
Labels: oncall: distributed, sharded_tensor

### 🚀 The feature, motivation and pitch

Those two functions are required by torchrec and should be part of the public API.

They currently lack good docstrings and unit tests, so those two itens must be addressed.

### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
Suggestion to throw a UserWarning when a user forgot .eval() mode during inference
Labels: module: nn, module: molly-guard, triaged

### 🚀 The feature, motivation and pitch

It happens really often that during inference time, one forgets to put the model in eval mode `model.eval()` beforehand. When the network has layers with different train/test behavior, such as Dropout / BatchNorm, this can be a source of bugs that are really hard to trace afterwards.

I was wondering if it's possible for PyTorch to throw a warning when the user has forgotten `.eval()` before doing inference. Not sure if this proposal has been raised before.

The only technical caveat is, how should PyTorch detect if the user is inferring the model or not. I am thinking of ways based on `.grad` checking -- e.g. check if a tensor is passed through the network the second time, and no gradient has been calculated for the whole model between the first and the second call. Perhaps another way would be to throw the warning only within `torch.no_grad()` environment.

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345
Whether 'targetSize' in inferExpandGeometryImpl needs to be checked when it is less than 0
Labels: module: error checking, triaged

### 🐛 Describe the bug

when I run below code
```
import torch
a = torch.randn(1,3)
a.expand([-2, 2, 1, 3])
```

I got error message as below:

> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/pytorch/torch/_tensor.py", line 341, in __repr__
>     return torch._tensor_str._str(self)
>   File "/home/pytorch/torch/_tensor_str.py", line 439, in _str
>     return _str_intern(self)
>   File "/home/pytorch/torch/_tensor_str.py", line 414, in _str_intern
>     tensor_str = _tensor_str(self, indent)
>   File "/home/pytorch/torch/_tensor_str.py", line 264, in _tensor_str
>     formatter = _Formatter(get_summarized_data(self) if summarize else self)
>   File "/home/pytorch/torch/_tensor_str.py", line 92, in __init__
>     tensor_view = tensor.reshape(-1)
> RuntimeError: Trying to create tensor with negative dimension -2: [-2, 2, 1, 3]

It seems to report 'reshape' has invalid parameter， actually expand op does.  It causes confuse sometime. Maybe torch should check 'targerSize' in inferExpandGeometryImpl to make sure it is greater than or equal to 0.

### Versions

'1.12.0a0+git0aa3c39'
NVFuser failing extremal opinfos
Labels: triaged, module: nvfuser

### 🐛 Describe the bug

[failure logs](https://gist.github.com/davidberard98/d6e30e7c888e04cd884397d0e5922139)

- [x] nn_functional_binary_cross_entropy_with_logits (https://github.com/pytorch/pytorch/pull/76768)
- [x] nn_functional_conv_transpose3d (#75363)
- [x] nn_functional_nll_loss (https://github.com/pytorch/pytorch/pull/76768)
- [x] nn_functional_prelu (https://github.com/pytorch/pytorch/pull/76768)
- [x] unique_consecutive (https://github.com/pytorch/pytorch/issues/76571)
- [x] unique  (https://github.com/pytorch/pytorch/issues/76571)

### Versions

https://github.com/pytorch/pytorch/runs/6215506399?check_suite_focus=true
CI, https://github.com/pytorch/pytorch/pull/75917

cc @jjsjann123
torch.bucketize doc typo on the left boundary when 'right=True'
Labels: module: docs, triaged, module: sorting and selection

### 📚 The doc issue

In https://pytorch.org/docs/stable/generated/torch.bucketize.html#torch.bucketize, the text says "If right is False (default), then the left boundary is closed. More formally, the returned index satisfies the following rules:", but the table below illustrates the opposite: left boundary is closed when right=True.
<img width="868" alt="Screen Shot 2022-04-25 at 1 48 46 PM" src="https://user-images.githubusercontent.com/45018570/165172387-ce93ac88-b2d5-4d3c-92f9-942a9c93b363.png">
.

### Suggest a potential alternative/fix

"If right is False (default), then the left boundary is open..."

cc @brianjo @mruberry
[checkpoint] Avoid loading whole tensor when resharding 
Labels: triaged, sharded_tensor

### 🚀 The feature, motivation and pitch


The FileSystem implementation of the storage layer uses torch.load / torch.save to serializer tensors.
While this is ok for saving, it's a problem when resharding during load as we can't do a partial read and load the whole tensor in memory first before copying a small part of it to its destination.

This could lead to OOM when resharding to a small memory target and wastes IO bandwidth.



### Alternatives

_No response_

### Additional context

_No response_
More understandable name column of the table of the profiling result.
Labels: oncall: profiler

### 🚀 The feature, motivation and pitch

When printing the profiled result like below, the name column is too narrow to show the understandable name.

```python
print(prof.key_averages())
```

For example, it's not clear what kind of elementwise_kernels are they in the screenshot below.

![Screenshot from 2022-04-22 23-19-22](https://user-images.githubusercontent.com/32722944/165097946-468d8f6c-609d-4637-9f96-2cc0ccdacaf5.png)

It seems that the upper bound of the column width is [fixed here](https://github.com/pytorch/pytorch/blob/master/torch/autograd/profiler_util.py#L688).
So I would like to add parameter to change the column width.
I'm considering simply adding `max_name_column_width` parameter to `table` method of `EventList` class.

### Alternatives

## Alternative 1
First alternative is implementing a way to show concise names at name column.

## Alternative 2
Alternative is exporting profiled result in other format and visualize with other tool or scripts.


### Additional context

_No response_

cc @ilia-cher @robieta @chaekit @gdankel @bitfort @ngimel @nbcsm @guotuofeng @guyang3532 @gaoteng-git
Unexpected _LinAlgError appeared only on my device
Labels: triaged, module: linear algebra

### 🐛 Describe the bug

The _LinAlgError message appears when running `torch.linalg.solve` with the given variables in the hess_grad.txt file. However, this error message only appears when I run the code locally on the JupyterLab of my anaconda but not on my friends' laptop (i.e. cannot be reproduced on other devices).

file storing the variables:
[hess_grad.txt](https://github.com/pytorch/pytorch/files/8553023/hess_grad.txt)


sample code:
```python
import torch
path = "hess_grad.txt"
checkpoint = torch.load(path)
torch.linalg.solve(checkpoint["hess"], checkpoint["grad"])
```

error message:
```
---------------------------------------------------------------------------
_LinAlgError                              Traceback (most recent call last)
Input In [23], in <cell line: 3>()
      1 path = "hess_grad.txt"
      2 checkpoint = torch.load(path)
----> 3 torch.torch.linalg.solve(checkpoint["hess"], checkpoint["grad"])

_LinAlgError: linalg.solve: The diagonal element 2 is zero, the solve could not be completed because the input matrix is singular.
```

### Versions

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.3.1 (x86_64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.3)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.7 (default, Sep 16 2021, 08:50:36)  [Clang 10.0.0 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.2
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl  
[conda] mkl                       2021.4.0           hecd8cb5_637  
[conda] mkl-service               2.4.0            py39h9ed2024_0  
[conda] mkl_fft                   1.3.1            py39h4ab4a9b_0  
[conda] mkl_random                1.2.2            py39hb2f4e1b_0  
[conda] mypy_extensions           0.4.3            py39hecd8cb5_1  
[conda] numpy                     1.21.5           py39h2e5f0a9_1  
[conda] numpy-base                1.21.5           py39h3b1a694_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] torch                     1.11.0                   pypi_0    pypi
[conda] torchaudio                0.11.0                   pypi_0    pypi
[conda] torchvision               0.12.0                   pypi_0    pypi

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
TorchScript: jit.script fails when using 'tolist' with int32 tensors.
Labels: oncall: jit

### 🐛 Describe the bug

**Description:**

I cannot use the 'tolist' builtin on a tensor with jit's scripting operation on an int32 tensor. Doing so will raise a RunTime error with the message "Output annotation element type and runtime tensor element type must match for tolist()". This is despite the int64 version working fine. This was found in a standard Google Colab environment, as well as my local machine.

**Expectations**

I would expect Int32Tensor.tolist() to be supported natively, with appropriate typing. Not throw a RunTime error. I might be wrong here, it might actually be a language feature. If this turns out to be the case,  and you are from the future, see the workaround.

**Workarounds**

For anyone else who encounters this bug, know that casting your tensors to int64 before calling tolist allows jit to compile correctly. 

**Reproduction**

Running the following code snippet is sufficient. It simply attempts to convert a tensor to a list and back.The int32 case will fail, but the int64 case will succeed.

```
import torch
from typing import List


@torch.jit.script
def test_tolist(item: torch.Tensor):
  """
  Convert input to a list, then back to an int64 tensor.
  """
  item: List[int] = item.tolist()
  item: torch.Tensor = torch.tensor(item, dtype=torch.int64)
  return item

test_int64 = torch.tensor([3, 2, 6], dtype=torch.int64)
test_int32 = torch.tensor([3, 2, 6], dtype=torch.int32)

test_tolist(test_int64) #Successful
test_tolist(test_int32) #Fails
```





### Versions

PyTorch version: 1.10.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0
Libc version: glibc-2.26

Python version: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: False
CUDA runtime version: 11.1.105
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.10.0+cu111
[pip3] torchaudio==0.10.0+cu111
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.11.0
[pip3] torchvision==0.11.1+cu111
[conda] Could not collect
torch._remove_batch_dim is interceptable by __torch_function__ / batch tensors don't print correctly
Labels: triaged, module: vmap, module: __torch_function__

### 🐛 Describe the bug

`_add_batch_dim` and `_remove_batch_dim` are supposed to be internal functions that are not user visible. However, `__torch_function__` makes it possible to observe these functions as well as their inputs/outputs, as they are defined in `native_functions.yaml` and correspondingly overrideable via `__torch_function__`. To avoid leaking in this way, I think we probably should have put a torch function handler on `vmap`, the higher order function, itself.

This being said, there is another problem: batch tensors don't print correctly.

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/users/ezyang/pytorch-tmp/torch/_vmap_internals.py", line 271, in wrapped
    batched_outputs = func(*batched_inputs)
  File "<stdin>", line 1, in <lambda>
  File "/data/users/ezyang/pytorch-tmp/torch/_tensor.py", line 338, in __repr__
    return torch._tensor_str._str(self)
  File "/data/users/ezyang/pytorch-tmp/torch/_tensor_str.py", line 443, in _str
    return _str_intern(self)
  File "/data/users/ezyang/pytorch-tmp/torch/_tensor_str.py", line 418, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/data/users/ezyang/pytorch-tmp/torch/_tensor_str.py", line 264, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/data/users/ezyang/pytorch-tmp/torch/_tensor_str.py", line 112, in __init__
    if value != torch.ceil(value):
RuntimeError: Batching rule not implemented for aten::is_nonzero. We could not generate a fallback.
>>> torch.vmap(lambda x: print(x))(torch.randn(2, 2))
```

which makes debugging problems involving batched tensor inputs quite irritating.

@zou3519 this is partially why I added a way of sniffing out batched tensors; random stuff doesn't work with them so I can't treat them uniformly as dense tensors.

### Versions

master

cc @zou3519 @hameerabbasi @rgommers @peterbell10
`torch.cuda.amp.GradScaler` may skip parameter synchronization required by post localSGD optimizer
Labels: oncall: distributed, module: amp (automated mixed precision)

### 🐛 Describe the bug

The following two designs may not work very well together.

-  When mixed precision is enabled by `torch.cuda.amp.GradScaler`, as noted in this [line](https://github.com/pytorch/pytorch/blob/bbc3cc6718d841665ab2677477d458ed97b75609/torch/cuda/amp/grad_scaler.py#L68) and this [line](https://github.com/pytorch/pytorch/blob/bbc3cc6718d841665ab2677477d458ed97b75609/torch/cuda/amp/grad_scaler.py#L86), if gradients contain infs/NaNs, then `optimizer.step()` can be skipped.
- Some [distributed optimizers](https://github.com/pytorch/pytorch/tree/77f23d64607142c0cba47f5148ebc1959c9de366/torch/distributed/optim) often run certain communications besides `step()` of the local optimizer.

A prominent example is [post_localSGD_optimizer](https://github.com/pytorch/pytorch/blob/77f23d64607142c0cba47f5148ebc1959c9de366/torch/distributed/optim/post_localSGD_optimizer.py#L78), which runs parameter averaging after `step()` of local optimizer. Since parameter/model averaging paradigm (e.g., local SGD, hierarchical SGD, SlowMo) does not synchronize parameters at every iteration, even if gradients have infs/NaNs and the local trainable parameters are not updated at the current iteration, it is still better to average the parameters than have such averaging skipped. Otherwise the parameters might become too stale (e.g., the last synchronization could be `2*T` iterations ago, where `T` denotes the averaging period). IMO, this might even cause the loss of NaN might occur for some cases.

Furthermore, this behavior of silently skipping `optimizer.step()` can also be a risk for some other (future) distributed optimizers if certain states (e.g., optimizer state, buffer) still need to be updated. For example, the internal step counter [here](https://github.com/pytorch/pytorch/blob/77f23d64607142c0cba47f5148ebc1959c9de366/torch/distributed/optim/functional_rmsprop.py#L91) can be less than the # of steps in a training loop, and same as the step counter in this [RFC](https://github.com/pytorch/pytorch/issues/74556). Additionally, not sure if any other optimizer wrapper may need to run certain communications even if no local optimizer step.

One relevant feature request is https://github.com/pytorch/pytorch/issues/67590, which proposes to signal that the scaler has been stepped and `optimizer.step()` has been actually called.

_Note that this issue may not be very serious for some cases in practice, because [such step skipping is expected to occur rarely after the first few iterations](https://github.com/pytorch/pytorch/blob/bbc3cc6718d841665ab2677477d458ed97b75609/torch/cuda/amp/grad_scaler.py#L93-L95)._

### Versions

PyTorch 1.11

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @mcarilli @ptrblck
Initial integration of ZenDNN as backend into PyTorch
Labels: feature, module: convolution, triaged

### 🚀 The feature, motivation and pitch

ZenDNN library optimizes basic math kernels used by Deep learning algorithms on AMD EPYC CPUs. As part of integration of ZenDNN into PyTorch we integrated a single op (convolution). We have support for other ops and optimization ready to be submitted.

This PR covers following
- Added ZenDNN as third-party library
    - ZenDNN is a submodule for pytorch
    - modified pytorch build infra to build with ZenDNN when USE_ZENDNN is set
      to 1
    - pytorch build reverts to default behavior if USE_ZENDNN environmental
      variable is set to zero or unset
- Added AOCL-blis as  dependency of ZenDNN
    - AOCL-blis is a submodule for pytorch
    - AOCL-blis is built if USE_ZENDNN is set to 1
- Added ZenDNN as backend to pytorch
    - added support for user level control to disable or enable zendnn backend
- Added following changes
    - added zendnn layout and ZendnnCPU dispatch key.
    - tensor conversion between dense and zendnn layouts
    - zendnn convolution as a new ConvBackend
        - when built with zendnn (USE_ZENDNN=1) and gradient calculations are
          disabled (torch.no_grad) convolution is computed using ZenDNN library)
        - backward propagation is not supported.
        - support is only added for fp32 data type
- Added few unit tests for pytorch + zendnn

URLs of dependent Library:
ZenDNN github: https://github.com/amd/ZenDNN
AMD-BLIS github: https://github.com/amd/blis
Corresponding Pull Request: https://github.com/pytorch/pytorch/pull/76242


### Alternatives

_No response_

### Additional context

_No response_
Observing a strange behavior - Row parallelism
Labels: module: numerical-stability, module: nn, triaged, module: correctness (silent)

### 🐛 Describe the bug

Hi there!
I am trying to manually reproduce this scheme, which corresponds to a model row parallelism paradigm:
![Screenshot from 2022-04-22 15-55-48](https://user-images.githubusercontent.com/49240599/164728838-3f78585b-1018-4366-a499-95133fdeaa89.png)
Therefore I have written a small snippet of code to reproduce the scheme above, the assertion passes for small ```batch_size```, ```hidden_dim``` and ```output_dim``` values (for example: ```batch_size = 1 \ hidden_dim = 4 \ output_dim = 2```) but strangely the assertion does not pass anymore for large values (specifically 10, 40, 20). What could explain this behavior?

Code snippet (tried on google colab):
```
import torch
import torch.nn as nn
import torch.nn.functional as F

batch_size = 10
hidden_dim = 40
output_dim = 20
sliced_dim = hidden_dim//2

dummy_mlp = nn.Linear(hidden_dim, output_dim, bias=False)
parameters = torch.nn.Parameter(torch.randn(output_dim, hidden_dim))

dummy_mlp.weight = parameters

dummy_input = torch.randn(batch_size, hidden_dim)

sliced_input_1, sliced_input_2 = torch.split(dummy_input, sliced_dim, dim=-1)

sliced_output_1 = F.linear(sliced_input_1, dummy_mlp.weight[:, :sliced_dim])
sliced_output_2 = F.linear(sliced_input_2, dummy_mlp.weight[:, sliced_dim:])

final_output = dummy_mlp(dummy_input)

assert torch.equal(final_output, sliced_output_1 + sliced_output_2)
```

Thank you very much in advance for your help!!

### Versions

Collecting environment information...
PyTorch version: 1.10.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0
Libc version: glibc-2.26

Python version: 3.7.13 (default, Mar 16 2022, 17:37:17)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.10.0+cu111
[pip3] torchaudio==0.10.0+cu111
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.11.0
[pip3] torchvision==0.11.1+cu111
[conda] Could not collect

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345
RuntimeError: bucket_count == per_bucket_sizes.size()INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1646755853042/work/torch/csrc/distributed/c10d/reducer.cpp":980, please report a bug to PyTorch. 
Labels: oncall: distributed, triaged

### 🐛 Describe the bug

```bash
bucket_count == per_bucket_sizes.size()INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1646755853042/work/torch/csrc/distributed/c10d/reducer.cpp":980, please report a bug to PyTorch. 
```

### Versions

```bash
python collect_env.py
Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.12 (default, Oct 12 2021, 13:49:34)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.2.152
GPU models and configuration: 
GPU 0: Tesla V100-PCIE-32GB
GPU 1: Tesla V100-PCIE-32GB
GPU 2: Tesla V100-PCIE-32GB
GPU 3: Tesla V100-PCIE-32GB

Nvidia driver version: 510.60.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl    defaults
[conda] cudatoolkit               10.2.89              hfd86e86_1    defaults
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.3.0           h06a4308_520    defaults
[conda] mkl-service               2.4.0            py38h7f8727e_0    defaults
[conda] mkl_fft                   1.3.1            py38hd3c417c_0    defaults
[conda] mkl_random                1.2.2            py38h51133e4_0    defaults
[conda] numpy                     1.21.2           py38h20f2e39_0    defaults
[conda] numpy-base                1.21.2           py38h79a1101_0    defaults
[conda] pytorch                   1.11.0          py3.8_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.11.0               py38_cu102    pytorch
[conda] torchvision               0.12.0               py38_cu102    pytorch
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
Update NCCL to 2.12
Labels: oncall: distributed, module: build, triaged

### 🐛 Describe the bug

The current PyTorch links NCCL v2.10.3-1 by default, which includes a critical bug that breaks the `CollNet` algorithm in distributed training: https://github.com/NVIDIA/nccl/issues/556. This bug has been fixed since v2.12.7-1: https://github.com/NVIDIA/nccl/commit/4ec992fab7d90691ad43bee6f7f6c34aa219934c.

Is there any timeline to update the NCCL version that PyTorch links to? If not I'd be happy to submit a PR.

### Versions

N/A

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @malfet @seemethere
[Feature request] Exclusive prefix sum, `torch.cumsum(input, dim=0, exclusive=True)`
Labels: high priority, module: cuda, triaged, enhancement, module: reductions

### 🚀 The feature, motivation and pitch

Currently `torch.cumsum` only supports inclusive prefix sum. However, the exclusive prefix sum is also needed in some cases.

Proposed to add a `exclusive` parameter to the `torch.cumsum` function
```py
input = torch.tensor([1, 1, 1])
torch.cumsum(input, dim=0, exclusive=True)   # [1, 2, 3]
torch.cumsum(input, dim=0, exclusive=False)  # [0, 1, 2]
```

Related issue from PyTorch Forums:
- [PyTorch equivalent of exclusive cumsum? - PyTorch Forums](https://discuss.pytorch.org/t/pytorch-equivalent-of-exclusive-cumsum/107259)

CUDA implementation from CUB:
- [CUB: cub::DeviceScan Struct Reference](https://nvlabs.github.io/cub/structcub_1_1_device_scan.html#a02b2d2e98f89f80813460f6a6ea1692b)

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @gchanan @zou3519 @ptrblck @ngimel