torch 1.12.1 cuda 10.2 runs slower than torch 1.8.2 cuda 10.2
Labels: module: performance, module: cudnn, module: cuda, triaged

## Issue description

Same code, using torch profile. It runs much slower on torch 1.12.1 than on torch 1.8.2 with cuda 10.2 and cudnn 7605. Torch are downloaded from https://download.pytorch.org/whl/cu102/torch_stable.html. Could anyone help me out of this problem?

## Code example

``` python
with torch.autograd.profiler.profile(enabled=True, use_cuda=True, record_shapes=False, profile_memory=False) as prof:
    self.train()  # forward and backward several times
print(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=100))
prof.export_chrome_trace('./resnet_profile.json')
```

1.8.2:

<html>
<body>
<!--StartFragment--><!DOCTYPE html>

Name | Self CPU % | Self CPU | CPU total % | CPU total | CPU time avg | Self CUDA | Self CUDA % | CUDA total | CUDA time avg | # of Calls
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
aten::cudnn_convolution_backward_input | 6.55% | 3.019s | 6.59% | 3.040s | 637.234us | 4.607s | 9.96% | 4.607s | 965.773us | 4770
aten::cudnn_convolution_backward_weight | 5.30% | 2.444s | 5.49% | 2.530s | 527.175us | 4.488s | 9.70% | 4.488s | 934.970us | 4800
aten::index_put_impl | 6.50% | 2.999s | 17.40% | 8.025s | 10.210ms | 4.361s | 9.43% | 4.974s | 6.328ms | 786
aten::cudnn_convolution | 5.33% | 2.459s | 5.39% | 2.485s | 517.661us | 3.964s | 8.57% | 3.971s | 827.318us | 4800
aten::sort | 8.51% | 3.925s | 8.51% | 3.925s | 4.994ms | 3.795s | 8.20% | 3.795s | 4.829ms | 786
SyncBatchNorm | 1.35% | 620.404ms | 14.65% | 6.754s | 2.186ms | 2.474s | 5.35% | 5.474s | 1.771ms | 3090
SyncBatchNormBackward | 1.42% | 655.297ms | 7.90% | 3.645s | 1.179ms | 2.376s | 5.14% | 3.739s | 1.210ms | 3090
aten::copy_ | 4.00% | 1.847s | 4.00% | 1.847s | 73.266us | 1.875s | 4.05% | 1.875s | 74.375us | 25206
aten::mul_ | 3.66% | 1.689s | 3.66% | 1.689s | 42.384us | 1.639s | 3.54% | 1.639s | 41.145us | 39846
aten::add_ | 3.64% | 1.678s | 3.64% | 1.678s | 40.636us | 1.297s | 2.80% | 1.297s | 31.395us | 41302
aten::fill_ | 0.80% | 367.642ms | 0.80% | 367.642ms | 17.658us | 1.218s | 2.63% | 1.218s | 58.498us | 20820
aten::scatter_ | 0.05% | 21.818ms | 0.05% | 23.170ms | 29.478us | 823.199ms | 1.78% | 823.199ms | 1.047ms | 786
aten::mul | 1.02% | 471.278ms | 1.09% | 502.562ms | 58.986us | 811.317ms | 1.75% | 811.317ms | 95.225us | 8520
aten::div | 1.71% | 790.098ms | 1.86% | 858.417ms | 40.404us | 779.192ms | 1.68% | 779.192ms | 36.675us | 21246
Optimizer.step#AdamW.step | 2.57% | 1.185s | 13.68% | 6.309s | 210.316ms | 695.652ms | 1.50% | 5.618s | 187.258ms | 30
aten::batch_norm_gather_stats_with_counts | 0.23% | 107.335ms | 0.29% | 132.311ms | 42.819us | 560.058ms | 1.21% | 560.058ms | 181.249us | 3090
aten::nonzero | 11.03% | 5.086s | 11.14% | 5.139s | 3.037ms | 554.498ms | 1.20% | 569.915ms | 336.829us | 1692
aten::addcmul_ | 1.25% | 576.361ms | 1.83% | 842.129ms | 64.680us | 539.481ms | 1.17% | 787.051ms | 60.449us | 13020
aten::sqrt | 1.02% | 470.158ms | 1.11% | 512.980ms | 39.399us | 529.064ms | 1.14% | 529.064ms | 40.635us | 13020
aten::batch_norm_backward_elemt | 2.47% | 1.138s | 2.70% | 1.247s | 403.576us | 489.826ms | 1.06% | 507.219ms | 164.149us | 3090
aten::batch_norm_backward_reduce | 1.47% | 675.860ms | 1.91% | 879.752ms | 284.709us | 477.165ms | 1.03% | 496.336ms | 160.626us | 3090
aten::threshold_backward | 0.89% | 410.714ms | 0.92% | 426.501ms | 116.530us | 418.105ms | 0.90% | 418.105ms | 114.236us | 3660
aten::zero_ | 0.95% | 436.176ms | 1.60% | 735.996ms | 41.371us | 407.443ms | 0.88% | 1.580s | 88.804us | 17790
aten::index | 0.21% | 96.017ms | 0.90% | 417.174ms | 246.557us | 379.260ms | 0.82% | 673.901ms | 398.286us | 1692
aten::batch_norm_stats | 0.20% | 92.754ms | 0.44% | 202.313ms | 65.473us | 354.815ms | 0.77% | 430.311ms | 139.259us | 3090
aten::addcdiv_ | 0.84% | 389.394ms | 1.38% | 636.954ms | 48.921us | 338.894ms | 0.73% | 582.971ms | 44.775us | 13020
aten::batch_norm_elemt | 0.20% | 93.432ms | 0.30% | 137.139ms | 44.382us | 307.737ms | 0.67% | 315.618ms | 102.142us | 3090
nccl:all_gather | 1.08% | 498.480ms | 1.96% | 902.411ms | 286.480us | 281.268ms | 0.61% | 586.863ms | 186.306us | 3150
aten::sum | 0.73% | 335.813ms | 0.84% | 386.259ms | 46.448us | 259.049ms | 0.56% | 279.370ms | 33.594us | 8316
aten::addcmul | 0.58% | 265.768ms | 0.58% | 265.768ms | 20.412us | 247.570ms | 0.54% | 247.570ms | 19.015us | 13020
aten::add | 0.41% | 191.072ms | 0.41% | 191.072ms | 37.421us | 245.389ms | 0.53% | 245.389ms | 48.059us | 5106
aten::addcdiv | 0.54% | 247.560ms | 0.54% | 247.560ms | 19.014us | 244.077ms | 0.53% | 244.077ms | 18.746us | 13020
aten::select | 0.58% | 267.881ms | 0.61% | 282.312ms | 25.938us | 221.090ms | 0.48% | 221.090ms | 20.313us | 10884
aten::eq | 0.43% | 196.792ms | 0.69% | 317.035ms | 30.900us | 216.781ms | 0.47% | 349.749ms | 34.089us | 10260
aten::threshold_ | 0.08% | 37.969ms | 0.08% | 37.969ms | 13.322us | 207.245ms | 0.45% | 207.245ms | 72.718us | 2850
aten::narrow | 0.39% | 181.472ms | 0.82% | 379.777ms | 15.197us | 162.169ms | 0.35% | 162.169ms | 6.489us | 24990
aten::_local_scalar_dense | 9.53% | 4.397s | 9.53% | 4.397s | 957.894us | 161.980ms | 0.35% | 161.980ms | 35.290us | 4590
aten::_cat | 0.43% | 199.887ms | 0.56% | 259.903ms | 27.243us | 161.344ms | 0.35% | 161.902ms | 16.971us | 9540
MulBackward0 | 0.49% | 227.946ms | 1.17% | 538.075ms | 232.329us | 149.274ms | 0.32% | 516.302ms | 222.928us | 2316
aten::reshape | 0.81% | 371.633ms | 0.95% | 436.767ms | 16.816us | 146.019ms | 0.32% | 243.907ms | 9.390us | 25974
aten::sub | 0.15% | 70.532ms | 0.17% | 79.130ms | 33.558us | 143.811ms | 0.31% | 143.811ms | 60.989us | 2358
aten::cat | 0.55% | 252.875ms | 1.11% | 512.778ms | 53.750us | 134.633ms | 0.29% | 296.535ms | 31.083us | 9540
aten::split | 0.45% | 205.602ms | 0.87% | 400.623ms | 64.826us | 120.865ms | 0.26% | 233.907ms | 37.849us | 6180
aten::_softmax_backward_data | 0.00% | 1.899ms | 0.01% | 3.120ms | 52.002us | 118.390ms | 0.26% | 172.045ms | 2.867ms | 60
aten::_log_softmax_backward_data | 0.00% | 1.489ms | 0.00% | 1.755ms | 29.254us | 117.674ms | 0.25% | 117.674ms | 1.961ms | 60
aten::to | 0.19% | 87.685ms | 2.73% | 1.260s | 110.987us | 117.138ms | 0.25% | 500.381ms | 44.075us | 11353
aten::_log_softmax | 0.00% | 2.072ms | 0.01% | 2.406ms | 40.108us | 110.746ms | 0.24% | 110.746ms | 1.846ms | 60
aten::cudnn_convolution_transpose | 0.11% | 50.498ms | 0.11% | 50.749ms | 563.874us | 107.779ms | 0.23% | 107.779ms | 1.198ms | 90
aten::_softmax | 0.01% | 6.843ms | 0.02% | 7.252ms | 120.867us | 94.688ms | 0.20% | 94.688ms | 1.578ms | 60
aten::remainder | 0.04% | 17.165ms | 0.04% | 20.687ms | 26.320us | 85.431ms | 0.18% | 85.431ms | 108.691us | 786
aten::rsub | 0.11% | 49.156ms | 0.12% | 53.415ms | 33.343us | 83.047ms | 0.18% | 83.047ms | 51.839us | 1602
aten::stack | 0.22% | 101.318ms | 0.54% | 248.895ms | 78.269us | 78.597ms | 0.17% | 158.489ms | 49.839us | 3180
aten::floor_divide | 0.03% | 11.563ms | 0.03% | 11.563ms | 14.711us | 72.163ms | 0.16% | 72.163ms | 91.811us | 786
aten::_cumsum | 0.08% | 37.536ms | 0.10% | 45.497ms | 28.942us | 71.103ms | 0.15% | 71.103ms | 45.231us | 1572
aten::slice_backward | 0.17% | 77.019ms | 0.30% | 136.770ms | 174.007us | 69.269ms | 0.15% | 883.591ms | 1.124ms | 786
torch::autograd::AccumulateGrad | 0.67% | 311.172ms | 2.34% | 1.078s | 82.773us | 67.385ms | 0.15% | 344.809ms | 26.483us | 13020
aten::cudnn_convolution_backward | 1.03% | 473.016ms | 13.11% | 6.047s | 1.260ms | 62.080ms | 0.13% | 9.170s | 1.910ms | 4800
Optimizer.zero_grad#AdamW.zero_grad | 0.24% | 112.693ms | 1.42% | 654.674ms | 21.822ms | 60.239ms | 0.13% | 632.327ms | 21.078ms | 30
aten::relu_ | 0.19% | 88.351ms | 0.27% | 126.320ms | 44.323us | 57.718ms | 0.12% | 264.962ms | 92.969us | 2850
aten::convolution | 0.14% | 66.392ms | 5.96% | 2.750s | 562.345us | 57.583ms | 0.12% | 4.337s | 887.004us | 4890
aten::cudnn_convolution_transpose_backward_input | 0.09% | 40.233ms | 0.09% | 40.669ms | 451.881us | 56.885ms | 0.12% | 56.885ms | 632.050us | 90
aten::is_nonzero | 0.12% | 55.350ms | 9.73% | 4.486s | 1.046ms | 55.870ms | 0.12% | 258.608ms | 60.281us | 4290
aten::abs | 0.09% | 43.417ms | 0.13% | 59.488ms | 37.842us | 54.711ms | 0.12% | 87.001ms | 55.344us | 1572
aten::dot | 0.10% | 45.942ms | 0.10% | 48.029ms | 61.106us | 54.031ms | 0.12% | 54.031ms | 68.741us | 786
aten::cudnn_convolution_transpose_backward_weight | 0.03% | 13.205ms | 0.03% | 14.851ms | 165.013us | 53.427ms | 0.12% | 53.427ms | 593.633us | 90
aten::_convolution | 0.17% | 77.291ms | 5.82% | 2.683s | 548.768us | 51.101ms | 0.11% | 4.280s | 875.229us | 4890
aten::item | 0.10% | 44.525ms | 9.63% | 4.441s | 967.594us | 46.883ms | 0.10% | 208.863ms | 45.504us | 4590
aten::mean | 0.06% | 27.828ms | 0.08% | 36.203ms | 40.226us | 45.969ms | 0.10% | 45.969ms | 51.077us | 900
aten::neg | 0.07% | 30.641ms | 0.10% | 47.496ms | 27.110us | 42.645ms | 0.09% | 76.275ms | 43.536us | 1752
aten::sgn | 0.09% | 42.741ms | 0.15% | 69.469ms | 44.192us | 41.335ms | 0.09% | 73.811ms | 46.954us | 1572
aten::cumsum | 0.12% | 54.457ms | 0.23% | 105.924ms | 67.382us | 40.295ms | 0.09% | 115.329ms | 73.364us | 1572
aten::zeros_like | 0.02% | 10.588ms | 0.11% | 50.934ms | 58.143us | 38.724ms | 0.08% | 54.422ms | 62.125us | 876
aten::nll_loss2d_backward | 0.01% | 3.131ms | 0.01% | 3.131ms | 52.189us | 37.951ms | 0.08% | 37.951ms | 632.521us | 60
aten::conv2d | 0.09% | 42.953ms | 5.93% | 2.736s | 569.969us | 35.841ms | 0.08% | 4.234s | 882.033us | 4800
aten::full | 0.10% | 45.472ms | 0.26% | 120.943ms | 39.140us | 32.619ms | 0.07% | 76.475ms | 24.749us | 3090
aten::sigmoid | 0.07% | 31.503ms | 0.08% | 35.420ms | 42.167us | 31.874ms | 0.07% | 31.874ms | 37.945us | 840
enumerate(DataLoader)#_MultiProcessingDataLoaderIter... | 0.05% | 25.112ms | 0.05% | 25.152ms | 838.390us | 25.075ms | 0.05% | 25.075ms | 835.821us | 30
CudnnConvolutionBackward | 0.29% | 132.425ms | 13.40% | 6.179s | 1.287ms | 24.718ms | 0.05% | 9.195s | 1.916ms | 4800
aten::relu | 0.04% | 17.842ms | 0.07% | 31.890ms | 39.371us | 23.776ms | 0.05% | 41.471ms | 51.198us | 810
aten::sigmoid_backward | 0.10% | 47.457ms | 0.11% | 51.341ms | 61.121us | 21.914ms | 0.05% | 21.914ms | 26.088us | 840
aten::zeros | 0.11% | 51.560ms | 0.53% | 246.205ms | 63.718us | 21.282ms | 0.05% | 1.012s | 261.991us | 3864
aten::clone | 0.04% | 19.877ms | 0.09% | 41.600ms | 43.064us | 20.823ms | 0.05% | 252.168ms | 261.043us | 966
aten::nll_loss2d_forward | 0.01% | 3.632ms | 0.01% | 3.632ms | 60.531us | 20.672ms | 0.04% | 20.672ms | 344.542us | 60
aten::select_backward | 0.07% | 32.220ms | 0.45% | 208.351ms | 147.140us | 18.403ms | 0.04% | 672.332ms | 474.811us | 1416
ReluBackward1 | 0.09% | 41.356ms | 0.92% | 425.500ms | 149.298us | 17.945ms | 0.04% | 418.488ms | 146.838us | 2850
aten::threshold | 0.02% | 11.310ms | 0.03% | 14.048ms | 17.344us | 17.695ms | 0.04% | 17.695ms | 21.845us | 810
aten::gt | 0.02% | 7.937ms | 0.04% | 16.825ms | 46.735us | 15.927ms | 0.03% | 32.166ms | 89.350us | 360
aten::set_ | 0.06% | 27.912ms | 0.06% | 27.912ms | 16.496us | 15.417ms | 0.03% | 15.417ms | 9.112us | 1692
aten::conj | 0.07% | 31.975ms | 0.07% | 31.975ms | 6.859us | 12.442ms | 0.03% | 12.442ms | 2.669us | 4662
AddBackward0 | 0.05% | 22.301ms | 0.05% | 22.301ms | 5.709us | 12.138ms | 0.03% | 12.138ms | 3.108us | 3906
ViewBackward | 0.07% | 34.431ms | 0.14% | 65.211ms | 33.441us | 10.773ms | 0.02% | 73.920ms | 37.908us | 1950
nccl:all_reduce | 0.20% | 92.087ms | 0.20% | 92.087ms | 29.802us | 10.416ms | 0.02% | 10.416ms | 3.371us | 3090
aten::upsample_nearest2d | 0.02% | 10.356ms | 0.02% | 10.379ms | 345.975us | 10.110ms | 0.02% | 10.110ms | 336.992us | 30
aten::upsample_bilinear2d | 0.00% | 906.354us | 0.00% | 1.629ms | 54.315us | 9.278ms | 0.02% | 9.278ms | 309.283us | 30
aten::value_selecting_reduction_backward | 0.05% | 22.869ms | 0.28% | 127.428ms | 162.122us | 7.721ms | 0.02% | 910.592ms | 1.159ms | 786
aten::flatten | 0.03% | 11.973ms | 0.12% | 53.415ms | 52.061us | 7.689ms | 0.02% | 161.333ms | 157.245us | 1026
SelectBackward | 0.11% | 50.893ms | 0.56% | 259.243ms | 183.082us | 7.288ms | 0.02% | 679.620ms | 479.958us | 1416
aten::upsample_bilinear2d_backward | 0.00% | 928.227us | 0.00% | 1.775ms | 59.166us | 7.160ms | 0.02% | 8.146ms | 271.533us | 30
aten::contiguous | 0.01% | 5.880ms | 0.03% | 12.937ms | 107.811us | 6.919ms | 0.01% | 116.119ms | 967.656us | 120
MeanBackward1 | 0.16% | 74.091ms | 0.35% | 162.211ms | 193.109us | 6.713ms | 0.01% | 108.535ms | 129.209us | 840

<!--EndFragment-->
</body>
</html>

1.12.1:

<html>
<body>
<!--StartFragment--><!DOCTYPE html>

Name | Self CPU % | Self CPU | CPU total % | CPU total | CPU time avg | Self CUDA | Self CUDA % | CUDA total | CUDA time avg | # of Calls
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
aten::convolution_backward | 4.93% | 5.875s | 6.82% | 8.119s | 1.660ms | 75.444s | 54.86% | 75.752s | 15.491ms | 4890
aten::cudnn_convolution | 1.68% | 2.004s | 2.20% | 2.621s | 545.968us | 25.200s | 18.32% | 25.342s | 5.280ms | 4800
record_param_comms | 0.43% | 507.603ms | 0.49% | 582.620ms | 60.494us | 5.162s | 3.75% | 5.165s | 536.287us | 9631
aten::index_put_impl | 0.13% | 157.533ms | 5.83% | 6.942s | 8.656ms | 3.749s | 2.73% | 4.449s | 5.548ms | 802
aten::nonzero | 0.91% | 1.089s | 27.39% | 32.617s | 2.967ms | 2.670s | 1.94% | 3.299s | 300.036us | 10994
aten::copy_ | 0.23% | 276.372ms | 0.79% | 940.697ms | 19.377us | 1.949s | 1.42% | 1.949s | 40.140us | 48548
aten::add | 0.09% | 102.464ms | 0.17% | 203.353ms | 37.982us | 1.662s | 1.21% | 1.662s | 310.495us | 5354
Optimizer.step#AdamW.step | 2.04% | 2.428s | 4.08% | 4.859s | 161.973ms | 1.587s | 1.15% | 4.869s | 162.306ms | 30
aten::sort | 0.07% | 86.208ms | 0.22% | 261.301ms | 325.812us | 1.285s | 0.93% | 1.320s | 1.646ms | 802
aten::index | 0.61% | 721.432ms | 22.94% | 27.316s | 2.485ms | 1.232s | 0.90% | 4.745s | 431.631us | 10994
aten::mul | 0.23% | 272.841ms | 0.35% | 412.120ms | 19.882us | 968.027ms | 0.70% | 968.027ms | 46.701us | 20728
aten::add_ | 0.46% | 545.468ms | 1.09% | 1.303s | 22.684us | 967.276ms | 0.70% | 1.829s | 31.849us | 57428
aten::scatter_ | 0.03% | 31.895ms | 0.03% | 39.452ms | 49.192us | 876.742ms | 0.64% | 881.933ms | 1.100ms | 802
aten::fill_ | 0.15% | 175.846ms | 0.28% | 329.190ms | 11.772us | 810.262ms | 0.59% | 810.262ms | 28.975us | 27964
aten::as_strided | 0.08% | 96.996ms | 0.08% | 96.996ms | 1.051us | 570.728ms | 0.41% | 570.728ms | 6.184us | 92286
aten::empty | 0.52% | 615.215ms | 0.64% | 766.584ms | 8.819us | 498.067ms | 0.36% | 498.067ms | 5.730us | 86926
aten::empty_strided | 0.12% | 140.033ms | 0.31% | 373.503ms | 15.024us | 494.714ms | 0.36% | 494.714ms | 19.900us | 24860
aten::mul_ | 0.24% | 280.976ms | 0.37% | 435.157ms | 10.917us | 491.003ms | 0.36% | 491.003ms | 12.318us | 39862
aten::_to_copy | 0.40% | 471.375ms | 0.79% | 945.325ms | 53.865us | 482.967ms | 0.35% | 1.273s | 72.531us | 17550
SyncBatchNorm | 1.34% | 1.598s | 25.56% | 30.436s | 9.850ms | 458.728ms | 0.33% | 8.360s | 2.706ms | 3090
aten::batch_norm_backward_elemt | 0.06% | 75.280ms | 0.20% | 232.941ms | 75.385us | 457.397ms | 0.33% | 487.502ms | 157.768us | 3090
aten::batch_norm_backward_reduce | 13.71% | 16.329s | 14.20% | 16.910s | 5.472ms | 435.912ms | 0.32% | 601.949ms | 194.806us | 3090
aten::threshold_backward | 0.07% | 85.410ms | 3.60% | 4.284s | 1.171ms | 409.248ms | 0.30% | 409.248ms | 111.816us | 3660
DistributedDataParallel.forward | 0.89% | 1.055s | 30.52% | 36.340s | 1.211s | 394.082ms | 0.29% | 36.663s | 1.222s | 30
aten::slice | 0.70% | 829.054ms | 0.72% | 855.069ms | 29.424us | 375.569ms | 0.27% | 510.767ms | 17.576us | 29060
aten::to | 0.21% | 252.151ms | 1.01% | 1.197s | 51.696us | 344.848ms | 0.25% | 1.618s | 69.840us | 23164
aten::div | 0.16% | 192.152ms | 0.22% | 260.268ms | 17.257us | 322.762ms | 0.23% | 322.762ms | 21.400us | 15082
aten::clamp_min_ | 0.03% | 34.953ms | 0.05% | 55.179ms | 19.361us | 316.585ms | 0.23% | 316.585ms | 111.082us | 2850
aten::batch_norm_elemt | 0.06% | 76.202ms | 0.27% | 326.538ms | 105.676us | 315.652ms | 0.23% | 556.380ms | 180.058us | 3090
aten::batch_norm_stats | 0.16% | 186.603ms | 0.35% | 421.129ms | 136.288us | 310.332ms | 0.23% | 437.433ms | 141.564us | 3090
aten::sum | 0.84% | 1.005s | 0.98% | 1.169s | 215.682us | 278.945ms | 0.20% | 317.240ms | 58.510us | 5422
aten::sqrt | 0.12% | 141.242ms | 0.16% | 195.275ms | 14.998us | 255.718ms | 0.19% | 255.718ms | 19.640us | 13020
aten::reshape | 0.18% | 212.281ms | 0.30% | 357.446ms | 19.893us | 252.506ms | 0.18% | 523.568ms | 29.139us | 17968
aten::t | 0.12% | 143.496ms | 0.26% | 305.038ms | 27.746us | 237.969ms | 0.17% | 368.440ms | 33.513us | 10994
Optimizer.zero_grad#AdamW.zero_grad | 0.25% | 292.486ms | 0.44% | 528.706ms | 17.624ms | 231.372ms | 0.17% | 510.154ms | 17.005ms | 30
aten::resize_ | 0.06% | 68.324ms | 0.06% | 76.434ms | 5.954us | 230.068ms | 0.17% | 230.068ms | 17.921us | 12838
aten::addcdiv_ | 0.09% | 104.086ms | 0.13% | 150.716ms | 11.576us | 227.215ms | 0.17% | 227.215ms | 17.451us | 13020
aten::zero_ | 0.27% | 319.079ms | 0.51% | 605.392ms | 24.309us | 220.124ms | 0.16% | 1.004s | 40.311us | 24904
aten::item | 0.17% | 207.628ms | 2.28% | 2.710s | 188.989us | 211.232ms | 0.15% | 405.139ms | 28.252us | 14340
autograd::engine::evaluate_function:  torch::autograd... | 0.87% | 1.033s | 1.64% | 1.956s | 150.192us | 194.219ms | 0.14% | 705.359ms | 54.175us | 13020
aten::_local_scalar_dense | 0.01% | 16.901ms | 2.10% | 2.502s | 174.510us | 193.907ms | 0.14% | 193.907ms | 13.522us | 14340
aten::addcmul_ | 0.08% | 94.889ms | 0.12% | 141.018ms | 10.831us | 191.320ms | 0.14% | 191.320ms | 14.694us | 13020
aten::batch_norm_gather_stats_with_counts | 0.10% | 117.458ms | 0.14% | 164.123ms | 53.114us | 186.357ms | 0.14% | 248.674ms | 80.477us | 3090
aten::narrow | 0.23% | 268.376ms | 0.89% | 1.061s | 42.540us | 185.883ms | 0.14% | 644.222ms | 25.841us | 24930
aten::sub | 0.04% | 45.214ms | 0.05% | 64.893ms | 16.063us | 179.324ms | 0.13% | 179.324ms | 44.387us | 4040
aten::mean | 0.04% | 41.849ms | 0.09% | 112.486ms | 124.984us | 170.373ms | 0.12% | 170.901ms | 189.890us | 900
torch.distributed.ddp.reducer::copy_bucket_to_grad | 3.06% | 3.646s | 3.25% | 3.875s | 297.603us | 144.493ms | 0.11% | 298.933ms | 22.960us | 13020
aten::view | 0.03% | 40.314ms | 0.03% | 40.314ms | 1.861us | 130.085ms | 0.09% | 130.085ms | 6.006us | 21660
aten::rsub | 0.12% | 137.232ms | 0.13% | 160.529ms | 98.243us | 126.236ms | 0.09% | 191.576ms | 117.244us | 1634
aten::empty_like | 0.25% | 295.137ms | 0.64% | 758.590ms | 61.315us | 123.312ms | 0.09% | 504.129ms | 40.748us | 12372
aten::flatten | 0.11% | 125.309ms | 0.15% | 183.384ms | 175.992us | 119.771ms | 0.09% | 304.967ms | 292.675us | 1042
aten::select | 0.25% | 297.481ms | 0.26% | 309.050ms | 22.015us | 119.229ms | 0.09% | 321.807ms | 22.924us | 14038
aten::_softmax_backward_data | 0.00% | 1.662ms | 0.00% | 3.234ms | 53.900us | 118.384ms | 0.09% | 173.740ms | 2.896ms | 60
aten::cat | 0.12% | 142.912ms | 0.18% | 215.138ms | 33.355us | 118.320ms | 0.09% | 121.942ms | 18.906us | 6450
aten::_log_softmax_backward_data | 0.00% | 851.000us | 0.00% | 1.358ms | 22.633us | 115.844ms | 0.08% | 115.844ms | 1.931ms | 60
aten::split | 0.24% | 281.333ms | 0.66% | 790.589ms | 127.927us | 111.768ms | 0.08% | 529.993ms | 85.759us | 6180
aten::_log_softmax | 0.00% | 1.441ms | 0.01% | 7.485ms | 124.750us | 109.909ms | 0.08% | 109.909ms | 1.832ms | 60
aten::remainder | 0.03% | 35.270ms | 0.03% | 41.520ms | 51.771us | 95.538ms | 0.07% | 95.538ms | 119.125us | 802
aten::_softmax | 0.00% | 1.275ms | 0.01% | 6.545ms | 109.083us | 93.144ms | 0.07% | 93.144ms | 1.552ms | 60
aten::zeros | 0.22% | 260.810ms | 0.54% | 639.160ms | 68.905us | 91.340ms | 0.07% | 748.781ms | 80.722us | 9276
aten::cumsum | 0.06% | 67.627ms | 0.07% | 83.266ms | 51.911us | 89.693ms | 0.07% | 94.311ms | 58.797us | 1604
aten::eq | 0.02% | 29.536ms | 0.03% | 40.440ms | 19.824us | 84.234ms | 0.06% | 84.234ms | 41.291us | 2040
aten::arange | 0.03% | 31.841ms | 0.06% | 69.181ms | 43.130us | 83.917ms | 0.06% | 168.721ms | 105.188us | 1604
torch::autograd::AccumulateGrad | 0.43% | 514.044ms | 0.60% | 717.056ms | 55.073us | 83.330ms | 0.06% | 299.909ms | 23.034us | 13020
aten::unflatten_dense_tensors | 0.17% | 203.059ms | 0.64% | 759.802ms | 12.663ms | 80.774ms | 0.06% | 362.612ms | 6.044ms | 60
SyncBatchNormBackward | 1.65% | 1.963s | 16.85% | 20.071s | 6.496ms | 80.482ms | 0.06% | 3.823s | 1.237ms | 3090
aten::_reshape_alias | 0.03% | 35.457ms | 0.03% | 35.457ms | 1.964us | 77.020ms | 0.06% | 77.020ms | 4.265us | 18058
aten::div_ | 0.02% | 22.501ms | 0.03% | 30.896ms | 35.842us | 74.688ms | 0.05% | 75.859ms | 88.003us | 862
aten::convolution | 0.08% | 94.326ms | 2.54% | 3.030s | 619.712us | 72.134ms | 0.05% | 25.755s | 5.267ms | 4890
aten::_convolution | 0.19% | 220.939ms | 2.47% | 2.936s | 600.423us | 68.615ms | 0.05% | 25.683s | 5.252ms | 4890
aten::transpose | 0.12% | 147.657ms | 0.14% | 161.542ms | 14.694us | 68.534ms | 0.05% | 130.471ms | 11.867us | 10994
aten::ge | 0.07% | 81.909ms | 0.10% | 116.833ms | 36.397us | 64.953ms | 0.05% | 64.953ms | 20.235us | 3210
aten::cudnn_convolution_transpose | 0.01% | 11.104ms | 0.03% | 34.225ms | 380.278us | 58.693ms | 0.04% | 67.167ms | 746.300us | 90
aten::squeeze | 0.07% | 81.836ms | 0.07% | 86.066ms | 18.043us | 56.443ms | 0.04% | 74.960ms | 15.715us | 4770
aten::full | 0.08% | 99.029ms | 0.13% | 153.570ms | 49.699us | 55.243ms | 0.04% | 90.889ms | 29.414us | 3090
aten::abs | 0.03% | 41.404ms | 0.06% | 75.755ms | 47.229us | 54.953ms | 0.04% | 99.147ms | 61.812us | 1604
aten::clone | 0.62% | 743.305ms | 0.95% | 1.134s | 307.879us | 53.085ms | 0.04% | 642.364ms | 174.461us | 3682
aten::dot | 0.02% | 28.681ms | 0.04% | 49.688ms | 61.955us | 46.876ms | 0.03% | 49.187ms | 61.330us | 802
aten::conv2d | 0.06% | 68.264ms | 2.56% | 3.051s | 635.609us | 41.811ms | 0.03% | 25.709s | 5.356ms | 4800
aten::neg | 0.01% | 12.326ms | 0.01% | 17.399ms | 19.506us | 40.134ms | 0.03% | 40.134ms | 44.993us | 892
aten::flatten_dense_tensors | 0.08% | 96.447ms | 0.10% | 115.561ms | 1.926ms | 39.576ms | 0.03% | 90.748ms | 1.512ms | 60
aten::set_ | 0.12% | 141.582ms | 0.12% | 141.582ms | 12.878us | 37.054ms | 0.03% | 37.054ms | 3.370us | 10994
aten::sgn | 0.01% | 10.734ms | 0.01% | 15.762ms | 19.653us | 36.918ms | 0.03% | 36.918ms | 46.032us | 802
autograd::engine::evaluate_function:  ConvolutionBack... | 0.14% | 163.685ms | 7.05% | 8.397s | 1.717ms | 36.218ms | 0.03% | 75.955s | 15.533ms | 4890
aten::sigmoid | 0.02% | 19.960ms | 0.02% | 26.123ms | 31.099us | 32.014ms | 0.02% | 32.014ms | 38.112us | 840
ConvolutionBackward0 | 0.07% | 82.317ms | 6.89% | 8.201s | 1.677ms | 30.754ms | 0.02% | 75.783s | 15.498ms | 4890
aten::relu_ | 0.13% | 153.385ms | 0.18% | 208.564ms | 73.180us | 26.268ms | 0.02% | 342.853ms | 120.299us | 2850
ReluBackward0 | 0.14% | 163.631ms | 3.74% | 4.448s | 1.215ms | 26.080ms | 0.02% | 435.328ms | 118.942us | 3660
autograd::engine::evaluate_function:  ReluBackward0 | 1.26% | 1.500s | 4.99% | 5.948s | 1.625ms | 25.104ms | 0.02% | 460.432ms | 125.801us | 3660
aten::contiguous | 0.03% | 34.204ms | 0.84% | 1.001s | 370.876us | 23.087ms | 0.02% | 288.258ms | 106.762us | 2700
autograd::engine::evaluate_function:  MulBackward0 | 0.22% | 257.082ms | 1.19% | 1.418s | 607.927us | 22.644ms | 0.02% | 517.881ms | 222.076us | 2332
aten::clamp_min | 0.01% | 11.062ms | 0.01% | 15.832ms | 19.546us | 21.387ms | 0.02% | 21.387ms | 26.404us | 810
MulBackward0 | 0.09% | 109.542ms | 0.17% | 204.256ms | 87.588us | 20.449ms | 0.01% | 378.814ms | 162.442us | 2332
autograd::engine::evaluate_function:  SyncBatchNormBa... | 18.30% | 21.798s | 35.16% | 41.869s | 13.550ms | 20.067ms | 0.01% | 3.843s | 1.244ms | 3090
aten::nll_loss2d_backward | 0.00% | 3.570ms | 0.01% | 7.052ms | 117.533us | 19.591ms | 0.01% | 38.506ms | 641.767us | 60
AsStridedBackward1 | 0.06% | 67.687ms | 0.19% | 220.483ms | 262.480us | 19.406ms | 0.01% | 90.206ms | 107.388us | 840
aten::select_backward | 0.13% | 158.524ms | 0.27% | 317.500ms | 221.718us | 18.842ms | 0.01% | 339.262ms | 236.915us | 1432
aten::nll_loss2d_forward | 0.00% | 3.116ms | 0.00% | 4.571ms | 76.183us | 18.613ms | 0.01% | 19.571ms | 326.183us | 60
aten::new_zeros | 0.04% | 51.710ms | 0.12% | 142.151ms | 86.572us | 17.260ms | 0.01% | 96.207ms | 58.591us | 1642
autograd::engine::evaluate_function:  AddBackward0 | 0.19% | 225.126ms | 0.19% | 226.791ms | 111.610us | 16.963ms | 0.01% | 25.421ms | 12.510us | 2032

<!--EndFragment-->
</body>
</html>

## System Info

PyTorch version: 1.8.2+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.18.4
Libc version: glibc-2.27

Python version: 3.8.6 (default, Oct  6 2020, 03:22:36)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-45-generic-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: 
GPU 0: Tesla V100-PCIE-32GB
GPU 1: Tesla V100-PCIE-32GB
GPU 2: Tesla V100-PCIE-32GB
GPU 3: Tesla V100-PCIE-32GB

Nvidia driver version: 450.36.06
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.0
[pip3] torch==1.8.2+cu102
[pip3] torch-scatter==2.0.5
[pip3] torchvision==0.9.2+cu102
[conda] Could not collect


cc @VitalyFedyunin @ngimel @csarofeen @ptrblck @xwang233