[Prims+NvFuser] Issue with aten.where.ScalarSelf
Labels: triaged, module: nvfuser

Running into **"Invoked with: <torch._C._nvfuser.FusionDefinition object at 0x7f6628a0d530>, <class 'TypeError'>, TypeError("where(): incompatible function arguments."** if I include aten.Where.ScalarSelf in the supported_op list. 


Full log here: 

skipping aten.embedding_dense_backward.default
skipping aten.addmm.default
skipping aten.native_layer_norm_backward.default
====== BERT_pytorch_forward_0 ======
Generating testing data...
aten2aten decomp:  aten.embedding.default <function embedding at 0x7f66317bb160>
aten2aten decomp:  aten.embedding.default <function embedding at 0x7f66317bb160>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.masked_fill.Scalar <function masked_fill_Scalar at 0x7f6631823790>
aten2aten decomp:  aten._softmax.default <function _softmax at 0x7f6631823c10>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f66317bbf70>
aten2aten decomp:  aten.std.correction <function std_decomposition at 0x7f66317bbd30>
aten2aten decomp:  aten.var.correction <function var_correction at 0x7f66317bbc10>
Partitions proposed:
partition 2 ['add_tensor_72', 'mul_tensor_47', 'sub_tensor_59', 'add_tensor_71', 'div_tensor_71']
partition 4 ['mul_tensor_46', 'sum_dim_int_list_35', 'sub_tensor_58', 'div_tensor_70']
partition 6 ['sub_tensor_57', 'sum_dim_int_list_34', 'div_tensor_69', 'exp_default_11']
partition 7 ['div_tensor_68', 'where_scalar_self_11', 'eq_scalar_11']
partition 8 ['mul_tensor_45', 'sub_tensor_56', 'add_tensor_68', 'div_tensor_67', 'add_tensor_69']
partition 10 ['sub_tensor_55', 'div_tensor_66', 'mul_tensor_44', 'sum_dim_int_list_33']
partition 13 ['mul_tensor_43', 'sub_tensor_54', 'add_tensor_65', 'div_tensor_65', 'add_tensor_66']
partition 15 ['mul_tensor_42', 'sum_dim_int_list_32', 'sub_tensor_53', 'div_tensor_64']
partition 17 ['exp_default_10', 'div_tensor_63', 'sub_tensor_52', 'sum_dim_int_list_31']
partition 18 ['div_tensor_62', 'eq_scalar_10', 'where_scalar_self_10']
partition 19 ['add_tensor_62', 'sub_tensor_51', 'mul_tensor_41', 'add_tensor_63', 'div_tensor_61']
partition 21 ['div_tensor_60', 'sub_tensor_50', 'sum_dim_int_list_30', 'mul_tensor_40']
partition 24 ['add_tensor_59', 'div_tensor_59', 'add_tensor_60', 'mul_tensor_39', 'sub_tensor_49']
partition 26 ['sum_dim_int_list_29', 'sub_tensor_48', 'div_tensor_58', 'mul_tensor_38']
partition 28 ['exp_default_9', 'sum_dim_int_list_28', 'sub_tensor_47', 'div_tensor_57']
partition 29 ['div_tensor_56', 'where_scalar_self_9', 'eq_scalar_9']
partition 30 ['div_tensor_55', 'add_tensor_57', 'add_tensor_56', 'sub_tensor_46', 'mul_tensor_37']
partition 32 ['sum_dim_int_list_27', 'mul_tensor_36', 'div_tensor_54', 'sub_tensor_45']
partition 35 ['div_tensor_53', 'add_tensor_54', 'mul_tensor_35', 'sub_tensor_44', 'add_tensor_53']
partition 37 ['sum_dim_int_list_26', 'sub_tensor_43', 'div_tensor_52', 'mul_tensor_34']
partition 39 ['exp_default_8', 'sum_dim_int_list_25', 'sub_tensor_42', 'div_tensor_51']
partition 40 ['div_tensor_50', 'where_scalar_self_8', 'eq_scalar_8']
partition 41 ['add_tensor_51', 'div_tensor_49', 'add_tensor_50', 'sub_tensor_41', 'mul_tensor_33']
partition 43 ['mul_tensor_32', 'div_tensor_48', 'sub_tensor_40', 'sum_dim_int_list_24']
partition 46 ['add_tensor_48', 'div_tensor_47', 'add_tensor_47', 'sub_tensor_39', 'mul_tensor_31']
partition 48 ['sum_dim_int_list_23', 'mul_tensor_30', 'div_tensor_46', 'sub_tensor_38']
partition 50 ['sum_dim_int_list_22', 'exp_default_7', 'div_tensor_45', 'sub_tensor_37']
partition 51 ['where_scalar_self_7', 'div_tensor_44', 'eq_scalar_7']
partition 52 ['sub_tensor_36', 'mul_tensor_29', 'add_tensor_45', 'div_tensor_43', 'add_tensor_44']
partition 54 ['mul_tensor_28', 'div_tensor_42', 'sub_tensor_35', 'sum_dim_int_list_21']
partition 57 ['add_tensor_42', 'div_tensor_41', 'add_tensor_41', 'sub_tensor_34', 'mul_tensor_27']
partition 59 ['mul_tensor_26', 'div_tensor_40', 'sub_tensor_33', 'sum_dim_int_list_20']
partition 61 ['sum_dim_int_list_19', 'exp_default_6', 'div_tensor_39', 'sub_tensor_32']
partition 62 ['where_scalar_self_6', 'div_tensor_38', 'eq_scalar_6']
partition 63 ['add_tensor_38', 'sub_tensor_31', 'mul_tensor_25', 'add_tensor_39', 'div_tensor_37']
partition 65 ['mul_tensor_24', 'div_tensor_36', 'sub_tensor_30', 'sum_dim_int_list_18']
partition 68 ['add_tensor_35', 'sub_tensor_29', 'mul_tensor_23', 'add_tensor_36', 'div_tensor_35']
partition 70 ['mul_tensor_22', 'div_tensor_34', 'sub_tensor_28', 'sum_dim_int_list_17']
partition 72 ['sum_dim_int_list_16', 'exp_default_5', 'div_tensor_33', 'sub_tensor_27']
partition 73 ['where_scalar_self_5', 'div_tensor_32', 'eq_scalar_5']
partition 74 ['add_tensor_33', 'div_tensor_31', 'add_tensor_32', 'sub_tensor_26', 'mul_tensor_21']
partition 76 ['div_tensor_30', 'sub_tensor_25', 'sum_dim_int_list_15', 'mul_tensor_20']
partition 79 ['div_tensor_29', 'add_tensor_29', 'sub_tensor_24', 'mul_tensor_19', 'add_tensor_30']
partition 81 ['mul_tensor_18', 'div_tensor_28', 'sub_tensor_23', 'sum_dim_int_list_14']
partition 83 ['sum_dim_int_list_13', 'exp_default_4', 'div_tensor_27', 'sub_tensor_22']
partition 84 ['div_tensor_26', 'where_scalar_self_4', 'eq_scalar_4']
partition 85 ['mul_tensor_17', 'sub_tensor_21', 'add_tensor_26', 'div_tensor_25', 'add_tensor_27']
partition 87 ['sub_tensor_20', 'div_tensor_24', 'mul_tensor_16', 'sum_dim_int_list_12']
partition 90 ['mul_tensor_15', 'sub_tensor_19', 'add_tensor_23', 'div_tensor_23', 'add_tensor_24']
partition 92 ['mul_tensor_14', 'sum_dim_int_list_11', 'sub_tensor_18', 'div_tensor_22']
partition 94 ['exp_default_3', 'sum_dim_int_list_10', 'sub_tensor_17', 'div_tensor_21']
partition 95 ['div_tensor_20', 'where_scalar_self_3', 'eq_scalar_3']
partition 96 ['add_tensor_20', 'div_tensor_19', 'add_tensor_21', 'mul_tensor_13', 'sub_tensor_16']
partition 98 ['sub_tensor_15', 'div_tensor_18', 'mul_tensor_12', 'sum_dim_int_list_9']
partition 101 ['sub_tensor_14', 'mul_tensor_11', 'add_tensor_18', 'div_tensor_17', 'add_tensor_17']
partition 103 ['sum_dim_int_list_8', 'mul_tensor_10', 'div_tensor_16', 'sub_tensor_13']
partition 105 ['exp_default_2', 'div_tensor_15', 'sub_tensor_12', 'sum_dim_int_list_7']
partition 106 ['div_tensor_14', 'eq_scalar_2', 'where_scalar_self_2']
partition 107 ['add_tensor_14', 'add_tensor_15', 'sub_tensor_11', 'div_tensor_13', 'mul_tensor_9']
partition 109 ['mul_tensor_8', 'div_tensor_12', 'sub_tensor_10', 'sum_dim_int_list_6']
partition 112 ['add_tensor_11', 'sub_tensor_9', 'mul_tensor_7', 'add_tensor_12', 'div_tensor_11']
partition 114 ['mul_tensor_6', 'div_tensor_10', 'sub_tensor_8', 'sum_dim_int_list_5']
partition 116 ['sum_dim_int_list_4', 'exp_default_1', 'div_tensor_9', 'sub_tensor_7']
partition 117 ['where_scalar_self_1', 'div_tensor_8', 'eq_scalar_1']
partition 118 ['mul_tensor_5', 'sub_tensor_6', 'add_tensor_9', 'div_tensor_7', 'add_tensor_8']
partition 120 ['sub_tensor_5', 'div_tensor_6', 'mul_tensor_4', 'sum_dim_int_list_3']
partition 123 ['add_tensor_6', 'sub_tensor_4', 'div_tensor_5', 'mul_tensor_3', 'add_tensor_5']
partition 125 ['sub_tensor_3', 'div_tensor_4', 'mul_tensor_2', 'sum_dim_int_list_2']
partition 127 ['sub_tensor_2', 'div_tensor_3', 'exp_default', 'sum_dim_int_list_1']
partition 128 ['where_scalar_self', 'div_tensor_2', 'eq_scalar']
partition 129 ['div_tensor_1', 'mul_tensor_1', 'sub_tensor_1', 'add_tensor_3', 'add_tensor_2']
partition 131 ['sum_dim_int_list', 'div_tensor', 'sub_tensor', 'mul_tensor']
partition 132 ['add_tensor', 'add_tensor_1']
num_partitions:  73
Eager execution time: 95.252 ms
Aten decomp time: 112.277 ms
aten2prim decomp:  aten.add.Tensor <function add at 0x7f66317a04c0>
aten2prim decomp:  aten.add.Tensor <function add at 0x7f66317a04c0>
aten2prim decomp:  aten.sub.Tensor <function sub at 0x7f663176a310>
aten2prim decomp:  aten.mul.Tensor <function _make_elementwise_binary_reference.<locals>._ref at 0x7f66317579d0>
aten2prim decomp:  aten.sum.dim_IntList <function sum at 0x7f66316fa280>
aten2prim decomp:  aten.div.Tensor <function div at 0x7f6631736280>
aten2prim decomp:  aten.sub.Tensor <function sub at 0x7f663176a310>
aten2prim decomp:  aten.add.Tensor <function add at 0x7f66317a04c0>
aten2prim decomp:  aten.mul.Tensor <function _make_elementwise_binary_reference.<locals>._ref at 0x7f66317579d0>
aten2prim decomp:  aten.div.Tensor <function div at 0x7f6631736280>
aten2prim decomp:  aten.add.Tensor <function add at 0x7f66317a04c0>
aten2prim decomp:  aten.div.Tensor <function div at 0x7f6631736280>
aten2prim decomp:  aten.eq.Scalar <function _make_elementwise_binary_reference.<locals>._ref at 0x7f66317363a0>
aten2prim decomp:  aten.where.self <function where at 0x7f663176adc0>
nvFuser 1st call execution time: 2703.377 ms
**BERT_pytorch_forward_0 failed! __exit__(): incompatible function arguments. The following argument types are supported:
    1. (self: torch._C._nvfuser.FusionDefinition, arg0: capsule, arg1: capsule, arg2: capsule) -> None**

Invoked with: <torch._C._nvfuser.FusionDefinition object at 0x7f6628a0d530>, <class 'TypeError'>, TypeError("where(): incompatible function arguments. The following argument types are supported:\n    1. (arg0: torch._C._nvfuser.TensorView, arg1: torch._C._nvfuser.TensorView, arg2: torch._C._nvfuser.TensorView) -> torch._C._nvfuser.TensorView\n    2. (arg0: torch._C._nvfuser.TensorView, arg1: torch._C._nvfuser.TensorView, arg2: torch._C._nvfuser.Val) -> torch._C._nvfuser.TensorView\n    3. (arg0: torch._C._nvfuser.TensorView, arg1: torch._C._nvfuser.Val, arg2: torch._C._nvfuser.TensorView) -> torch._C._nvfuser.TensorView\n    4. (arg0: torch._C._nvfuser.TensorView, arg1: torch._C._nvfuser.Val, arg2: torch._C._nvfuser.Val) -> torch._C._nvfuser.TensorView\n    5. (arg0: torch._C._nvfuser.Val, arg1: torch._C._nvfuser.TensorView, arg2: torch._C._nvfuser.TensorView) -> torch._C._nvfuser.TensorView\n    6. (arg0: torch._C._nvfuser.Val, arg1: torch._C._nvfuser.TensorView, arg2: torch._C._nvfuser.Val) -> torch._C._nvfuser.TensorView\n    7. (arg0: torch._C._nvfuser.Val, arg1: torch._C._nvfuser.Val, arg2: torch._C._nvfuser.TensorView) -> torch._C._nvfuser.TensorView\n    8. (arg0: torch._C._nvfuser.Val, arg1: torch._C._nvfuser.Val, arg2: torch._C._nvfuser.Val) -> torch._C._nvfuser.Val\n\nInvoked with: <torch._C._nvfuser.TensorView object at 0x7f6621060ef0>, tensor([[[[-.......e+09,\n           -1.0000e+09, -1.0000e+09]]]], device='cuda:0'), <torch._C._nvfuser.TensorView object at 0x7f66210610f0>"), <traceback object at 0x7f66210198c0>

non_fusible_op {'aten.sqrt.default', 'aten.mean.dim', 'aten.to.dtype', 'aten.amax.default', 'aten.gt.Scalar', '<built-in function getitem>', 'aten.add.Tensor', 'aten.index_select.default', 'aten._unsafe_view.default', 'aten.gelu.default', 'aten.t.default', 'aten.repeat.default', 'aten.reshape.default', 'aten.slice.Tensor', 'aten.unsqueeze.default', 'aten.addmm.default', 'aten.clone.default', 'aten.view.default', 'aten.bmm.default', 'aten.permute.default', 'aten.expand.default'}
BERT_pytorch_forward_0	73	95.25213693268597	112.27697692811489	None	None	None