Repository: apache/tvm

Issue:
[TOPI] Add dilation argument to conv2d and depthwise_conv2d
Currently `dilate` is a separate operation and one has to call `dilate` + `conv2d` to compose dilated convolution. 
The problems:
* Sometimes it is better to combine both into a single operator because we may want to write the expresion of dilated convolution directly instread of constructing a dilated kernel and then call `compute_inline`.
* The workload in AutoTVM does not reflect dilation. For example, we will use the same configuration for 5x5 kernel and 3x3 kernel with dilation=2 because they have the same workload.

This PR adds `dilation` as a new argument to `conv2d` / `depthwise_conv2d`. If dilation is used, it will call `nn.dilate` internally in conv2d (like `nn.pad`). The workload is also updated. This will invalidate AutoTVM logs generated previously.

cc @merrymercy @tqchen @masahi @nishi-t @anijain2305 @Huyuwei 


Discussion:
Currently, the log format of autotvm depends on arguments. So every time we change arguments, we will invalidate all old log records. Should we use another design? @tqchen 

Since this PR will break old logs, you should provide a log converter for other users.
I can test you converter and update the logs in TopHub.
I think in this case, adding dilation makes sense, because this makes things consistent with the high level op. We do need to provide a quick upgrade path though, please try make API consistent with high level op https://docs.tvm.ai/langref/relay_op.html#level-2-definitions
@merrymercy Are there cases where dilation is used in tophub logs? To convert old logs to new workload, I will add a dilation field to all old logs. Since we don't have dilation previously, if dilation is used we will have dilated kernel shape in the workload. 
No dilation is used in tophub. 
here is the converter for old logs
https://gist.github.com/vinx13/6c9231843a5833f31de7eab938b345b1
@vinx13 please check the CI errors, then ask for review again
the test of dilated conv2d nhwc is incorrect, see https://github.com/dmlc/tvm/blob/42dc24a310170577f929f648f477ca2567c8bc9a/topi/tests/python/test_topi_conv2d_nhwc.py#L16
https://github.com/dmlc/tvm/blob/42dc24a310170577f929f648f477ca2567c8bc9a/topi/python/topi/nn/conv2d.py#L277
the dilation should be [dilation_h, dilation_w, 1, 1]

@tqchen How can I update pickle-memorized reference data?
Currently x86 backend in under refactor #1993 .
I think it is better to merge that first. So you can stop all your changes on `topi/x86/*`.

To update a pickle-memorized data, a trick is to rename the string.
https://github.com/dmlc/tvm/blob/42dc24a310170577f929f648f477ca2567c8bc9a/topi/tests/python/test_topi_conv2d_nhwc.py#L23
@merrymercy I have looked into the error in http://ci.tvm.ai:8080/blue/organizations/jenkins/tvm/detail/PR-1970/21/pipeline/232#step-255-log-1230. There is one failing case of nchw conv2d on nvptx because tophub logs are invalidated in this pr. The fallback schedule from `fallback_with_reference_log` produced wrong conv2d result.
There are some errors with nvptx backend. We found some cases before.

The solution can be
1. Update llvm in CI to 6.0
2. Disable that test case (for nvptx)
> There are some errors with nvptx backend. We found some cases before.
> 
> The solution can be
> 
> 1. Update llvm in CI to 6.0
> 2. Disable that test case (for nvptx)

I tried llvm-6.0 locally but the error still occurs. This error can be fixed by updating tophub logs with dilation arg, or disabling fallback schedule generated from reference log
OK. We will first merge #2034, and then merge this PR.

In the meanwhile, you can do update to tophub

1. Use your converter to convert all latest logs under https://github.com/uwsampl/tvm-distro/tree/master/tophub
    - arm_cpu_v0.03 -> arm_cpu_v0.04, cuda_v0.03 -> cuda_v0.04,  opencl_v0.01 -> opencl_v0.02, ..., the only special case is **vta_v0.01 -> vta_v0.04**
    - Note there are also dense layer workloads. So your convert should only convert conv layers.
    - Then I will merge your PR
2. Update tophub versions for all backends in tvm
    -  Update them to your version
 https://github.com/dmlc/tvm/blob/b840e9602e357c50124d0c7fb131c52321062570/python/tvm/autotvm/tophub.py#L22-L31
    - Then you should be able to pass your test
Can you also help to update the style of cuda winograd and cuda int8 according to #2034 ?
So we can delete all strange logic in things like `_conv2d_arg_to_workloads`
Thanks @merrymercy @vinx13 this is now merged

Resolving patch:
diff --git a/topi/python/topi/arm_cpu/conv2d.py b/topi/python/topi/arm_cpu/conv2d.py
--- a/topi/python/topi/arm_cpu/conv2d.py
+++ b/topi/python/topi/arm_cpu/conv2d.py
@@ -2,6 +2,8 @@
 """Conv2D schedule for ARM CPU"""
 from __future__ import absolute_import as _abs
 
+import warnings
+
 import numpy as np
 
 import tvm
@@ -522,7 +524,10 @@ def _alter_conv2d_layout_arm(attrs, inputs, tinfos):
     out_dtype = attrs["out_dtype"]
     out_dtype = tinfos[0].dtype if out_dtype == "same" else out_dtype
 
-    if layout != 'NCHW' or groups != 1 or dilation != (1, 1):
+    if layout != 'NCHW' or groups != 1:
+        return None
+    if dilation != (1, 1):
+        warnings.warn("Does not support weight pre-transform for dilated convolution.")
         return None
 
     data, kernel = tinfos[0:2]
@@ -531,7 +536,7 @@ def _alter_conv2d_layout_arm(attrs, inputs, tinfos):
 
     # query config of this workload
     workload = autotvm.task.args_to_workload(
-        [data, kernel, strides, padding, layout, out_dtype], conv2d)
+        [data, kernel, strides, padding, dilation, layout, out_dtype], conv2d)
     target = tvm.target.current_target()
     dispatch_ctx = autotvm.DispatchContext.current
     cfg = dispatch_ctx.query(target, workload)
@@ -548,7 +553,7 @@ def _alter_conv2d_layout_arm(attrs, inputs, tinfos):
         new_data = data
         new_kernel = tvm.placeholder((CO // VC, CI, KH, KW, VC), dtype=kernel.dtype)
         new_workload = autotvm.task.args_to_workload(
-            [new_data, new_kernel, strides, padding, 'NCHW', out_dtype], conv2d)
+            [new_data, new_kernel, strides, padding, dilation, 'NCHW', out_dtype], conv2d)
         dispatch_ctx.update(target, new_workload, cfg)
 
         return sym.conv2d(*copy_inputs, **new_attrs)
@@ -574,7 +579,8 @@ def _alter_conv2d_layout_arm(attrs, inputs, tinfos):
         new_weight = tvm.placeholder((KH + tile_size - 1, KH + tile_size -1, CO // VC, CI, VC),
                                      kernel.dtype)
         new_workload = autotvm.task.args_to_workload(
-            [new_data, new_weight, strides, padding, new_attrs['layout'], out_dtype, tile_size],
+            [new_data, new_weight, strides, padding, dilation,
+             new_attrs['layout'], out_dtype, tile_size],
             conv2d_winograd_without_weight_transform)
         dispatch_ctx.update(target, new_workload, cfg)
 
diff --git a/topi/python/topi/cuda/conv2d_winograd.py b/topi/python/topi/cuda/conv2d_winograd.py
--- a/topi/python/topi/cuda/conv2d_winograd.py
+++ b/topi/python/topi/cuda/conv2d_winograd.py
@@ -375,6 +375,8 @@ def _alter_conv2d_layout(attrs, inputs, tinfos):
             new_attrs['out_layout'] = new_layout
             new_attrs['kernel_layout'] = 'OIHW4o4i'
             ic_block_factor = oc_block_factor = 4
+
+            # Store the same config for the altered operator (workload)
             new_data = tvm.placeholder((N, CI // ic_block_factor, H, W, ic_block_factor),
                                        dtype=data.dtype)
             new_kernel = tvm.placeholder((CO // oc_block_factor, CI // ic_block_factor, KH, KW,\
@@ -387,7 +389,9 @@ def _alter_conv2d_layout(attrs, inputs, tinfos):
             return sym.conv2d(*copy_inputs, **new_attrs)
 
         if attrs.get_int_tuple("dilation") != (1, 1):
+            warnings.warn("Does not support weight pre-transform for dilated convolution.")
             return None
+
         # pre-compute weight transformation in winograd
         tile_size = _infer_tile_size(tinfos[0], tinfos[1])
 
@@ -397,6 +401,7 @@ def _alter_conv2d_layout(attrs, inputs, tinfos):
         copy_inputs[1] = weight
         new_attrs['tile_size'] = tile_size
 
+        # Store the same config for the altered operator (workload)
         new_data = data
         new_weight = tvm.placeholder((KH + tile_size - 1, KW + tile_size - 1, CI, CO),
                                      dtype=kernel.dtype)
diff --git a/topi/python/topi/mali/conv2d.py b/topi/python/topi/mali/conv2d.py
--- a/topi/python/topi/mali/conv2d.py
+++ b/topi/python/topi/mali/conv2d.py
@@ -440,7 +440,7 @@ def _schedule_winograd(cfg, s, op):
 
 ##### REGISTER TOPI COMPUTE / SCHEDULE FOR WINOGRAD WITH WEIGHT TRANSFORM #####
 @autotvm.register_topi_compute(conv2d_winograd_without_weight_transform, 'mali', ['winograd'])
-def conv2d_winograd_ww(cfg, data, kernel, strides, padding, layout, out_dtype, tile_size):
+def conv2d_winograd_ww(cfg, data, kernel, strides, padding, dilation, layout, out_dtype, tile_size):
     """TOPI compute callback"""
     return _decl_winograd(cfg, data, kernel, strides, padding, dilation, layout, out_dtype,
                           tile_size)


Test patch: