Package: tvdcn
Torchvision+ Deformable Convolution NetworksThis package contains the PyTorch implementations of theDeformable Convolutionoperation
(the commonly usedtorchvision.ops.deform_conv2d) proposed inhttps://arxiv.org/abs/1811.11168,
and theTransposed Deformable Convolutionproposed inhttps://arxiv.org/abs/2210.09446(currently without interpolation kernel scaling).
It also supports their1Dand3Dequivalences, which are not available intorchvision(thus the name).HighlightsSupported operators:(All are implemented in C++/Cuda)tvdcn.ops.deform_conv1dtvdcn.ops.deform_conv2d(faster thantorchvision.ops.deform_conv2dby at least 10% during forward pass
on ourQuadro RTX 5000according tothis test)tvdcn.ops.deform_conv3dtvdcn.ops.deform_conv_transpose1dtvdcn.ops.deform_conv_transpose2dtvdcn.ops.deform_conv_transpose3dAnd the followingsupplementary operators(maskactivation proposed inhttps://arxiv.org/abs/2211.05778):tvdcn.ops.mask_softmax1dtvdcn.ops.mask_softmax2dtvdcn.ops.mask_softmax3dBothoffsetandmaskcan be turned off, and can be applied in separate groups.All thenn.Modulewrappers for these operators are implemented,
everything is@torch.jit.script-able! Please checkUsage.Note:We don't care much aboutonnxexportation, but if you do, you can check this repo:https://github.com/masamitsu-murase/deform_conv2d_onnx_exporter.Requirementstorch>=2.1.0(torch>=1.9.0if installed from source)InstallationFrom PyPI:tvdcnprovides some prebuilt wheels onPyPI.
Run this command to install:pip install tvdcnSincePyTorchis migrating to Cuda 12 versions,
our Linux and Windows wheels are built withCuda 12.1and won't be compatible with older versions.Linux/WindowsMacOSPython version:3.8-3.113.8-3.11PyTorch version:torch==2.1.0torch==2.1.0Cuda version:12.1-GPU CCs:5.0,6.0,6.1,7.0,7.5,8.0,8.6,9.0+PTX-When the Cuda versions oftorchandtvdcnmismatch, you will see an error like this:RuntimeError: Detected that PyTorch and Extension were compiled with different CUDA versions.
PyTorch has CUDA Version=11.8 and Extension has CUDA Version=12.1.
Please reinstall the Extension that matches your PyTorch install.If you see this error instead, that means there are other issues related to Python, PyTorch, device arch, e.t.c.
Please proceed toinstructions to build from source, all steps are super easy.RuntimeError: Couldn't load custom C++ ops. Recompile C++ extension with:
     python setup.py build_ext --inplaceFrom Source:For installing from source, you need a C++ compiler (gcc/msvc) and a Cuda compiler (nvcc) with C++17 features
enabled.
Clone this repo and execute the following command:pip install .Or just compile the binary for inplace usage:python setup.py build_ext --inplaceA binary (.sofile for Unix and.pydfile for Windows) should be compiled inside thetvdcnfolder.
To check if installation is successful, try:importtvdcnprint('Library loaded successfully:',tvdcn.has_ops())print('Compiled with Cuda:',tvdcn.with_cuda())Note:We use soft Cuda version compatibility checking between the built binary and the installed PyTorch,
which means only major version matching is required.
However, we suggest building the binaries with the same Cuda version with installed PyTorch's Cuda version to prevent
any possible conflict.UsageOperators:Functionally, the package offers 6 functions (listed inHighlights) much similar totorchvision.ops.deform_conv2d.
However, the order of parameters is slightly different, so be cautious
(checkthis comparison).torchvisiontvdcnimporttorch,torchvisioninput=torch.rand(4,3,10,10)kh,kw=3,3weight=torch.rand(5,3,kh,kw)offset=torch.rand(4,2*kh*kw,8,8)mask=torch.rand(4,kh*kw,8,8)bias=torch.rand(5)output=torchvision.ops.deform_conv2d(input,offset,weight,bias,stride=(1,1),padding=(0,0),dilation=(1,1),mask=mask)print(output)importtorch,tvdcninput=torch.rand(4,3,10,10)kh,kw=3,3weight=torch.rand(5,3,kh,kw)offset=torch.rand(4,2*kh*kw,8,8)mask=torch.rand(4,kh*kw,8,8)bias=torch.rand(5)output=tvdcn.ops.deform_conv2d(input,weight,offset,mask,bias,stride=(1,1),padding=(0,0),dilation=(1,1),groups=1)print(output)Specifically, the signatures ofdeform_conv2danddeform_conv_transpose2dlook like these:defdeform_conv2d(input:Tensor,weight:Tensor,offset:Optional[Tensor]=None,mask:Optional[Tensor]=None,bias:Optional[Tensor]=None,stride:Union[int,Tuple[int,int]]=1,padding:Union[int,Tuple[int,int]]=0,dilation:Union[int,Tuple[int,int]]=1,groups:int=1)->Tensor:...defdeform_conv_transpose2d(input:Tensor,weight:Tensor,offset:Optional[Tensor]=None,mask:Optional[Tensor]=None,bias:Optional[Tensor]=None,stride:Union[int,Tuple[int,int]]=1,padding:Union[int,Tuple[int,int]]=0,output_padding:Union[int,Tuple[int,int]]=0,dilation:Union[int,Tuple[int,int]]=1,groups:int=1)->Tensor:...Ifoffset=Noneandmask=None, the executed operators are identical to conventional convolution.Neural Network Layers:Thenn.Modulewrappers are:tvdcn.ops.DeformConv1dtvdcn.ops.DeformConv2dtvdcn.ops.DeformConv3dtvdcn.ops.DeformConvTranspose1dtvdcn.ops.DeformConvTranspose2dtvdcn.ops.DeformConvTranspose3dThey are subclasses of thetorch.nn.modules._ConvNd,
but you have to specifyoffsetand optionallymaskas extra inputs for theforwardfunction.
For example:importtorchfromtvdcnimportDeformConv2dinput=torch.rand(2,3,64,64)offset=torch.rand(2,2*3*3,62,62)# if mask is None, perform the original deform_conv without modulation (v2)mask=torch.rand(2,1*3*3,62,62)conv=DeformConv2d(3,16,kernel_size=(3,3))output=conv(input,offset,mask)print(output.shape)Additionally, following many other implementations out there, we also implemented thepackedwrappers:tvdcn.ops.PackedDeformConv1dtvdcn.ops.PackedDeformConv2dtvdcn.ops.PackedDeformConv3dtvdcn.ops.PackedDeformConvTranspose1dtvdcn.ops.PackedDeformConvTranspose2dtvdcn.ops.PackedDeformConvTranspose3dThese are easy-to-use classes that contain ordinary convolution layers with appropriate hyperparameters to generateoffset(andmaskif initialized withmodulated=True);
but that means less customization.
The only tunable hyperparameters that effect these supplementary conv layers areoffset_groupsandmask_groups,
which have been decoupled from and behave somewhat similar togroups.To use the softmax activation for mask proposed inDeformable Convolution v3,
setmask_activation='softmax'.offset_activationandmask_activationalso accept anynn.Module.importtorchfromtvdcnimportPackedDeformConv1dinput=torch.rand(2,3,128)conv=PackedDeformConv1d(3,16,kernel_size=5,modulated=True,mask_activation='softmax')# jit scriptingscripted_conv=torch.jit.script(conv)print(scripted_conv)output=scripted_conv(input)print(output.shape)Note:For transposed packed modules, we are generatingoffsetandmaskwith pointwise convolution
as we haven't found a better way to do it.Do check theexamplesfolder, maybe you can find something helpful.AcknowledgementsThisfor funproject is directly modified and extended fromtorchvision.ops.deform_conv2d.LicenseThe code is released under the MIT license. SeeLICENSE.txtfor details.