Enforce device ownership in SpatialConvolutionMM{_BHWD}