Change:
max pooling functions are not consistent with max functions. Below an example, every max pooling (be it 1d, 2d or 3d, adaptive or not) acts the same, on cpu or on cuda. Essentially, there are two fondamental differences : max pooling of all values is while for it's max pooling of nan and valid values is valid values, which means s get ignored, while for , as soon as there is a value, the result is . More generally, choosing explicetely how to deal with as in numpy () could be a solution, but maybe this is related to CuDNN's max pooling ? Built from latest sources (as of 05/17) PyTorch version: 0.5.0a0+ Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 CMake version: version 3.9.4 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 9.1.85 GPU models and configuration: GPU 0: Quadro M1000M Nvidia driver version: 390.30 cuDNN version: Probably one of the following: /usr/lib/x8664-linux- /usr/lib/x8664-linux-gnu/libcudnnstaticv7.a Versions of relevant libraries: [conda] magma-cuda91 2.3.0 1 pytorch [conda] torch 0.5.0a0+ <pip[conda] torch 0.3.1b0+ <pip[conda] torch 0.5.0a0+ <pip
Hi, . Thank you for providing the example. I'll take a look.
Hmhm. I seem to get a double free in the ...
Another question: Do we really want to provide an option to ignore NaN? My view is that "if you have nan in your net, you're screwed", so I would just return NaN there, personally.
My practical usecase is for KITTI groundtruth depthmaps and FlowMaps which are 2D sparse arrays. For an algorithm that use FlowNet-like architecture that outputs predictions at multiple scale levels, we can either compare predictions to downscaled GT or upscaled predictions to GT. The first being obviously less computationally expensive, the ignore NaN would help downscaling such sparse 2D maps. For the moment we do something I find ugly, you can see it Essentially it zeroes the s, takes the map and construct two maps of positive and negative values which are then maxpooled and back together. I am actually open for a strict "no-nan" policy on pooling functions, but in that case better enforce it before someone writes a code that tries to benefit from maxpooling ignore s feature/bug and if you have a clever way of pooling sparse 2D tensors, I'm open to it, but I guess it's a topic for pytorch forums ;)
Personally, I think it is more sane have NaN -NaN in the pooling and offer a parametrizable (where you get to pick the values) that does or so.
So the proposed fix does NaN -NaN similar to max. I didn't try to fix gradients for the nan case. This would involve keeping the values and I don't think that is worth it (in particular because I would not expect the pooling layer to be last, and otherwise we'd probably get NaN as grad_out). If you are reasonably happy with it, I'd move it to a PR.
+1 for NaN -NaN, "abyssus abyssum invocat" We could also add an optional mask which would be a ByteTensor of the same size, specifying whether or not the considered pixel is used for the pooling, that could be used for any kind of pulling. An functionality would then be to provide the mask
I could work with that. to continue to be picky, the potential drawback is that it won't work for other pooling methods such as average pooling or median pooling. The problem here is that the max operation inherently ignores nonmax values which can be leveraged for "ignore some pixels" operation, but it has a "non universal" feel to it since it woun't be as easy for other kinds of 2D operations.
To be clear, the fix is good for me, but I figured a related discussion on selective pooling (whether to ditch NaN values or anything you want to ignore) could happen (maybe not on this issue ?) It would be convenient for my use case, but maybe for others too.
fixed via

Discussion:
for(ih = 0; ih < kH; ih++) { for(iw = 0; iw < kW; iw++) { T val = ptr_input[iw*istrideW]; <del> if (val > max) { </del> <ins> if ((val > max) || THCNumerics<T>::isnan(val)) { </ins> max = val; argmax = (ih+istartH)*isizeW + iw+istartW; }

#include "THCTensor.hpp" #include "THCHalf.h" #include "THCHalfAutoNumerics.cuh" <ins> #include "THCNumerics.cuh" </ins> #include "common.h" // kernels borrowed from Caffe

bottom_data += (n * channels + c) * height * width; for (int h = hstart; h < hend; h += dilation_h) { for (int w = wstart; w < wend; w += dilation_w) { <del> if (ScalarConvert<Dtype, AccType>::to(bottom_data[h * width + w]) > maxval) { </del> <ins> Dtype val = bottom_data[h * width + w]; if ((ScalarConvert<Dtype, AccType>::to(val) > maxval) || THCNumerics<Dtype>::isnan(val)) { </ins> maxidx = h * width + w; <del> maxval = ScalarConvert<Dtype, AccType>::to(bottom_data[maxidx]); </del> <ins> maxval = ScalarConvert<Dtype, AccType>::to(val); </ins> } } }

for(ih = 0; ih < kH; ++ih) { for(iw = 0; iw < kW; ++iw) { T val = ptr_input[ih*istrideH + iw*istrideW]; <del> if (val > max) { </del> <ins> if ((val > max) || THCNumerics<T>::isnan(val)) { </ins> max = val; argmax = (it+istartT)*isizeH*isizeW + (ih+istartH)*isizeW + iw+istartW; }

index = t * inputH * inputW + h * inputW + w; Dtype val = inputData[index]; <del> if (max < val) </del> <ins> if ((max < val) || THCNumerics<Dtype>::isnan(val)) </ins> { max = val; maxIndex = index;

for(iw = 0; iw < kW; iw++) { real val = *(ip + ih*istrideH + iw*istrideW); <del> if (val > maxval) </del> <ins> if ((val > maxval) || isnan(val)) </ins> { maxval = val; maxindex = (ih+istartH)*isizeW + (iw+istartW);

{ tcntr = y*iwidth + x; real val = *(ip + tcntr); <del> if (val > maxval) </del> <ins> if ((val > maxval) || isnan(val)) </ins> { maxval = val; maxindex = tcntr;

for(iw = 0; iw < kW; iw++) { real val = *(ip + it*istrideT + ih*istrideH + iw*istrideW); <del> if (val > maxval) </del> <ins> if ((val > maxval) || isnan(val)) </ins> { maxval = val; maxindex = (it+istartT)*isizeH*isizeW + (ih+istartH)*isizeW + (iw+istartW);

{ index = z * iwidth * iheight + y * iwidth + x; real val = ip[index]; <del> if (val > maxval) </del> <ins> if ((val > maxval) || isnan(val)) </ins> { maxval = val; maxindex = index;

def test_AdaptiveMaxPool3d_indices_cuda(self, dtype=torch.float): self._test_maxpool_indices(3, adaptive=True, device="cuda", dtype=dtype) <ins> @staticmethod def _test_max_pool_nan(self, device, dtype=torch.float): for adaptive in ['', 'adaptive_']: for num_dim in [1, 2, 3]: fn_name = '{}max_pool{}d'.format(adaptive, num_dim) fn = getattr(F, fn_name) x = torch.full([1, 1] + num_dim * [3], float('nan')) res = fn(x, 1 if adaptive else 3) self.assertTrue(math.isnan(res.item())) @unittest.skipIf(not TEST_CUDA, "CUDA unavailable") @repeat_test_for_types(ALL_TENSORTYPES) def test_max_pool_nan_cuda(self, dtype=torch.float): self._test_max_pool_nan(self, device="cuda", dtype=dtype) def test_max_pool_nan(self, dtype=torch.float): self._test_max_pool_nan(self, device="cpu") </ins> def _test_scatter(self, tensor): x = torch.tensor(tensor, requires_grad=True) result = dp.scatter(x, (0, 1))