Select correct device before freeing CUDA buffer