Repository: cupy/cupy
Pull request: Add APIs for creating NumPy arrays backed by pinned memory
Close #3625. 

This PR adds a few convenience functions to the `cupyx` namespace for allocating NumPy arrays backed by pinned/pagelocked memory, see the discussion around https://github.com/cupy/cupy/issues/3625#issuecomment-704384736. Currently the provided APIs are as follows:
- `cupyx.empty_pinned()`
- `cupyx.empty_like_pinned()`
- `cupyx.zeros_pinned()`
- `cupyx.zeros_like_pinned()`

with exactly the semantics as their NumPy counterparts.

Ideally, we can do better than this if we use NumPy's memory hooks to always allocate NumPy arrays from pinned memory (https://github.com/numpy/numpy/issues/17467), however it's still a WIP, so we can only revisit it when it's in.

TODO:
- [x] Decide if the name/API is appropriate or not
- [x] Add docstrings 
- [x] Fix order = 'K'
- [x] Add tests

Problem statement:
Pinned memory allocation returns odd size
It looks like CuPy allocates more bytes than expected when calling `cupy.cuda.alloc_pinned_memory`. Any ideas why that might be?

```python
In [1]: import numpy 
   ...: import cupy                                                             

In [2]: a = cupy.arange(1_000_000)                                              

In [3]: b = numpy.asarray(cupy.cuda.alloc_pinned_memory(a.nbytes)).view(a.dtype)
   ...:                                                                         

In [4]: b.nbytes                                                                
Out[4]: 8388608

In [5]: a.nbytes                                                                
Out[5]: 8000000
```

* Conditions (you can just paste the output of `python -c 'import cupy; cupy.show_config()'`)
  - CuPy version: 7.6.0
  - OS/Platform: Ubuntu 18.04.4 LTS
  - CUDA version: 10.2
  - cuDNN/NCCL version (if applicable): 7.6.5/2.5.7.1

```$ python -c 'import cupy; cupy.show_config()'
CuPy Version          : 7.6.0
CUDA Root             : /datasets/jkirkham/miniconda/envs/rapids15dev
CUDA Build Version    : 10020
CUDA Driver Version   : 10020
CUDA Runtime Version  : 10020
cuBLAS Version        : 10202
cuFFT Version         : 10102
cuRAND Version        : 10102
cuSOLVER Version      : (10, 3, 0)
cuSPARSE Version      : 10301
NVRTC Version         : (10, 2)
cuDNN Build Version   : 7605
cuDNN Version         : 7605
NCCL Build Version    : 2406
NCCL Runtime Version  : 2507
CUB Version           : None
cuTENSOR Version      : None
```

Hints from issue discussion:
I think this was the reason:
https://github.com/cupy/cupy/blob/725c350884142c85c78df13a111f29b7c508540e/cupy/cuda/pinned_memory.pyx#L296-L298
So what's the best way to use this? Just trim the array viewing the allocation afterwards? Or is there a better way for one to be allocating pinned memory?
Hi John, sorry I dropped the ball. This is usually what I do
```python
>>> mem = cupy.cuda.alloc_pinned_memory(a.nbytes)
>>> b = numpy.frombuffer(mem, a.dtype, a.size)
>>> b.nbytes
8000000
```
Perhaps the core devs can recommend better approaches?
Another way:
```python
>>> mem = cupy.cuda.alloc_pinned_memory(a.nbytes)
>>> c = numpy.ndarray((a.size,), dtype=a.dtype, buffer=mem)
>>> c.nbytes
8000000
```
We were not aware of this use case of pinned memory allocator. When do you need this?
@kmaehashi It is very useful to back NumPy arrays by pinned memory. I use this very often when knowing in advance there'll be frequent device-host transfer following.
Thanks Leo! That makes sense 🙂

Yeah this is the same thing we were looking at. Having a NumPy array is a bit easier to work with.
Related to #4080, I'm wondering if it's better to provide this (equivalent to Leo's snippet) as an API under cupyx.
https://github.com/cupy/cupy/issues/3625#issuecomment-664135469

PyCUDA has an API for this feature:
https://documen.tician.de/pycuda/driver.html#pagelocked-allocation
That would be very helpful 🙂
> if it's better to provide this (equivalent to Leo's snippet) as an API under cupyx

I sent #4870 (still WIP) to address this need.

Commit message: move _get_strides_for_order_K to cupy.core.internal

Patch to review:
diff --git a/cupy/_creation/basic.py b/cupy/_creation/basic.py
index d633fbe39ef..aa6a633ac6a 100644
--- a/cupy/_creation/basic.py
+++ b/cupy/_creation/basic.py
@@ -1,7 +1,8 @@
 import cupy
 import numpy
 
-from cupy.core.core import _update_order_char, _get_strides_for_order_K
+from cupy.core.core import _update_order_char
+from cupy.core.internal import _get_strides_for_order_K
 
 
 def empty(shape, dtype=float, order='C'):
@@ -22,7 +23,9 @@ def empty(shape, dtype=float, order='C'):
     return cupy.ndarray(shape, dtype, order=order)
 
 
-def _new_like_order_and_strides(a, dtype, order, shape=None):
+def _new_like_order_and_strides(
+        a, dtype, order, shape=None, *, get_memptr=True,
+        get_char=_update_order_char):
     """
     Determine order and strides as in NumPy's PyArray_NewLikeArray.
 
@@ -40,12 +43,12 @@ def _new_like_order_and_strides(a, dtype, order, shape=None):
     if order == 'K' and shape is not None and len(shape) != a.ndim:
         return 'C', None, None
 
-    order = chr(_update_order_char(a, ord(order)))
+    order = chr(get_char(a, ord(order)))
 
     if order == 'K':
         strides = _get_strides_for_order_K(a, numpy.dtype(dtype), shape)
         order = 'C'
-        memptr = cupy.empty(a.size, dtype=dtype).data
+        memptr = cupy.empty(a.size, dtype=dtype).data if get_memptr else None
         return order, strides, memptr
     else:
         return order, None, None
diff --git a/cupy/core/core.pyx b/cupy/core/core.pyx
index d9696525904..a32c4ec11b5 100644
--- a/cupy/core/core.pyx
+++ b/cupy/core/core.pyx
@@ -444,7 +444,7 @@ cdef class ndarray:
         order_char = _update_order_char(self, order_char)
 
         if order_char == b'K':
-            strides = _get_strides_for_order_K(self, dtype)
+            strides = internal._get_strides_for_order_K(self, dtype)
             newarray = _ndarray_init(self._shape, dtype)
             # TODO(niboshi): Confirm update_x_contiguity flags
             newarray._set_shape_and_strides(self._shape, strides, True, True)
@@ -1835,20 +1835,6 @@ cpdef int _update_order_char(ndarray x, int order_char):
     return order_char
 
 
-cpdef strides_t _get_strides_for_order_K(ndarray x, dtype, shape=None):
-    cdef strides_t strides
-    # strides used when order='K' for astype, empty_like, etc.
-    stride_and_index = [
-        (abs(s), -i) for i, s in enumerate(x.strides)]
-    stride_and_index.sort()
-    strides.resize(x.ndim)
-    stride = dtype.itemsize
-    for s, i in stride_and_index:
-        strides[-i] = stride
-        stride *= shape[-i] if shape else x.shape[-i]
-    return strides
-
-
 _HANDLED_TYPES = (ndarray, numpy.ndarray)
 
 
diff --git a/cupy/core/internal.pxd b/cupy/core/internal.pxd
index 612871700af..92e2aab9a03 100644
--- a/cupy/core/internal.pxd
+++ b/cupy/core/internal.pxd
@@ -57,3 +57,5 @@ cpdef Py_ssize_t _normalize_axis_index(
 
 cpdef tuple _normalize_axis_indices(
     axes, Py_ssize_t ndim, cpp_bool sort_axes=*)
+
+cpdef strides_t _get_strides_for_order_K(x, dtype, shape=*)
diff --git a/cupy/core/internal.pyx b/cupy/core/internal.pyx
index 346a4531bd8..76f2bb509c8 100644
--- a/cupy/core/internal.pyx
+++ b/cupy/core/internal.pyx
@@ -437,3 +437,18 @@ cpdef tuple _normalize_axis_indices(
         res.append(axis)
 
     return tuple(sorted(res) if sort_axes else res)
+
+
+cpdef strides_t _get_strides_for_order_K(x, dtype, shape=None):
+    # x here can be either numpy.ndarray or cupy.ndarray
+    cdef strides_t strides
+    # strides used when order='K' for astype, empty_like, etc.
+    stride_and_index = [
+        (abs(s), -i) for i, s in enumerate(x.strides)]
+    stride_and_index.sort()
+    strides.resize(x.ndim)
+    stride = dtype.itemsize
+    for s, i in stride_and_index:
+        strides[-i] = stride
+        stride *= shape[-i] if shape else x.shape[-i]
+    return strides
diff --git a/cupyx/__init__.py b/cupyx/__init__.py
index a7b032117f7..a80f06076b3 100644
--- a/cupyx/__init__.py
+++ b/cupyx/__init__.py
@@ -16,3 +16,8 @@
 from cupyx._ufunc_config import seterr  # NOQA
 from cupy.core.syncdetect import allow_synchronize  # NOQA
 from cupy.core.syncdetect import DeviceSynchronized  # NOQA
+
+from cupyx._pinned_array import empty_pinned  # NOQA
+from cupyx._pinned_array import empty_like_pinned  # NOQA
+from cupyx._pinned_array import zeros_pinned  # NOQA
+from cupyx._pinned_array import zeros_like_pinned  # NOQA
diff --git a/cupyx/_pinned_array.py b/cupyx/_pinned_array.py
new file mode 100644
index 00000000000..e6001a2bb67
--- /dev/null
+++ b/cupyx/_pinned_array.py
@@ -0,0 +1,157 @@
+import numpy
+
+from cupy import cuda
+from cupy._creation.basic import _new_like_order_and_strides
+from cupy.core import internal
+
+
+def _update_shape(a, shape):
+    if shape is None and a is not None:
+        shape = a.shape
+    elif isinstance(shape, int):
+        shape = (shape,)
+    else:
+        shape = tuple(shape)
+    return shape
+
+
+def _update_order_char(x, order_char):
+    # This is a pure Python version of cupy.core.core._update_order_char()
+    # update order_char based on array contiguity
+    if order_char == ord('A'):
+        if x.flags.f_contiguous:
+            order_char = ord('F')
+        else:
+            order_char = ord('C')
+    elif order_char == ord('K'):
+        if x.flags.f_contiguous:
+            order_char = ord('F')
+        elif x.flags.c_contiguous:
+            order_char = ord('C')
+    return order_char
+
+
+def empty_pinned(shape, dtype=float, order='C'):
+    """Returns an array without initializing the elements.
+
+    This is a convenience function which is just :func:`numpy.empty`,
+    except that the underlying memory is allocated from CuPy's pinned
+    memory pool.
+
+    Args:
+        shape (int or tuple of ints): Dimensionalities of the array.
+        dtype: Data type specifier.
+        order ({'C', 'F'}): Row-major (C-style) or column-major
+            (Fortran-style) order.
+
+    Returns:
+        numpy.ndarray: A new array with elements not initialized.
+
+    .. seealso:: :func:`numpy.empty`
+
+    """
+    shape = _update_shape(None, shape)
+    nbytes = internal.prod(shape) * numpy.dtype(dtype).itemsize
+    mem = cuda.alloc_pinned_memory(nbytes)
+    out = numpy.ndarray(shape, dtype=dtype, buffer=mem, order=order)
+    return out
+
+
+def empty_like_pinned(a, dtype=None, order='K', subok=None, shape=None):
+    """Returns a new array with same shape and dtype of a given array.
+
+    This is a convenience function which is just :func:`numpy.empty_like`,
+    except that the underlying memory is allocated from CuPy's pinned
+    memory pool.
+
+    This function currently does not support ``subok`` option.
+
+    Args:
+        a (numpy.ndarray or cupy.ndarray): Base array.
+        dtype: Data type specifier. The data type of ``a`` is used by default.
+        order ({'C', 'F', 'A', or 'K'}): Overrides the memory layout of the
+            result. ``'C'`` means C-order, ``'F'`` means F-order, ``'A'`` means
+            ``'F'`` if ``a`` is Fortran contiguous, ``'C'`` otherwise.
+            ``'K'`` means match the layout of ``a`` as closely as possible.
+        subok: Not supported yet, must be None.
+        shape (int or tuple of ints): Overrides the shape of the result. If
+            ``order='K'`` and the number of dimensions is unchanged, will try
+            to keep order, otherwise, ``order='C'`` is implied.
+
+    Returns:
+        numpy.ndarray: A new array with same shape and dtype of ``a`` with
+        elements not initialized.
+
+    .. seealso:: :func:`numpy.empty_like`
+
+    """
+    # We're kinda duplicating the code here because order='K' needs special
+    # treatment: strides need to be computed
+    if subok is not None:
+        raise TypeError('subok is not supported yet')
+    if dtype is None:
+        dtype = a.dtype
+    shape = _update_shape(a, shape)
+    order, strides, _ = _new_like_order_and_strides(
+        a, dtype, order, shape, get_memptr=False, get_char=_update_order_char)
+    nbytes = internal.prod(shape) * numpy.dtype(dtype).itemsize
+    mem = cuda.alloc_pinned_memory(nbytes)
+    out = numpy.ndarray(shape, dtype=dtype, buffer=mem,
+                        strides=strides, order=order)
+    return out
+
+
+def zeros_pinned(shape, dtype=float, order='C'):
+    """Returns a new array of given shape and dtype, filled with zeros.
+
+    This is a convenience function which is just :func:`numpy.zeros`,
+    except that the underlying memory is allocated from CuPy's pinned
+    memory pool.
+
+    Args:
+        shape (int or tuple of ints): Dimensionalities of the array.
+        dtype: Data type specifier.
+        order ({'C', 'F'}): Row-major (C-style) or column-major
+            (Fortran-style) order.
+
+    Returns:
+        numpy.ndarray: An array filled with zeros.
+
+    .. seealso:: :func:`numpy.zeros`
+
+    """
+    out = empty_pinned(shape, dtype, order)
+    numpy.copyto(out, 0, casting='unsafe')
+    return out
+
+
+def zeros_like_pinned(a, dtype=None, order='K', subok=None, shape=None):
+    """Returns an array of zeros with same shape and dtype as a given array.
+
+    This is a convenience function which is just :func:`numpy.zeros_like`,
+    except that the underlying memory is allocated from CuPy's pinned
+    memory pool.
+
+    This function currently does not support ``subok`` option.
+
+    Args:
+        a (numpy.ndarray or cupy.ndarray): Base array.
+        dtype: Data type specifier. The dtype of ``a`` is used by default.
+        order ({'C', 'F', 'A', or 'K'}): Overrides the memory layout of the
+            result. ``'C'`` means C-order, ``'F'`` means F-order, ``'A'`` means
+            ``'F'`` if ``a`` is Fortran contiguous, ``'C'`` otherwise.
+            ``'K'`` means match the layout of ``a`` as closely as possible.
+        subok: Not supported yet, must be None.
+        shape (int or tuple of ints): Overrides the shape of the result. If
+            ``order='K'`` and the number of dimensions is unchanged, will try
+            to keep order, otherwise, ``order='C'`` is implied.
+
+    Returns:
+        numpy.ndarray: An array filled with zeros.
+
+    .. seealso:: :func:`numpy.zeros_like`
+
+    """
+    out = empty_like_pinned(a, dtype, order, subok, shape)
+    numpy.copyto(out, 0, casting='unsafe')
+    return out
diff --git a/docs/source/reference/ext.rst b/docs/source/reference/ext.rst
index 57970c66c70..d3f6ddf84ab 100644
--- a/docs/source/reference/ext.rst
+++ b/docs/source/reference/ext.rst
@@ -13,3 +13,7 @@ CuPy-specific functions are placed under ``cupyx`` namespace.
    cupyx.scatter_add
    cupyx.scatter_max
    cupyx.scatter_min
+   cupyx.empty_pinned
+   cupyx.empty_like_pinned
+   cupyx.zeros_pinned
+   cupyx.zeros_like_pinned
diff --git a/docs/source/reference/memory.rst b/docs/source/reference/memory.rst
index 0e65592459c..f21a168d71f 100644
--- a/docs/source/reference/memory.rst
+++ b/docs/source/reference/memory.rst
@@ -16,6 +16,10 @@ There are two different memory pools in CuPy:
 
 See :doc:`cuda` for the details of memory management APIs.
 
+For using pinned memory more conveniently, we also provide a few high-level APIs in the ``cupyx`` namespace,
+including :func:`cupyx.empty_pinned`, :func:`cupyx.empty_like_pinned`, :func:`cupyx.zeros_pinned`, and
+:func:`cupyx.zeros_like_pinned`. They return NumPy arrays backed CuPy's pinned memory pool.
+
 Memory Pool Operations
 ----------------------
 
diff --git a/tests/cupyx_tests/test_pinned_array.py b/tests/cupyx_tests/test_pinned_array.py
new file mode 100644
index 00000000000..339ad20a87d
--- /dev/null
+++ b/tests/cupyx_tests/test_pinned_array.py
@@ -0,0 +1,428 @@
+import unittest
+
+import numpy
+import pytest
+
+import cupy
+from cupy import testing
+from cupy.testing._helper import _wraps_partial
+import cupyx
+
+
+def numpy_cupyx_array_equal(target_func, name='func'):
+    _mod = (cupy, numpy)
+
+    _numpy_funcs = {
+        'empty': numpy.empty,
+        'empty_like': numpy.empty_like,
+        'zeros': numpy.zeros,
+        'zeros_like': numpy.zeros_like,
+    }
+
+    _cupy_funcs = {
+        'empty': cupyx.empty_pinned,
+        'empty_like': cupyx.empty_like_pinned,
+        'zeros': cupyx.zeros_pinned,
+        'zeros_like': cupyx.zeros_like_pinned,
+    }
+
+    def _get_test_func(xp, func):
+        if xp is numpy:
+            return _numpy_funcs[func]
+        elif xp is cupy:
+            return _cupy_funcs[func]
+        else:
+            assert False
+
+    def _check_pinned_mem_used(a, xp):
+        if xp is cupy:
+            assert isinstance(a.base, cupy.cuda.PinnedMemoryPointer)
+
+    def decorator(impl):
+        @_wraps_partial(impl, name)
+        def test_func(self, *args, **kw):
+            out = []
+            for xp in _mod:
+                func = _get_test_func(xp, target_func)
+                kw[name] = func
+                a = impl(self, *args, **kw)
+                _check_pinned_mem_used(a, xp)
+                out.append(a)
+            numpy.testing.assert_array_equal(*out)
+        return test_func
+    return decorator
+
+
+# test_empty_scalar_none is removed
+# test_zeros_scalar_none is removed
+class TestBasic(unittest.TestCase):
+    @testing.for_CF_orders()
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty')
+    def test_empty(self, dtype, order, func):
+        a = func((2, 3, 4), dtype=dtype, order=order)
+        a.fill(0)
+        return a
+
+    @testing.slow
+    def test_empty_huge_size(self):
+        a = cupyx.empty_pinned((1024, 2048, 1024), dtype='b')
+        a.fill(123)
+        assert (a == 123).all()
+        # Free huge memory for slow test
+        del a
+        cupy.get_default_pinned_memory_pool().free_all_blocks()
+
+    @testing.slow
+    def test_empty_huge_size_fill0(self):
+        a = cupyx.empty_pinned((1024, 2048, 1024), dtype='b')
+        a.fill(0)
+        assert (a == 0).all()
+        # Free huge memory for slow test
+        del a
+        cupy.get_default_pinned_memory_pool().free_all_blocks()
+
+    @testing.for_CF_orders()
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty')
+    def test_empty_scalar(self, dtype, order, func):
+        a = func((), dtype=dtype, order=order)
+        a.fill(0)
+        return a
+
+    @testing.for_CF_orders()
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty')
+    def test_empty_int(self, dtype, order, func):
+        a = func(3, dtype=dtype, order=order)
+        a.fill(0)
+        return a
+
+    @testing.slow
+    def test_empty_int_huge_size(self):
+        a = cupyx.empty_pinned(2 ** 31, dtype='b')
+        a.fill(123)
+        assert (a == 123).all()
+        # Free huge memory for slow test
+        del a
+        cupy.get_default_pinned_memory_pool().free_all_blocks()
+
+    @testing.slow
+    def test_empty_int_huge_size_fill0(self):
+        a = cupyx.empty_pinned(2 ** 31, dtype='b')
+        a.fill(0)
+        assert (a == 0).all()
+        # Free huge memory for slow test
+        del a
+        cupy.get_default_pinned_memory_pool().free_all_blocks()
+
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty_like')
+    def test_empty_like(self, dtype, order, func):
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        b = func(a, order=order)
+        b.fill(0)
+        return b
+
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty_like')
+    def test_empty_like_contiguity(self, dtype, order, func):
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        b = func(a, order=order)
+        b.fill(0)
+        if order in ['f', 'F']:
+            assert b.flags.f_contiguous
+        else:
+            assert b.flags.c_contiguous
+        return b
+
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty_like')
+    def test_empty_like_contiguity2(self, dtype, order, func):
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        a = numpy.asfortranarray(a)
+        b = func(a, order=order)
+        b.fill(0)
+        if order in ['c', 'C']:
+            assert b.flags.c_contiguous
+        else:
+            assert b.flags.f_contiguous
+        return b
+
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty_like')
+    def test_empty_like_contiguity3(self, dtype, order, func):
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        # test strides that are both non-contiguous and non-descending
+        a = a[:, ::2, :].swapaxes(0, 1)
+        b = func(a, order=order)
+        b.fill(0)
+        if order in ['k', 'K', None]:
+            assert not b.flags.c_contiguous
+            assert not b.flags.f_contiguous
+        elif order in ['f', 'F']:
+            assert not b.flags.c_contiguous
+            assert b.flags.f_contiguous
+        else:
+            assert b.flags.c_contiguous
+            assert not b.flags.f_contiguous
+        return b
+
+    @testing.for_all_dtypes()
+    @testing.gpu
+    def test_empty_like_K_strides(self, dtype):
+        # test strides that are both non-contiguous and non-descending;
+        # also test accepting cupy.ndarray
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        a = a[:, ::2, :].swapaxes(0, 1)
+        b = numpy.empty_like(a, order='K')
+        b.fill(0)
+
+        # GPU case
+        ag = testing.shaped_arange((2, 3, 4), cupy, dtype)
+        ag = ag[:, ::2, :].swapaxes(0, 1)
+        bg = cupyx.empty_like_pinned(ag, order='K')
+        bg.fill(0)
+
+        # make sure NumPy and CuPy strides agree
+        assert b.strides == bg.strides
+
+    @testing.with_requires('numpy>=1.19')
+    @testing.for_all_dtypes()
+    def test_empty_like_invalid_order(self, dtype):
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        with pytest.raises(ValueError):
+            cupyx.empty_like_pinned(a, order='Q')
+
+    def test_empty_like_subok(self):
+        a = testing.shaped_arange((2, 3, 4), numpy)
+        with pytest.raises(TypeError):
+            cupyx.empty_like_pinned(a, subok=True)
+
+    @testing.for_CF_orders()
+    def test_empty_zero_sized_array_strides(self, order):
+        a = numpy.empty((1, 0, 2), dtype='d', order=order)
+        b = cupyx.empty_pinned((1, 0, 2), dtype='d', order=order)
+        assert b.strides == a.strides
+
+    @testing.for_CF_orders()
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='zeros')
+    def test_zeros(self, dtype, order, func):
+        return func((2, 3, 4), dtype=dtype, order=order)
+
+    @testing.for_CF_orders()
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='zeros')
+    def test_zeros_scalar(self, dtype, order, func):
+        return func((), dtype=dtype, order=order)
+
+    @testing.for_CF_orders()
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='zeros')
+    def test_zeros_int(self, dtype, order, func):
+        return func(3, dtype=dtype, order=order)
+
+    @testing.for_CF_orders()
+    def test_zeros_strides(self, order):
+        a = numpy.zeros((2, 3), dtype='d', order=order)
+        b = cupyx.zeros_pinned((2, 3), dtype='d', order=order)
+        assert b.strides == a.strides
+
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='zeros_like')
+    def test_zeros_like(self, dtype, order, func):
+        a = numpy.ndarray((2, 3, 4), dtype=dtype)
+        return func(a, order=order)
+
+    def test_zeros_like_subok(self):
+        a = numpy.ndarray((2, 3, 4))
+        with pytest.raises(TypeError):
+            cupyx.zeros_like_pinned(a, subok=True)
+
+
+@testing.parameterize(
+    *testing.product({
+        'shape': [4, (4, ), (4, 2), (4, 2, 3), (5, 4, 2, 3)],
+    })
+)
+class TestBasicReshape(unittest.TestCase):
+
+    @testing.with_requires('numpy>=1.17.0')
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty_like')
+    def test_empty_like_reshape(self, dtype, order, func):
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        b = func(a, order=order, shape=self.shape)
+        b.fill(0)
+        return b
+
+    @testing.for_CF_orders()
+    @testing.for_all_dtypes()
+    @testing.gpu
+    def test_empty_like_reshape_cupy_only(self, dtype, order):
+        a = testing.shaped_arange((2, 3, 4), cupy, dtype)
+        b = cupyx.empty_like_pinned(a, shape=self.shape)
+        b.fill(0)
+        c = cupyx.empty_pinned(self.shape, order=order, dtype=dtype)
+        c.fill(0)
+        numpy.testing.assert_array_equal(b, c)
+
+    @testing.with_requires('numpy>=1.17.0')
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty_like')
+    def test_empty_like_reshape_contiguity(self, dtype, order, func):
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        b = func(a, order=order, shape=self.shape)
+        b.fill(0)
+        if order in ['f', 'F']:
+            assert b.flags.f_contiguous
+        else:
+            assert b.flags.c_contiguous
+        return b
+
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @testing.gpu
+    def test_empty_like_reshape_contiguity_cupy_only(self, dtype, order):
+        a = testing.shaped_arange((2, 3, 4), cupy, dtype)
+        b = cupyx.empty_like_pinned(a, order=order, shape=self.shape)
+        b.fill(0)
+        c = cupyx.empty_pinned(self.shape)
+        c.fill(0)
+        if order in ['f', 'F']:
+            assert b.flags.f_contiguous
+        else:
+            assert b.flags.c_contiguous
+        numpy.testing.assert_array_equal(b, c)
+
+    @testing.with_requires('numpy>=1.17.0')
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty_like')
+    def test_empty_like_reshape_contiguity2(self, dtype, order, func):
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        a = numpy.asfortranarray(a)
+        b = func(a, order=order, shape=self.shape)
+        b.fill(0)
+        shape = self.shape if not numpy.isscalar(self.shape) else (self.shape,)
+        if (order in ['c', 'C'] or
+                (order in ['k', 'K', None] and len(shape) != a.ndim)):
+            assert b.flags.c_contiguous
+        else:
+            assert b.flags.f_contiguous
+        return b
+
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @testing.gpu
+    def test_empty_like_reshape_contiguity2_cupy_only(self, dtype, order):
+        a = testing.shaped_arange((2, 3, 4), cupy, dtype)
+        a = cupy.asfortranarray(a)
+        b = cupyx.empty_like_pinned(a, order=order, shape=self.shape)
+        b.fill(0)
+        c = cupyx.empty_pinned(self.shape)
+        c.fill(0)
+        shape = self.shape if not numpy.isscalar(self.shape) else (self.shape,)
+        if (order in ['c', 'C'] or
+                (order in ['k', 'K', None] and len(shape) != a.ndim)):
+            assert b.flags.c_contiguous
+        else:
+            assert b.flags.f_contiguous
+        numpy.testing.assert_array_equal(b, c)
+
+    @testing.with_requires('numpy>=1.17.0')
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='empty_like')
+    def test_empty_like_reshape_contiguity3(self, dtype, order, func):
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        # test strides that are both non-contiguous and non-descending
+        a = a[:, ::2, :].swapaxes(0, 1)
+        b = func(a, order=order, shape=self.shape)
+        b.fill(0)
+        shape = self.shape if not numpy.isscalar(self.shape) else (self.shape,)
+        if len(shape) == 1:
+            assert b.flags.c_contiguous
+            assert b.flags.f_contiguous
+        elif order in ['k', 'K', None] and len(shape) == a.ndim:
+            assert not b.flags.c_contiguous
+            assert not b.flags.f_contiguous
+        elif order in ['f', 'F']:
+            assert not b.flags.c_contiguous
+            assert b.flags.f_contiguous
+        else:
+            assert b.flags.c_contiguous
+            assert not b.flags.f_contiguous
+        return b
+
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @testing.gpu
+    def test_empty_like_reshape_contiguity3_cupy_only(self, dtype, order):
+        a = testing.shaped_arange((2, 3, 4), cupy, dtype)
+        # test strides that are both non-contiguous and non-descending
+        a = a[:, ::2, :].swapaxes(0, 1)
+        b = cupyx.empty_like_pinned(a, order=order, shape=self.shape)
+        b.fill(0)
+        shape = self.shape if not numpy.isscalar(self.shape) else (self.shape,)
+        if len(shape) == 1:
+            assert b.flags.c_contiguous
+            assert b.flags.f_contiguous
+        elif order in ['k', 'K', None] and len(shape) == a.ndim:
+            assert not b.flags.c_contiguous
+            assert not b.flags.f_contiguous
+        elif order in ['f', 'F']:
+            assert not b.flags.c_contiguous
+            assert b.flags.f_contiguous
+        else:
+            assert b.flags.c_contiguous
+            assert not b.flags.f_contiguous
+
+        c = cupyx.zeros_pinned(self.shape)
+        c.fill(0)
+        testing.assert_array_equal(b, c)
+
+    @testing.with_requires('numpy>=1.17.0')
+    @testing.for_all_dtypes()
+    @testing.gpu
+    def test_empty_like_K_strides_reshape(self, dtype):
+        # test strides that are both non-contiguous and non-descending
+        a = testing.shaped_arange((2, 3, 4), numpy, dtype)
+        a = a[:, ::2, :].swapaxes(0, 1)
+        b = cupyx.empty_like_pinned(a, order='K', shape=self.shape)
+        b.fill(0)
+
+        # GPU case
+        ag = testing.shaped_arange((2, 3, 4), cupy, dtype)
+        ag = ag[:, ::2, :].swapaxes(0, 1)
+        bg = cupyx.empty_like_pinned(ag, order='K', shape=self.shape)
+        bg.fill(0)
+
+        # make sure NumPy and CuPy strides agree
+        assert b.strides == bg.strides
+        return
+
+    @testing.with_requires('numpy>=1.17.0')
+    @testing.for_orders('CFAK')
+    @testing.for_all_dtypes()
+    @numpy_cupyx_array_equal(target_func='zeros_like')
+    def test_zeros_like_reshape(self, dtype, order, func):
+        a = numpy.ndarray((2, 3, 4), dtype=dtype)
+        return func(a, order=order, shape=self.shape)
+
+    @testing.for_CF_orders()
+    @testing.for_all_dtypes()
+    @testing.gpu
+    def test_zeros_like_reshape_cupy_only(self, dtype, order):
+        a = testing.shaped_arange((2, 3, 4), cupy, dtype)
+        b = cupyx.zeros_like_pinned(a, shape=self.shape)
+        c = cupyx.zeros_pinned(self.shape, order=order, dtype=dtype)
+        numpy.testing.assert_array_equal(b, c)


Code review comments:
diff_hunk: @@ -0,0 +1,157 @@
+import numpy
+
+from cupy import cuda
+from cupy._creation.basic import _new_like_order_and_strides
+from cupy.core import internal
+
+
+def _update_shape(a, shape):
+    if shape is None and a is not None:
+        shape = a.shape
+    elif isinstance(shape, int):
+        shape = (shape,)
+    else:
+        shape = tuple(shape)
+    return shape
+
+
+def _update_order_char(x, order_char):
+    # This is a pure Python version of cupy.core.core._update_order_char()
+    # update order_char based on array contiguity
+    if order_char == ord('A'):
+        if x.flags.f_contiguous:
+            order_char = ord('F')
+        else:
+            order_char = ord('C')
+    elif order_char == ord('K'):
+        if x.flags.f_contiguous:
+            order_char = ord('F')
+        elif x.flags.c_contiguous:
+            order_char = ord('C')
+    return order_char
+
+
+def empty_pinned(shape, dtype=float, order='C'):
+    """Returns an array without initializing the elements.
+
+    This is a convenience function which is just :func:`numpy.empty`,
+    except that the underlying memory is allocated from CuPy's pinned
+    memory pool.
path: cupyx/_pinned_array.py
text: @user1:
Users may disable the memory pool, so:

```suggestion
    except that the underlying memory is allocated as a pinned memory.
```


@author:
Done in 1bc4e3c with minor changes.

diff_hunk: @@ -0,0 +1,157 @@
+import numpy
+
+from cupy import cuda
+from cupy._creation.basic import _new_like_order_and_strides
+from cupy.core import internal
+
+
+def _update_shape(a, shape):
+    if shape is None and a is not None:
+        shape = a.shape
+    elif isinstance(shape, int):
+        shape = (shape,)
+    else:
+        shape = tuple(shape)
+    return shape
+
+
+def _update_order_char(x, order_char):
+    # This is a pure Python version of cupy.core.core._update_order_char()
path: cupyx/_pinned_array.py
text: @author:
We could relax the typing in `cupy.core.core._update_order_char()` so that it can be used here, as I want to accept both cupy and numpy arrays. However, removing the `ndarray` typing increases the overhead a little bit, as we have to access Python attributes (`x.flags.f_contiguous`) instead of C attributes (`x._f_contiguous`) which is absent in `numpy.ndarray`:
|  | time |
|--- | --- |
| original `cupy.core.core._update_order_char()` | 0.13 us |
| remove the `ndarray` typing from `x` (but still kept in Cython) | 0.48 us |
| full Python implementation (as done above) | 0.90 us |

Let me know if this overhead is acceptable for the purpose of reducing code duplication. 

@user1:
How about changing the original `_update_order_char` function to take `bool c_contiguous, bool f_continguous`  instead of `ndarray`?

@author:
Good point, @user1! I should have thought about it...Done in b833b24. No performance regression is observed, with a caveat that in Python the (likely small) overhead is swept into function parameter preparation (`c_contiguous` and `f_contiguous`).