Repository: numpy/numpy
Pull request: MAINT: Add missing `dtype` overloads for object- and ctypes-based inputs
closes https://github.com/numpy/numpy/issues/19467

This PR modifies a number of overloads in the `np.dtype` constructor, adding support for the various 
`ctypes._SimpleCData` subclasses. Secondly, it adds a catchall overload for `Type[object]` which was 
previously missing.

Examples
----------
``` python
import ctypes
import numpy as np
from typing import TYPE_CHECKING

dtype_object = np.dtype(object)
dtype_c_long = np.dtype(ctypes.c_long)

if TYPE_CHECKING:
    reveal_type(dtype_object)  # Revealed type is "numpy.dtype[numpy.object_]"
    reveal_type(dtype_c_long)  # Revealed type is "numpy.dtype[numpy.signedinteger[numpy.typing._64Bit]]"
```

Problem statement:
Incorrect dtype-ctypes static typing
I have some c++ - python compatible features and use ctypes lib for creating dtypes dynamically.
np.dtype typing is incorrect when ctypes type is argument.

### Reproducing code example:

```python
# myproblem.py
import ctypes
from typing import TYPE_CHECKING

import numpy as np


dtype_int = np.dtype(ctypes.c_int)
dtype_size_t = np.dtype(ctypes.c_size_t)
print(repr(dtype_int))  # prints `dtype('int32')>` as expected on my machine
print(repr(dtype_size_t))  # prints `dtype('uint64')` as expected on my machine
if TYPE_CHECKING:
    reveal_type(dtype_int)  # something like `numpy.dtype[numpy.signedinteger[numpy.typing._32Bit]]` is expected
    reveal_type(dtype_size_t)  # something like `numpy.dtype[numpy.unsignedinteger[numpy.typing._64Bit]]` is expected
```

### Mypy error message:

```
$ mypy test.py
myproblem.py:8: error: Value of type variable "_DTypeScalar_co" of "dtype" cannot be "c_int"
myproblem.py:9: error: Value of type variable "_DTypeScalar_co" of "dtype" cannot be "c_size_t"
myproblem.py:13: note: Revealed type is "numpy.dtype[ctypes.c_int*]"
myproblem.py:14: note: Revealed type is "numpy.dtype[ctypes.c_size_t*]"
Found 2 errors in 1 file (checked 1 source file)
```

### NumPy/Python version information:

```
>>> import sys; from importlib.metadata import version; print(sys.version, version('mypy'), version('numpy'), sep='\n')
3.9.1 (default, Jul  7 2021, 22:39:47)
[Clang 12.0.5 (clang-1205.0.22.9)]
0.910
1.21.0
```

Hints from issue discussion:
Ah, I though we had a catchall overload for miscellaneous `type`objects, but it seems to have been removed at some point.

In any case, https://github.com/numpy/numpy/pull/19468 should add the necasary overloads for the `ctypes` types.

Commit message: TST: Update the `dtype` typing tests

Patch to review:
diff --git a/numpy/__init__.pyi b/numpy/__init__.pyi
index ca0c79b1719f..104acd7226ce 100644
--- a/numpy/__init__.pyi
+++ b/numpy/__init__.pyi
@@ -2,6 +2,7 @@ import builtins
 import os
 import sys
 import mmap
+import ctypes as ct
 import array as _array
 import datetime as dt
 from abc import abstractmethod
@@ -911,7 +912,7 @@ class dtype(Generic[_DTypeScalar_co]):
     # other special cases. Order is sometimes important because of the
     # subtype relationships
     #
-    # bool < int < float < complex
+    # bool < int < float < complex < object
     #
     # so we have to make sure the overloads for the narrowest type is
     # first.
@@ -929,51 +930,51 @@ class dtype(Generic[_DTypeScalar_co]):
     @overload
     def __new__(cls, dtype: Type[bytes], align: bool = ..., copy: bool = ...) -> dtype[bytes_]: ...
 
-    # `unsignedinteger` string-based representations
+    # `unsignedinteger` string-based representations and ctypes
     @overload
-    def __new__(cls, dtype: _UInt8Codes, align: bool = ..., copy: bool = ...) -> dtype[uint8]: ...
+    def __new__(cls, dtype: _UInt8Codes | Type[ct.c_uint8], align: bool = ..., copy: bool = ...) -> dtype[uint8]: ...
     @overload
-    def __new__(cls, dtype: _UInt16Codes, align: bool = ..., copy: bool = ...) -> dtype[uint16]: ...
+    def __new__(cls, dtype: _UInt16Codes | Type[ct.c_uint16], align: bool = ..., copy: bool = ...) -> dtype[uint16]: ...
     @overload
-    def __new__(cls, dtype: _UInt32Codes, align: bool = ..., copy: bool = ...) -> dtype[uint32]: ...
+    def __new__(cls, dtype: _UInt32Codes | Type[ct.c_uint32], align: bool = ..., copy: bool = ...) -> dtype[uint32]: ...
     @overload
-    def __new__(cls, dtype: _UInt64Codes, align: bool = ..., copy: bool = ...) -> dtype[uint64]: ...
+    def __new__(cls, dtype: _UInt64Codes | Type[ct.c_uint64], align: bool = ..., copy: bool = ...) -> dtype[uint64]: ...
     @overload
-    def __new__(cls, dtype: _UByteCodes, align: bool = ..., copy: bool = ...) -> dtype[ubyte]: ...
+    def __new__(cls, dtype: _UByteCodes | Type[ct.c_ubyte], align: bool = ..., copy: bool = ...) -> dtype[ubyte]: ...
     @overload
-    def __new__(cls, dtype: _UShortCodes, align: bool = ..., copy: bool = ...) -> dtype[ushort]: ...
+    def __new__(cls, dtype: _UShortCodes | Type[ct.c_ushort], align: bool = ..., copy: bool = ...) -> dtype[ushort]: ...
     @overload
-    def __new__(cls, dtype: _UIntCCodes, align: bool = ..., copy: bool = ...) -> dtype[uintc]: ...
+    def __new__(cls, dtype: _UIntCCodes | Type[ct.c_uint], align: bool = ..., copy: bool = ...) -> dtype[uintc]: ...
     @overload
-    def __new__(cls, dtype: _UIntPCodes, align: bool = ..., copy: bool = ...) -> dtype[uintp]: ...
+    def __new__(cls, dtype: _UIntPCodes | Type[ct.c_void_p] | Type[ct.c_size_t], align: bool = ..., copy: bool = ...) -> dtype[uintp]: ...
     @overload
-    def __new__(cls, dtype: _UIntCodes, align: bool = ..., copy: bool = ...) -> dtype[uint]: ...
+    def __new__(cls, dtype: _UIntCodes | Type[ct.c_ulong], align: bool = ..., copy: bool = ...) -> dtype[uint]: ...
     @overload
-    def __new__(cls, dtype: _ULongLongCodes, align: bool = ..., copy: bool = ...) -> dtype[ulonglong]: ...
+    def __new__(cls, dtype: _ULongLongCodes | Type[ct.c_ulonglong], align: bool = ..., copy: bool = ...) -> dtype[ulonglong]: ...
 
-    # `signedinteger` string-based representations
+    # `signedinteger` string-based representations and ctypes
     @overload
-    def __new__(cls, dtype: _Int8Codes, align: bool = ..., copy: bool = ...) -> dtype[int8]: ...
+    def __new__(cls, dtype: _Int8Codes | Type[ct.c_int8], align: bool = ..., copy: bool = ...) -> dtype[int8]: ...
     @overload
-    def __new__(cls, dtype: _Int16Codes, align: bool = ..., copy: bool = ...) -> dtype[int16]: ...
+    def __new__(cls, dtype: _Int16Codes | Type[ct.c_int16], align: bool = ..., copy: bool = ...) -> dtype[int16]: ...
     @overload
-    def __new__(cls, dtype: _Int32Codes, align: bool = ..., copy: bool = ...) -> dtype[int32]: ...
+    def __new__(cls, dtype: _Int32Codes | Type[ct.c_int32], align: bool = ..., copy: bool = ...) -> dtype[int32]: ...
     @overload
-    def __new__(cls, dtype: _Int64Codes, align: bool = ..., copy: bool = ...) -> dtype[int64]: ...
+    def __new__(cls, dtype: _Int64Codes | Type[ct.c_int64], align: bool = ..., copy: bool = ...) -> dtype[int64]: ...
     @overload
-    def __new__(cls, dtype: _ByteCodes, align: bool = ..., copy: bool = ...) -> dtype[byte]: ...
+    def __new__(cls, dtype: _ByteCodes | Type[ct.c_byte], align: bool = ..., copy: bool = ...) -> dtype[byte]: ...
     @overload
-    def __new__(cls, dtype: _ShortCodes, align: bool = ..., copy: bool = ...) -> dtype[short]: ...
+    def __new__(cls, dtype: _ShortCodes | Type[ct.c_short], align: bool = ..., copy: bool = ...) -> dtype[short]: ...
     @overload
-    def __new__(cls, dtype: _IntCCodes, align: bool = ..., copy: bool = ...) -> dtype[intc]: ...
+    def __new__(cls, dtype: _IntCCodes | Type[ct.c_int], align: bool = ..., copy: bool = ...) -> dtype[intc]: ...
     @overload
-    def __new__(cls, dtype: _IntPCodes, align: bool = ..., copy: bool = ...) -> dtype[intp]: ...
+    def __new__(cls, dtype: _IntPCodes | Type[ct.c_ssize_t], align: bool = ..., copy: bool = ...) -> dtype[intp]: ...
     @overload
-    def __new__(cls, dtype: _IntCodes, align: bool = ..., copy: bool = ...) -> dtype[int_]: ...
+    def __new__(cls, dtype: _IntCodes | Type[ct.c_long], align: bool = ..., copy: bool = ...) -> dtype[int_]: ...
     @overload
-    def __new__(cls, dtype: _LongLongCodes, align: bool = ..., copy: bool = ...) -> dtype[longlong]: ...
+    def __new__(cls, dtype: _LongLongCodes | Type[ct.c_longlong], align: bool = ..., copy: bool = ...) -> dtype[longlong]: ...
 
-    # `floating` string-based representations
+    # `floating` string-based representations and ctypes
     @overload
     def __new__(cls, dtype: _Float16Codes, align: bool = ..., copy: bool = ...) -> dtype[float16]: ...
     @overload
@@ -983,11 +984,11 @@ class dtype(Generic[_DTypeScalar_co]):
     @overload
     def __new__(cls, dtype: _HalfCodes, align: bool = ..., copy: bool = ...) -> dtype[half]: ...
     @overload
-    def __new__(cls, dtype: _SingleCodes, align: bool = ..., copy: bool = ...) -> dtype[single]: ...
+    def __new__(cls, dtype: _SingleCodes | Type[ct.c_float], align: bool = ..., copy: bool = ...) -> dtype[single]: ...
     @overload
-    def __new__(cls, dtype: _DoubleCodes, align: bool = ..., copy: bool = ...) -> dtype[double]: ...
+    def __new__(cls, dtype: _DoubleCodes | Type[ct.c_double], align: bool = ..., copy: bool = ...) -> dtype[double]: ...
     @overload
-    def __new__(cls, dtype: _LongDoubleCodes, align: bool = ..., copy: bool = ...) -> dtype[longdouble]: ...
+    def __new__(cls, dtype: _LongDoubleCodes | Type[ct.c_longdouble], align: bool = ..., copy: bool = ...) -> dtype[longdouble]: ...
 
     # `complexfloating` string-based representations
     @overload
@@ -1001,9 +1002,9 @@ class dtype(Generic[_DTypeScalar_co]):
     @overload
     def __new__(cls, dtype: _CLongDoubleCodes, align: bool = ..., copy: bool = ...) -> dtype[clongdouble]: ...
 
-    # Miscellaneous string-based representations
+    # Miscellaneous string-based representations and ctypes
     @overload
-    def __new__(cls, dtype: _BoolCodes, align: bool = ..., copy: bool = ...) -> dtype[bool_]: ...
+    def __new__(cls, dtype: _BoolCodes | Type[ct.c_bool], align: bool = ..., copy: bool = ...) -> dtype[bool_]: ...
     @overload
     def __new__(cls, dtype: _TD64Codes, align: bool = ..., copy: bool = ...) -> dtype[timedelta64]: ...
     @overload
@@ -1011,11 +1012,11 @@ class dtype(Generic[_DTypeScalar_co]):
     @overload
     def __new__(cls, dtype: _StrCodes, align: bool = ..., copy: bool = ...) -> dtype[str_]: ...
     @overload
-    def __new__(cls, dtype: _BytesCodes, align: bool = ..., copy: bool = ...) -> dtype[bytes_]: ...
+    def __new__(cls, dtype: _BytesCodes | Type[ct.c_char], align: bool = ..., copy: bool = ...) -> dtype[bytes_]: ...
     @overload
     def __new__(cls, dtype: _VoidCodes, align: bool = ..., copy: bool = ...) -> dtype[void]: ...
     @overload
-    def __new__(cls, dtype: _ObjectCodes, align: bool = ..., copy: bool = ...) -> dtype[object_]: ...
+    def __new__(cls, dtype: _ObjectCodes | Type[ct.py_object], align: bool = ..., copy: bool = ...) -> dtype[object_]: ...
 
     # dtype of a dtype is the same dtype
     @overload
@@ -1040,7 +1041,7 @@ class dtype(Generic[_DTypeScalar_co]):
         align: bool = ...,
         copy: bool = ...,
     ) -> dtype[Any]: ...
-    # Catchall overload
+    # Catchall overload for void-likes
     @overload
     def __new__(
         cls,
@@ -1048,6 +1049,14 @@ class dtype(Generic[_DTypeScalar_co]):
         align: bool = ...,
         copy: bool = ...,
     ) -> dtype[void]: ...
+    # Catchall overload for object-likes
+    @overload
+    def __new__(
+        cls,
+        dtype: Type[object],
+        align: bool = ...,
+        copy: bool = ...,
+    ) -> dtype[object_]: ...
 
     @overload
     def __getitem__(self: dtype[void], key: List[str]) -> dtype[void]: ...
diff --git a/numpy/typing/tests/data/fail/dtype.py b/numpy/typing/tests/data/fail/dtype.py
index 7d419a1d1e5f..0f3810f3c014 100644
--- a/numpy/typing/tests/data/fail/dtype.py
+++ b/numpy/typing/tests/data/fail/dtype.py
@@ -18,5 +18,3 @@ class Test2:
         "field2": (int, 3),
     }
 )
-
-np.dtype[np.float64](np.int64)  # E: Argument 1 to "dtype" has incompatible type
diff --git a/numpy/typing/tests/data/reveal/dtype.py b/numpy/typing/tests/data/reveal/dtype.py
index eb1489bf3053..364d1dcab306 100644
--- a/numpy/typing/tests/data/reveal/dtype.py
+++ b/numpy/typing/tests/data/reveal/dtype.py
@@ -1,3 +1,4 @@
+import ctypes as ct
 import numpy as np
 
 dtype_U: np.dtype[np.str_]
@@ -23,6 +24,15 @@
 reveal_type(np.dtype(bool))  # E: numpy.dtype[numpy.bool_]
 reveal_type(np.dtype(str))  # E: numpy.dtype[numpy.str_]
 reveal_type(np.dtype(bytes))  # E: numpy.dtype[numpy.bytes_]
+reveal_type(np.dtype(object))  # E: numpy.dtype[numpy.object_]
+
+# ctypes
+reveal_type(np.dtype(ct.c_double))  # E: numpy.dtype[{double}]
+reveal_type(np.dtype(ct.c_longlong))  # E: numpy.dtype[{longlong}]
+reveal_type(np.dtype(ct.c_uint32))  # E: numpy.dtype[{uint32}]
+reveal_type(np.dtype(ct.c_bool))  # E: numpy.dtype[numpy.bool_]
+reveal_type(np.dtype(ct.c_char))  # E: numpy.dtype[numpy.bytes_]
+reveal_type(np.dtype(ct.py_object))  # E: numpy.dtype[numpy.object_]
 
 # Special case for None
 reveal_type(np.dtype(None))  # E: numpy.dtype[{double}]


Code review comments:
diff_hunk: @@ -929,51 +930,51 @@ class dtype(Generic[_DTypeScalar_co]):
     @overload
     def __new__(cls, dtype: Type[bytes], align: bool = ..., copy: bool = ...) -> dtype[bytes_]: ...
 
-    # `unsignedinteger` string-based representations
+    # `unsignedinteger` string-based representations and ctypes
     @overload
-    def __new__(cls, dtype: _UInt8Codes, align: bool = ..., copy: bool = ...) -> dtype[uint8]: ...
+    def __new__(cls, dtype: _UInt8Codes | Type[ct.c_uint8], align: bool = ..., copy: bool = ...) -> dtype[uint8]: ...
     @overload
-    def __new__(cls, dtype: _UInt16Codes, align: bool = ..., copy: bool = ...) -> dtype[uint16]: ...
+    def __new__(cls, dtype: _UInt16Codes | Type[ct.c_uint16], align: bool = ..., copy: bool = ...) -> dtype[uint16]: ...
     @overload
-    def __new__(cls, dtype: _UInt32Codes, align: bool = ..., copy: bool = ...) -> dtype[uint32]: ...
+    def __new__(cls, dtype: _UInt32Codes | Type[ct.c_uint32], align: bool = ..., copy: bool = ...) -> dtype[uint32]: ...
     @overload
-    def __new__(cls, dtype: _UInt64Codes, align: bool = ..., copy: bool = ...) -> dtype[uint64]: ...
+    def __new__(cls, dtype: _UInt64Codes | Type[ct.c_uint64], align: bool = ..., copy: bool = ...) -> dtype[uint64]: ...
     @overload
-    def __new__(cls, dtype: _UByteCodes, align: bool = ..., copy: bool = ...) -> dtype[ubyte]: ...
+    def __new__(cls, dtype: _UByteCodes | Type[ct.c_ubyte], align: bool = ..., copy: bool = ...) -> dtype[ubyte]: ...
     @overload
-    def __new__(cls, dtype: _UShortCodes, align: bool = ..., copy: bool = ...) -> dtype[ushort]: ...
+    def __new__(cls, dtype: _UShortCodes | Type[ct.c_ushort], align: bool = ..., copy: bool = ...) -> dtype[ushort]: ...
     @overload
-    def __new__(cls, dtype: _UIntCCodes, align: bool = ..., copy: bool = ...) -> dtype[uintc]: ...
+    def __new__(cls, dtype: _UIntCCodes | Type[ct.c_uint], align: bool = ..., copy: bool = ...) -> dtype[uintc]: ...
     @overload
-    def __new__(cls, dtype: _UIntPCodes, align: bool = ..., copy: bool = ...) -> dtype[uintp]: ...
+    def __new__(cls, dtype: _UIntPCodes | Type[ct.c_void_p] | Type[ct.c_size_t], align: bool = ..., copy: bool = ...) -> dtype[uintp]: ...
path: numpy/__init__.pyi
text: @author:
To verify: are there any situations where `ctypes.c_void_p` and `ctypes.c_size_t` don't map to `np.uintp`?

@user1:
Yes there are, platforms where `uint_ptr_t != size_t`.

Numpy is probably already broken on those platforms though.

@author:
> Numpy is probably already broken on those platforms though.

Based on this I feel it would be adequate to add a comment for these cases and call it a day (https://github.com/numpy/numpy/pull/19468/commits/e7cd1e78ce728e3b745cea5a34c7fdea0f5c378e).

@user1:
Just to check... Is this actually necessary? `ct.c_size_t` is just a type alias for one of the other C types, unlike `ctypes.c_void_p` which is actually a different type.

Is `uintp` considered a distinct type by mypy, or is it also treated as an alias of another type? Because if they're both aliases, you can just leave out `ct.c_size_t` and it should do the right thing automatically.

@author:
> Just to check... Is this actually necessary?

Unfortunately it is, as the various `ctypes`aliases are all annotated as distinct types in its [stub file](https://github.com/python/typeshed/blob/0cd4ee39a2fbe17001b1a98929158ffc0b11f664/stdlib/ctypes/__init__.pyi#L227-L250).
Mypy does have a [builtin plugin](https://github.com/python/mypy/blob/master/mypy/plugins/ctypes.py) for dealing with `ctypes.Array`, but there doesn't seem to be anything
for dealing with type aliases (or the lack thereof).

> Is `uintp` considered a distinct type by mypy,  or is it also treated as an alias of another type?

No, on numpy's side all `np.unsignedinteger` subclasses are considered interchangeable (as long as 
they're of the size). So on a 64-bit platform `np.uint64` and `np.uintp` are equivalent as far as mypy 
is concerned. 

So with this in mind, I'd say that the `np.generic` subclasses are currently a bit easier to use with typing 
than `ctypes._SimpleCData`.

https://github.com/numpy/numpy/blob/89c80ba606f4346f8df2a31cfcc0e967045a68ed/numpy/__init__.pyi#L3287-L3298