Repository: nv-legate/cunumeric

Issue:
cunumeric does not issue unimplemented warnings for functions in submodules
For example, calling unsupported functions in the `random` module silently fall back to numpy instead of issuing a warning.


Discussion:
Tentatively assigning to @bryevdv, who mostly recently touched the module-wrapping logic. Hopefully there's a simple extension we could make.
Do we want to warn only on first use, or every call? 
Or for that matter, do we want to only warn on _any_ first use, with an actionable message about generating API coverage report output?
For modules where we have partial coverage, I like the current behavior of warning for each missing function separately, but only the first time it's called.

If we're talking about a whole module missing, then it would be nice to warn once, on the first use of any function in that module. But if that's too hard, then warning on the first use of each function from the module would still be OK IMHO.

Noting the possibility of API coverage reporting could be an option in all scenarios I assume.
So, there already should be a warning emitted:

https://github.com/nv-legate/cunumeric/blob/35ebbb741838c34f0f0cb9d96ad0077cbb424d14/cunumeric/coverage.py#L178-L189

with unit  tests [here (standard funcs)](https://github.com/nv-legate/cunumeric/blob/5f17dd6af24441484f33564c09a1c78892b94a89/tests/unit/cunumeric/test_coverage.py#L242-L263) and [here (ufuncs)](https://github.com/nv-legate/cunumeric/blob/5f17dd6af24441484f33564c09a1c78892b94a89/tests/unit/cunumeric/test_coverage.py#L286-L305)
@rohany can you provide a concrete minimal reproducer?
Here is an example that does print a warning, for instance:
```
import cunumeric as np
A = np.diag([1.0, 2, 3])
print(np.flipud(A))
```
This yields:
```
dev310 ❯ legate foo.py                            
foo.py:3: RuntimeWarning: cuNumeric has not implemented numpy.flipud and is falling back to canonical numpy. You may notice significantly decreased performance for this function call.
  print(np.flipud(A))
[[0. 0. 3.]
 [0. 2. 0.]
 [1. 0. 0.]]

~/tmp
```
Note that  currently warnings only emitted when coverage reporting is _not_ turned on. i.e. if you have added the command line option `cunumeric:report:coverage`, then there won't (currently) be any warning:
```
dev310 ❯ legate foo.py  -cunumeric:report:coverage
[[0. 0. 3.]
 [0. 2. 0.]
 [1. 0. 0.]]
cuNumeric API coverage: 5/6 (83.33333333333334%)

~/tmp

```
This could be changed if desired. 
Noting the current behavior does not de-dupe so we could also certainly improve that, although the advantage of not de-duping is that the user gets line numbers:
```
dev310 ❯ legate foo.py                                                               
foo.py:3: RuntimeWarning: cuNumeric has not implemented numpy.flipud and is falling back to canonical numpy. You may notice significantly decreased performance for this function call.
  np.flipud(A)
foo.py:4: RuntimeWarning: cuNumeric has not implemented numpy.flipud and is falling back to canonical numpy. You may notice significantly decreased performance for this function call.
  np.flipud(A)
foo.py:5: RuntimeWarning: cuNumeric has not implemented numpy.flipud and is falling back to canonical numpy. You may notice significantly decreased performance for this function call.
  np.flipud(A)
foo.py:6: RuntimeWarning: cuNumeric has not implemented numpy.flipud and is falling back to canonical numpy. You may notice significantly decreased performance for this function call.
  np.flipud(A)
foo.py:7: RuntimeWarning: cuNumeric has not implemented numpy.flipud and is falling back to canonical numpy. You may notice significantly decreased performance for this function call.
  np.flipud(A)
```
so maybe it is better to keep as it is.
Try calling `cunumeric.random.permutation`. This shouldn't issue a warning, but `permutation` is unimplemented by `cunumeric`.
@rohany can you please provide an actual [complete, minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example), so as to avoid any and all speculation and ambiguity?

Also can clarify this statement:

>  This shouldn't issue a warning, but permutation is unimplemented by cunumeric.

Did you mean it *should* issue a warning?
```
Welcome to Legion Python interactive console
>>> import cunumeric
>>> cunumeric.random.permutation(5)
array([1, 4, 2, 3, 0])
```
> Did you mean it should issue a warning?

Sorry I was confusing. I meant that I expect a warning, but running that code is not going to raise a warning (unless something changed in the meantime).

OK now we are getting somewhere.

@manopapad I think coverage just does not work with `np.random` in general. I'm not sure if it never worked, or if it was recently broken somewhere. This will require some investigation. The coverage codepaths are already fairly convoluted and unpleasantly special-cased.
Maybe that was too pessimistic. I think some things _are_ getting wrapped properly, e.g `np.random.normal` but not others, like `np.random.permutation`. The proximate cause for that is:
```
>>> isinstance(m.permutation, (FunctionType, MethodType, MethodDescriptorType))
False
```
Which I guess probably has something to do with this odd type not being covered by the above:
```
>>> type(m.permutation)
<class 'builtin_function_or_method'>
```
This returns True, but I'll have to see if just adding this to `should_wrap` is a good idea or not:
```
>>> isinstance(m.permutation, types.BuiltinMethodType)
True
```
Yes, well, this diff "fixes" 
```diff
diff --git a/cunumeric/coverage.py b/cunumeric/coverage.py
index f8f4446a..acf5ba01 100644
--- a/cunumeric/coverage.py
+++ b/cunumeric/coverage.py
@@ -17,7 +17,7 @@ from __future__ import annotations
 import warnings
 from dataclasses import dataclass
 from functools import wraps
-from types import FunctionType, MethodDescriptorType, MethodType, ModuleType
+from types import BuiltinMethodType, FunctionType, MethodDescriptorType, MethodType, ModuleType
 from typing import Any, Container, Mapping, Optional, cast
 
 import numpy as np
@@ -230,7 +230,7 @@ def clone_module(
         # Only need to wrap things that are in the origin module to begin with
         if attr not in origin_module.__dict__:
             continue
-        if isinstance(value, (FunctionType, lgufunc)):
+        if isinstance(value, (BuiltinMethodType, FunctionType, lgufunc)):
             wrapped = implemented(
                 cast(AnyCallable, value), mod_name, attr, reporting=reporting
             )
@@ -239,7 +239,7 @@ def clone_module(
     from numpy import ufunc as npufunc
 
     for attr, value in missing.items():
-        if isinstance(value, (FunctionType, npufunc)):
+        if isinstance(value, (BuiltinMethodType, FunctionType, npufunc)):
             wrapped = unimplemented(value, mod_name, attr, reporting=reporting)
             new_globals[attr] = wrapped
         else:
@@ -247,7 +247,7 @@ def clone_module(
 
 
 def should_wrap(obj: object) -> bool:
-    return isinstance(obj, (FunctionType, MethodType, MethodDescriptorType))
+    return isinstance(obj, (BuiltinMethodType, FunctionType, MethodType, MethodDescriptorType))
 
 
 def clone_np_ndarray(cls: type) -> type:
```
produces:
```
dev310 ❯ legate foo.py
foo.py:2: RuntimeWarning: cuNumeric has not implemented numpy.random.permutation and is falling back to canonical numpy. You may notice significantly decreased performance for this function call.
  np.random.permutation(5)
```

But unfortunately causes other things to explode: 
```
   TypeError: ndarray.__array__() missing 1 required positional argument: 'self'
   /home/bryan/work/legate.core/legate/core/context.py:277: RuntimeWarning: cuNumeric has not implemented numpy.ndarray.__new__ and is falling back to canonical numpy. You may notice significantly decreased performance for this function call.
     result = func(*args, **kwargs)
```

I guess I am really not sure why  `permutation` has the type `builtin_function_or_method` to begin with. That seems wrong, but is certainly a huge pain, regardless. 

Edit: well, if I had to guess, it probably has that type because it is probably a thin wrapper around a C function.

Resolving patch:
diff --git a/cunumeric/coverage.py b/cunumeric/coverage.py
--- a/cunumeric/coverage.py
+++ b/cunumeric/coverage.py
@@ -17,7 +17,13 @@
 import warnings
 from dataclasses import dataclass
 from functools import wraps
-from types import FunctionType, MethodDescriptorType, MethodType, ModuleType
+from types import (
+    BuiltinFunctionType,
+    FunctionType,
+    MethodDescriptorType,
+    MethodType,
+    ModuleType,
+)
 from typing import Any, Container, Mapping, Optional, cast
 
 import numpy as np
@@ -194,7 +200,9 @@ def wrapper(*args: Any, **kwargs: Any) -> Any:
 
 
 def clone_module(
-    origin_module: ModuleType, new_globals: dict[str, Any]
+    origin_module: ModuleType,
+    new_globals: dict[str, Any],
+    include_builtin_function_type: bool = False,
 ) -> None:
     """Copy attributes from one module to another, excluding submodules
 
@@ -230,7 +238,10 @@ def clone_module(
         # Only need to wrap things that are in the origin module to begin with
         if attr not in origin_module.__dict__:
             continue
-        if isinstance(value, (FunctionType, lgufunc)):
+        if isinstance(value, (FunctionType, lgufunc)) or (
+            include_builtin_function_type
+            and isinstance(value, BuiltinFunctionType)
+        ):
             wrapped = implemented(
                 cast(AnyCallable, value), mod_name, attr, reporting=reporting
             )
@@ -239,7 +250,10 @@ def clone_module(
     from numpy import ufunc as npufunc
 
     for attr, value in missing.items():
-        if isinstance(value, (FunctionType, npufunc)):
+        if isinstance(value, (FunctionType, npufunc)) or (
+            include_builtin_function_type
+            and isinstance(value, BuiltinFunctionType)
+        ):
             wrapped = unimplemented(value, mod_name, attr, reporting=reporting)
             new_globals[attr] = wrapped
         else:
diff --git a/cunumeric/random/__init__.py b/cunumeric/random/__init__.py
--- a/cunumeric/random/__init__.py
+++ b/cunumeric/random/__init__.py
@@ -25,7 +25,7 @@
 else:
     from cunumeric.random.legacy import *
 
-clone_module(_nprandom, globals())
+clone_module(_nprandom, globals(), include_builtin_function_type=True)
 
 del clone_module
 del _nprandom


Test patch: