Issue 1:
Issue

I am implementing custom dtypes for symbolic types: Variable, Expression, Formula, where the algebra is more or less:

<ExpressionOperand> ::= <Variable> | <Expression>

<math_op> ::= + | - | * | - | etc...

<logical_op> ::= == | != | < | <= | etc...

<ExpressionOperand> <math_op> <ExpressionOperand> -> <Expression>

<ExpressionOperand> <logical_op> <ExpressionOperand> -> <Formula>

I can more or less implement this for most of the operators by augmenting the existing UFuncs using PyUFunc_RegisterLoopForType.

However, for some builtin operations, like dotfunc, it seems that the C API is rigidly fixed to only providing closed operations for dot, matmul, etc., so I cannot define a meaningful dot produce for Variable, since the result of this operation should produce a Expression:

RobotLocomotion/drake#8452 (comment)

Rambling

I am currently inspecting the source code for v1.11.0 (what I currently have on Ubuntu 16.04 LTS), and it seems that I can use __numpy_ufunc__ (the old version of __array_ufunc__ which was solidified in v1.13.0, it seems?), because:

array_dot (methods.c) calls PyUFunc_CheckOverride first to check if numpy.core.multiarray.dot can be overridden by a ndarray subclass

If no override is found, it uses PyArray_MatrixProduct2, which then calls dotfunc

TBH, it seems that PyUFunc_CheckOverride could possibly be generalized to see if a compatible user loop has been registered for the PyUFuncObject, if no __array_ufunc__ is available.

Looking at a recent master (c486d8d), it looks like this functionality is removed from array_dot, so I'm not sure where it's gone now.

Seeing this footnote in the __array_ufunc__ documentation, it seems that matmul is under consideration, but not dot?

I am still generally concerned of subclassing ndarray for (a) version considerations (it seems that this feature is still relatively volatile) and (b) forcing downstream users to care about which ndarray subclass they use.

(That being said, is there a way to create a custom dtype, and then indicate which ndarray subclass should be used?)

Issue 2:
With NumPy master, it is currently impossible to override ndarray @ other if other also implements __array_ufunc__.

Consider:

import numpy as np

np.__version__ # '1.13.0.dev0+9ef5891'

class OptOut:
__array_ufunc__ = None
def __rmatmul__(self, other):
return 'rmatmul'

class OtherArray:
def __array_ufunc__(self, *args, **kwargs):
return 'array_ufunc'
def __rmatmul__(self, other):
return 'rmatmul'

array = np.arange(3)
opt_out = OptOut()
other_array = OtherArray()

# OptOut works as expected:
array * opt_out # TypeError: unsupported operand type(s) for *: 'numpy.ndarray' and 'OptOut'
array @ opt_out # 'rmatmul'

# But OtherArray does not:
array * other_array # 'array_ufunc'
array @ other_array # TypeError: Object arrays are not currently supported

This is problematic. We definitely need a way to allow for overriding @ from the second argument, whether than means breaking the rules on __array_ufunc__ (allowing matmul to be passed even though it isn't a ufunc) or breaking the rules for __matmul__ (to call __rmatmul__ or allow __rmatmul__ to be called by Python, even when __array_ufunc__ is implemented on the other argument).