> Just a pointer would not keep the NumPy object alive, I think.

No. We would have to do reference counting in addition. I'd really like
to stay away from such things.