I'm hesitant to change behavior in this PR, other than performance.

I don't really understand why buffer is supported here at all.  Does
memoryview conform to the buffer interface?  Buffer is deprecated in
3.0.

I think if you look closely at these suppressed exceptions, most are
sloppy to ignore.  __array_struct__ doesn't work quite the way I expect
either.