Eliminate numpy dependency

We only used numpy for one little thing, and it seems we don't actually
need it.