Remove dependency on numpy

The dependency was merely a convenience and functionally not necessary. By
removing it we gain pypy compatibility and make the package more light-weight
in general.