Remove rsa dependency, which is not needed (fixes #1834)

We don't use it directly and dependent libraries are coded
to use pycrypto if it is available.