Fix bugs in RSA::setKey(filename)

Fixes issues reported in opentibia/server:
https://github.com/opentibia/server/issues/22