RDP security fixes

* limit key size to 2048 bit
* fix problems with keys > 256 bit