[lib] remove duplicated code in rsa.c

    - remove duplicated code
    - better code style