In order to break RSA encryption, you have to factor Mod, from the public key. So breaking RSA is as difficult as factoring. It is astronomically unlikely, however. Assuming a uniform distribution of probability, the chances of a solution before that time are less than one percent. In october 1992 Arjen Lenstra and Dan Bernstein, factored a 522-bit number into primes using the Number Field Sieve.
Addition is considerably faster than most CPU instructions.
However, such devices should be easily defeated by simply using a ludicrous length for your key, which exceeds expectation and therefore the hardware limitations of any reasonable device. It would be silly of anyone to build a device handling in hardware numbers large enough to assure that that kind of time were required anyway. It will take you a few minutes to read it, and that's time for your machine to decrypt the next one. It will take a few seconds to encrypt a letter, but again, it will take you a few minutes to write it, and you can write the next one while the encryption is taking place. At 2048 bits, the rate of encryption of a good computer exceeds human writing and human reading speed by a wide margin.