prioritisetransaction RPC: Restore compatibility with existing implementations by using satoshis for fee offset rather than BTC

#4531 was merged changing the unit from satoshis to RPC on the premise of prioritisetransaction being a newly introduced RPC, but in fact it has been around and in use out-of-tree since v0.6.1. I don't think we should bring it into 0.10 incompatibly.
Instead of SetHex, a function similar to ParseHashStr() should be used to report invalid values.