Move vi mode stuff to .inputrc and use rvm's version of it's sourcing thing.