[rst mode] Better innerMode implementation

Also fixes CodeMirror.innerMode to give up on null or fixpoint

Issue #1704