Better errors when formatmessage fails in railgun

Previous change stopped the session from crashing on NULL, but this change
actually has more realistic messages coming back intead of always
indicating that things happened correctly. It's still up to the caller to
check the return code to make sure it's ok