Better errors when formatmessage fails in railgun

Previous change stopped the session from crashing on NULL, but this change actually has more realistic messages coming back intead of always indicating that things happened correctly. It's still up to the caller to check the return code to make sure it's ok.