prevent segv when illegal nested message
[encode] refactor encoding message
[message] improve zval handling