malloc compile warning and error handling fixes

fixed a compile warning and added a check for malloc() failures

the call to sml_value_init() should probably moved upwards, before sml_buf_get_next_type(), but I don't know if this will make anything better.