Try to convert first with to_hash, then to_h

rb_convert_type doesn't return if conversion fails, so use
rb_check_convert_type and the raise vi rb_convert_type. Make sure, that
this behaviour is consisten across all generator implementations.

Fixes https://github.com/flori/json/issues/162