Fix return code of gds_inline_rbtree_add

and use 'int' instead of 'int8_t' as return type