rbd.c: fix error handling for custom functions

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>