Fixes and enhancements for rbtree

Cherry-pick as you see fit.  I need the enhancements for my rlm_ddds work, but if unacceptable I can just do my own lock handling outside the rbtree.