[rtree] move size template argument to internal data structure
[rtree] use different algorithm for hilbert numbers