This implementation is simple hence elegant, yet it looks like it will consume quite some memory as the number of items grows. Also the recursivity will soon cause some concurrency issues sooner or later.
It could be worth checking whether an implementation storing two TreeSet (sorting over X coordinates in one, and Y coordinates in the other) does not improve memory (and maybe also cpu) performances. Binary searches on these sets should offer you a similar complexity and a much better memory footprint.