[MRG] fix malloc errors in trees

There were still a few unchecked mallocs and constructs were the checks would actually cause double `free`. I also fixed some other minor stuff. See commit log for details.

I was actually trying to optimize the sorting by implementing three-way quicksort, but when I tried to cache some stuff in a new `malloc`'d array, the tree code started crashing.

Ping @glouppe.