Add queue for freed temporary points in PointBuffer so we don't do tons
of expansion/reallocation when using temps.