gc improvement: use both depth-first and breadth-first marking

mark depth-first up to some limit, and queue objects after that. this
avoids copying long arrays of pointers into the mark queue, which can
waste a lot of memory. I also managed to flatten most of the marking
into a single function, which seems to make marking faster in general.