Fix memory bug when using AsyncWrapper

Even when nothing was logged the background worker in the AsyncWrapper
would construct empty arrays, over time slowly consuming memory.
fixes #177