Assert we're not leaking memory

As suggested by @kacfengine.