Moved check for heap allocation to end of unref

This is for #316