Ensure accessory view is released by delegate