Fix element removal for inline/inverse collections