Remove user association from address instead of setting deleted at flag