[core] Remove user association in guardian_factory