To control whether Google can collect contact data in Google products other than Gmail, do this:
Go to your Google account. From Gmail, click your profile picture in the top right and select Manage your Google Account.
Switch to People & sharing.
Under Contacts, click the options Contact info saved from interactions and Contact info from your devices and toggle them on or off. Google automatically saves your changes.