No, you don't have to give Duo access to your contacts. However, if you want to make video calls with people who aren't in your contacts list, you'll need to enter their phone numbers manually.