Go to your Settings > Privacy and Security > Groups & channels > Choose My Contacts OR Choose who to always allow or never allow. See the screenshots below for a more detailed explanation.
Step 1: In your Telegram account, click on the hamburger button and choose Settings
Step 2: In Settings, choose Privacy and Security
Step 3: In Privacy and Security, choose Groups and Channels
Step 4: Choose who to allow to add you to groups and channels. In this example, only the users contacts can add them to group and channels. Click SAVE.
For more details on this, visit Telegram's Official FAQ page here https://telegram.org/faq