Removing an authorized user on a credit card won't have any impact on the primary cardholder's credit score. It may decrease the authorized user’s credit score because it will decrease their available credit and their average age of accounts, two main credit score factors.