You are right to acknowledge his need to be held. He will need special coaxing, however, so that your needs are met, too. Certainly you can arrange to lessen the amount of time spent in the car with him, but you cannot be held hostage. You must be able to do the shopping and other errands your family needs. I assure you, this is not the first time I have heard this distress call.
Some babies hate being restricted in a car seat, while others almost purr for a drive.
Your son may also show other specific preferences as he grows, some of which will likely even make parenting easier instead of harder. The good news, as well as the bad news, is that he is already a child with specific likes and dislikes. Begin a program of desensitization. Build positive experiences around the car seat and driving in the car. You may want to purchase some of the special products on the market for babies that are made specifically designed for your child's visual development.