I think people conflate two different technologies when talking about self-driving vehicles.

First, there's the self-driving car for the dense urban environment, which is basically a robo-taxi. You don't buy or own these cars, you hail them and they take you where you need to go. Basically mass transit with the option to go wherever you need instead of pre-defined stops, combining the best aspects of a taxi and a bus. There's no reason these need to be cars at all, aside from the fact that we have all the infrastructure for cars. They could just as well be small trains/"horizontal elevators", the tubes from Futurama, or whatever else your local sci-fi author can come up with. These types of vehicles could be centrally controlled and you could indeed replace complex AI with expensive infrastructure, but that is something we'd like to avoid (who knows if it's even politically possible), hence the push for using existing infrastructure. As an added bonus, these types of fully self-driving vehicles can take over tasks like delivery.

The second type of self-driving car is the car for every other environment, basically the same cars we have now with an autopilot add-on. These vehicle would be for people who live in small towns, suburbs, rural areas or in general travel outside of dense cities. It makes more sense to own vehicles like these, but they must cope with a larger variety of environments. Personally, I would never buy a car that does not have some form of manual override so I could drive it myself. If the AI is advanced enough this manual override doesn't have to be a steering wheel, but for a car I own, having the ability to violate the law, crash it or tell it to go someplace it doesn't want to (or, god-forbid, run someone over intentionally) is essential. I suspect that making an AI that can cope with all kinds of driving conditions in the country is a task so monumentally complex that cars (the kind you own) without manual override mode will basically never be a thing, both because it would be expensive and because I don't think the kind of people who would need to buy a car to own would accept not having full control. A couple of instances of the AI failing in some basic way (failure to drive into a small garage, for example) would be enough to convince people that a manual mode is essential.

What percentage of cars in the future will be of the first type and which will be of the second type is up for debate, but regardless I think the second type of car will be something you buy out of necessity or a luxury good, and overall car ownership will go down.

However, with technologies like self-driving electric delivery vans, telepresence, improved solar power and battery storage as well as better internet coverage, it's not obvious to me that the future lies in dense cities. I speculate that a certain tipping point will come in the near future where less and less people will need to live in a large city for work and migrate to small and medium sized cities for better living conditions and larger homes. As technology advances, it will become more and more comfortable (and possible) to live in a less dense environment and population will spread out. I'll admit that this involves a bit of wishful thinking since I hate big cities, but the alternative where population keeps concentrating in urban areas seems like a terrible dystopia to me.