Taxi drivers are underpaid. The cost of gas is high and the fares are low. Let's just call them opportunistic. If you were Greek, they would know that they could not get away with it, but a foreigner will never know the difference, and they assume you are rich and whatever they charge extra is just a drop in the bucket for you. You should not have to pay more then 2000, give or take a few hundred drachma, to any hotel downtown.
Most taxi drivers are fine people.
Try talking to him. Many speak English. Some have spent years in America. Some of my closest friends are taxi drivers. The taxi stand is across from the terminal.