In order to enter the city, the quest Song of the Elves must be completed. While outside Prifddinas, it will appear as if there's nothing there when looking from outside the city walls. This is revealed by Baxtorian to be one of the security measures put in place during the God Wars.