By the 17th century , the Portuguese , followed by the Swedish , Dutch , French , British and Danish , had constructed forts in the city .
The Portuguese built a fort in the town .