I couldn't find info on the word being used as a derogatory word, specifically but the Romans did use "barbarus" for uncivilised people and it became a common term for all foreigners after the Augustus age.