It is because of the negative context. For example, the word "idiot" is also offensive. Although it means the same thing as the phrase "private person", just in another language. By the way, once upon a time in the U.S., the offensive word was just that "black". And blacks themselves were called blacks.