Anchovies are cured in salt and vinegar for a period of time and that’s where the saltiness and their signature taste comes from. As they cure, they absorb some of this salt. That’s why even if you rinse them, they’ll still taste quite salty.