What unites them is a tradition, culture, history , destiny maybe, but not genetics. If you do assume for a moment that Semites are a special race, consider also the implication that this so-called race comprises both Jews and Arabs. One often talks of the kinship between these two, who are now at loggerheads with each other. So did the Nazis who killed the Jews and invited cooperation from the Arabs. If there is any substance to the term, it is only to denote a specifically anti-Jewish movement.
Antisemitism is a generic term which signifies a singular attitude to a particular group of people.