They may have a different language, different customs. They may seem insular, clannish, self-supporting, not dealing that much with the world surrounding them but rather just with each other. They may also seem better off financially than their neighbors. As individuals, they did no harm to their neighbors, except they have the bad luck to be both different and successful.
Armed men come in the middle of the night.
They are granted but a few minutes to collect what belongings they can carry, and are marched off. They are taken in the dead of winter, but few have the winter clothing needed for the trek to the camp many hundreds of miles away. Many freeze to death. Old age, illness count for nothing. Their guards push them ever onward.