The sign-swapping approach is used only when you test the intercept. When you test other covariates, you simply use a permutation.