Only check swap-size if swap=yes

Tweak the logic so a check of swap-size is only done if swap=yes