While women might notice, it's more about how clean and well-maintained your clothes are. It's perfectly fine to have a "uniform" or favorite outfit, as long as it's cared for.