Yes, baby shower brunches are often less formal, so casual, comfortable outfits are perfectly acceptable.