Yes, they can. However, this question is not as simple as that. While even some bags will be made with water-resistant materials, what you want to avoid is the chalk itself getting wet.
If this happens, depending on how much water seeps inside, you may have to get new chalk. When climbing chalk is mixed with water, you get a messy paste (as you would expect). Let’s say that it is not precisely usable after that.