During your ascent, it is possible that you may accidentally inhale some of the chalk in your bag. If this happens, then there is no reason to panic. Since climbing chalk is not loaded with toxic chemicals or anything like that, you should be fine.
The only concern would be if you had constant exposure to the chalk. With enough dust digested, long-term health problems could persist. However, that is not very likely to occur, so it is not something that you should stress about.