If you haven’t had chicken pox before then, it is highly likely that if you are looking after a child with chicken pox, then you may catch it.
If you have had chicken pox yourself, then your immunity is usually lifelong and therefore, there is no reason for you to stay off work other than to look after your child.