My H is not particularly good with kids, especially other peoples'. I hear you. I understand what you are feeling. Read Brokens post this am. It's more encouragement for you.
Withdrawal does end.
Things come back into focus. Your pain will lessen and it will fade. It happened for me. It's weird freedom, I've learned that sometimes even when our hearts don't seem to agree with us, we NEED to go with our head. The heart will follow and it WILL agree with your head.