Caffeine may take about an hour to reach the baby through breast milk after you consume it. It may make your baby fussy and irritable (20).