Lethargy in a baby is a serious concern. It may be difficult to tell whether a baby is lethargic or just drowsy from sleeping. If they seem less alert, don't smile, are too weak to cry, have difficulty waking up, or show a fever greater than 102 degrees F, they should be immediately brought to a healthcare professional.