The mother will care for it even if you have handled it. Observe the nest from a distance to see if the mother returns. Some animals, like rabbits , only come to the nest in the late evening and early morning, so you won't see them in the daytime. A baby bunny as big as a tennis ball can be released in a safe place. They mature very quickly and aren't helpless.
Deer leave their fawns for hours in places they consider safe.
So if you see a fawn that appears to be abandoned, the doe may return for it later. If so, call for assistance without delay. Do keep them warm. That is more important than feeding them. Feeding milk to an orphaned animal will cause severe diarrhea, and usually results in death.