This is not usually the case if the bird is fully feathered. Almost all baby birds, when they leave the nest, can barely fly, often end up on the ground. It may take the baby several days before it is strong enough to fly back up into the trees where it is safe. This is a natural process. It's okay to touch a baby bird because the mother will NOT reject it.
Most birds have very little sense of smell and only care how their babies look and sound.
The easiest way to tell if a baby needs to be rescued is to feel its chest. A baby cared for by its parents will be quite plump and the meat on either side of its breastbone will be rounded and even with the breastbone. If the meat is sunken in on either side of the breastbone, the bone sticks out like the keel of a boat and you can pinch the bone between your thumb and first finger, this baby has probably lost his parents to a predator and should be taken to a rehabilitator immediately. DO NOT give the baby bird water with an eyedropper. This is a good way to choke or even drown it.