It’s hard for you to see baby squirrels since they don't leave the nest until they are fully furred and can survive on their own. The only chance you’ll come across baby squirrels will be if their mother died and leave them helpless. Of Course they’ll wander out of the nest in search of what to eat.  Secondly, if baby squirrels should fall off the nest or wander out, they can be preyed on by bigger animals.