There are some people who have a wide range of sugar intolerances, but overall it is seldom a problem. Yes, humans can get along on just about any diet they want. Vegans, those vegetarians who do without any animal products, can have perfectly adequate diets and can get all the calcium they need from plant sources. You might want to check out my Links for more information. That said, very few people except the most militant vegans believe all the horror stories about milk.
Most people can and do use milk as part of a healthy diet.
The choice is totally yours.