All of these things can affect the quality of the experience, and when times get rough in the lab, as they almost always do for grad students at some point, these other factors can make the difference between sticking it out and quitting. You do not have to give them a reason for choosing to go somewhere else. They will be curious about where you did go, so you can tell them if you like. Remember, some of these people you will meet again, and it pays to be polite. Yes, you can.
Graduate schools are pretty good at selecting students who can succeed.
If they let you in, they think you're a good bet. This can be more difficult to deal with than fear of failure. Relax a bit. Your first experience with grad school may bring up these fears, especially impostor syndrome, as you find yourself working very hard and coming up only average in your class. Concentrate on holding your own in the group, and learning from each other if the culture of your department allows that.