At some schools, this is a problem. I did like physics. It's physics that keeps the set from falling down. Sorry, without them, no plays to work on. Do they have other classes you would like to take.
College is the time to explore various subjects to see what else is out there.
Think about this. If you're closer, they'll come visit, take you to dinner, go home. Don't simply follow your friends. Make your own, informed, decisions. I'm sure all this may ruffle some feathers.