It depends on the context. If there's a group of strangers quietly enjoying a brunch together, yes, it would be kind of rude to just insert yourself into their space. But if you're at a party or in a more open environment, no, there's nothing wrong with talking to people!