Good manners dictates that you do not exclude those around you if at all possible, and in my country this would be considered quite offensive.
i've experienced some people assuming the opposite in a crazy way.