The number of players are adjusted not only to the size of the room, but also to the amount of riddles and other challenges to face. That means the rooms are prepared for certain amount of people, and going over that limit might decrease the quality of the game.