The Brown family decided to congregate around the one table with the Greenhithe Mob by the window