A group of men are sitting and standing a courtyard, some of them are reading books and some are talking.
Men are reading and milling about.
A group of men standing together is "milling about" and both sentences having the men reading.
People milling about would imply that they are talking, and reading books would indicate that they are reading.
If some men are reading then the men are reading.