Also, some men and boys will not go out with their chests uncovered. The more Orthodox a movement is, the greater the concern about modesty in swimming attire. Hence, it is difficult to give a broad rule. Similarly, men should wear boxer-style trunks. Lastly, remember that it is not a good idea in general to start dressing very young children in ways that mimic sexually suggestive adult dress.
Many people do so for other reasons as well, such as to protect one's skin from the sun.
There is less of a concern around the immediate family. The FAQ is a collection of documents that is an attempt to answer questions that are continually asked on the soc.culture.jewish family of newsgroups. It was written by cooperating laypeople from the various Judaic movements. Hopefully, the FAQ will provide the answer to your questions. The FAQ maintainer will endeavor to direct your query to an appropriate individual that can answer it.