There may be a myriad of reasons why one will sink while trying to float on water. It may be that their body is in the wrong position or perhaps they’re employing the wrong technique. And no, your swimwear like your swim trunks (if you’re a male) or your swimsuit (if you’re a female) has nothing to do with this unless you’re wearing jeans and cotton shirts while swimming (which is highly unlikely).  But in general, the reason why your body sinks while trying to float is based on relative density.
To keep things simple, let’s take a look at fat and muscle tissues. Fat tissues are less dense than water and will float easily on its surface. Think of the oil (from barges, oil rigs, etc.) that spill on the water’s surface and notice how they float instead of mixing with the water. So basically, fat’s density is lesser relative to the density of the water. It’s the other way around for muscle tissues. Muscle tissues have higher density levels relative to water, which means that muscle tissues will sink much faster and will require more effort to float.
This is why you’ll often hear that fat people float much easier than muscular people simply because they have more fat tissues than muscle tissues in their bodies. The same applies to the assumption that women float better than males, again because they have more fat tissues than muscle tissues.