The answer to this comes down to basic physics. Rapidly-expanding gas is responsible for the bulk of the sound’s volume, along with the miniature sonic boom that accompanies a bullet traveling faster than the speed of sound (which is most bullets).
With a silencer, you can bring that volume down to a level that’s safe enough to be exposed to without hearing protection. Most .223/5.56 suppressors reduce sound between 30-40 dB.