On its own, not wearing a hearing aid is not likely to worsen the physiological causes of hearing loss, such as damage to the tiny hairs on the inside of the ear that allow you to hear. It’s possible to damage your hearing further if you use other means to compensate for your hearing loss, such as listening to extremely loud music or standing very close to speakers. Some inexpensive hearing amplifiers that you can buy at drugstores look like hearing aids but are not approved by the FDA and can sometimes damage your hearing further by overamplifying sounds.