Overfishing, pollution, and habitat destruction significantly reduce fish populations, making certain species rarer. These factors disrupt the natural balance and can lead to the endangerment or extinction of fish species, further increasing their rarity.