Any sort of loud, sustained sound can cause damage to your hearing, even for short periods of time. If you're playing acoustic drums that haven't been dampened, then you definitely want to be weary of protecting your hearing. If you play drums, you usually always want to be wearing hearing protection.