Use a priority iq for dirs to temper memory growth