Remove a temorary listener after flushed in Queue