Revert "Queue more idle work till the buffer is full on avalon."

This reverts commit ab5093bea3a6204068f0a80b48ec441f9564beda.