Revert "mipmap cache: safety margin around full buffers. avoids deadlocks for me."

This reverts commit f225278d7981fefdaf9ae1c205d33dbd06f619c1.