my-ibuffer: Fix performance problem