Swap directions when scoring for LRU optimization