Fix a bug where set was incorrect when updating LRU due to fill