Only remove actualy failed blocks from setBlockIndexValid