Change:
The speedup obtained by the optimizations from which hit stable in 1.23 were mostly lost again from 1.25 onwards, because the compiler started emitting a branch instead of a cmov. 1.24 1.25 From godbolt: Bench comparison with 1.23 nightly (2017-11-20, 9 days after PR merge) and status quo with 1.30 nightly (2018-08-15)
The likely culprit is the upgrade from LLVM 4 to 6. The upgrade was merged on the 10th of Feb. The binary search was fast on the 9th and slow on the 12th. The optimization did hit stable (if only for 2 releases), so does it not need the label?
what do you mean for only 2 releases? Isn't the slowdown present in stable through nightly? Reading the assembly from godbolt it seems to me like there have been some changes but the code causing the slowdown presented in this issue is still there. If that is the case, then it would need that label, but if it was fixed in current stable, then we can close the ticket, as I do not believe we cut point releases for previous stable releases.
I mean it was fast in 1.23 stable and slow from 1.25 stable on, so fast for 2 stable releases, then regressed.
triage: P-medium, since I don't think investigating this takes priority over other things currently being juggled. but also self-assigning since I'm curious and I think I can do some bisection as a background task, assuming I can reproduce the perf regression locally.
discussed in T-compiler meeting. Re-prioritizing as P-high just for evaluating the scope of the regression (though some good work on that is already evident in this comment thread).
Not sure why I previously removed I-slow; I'm putting it back now.
Hey what is the distinction you are making in the columns with the prefix "old" and "new" in the tables you present? That is, I would have thought that "old" somehow corresponds to the state prior to the changes from PR and "new" corresponds to after PR landed; but you already established that PR landed in between the two versions that each have "old" and "new" labels applied to them. So I don't understand what difference has been applied to either the code or the benchmark in between the two columns there. (Hypothesis: Is "old" testing the binary search implementation prior to PR , and "new" testing the binary search implementation that was by PR ?)
(In any case I definitely am able to reproduce a regression on my Linux desktop on between nightly-2018-02-09 and 2018-02-12, as anticipated/noted by ; there's a fair amount of noise between runs when looking at binarysearchl3 in my own machine, but binarysearchl1 is much more consistent in illustrating a 2x regression for me.)
Hypothesis: Is "old" testing the binary search implementation prior to PR , and "new" testing the binary search implementation that was by PR ? That is exactly it
Okay so I've done some preliminary analysis of the scope of the problem. On my desktop linux machine, I built the benchmark suite in atop the nightly compilers from 2018-02-09 and 2018-02-12. Then I did three runs of the suite on each compiler (for a total of six runs). Update: On the LLVM ticket associated with this issue that was subsequently filed, someone noted that I didn't indicate the processor model where I gathered this data. Here that is, in a details block. <details</detailsHere is the raw data from that exercise: Then I put it into a spreadsheet, took the min (of the three runs) of the reported ns/iter for each benchmark, and asked to see the ratio (i.e. slowdown, where 2.0 is a 2x slowdown, 1.0 is no change, and 0.5 is a 2x speedup), comparing the min for 2018-02-09 to 2018-02-12. (One might argue I should use average instead of minimum... but for understanding compiler output, I go along with the argument presented that usually the outliers are much slower than normal, rather than faster, and so taking the minimum is more likely to filter out transient noise rather than artificial improvement) Finally, when looking at the slowdown ratio, I asked for the values 1.2 to be highlighted in red (so that we'd pay special attention to a 1.2x slowdown) and the values < 0.9 to be highlighted in green (so that we'd also notice reasonable speedups). Here is that spreadsheet:
A number of benchmarks improved significantly (i.e. the slowdown ratio is <0.9) over the time period specified. I'm not going to list them here, but there are ~44 of them. But here are the 9 cases that regressed: 1.24x 1.51x 1.30x 2.43x 2.19x 2.16x 2.16x 1.31x 1.30x Obviously the latter six cases are the ones that were already being pointed out by this bug. (The other three entries in the table may or may not have the same core cause.) Obviously the libcore/benches benchmark should not be the sole decider about whether a regression is significant, but based on these results, the impact does seem to be somewhat restricted in scope...
Also, there are some pretty interesting discussions among LLVM developers about when they turn cmov into branches. It sounds there are a number of scenarios where this exhibits better performance due to (I think) either 1. the processor's branch prediction or 2. an expensive test condition that you'd rather evaluate just once
In particular, in the LLVM developer discussions linked above, I saw mention of the following flag: I gave that a whirl, and the results seem promising: <details
To be clear: I am not suggesting that we should turn the pass off by default in . But: we should check whether the improved version from is in our LLVM (and if not, consider cherry-picking it over), and if we can reproduce this issue on the current LLVM version, then we should file an issue with them upstream.
Just checked, that patch is in our LLVM, and this still reproduces with current LLVM (running generated IR through llc).
I misunderstood what this pass is intended to do. It's not interested in hoisting anything into the branches, the only point is to remove the dependency on the branch condition through speculation.
I've filed an LLVM issue:
Okay at this point I think I've convinced myself that the scope of this problem is somewhat limited, So I'm going to downgrade this to P-medium. It also isn't really going to be addressed on our end (except perhaps via cherry-picking a patch for ). So I'm tempted to close it, but I'll maybe wait to let other members of weigh in before I do that.
I'll leave myself assigned for now, but mostly as a reminder to close it in the future. If someone else wants to take this issue and run with it, feel free to assign yourself and unassign me.
I think the main actionable point here is to try and change the binary search implementation in a way that does not run into this LLVM issue. Not sure how that would look like, but something that interchanges the order of the compare and the load (i.e. already load for the next iteration) might work, because this is a pattern that LLVM specifically checks for.
Putting this on T-libs if they can find a libs-level solution.
If somebody knows a workaround, I'm interested. Intersection queries in tantivy are roughly 10% slower because of this. In nightly, one can force cmov with inline . For instance, in the case of binary search. (I personally cannot use nightly.)
have you tried passing to rustc? I noted that this seems to recover most of the lost performance on the relevant slice benchmarks.
Unassigning self: I don't think there's much for me to do here from the viewpoint of compiler, apart from hypothetically turning off by default, which seems like a bad idea.
I'm not fluent in asm but it looks as though the problem still exists. 24: 55-nightly: +-%3E+Result%3Cusize,+usize%3E+%7B%0A++++slice.binarysearch(%26key)%0A%7D'),l:'5',n:'0',o:'Rust+source+%231',t:'0')),k:33.,l:'4',n:'0',o:'',s:0,t:'0'),(g:!((h:compiler,i:(compiler:r1240,filters:(b:'0',binary:'1',commentOnly:'0',demangle:'0',directives:'0',execute:'1',intel:'0',libraryCode:'0',trim:'1'),fontScale:14,fontUsePx:'0',j:1,lang:rust,libs:!(),options:'-O',selection:(endColumn:24,endLineNumber:19,positionColumn:1,positionLineNumber:10,selectionStartColumn:24,selectionStartLineNumber:19,startColumn:1,startLineNumber:10),source:1),l:'5',n:'0',o:'rustc+1.24.0+(Editor+%231,+Compiler+%231)+Rust',t:'0')),k:33.,l:'4',n:'0',o:'',s:0,t:'0'),(g:!((h:compiler,i:(compiler:nightly,filters:(b:'0',binary:'1',commentOnly:'0',demangle:'0',directives:'0',execute:'1',intel:'0',libraryCode:'0',trim:'1'),fontScale:14,fontUsePx:'0',j:2,lang:rust,libs:!(),options:'-O',selection:(endColumn:24,endLineNumber:24,positionColumn:1,positionLineNumber:16,selectionStartColumn:24,selectionStartLineNumber:24,startColumn:1,startLineNumber:16),source:1),l:'5',n:'0',o:'rustc+nightly+(Editor+%231,+Compiler+%232)+Rust',t:'0')),k:33.,l:'4',n:'0',o:'',s:0,t:'0')),l:'2',n:'0',o:'',t:'0')),version:4)
Note that significantly changed the binary_search implementation in the meantime so they are no longer comparable. Instead you'll have to copy the old implementation into godbolt to retain comparability. That said, the issue still exists there too.
I have tested several older revisions of the binarysearch and partitionpoint implementations. All of them had their conditional moves converted to cmp + branch due to this llvm change. suggests that the primary heuristic which turns off this optimization only looks for binary tree searches (unpredictable pointer chasing) but not binary searches (unpredictable pointer arithmetic). Another heuristic related to cost modelling has been . Modifying its threshold via does revert the old binary search implementation (see previous comment) to conditional moves. This doesn't work for the current, short-circuiting implementation from (but running that one on rust 1.24 does produce cmovs). I don't think there's anything we could do on the libs side to trigger that heuristic with its current default (4). The options I currently see are: for an llvm fix (might require a new intrinsic) -case x86 with assembly llvm-args Only option 2 would be a pure libs thing. Some of those options would still require changes to the current implementation and benchmark runs. The old approach seems to be easier to turn into cmovs than current one. Or separate implementations and may be needed.
We discussed this issue in our last libs team meeting and concluded that there's nothing left for libs to do, and that we need to wait for the fix to happen upstream in Our plan is to hand this over to the compiler team for tracking as discussed in https://rust- For now we've tagged both teams and nominated it so that the compiler team can confirm the handover in their next meeting.
I'm happy to confirm handover to T-compiler, but the prioritization of this (currently P-low; P-medium as of 8 days ago) essentially means that we're never going to prioritize it for discussion in a meeting. It will just be task hanging out there for interested people to look at and discuss. So, I just wanted to confirm: Is that an acceptable outcome from the point of view of you all? Or do you think this should be assigned higher priority?
That's roughly the outcome we were expecting. We do feel that this would be worthwhile to have on the radar of a team whose primary purview is LLVM-related fixes that would benefit Rust, but we also recognize that that's a "somebody should", and we don't have a somebody to offer.
What Josh said, I've gone ahead and removed the label, thanks for the confirmation ^_^.
Could something like be used in the interim to force the usage of cmov to recover the performance?
Is profile guided optimization able to do anything in this case maybe?
The x86 backend should now respect for cmov conversion, so if we can expose that in rustc in some way, we can use that.
There's what to do about on IRLO. Adding an was mentioned too.
See for a potential fix.

Discussion:
#![stable(feature = "rust1", since = "1.0.0")] use crate::cmp::Ordering::{self, Equal, Greater, Less}; <del> use crate::intrinsics::{exact_div, unchecked_sub}; </del> <ins> use crate::intrinsics::{exact_div, select_unpredictable, unchecked_sub}; </ins> use crate::mem::{self, SizedTypeProperties}; use crate::num::NonZero; use crate::ops::{Bound, OneSidedRange, Range, RangeBounds};

where F: FnMut(&'a T) -> Ordering, { <del> // INVARIANTS: // - 0 <= left <= left + size = right <= self.len() // - f returns Less for everything in self[..left] // - f returns Greater for everything in self[right..] </del> let mut size = self.len(); <del> let mut left = 0; let mut right = size; while left < right { let mid = left + size / 2; // SAFETY: the while condition means `size` is strictly positive, so // `size/2 < size`. Thus `left + size/2 < left + size`, which // coupled with the `left + size <= self.len()` invariant means // we have `left + size/2 < self.len()`, and this is in-bounds. </del> <ins> if size == 0 { return Err(0); } let mut base = 0usize; // This loop intentionally doesn't have an early exit if the comparison // returns Equal. We want the number of loop iterations to depend *only* // on the size of the input slice so that the CPU can reliably predict // the loop count. while size > 1 { let half = size / 2; let mid = base + half; // SAFETY: the call is made safe by the following inconstants: // - `mid >= 0`: by definition // - `mid < size`: `mid = size / 2 + size / 4 + size / 8 ...` </ins> let cmp = f(unsafe { self.get_unchecked(mid) }); <del> // This control flow produces conditional moves, which results in // fewer branches and instructions than if/else or matching on // cmp::Ordering. // This is x86 asm for u8: https://rust.godbolt.org/z/698eYffTx. left = if cmp == Less { mid + 1 } else { left }; right = if cmp == Greater { mid } else { right }; if cmp == Equal { // SAFETY: same as the `get_unchecked` above unsafe { hint::assert_unchecked(mid < self.len()) }; return Ok(mid); } size = right - left; </del> <ins> // Binary search interacts poorly with branch prediction, so force // the compiler to use conditional moves if supported by the target // architecture. base = select_unpredictable(cmp == Greater, base, mid); // This is imprecise in the case where `size` is odd and the // comparison returns Greater: the mid element still gets included // by `size` even though it's known to be larger than the element // being searched for. // // This is fine though: we gain more performance by keeping the // loop iteration count invariant (and thus predictable) than we // lose from considering one additional element. size -= half; </ins> } <del> // SAFETY: directly true from the overall invariant. // Note that this is `<=`, unlike the assume in the `Ok` path. unsafe { hint::assert_unchecked(left <= self.len()) }; Err(left) </del> <ins> // SAFETY: base is always in [0, size) because base <= mid. let cmp = f(unsafe { self.get_unchecked(base) }); if cmp == Equal { // SAFETY: same as the `get_unchecked` above. unsafe { hint::assert_unchecked(base < self.len()) }; Ok(base) } else { let result = base + (cmp == Less) as usize; // SAFETY: same as the `get_unchecked` above. // Note that this is `<=`, unlike the assume in the `Ok` path. unsafe { hint::assert_unchecked(result <= self.len()) }; Err(result) } </ins> } /// Binary searches this slice with a key extraction function.

assert_eq!(b.binary_search(&8), Err(5)); let b = [(); usize::MAX]; <del> assert_eq!(b.binary_search(&()), Ok(usize::MAX / 2)); </del> <ins> assert_eq!(b.binary_search(&()), Ok(usize::MAX - 1)); </ins> } #[test] fn test_binary_search_by_overflow() { let b = [(); usize::MAX]; <del> assert_eq!(b.binary_search_by(|_| Ordering::Equal), Ok(usize::MAX / 2)); </del> <ins> assert_eq!(b.binary_search_by(|_| Ordering::Equal), Ok(usize::MAX - 1)); </ins> assert_eq!(b.binary_search_by(|_| Ordering::Greater), Err(0)); assert_eq!(b.binary_search_by(|_| Ordering::Less), Err(usize::MAX)); }

let b = [1, 1, 2, 2, 3, 3, 3]; assert_eq!(b.binary_search(&1), Ok(1)); assert_eq!(b.binary_search(&2), Ok(3)); <del> assert_eq!(b.binary_search(&3), Ok(5)); </del> <ins> assert_eq!(b.binary_search(&3), Ok(6)); </ins> let b = [1, 1, 1, 1, 1, 3, 3, 3, 3]; assert_eq!(b.binary_search(&1), Ok(4)); <del> assert_eq!(b.binary_search(&3), Ok(7)); </del> <ins> assert_eq!(b.binary_search(&3), Ok(8)); </ins> let b = [1, 1, 1, 1, 3, 3, 3, 3, 3]; <del> assert_eq!(b.binary_search(&1), Ok(2)); assert_eq!(b.binary_search(&3), Ok(4)); </del> <ins> assert_eq!(b.binary_search(&1), Ok(3)); assert_eq!(b.binary_search(&3), Ok(8)); </ins> } #[test]