Change:
<!-- Thank you for filing a bug report! Please provide a short summary of the bug, along with any information you feel relevant to replicating the bug. --The following code: Both functions should be identical, returning the 128-bit value in RAX and RDX per the sysv-64 ABI, which is the default for the environment godbolt uses by default. Instead, when returning the value without specifying any ABI, the compiler chooses to return the value using an out pointer argument. Note: This affects , which is a particularly useful type and affects a significant portion of , such as and This has improved and regressed since 1.43.0, the first version I could find which would use 128 bits for the type. All versions tested used the expected code generation for the function. All tests done using and no target overrides, which in practice is a x86_64 linux machine using the sysv-64 ABI. ..= 1.42.0: niche not used, cannot be compared 43.0 ..= 1.47.0: codegen like today, uses out ptr for return 48.0 ..= 1.60.0: returns using an LLVM , which produces the most desirable code on sysv-64 61.0 .. (current 2022-05-28 nightly): poor codegen that uses an out ptr for return label +regression-from-stable-to-stable +A-codegen
To be clear, it's debatable if this is a regression (if it is, it's just a performance issue), since we make no guarantee about the ABI of . That said, this likely applies to all 128 bit types, so could have a non-negligible impact on overall performance.
This is / , reverted by /
This is interesting, because tuples or structs seem to have some exceptions for two usize-sized elements specifically. and return by value, while , , and (and all arrays) return with a pointer. Additionally structs containing two usize values exactly are returned by value (both with and without . The niche usage is specifically inhibiting this optimization, since otherwise the data returned would look like a in the or cases.
Performance regression is a regression. And I think this is a very bad one. If I replace the with then it even starts to loading from memory. I don't think this is related to niche though; still have the good old codegen. Somehow ceased to be passed as a scalar pair, while still does.
Ah, or don't reproduce it, but do return via an out ptr.
Okay, I located the reason. is only produced if all variant' data part are scalar and have the same . and are all which matches that of , so it's treated as a scalar pair. However have different integer size, and have which never matches with any integer, so is not produced and it's treated as . Before , something this small is passed in registers, so it is optimized to be the same as if it's passed as scalar pair, but it's not, it's actually just an small aggregated passed directly. forces it to be an indirect passing, causing the perf regression. Given the motivation of is to workaround LLVM inability to autovectorize things passed in register, I think a fix would to be use some heurstics to determine if a 2-usize type can be a target of autovectorization -- if so, pass indirectly, otherwise, pass by value.
Look at which was a more targeted fix. Maybe it could be reopened ? EDIT: Looking into it right now.
Thanks for the pointer. That PR looks like a better approach to me. I think the compile-time perf regression in that PR results from excessive calls when the type is smaller or equal to pointer size (which includes almost all primitives!). I suspect that if you change the nesting of ifs, so that it's only called on types of size between 1usize and 2usize, then the perf regression should be mitigated.
I've reopened as I've confirmed that with the PR the regression is fixed and hopefully the perf regression of the PR is also fixed.
Assigning priority as discussed in the of the Prioritization Working Group. label -I-prioritize +P-high +T-compiler
This issue only dubiously shows a codegen regression (it is Not Great, to be clear, but it's not clear from this example and my knowledge of how common x86 microarchitectures work that it's Actively Awful). If it is, it does not show it is a performance impact in practical cases, as it is still very few instructions and then a simple . This requires a benchmark to show what we are actually trying to improve on in terms of runtime, or comparison for multiple values, not just the zero cases, or even just something that shows that the impact is more significant than a xorps and mov where a xor would do.
An alternative would be experimenting more with this in context of other code we expect it to live in, because LLVM eagerly inlining can make problems go away, as cited here: Small functions like this, in particular, will tend to be inlined by any backend, precisely because Rust functions are not made external by default. Thus they do not have to respect the System V ABI. Or any ABI, for that matter, and they can make things up as they go along. Constant folding becomes quite practical to do. So, many functions offer seemingly terrible codegen in isolation that doesn't actually come up as a problem in actual practice. This is much more of an issue when we're mostly touching how they do parameter passing, which is precisely what inlining can make disappear, which is why I bring it up.
BTW, in our project, we also noticed performance degradation because of this regression (small 16B s returned via the stack instead of registers), so I vote for a quick fix. Ideally, all return values of up to 16B size should be returned in registers (at least on x64 and arm64 and at least for integer/pointer values, not sure about floats and the like). Thanks.
I would sincerely appreciate it if this was quantified in some manner or if you could give a more complex example. It's not that I don't think you are correct, it's just that unfortunately the quickest fix is to simply revert the changes that regressed this outright. But reverting the changes will simply regress the performance in other areas, and the losses there were quite significant. That's not acceptable, but this can still be fixed. We need to find something else instead as a solution, and more benchmarks or even a nuanced array of codegen examples means we are less likely to perform an overly sophomoric "fix" that could make things even worse for real code at the expense of improving academic cases. It will be fixed, but not with the quickest fix, except in the sense of making things faster. Honestly, the question I have is why it isn't e.g. returned in a single xmm register instead. It doesn't make sense to me that LLVM would drop a value that can clearly already fit in a register on the stack, ever.
It's the rustc that makes return value indirect, instead of LLVM.
The ABI is decided by rustc here, not LLVM. If rustc tells LLVM to use an sret argument, then it must do so. Returning an i64 pair in an XMM register is also a bad idea because it requires GPR to XMM moves, even if we ignore issues related to the target-feature dependence of XMM registers.
Yes, we need to quantify this, but it's not that trivial. Our project is fairly big and the benchmark directly or indirectly touches 200K+ LOCs of our code and likely millions of LOCs of code from other crates, so it's not easy to quantify. I'd estimate the average performance loss 5% across the board from 1.60 -1.62. We do use quite a few functions returning s in tight loops (fallible value matchers, pushing results to bitmap sinks). I can try to create a manageable repro case with this, if it would help.
Yes, one of the smaller loops that is like what you use, if it can show an even somewhat similar regression (may need generous application of ), would be excellent. Hm. Honestly, I guess that after the amount of noodling around I have done with LLVMIR and the way the LangRef phrases 's documentation, implying it is only a polite suggestion, I didn't fully grok that LLVM really interprets as a true unconditional directive.
I've tried to implement something with , unfortunately, Rust happily optimizes it out, so I have to find a different approach (didn't have much time yet, sorry). What we also see and that is probably at least equally big problem is the handling of larger s, where a lot of unnecessary memory copying is in play. After doing some -based optimizations, these memcpy() calls are the hottest function in the profile. What happens: Calling an async function in fact calls a stub producing a (which is in fact a generator, which is later polled). This is stored in the parent's async function "stack", which is simply an aggregated of the parent function and the called function (with some layout optimizations). Unfortunately, instead of telling the generator to materialize the directly into the proper place, the is materialized on the regular stack and then copied from the stack to the caller's . Now, we have a loop calling a request handler which supports various request types (dispatched by a statement) where one of them produces a large . Then, the call to the request handler materializes a by writing a few words to the stack and then this (practically empty) space is copied in full from the stack to the parent . This wastes cycles like there is no tomorrow - instead of materializing the callee future on the stack, the async functions should materialize the callee future directly in-place in the caller's future. That would save the copying (and, especially, copying of still uninitialized space!). I'm wondering if that has something to do with the issue as well or not. In any case, I couldn't find a specific issue for this problem in issues. Are you aware of this problem already or not? I'm definitely no expert in compiler building, but if I understand it correctly, the LLVM attribute takes a location where to materialize the return value. So it should be possible to specify the location inside of the caller's future instead of the stack as such (and thus greatly optimize async function start). If not, possibly, there is a different LLVM feature which could be used instead (or the ABI modelled differently, by passing a hidden parameter where to materialize the generator).
Note that this also happens for . I saw some suggestion that single-pointer Error types (, , , ...) could work around this bug by storing their pointers in a when not running under miri. Ignoring the fact that this is gross, it would just move the problem so that you get the bad codegen for cases where is a pointer/ref/etc instead (stuff like ). So, just so nobody else is confused, this does not seem to be a case where you can abandon your morals in the name of performance in order to work around this, you'll just hit it in different cases. Anyway, I do think it's kinda bad[^small]. Most Rust code is very result heavy. I think quantifying this will be very hard though -- For example, rust code often looks like: The code is clearly worse than -- those reg/reg movs are free (on most modernish machines), but the stack ops are reads/writes[^4] to memory. Sadly, I'm not sure this is something that will be easy to measure... [^4]: Well, the writes are implied, since they'd be in the callee. Notably, in particular would be a problem if used too aggressively -- it currently basically forces stack spilling of the argument (see ), so you'd need to use it in such a way that this overhead didn't overshadow the ABI wonkiness. Constructing a benchmark out of something like the linked godbolt but with more cases might be an okay thing to measure in a benchmark, with some caveats like measuring min time rather than avg (but I'd suspect it not to still kinda suck for all the normal reasons benchmarks suck[^1]) For codegen issues, something that models the CPU resources like llvm-mca may be the best bet, but it tends to be terrible at things with calls (so, as always, real measurements artificial benchmarks). For this case, because they both have basically equivalent calls it might be fine, and MCA does seem to suggest that the code will perform decidedly better than , at least on the CPU it's simulating. (But yeah, don't read into how much less, due to the calls) [^small]: Well, I'd expect this to be pretty small in terms of absolute impact, it just may be a small overhead to a fairly large amount of code. [^1]: TBH, I've all but given up[^3] on artificial benchmarks at this point unless carefully constructed. Processors are getting very smart at making running the same code in a loop repeatedly (the way you do in a benchmark) an inaccurate way to measure time that that code will normally take to execute. [^3]: Which is to say I still use them, they just make me sad.
Hmm. It kinda sucks that we don't use higher-level information from MIR or thereabouts to annotate types with "wants vectorization" and "doesn't want vectorization". MIR is about where we know which cases of would prefer to go into a vector register and which would be just as happy being passed in two general registers.
note about that confused me a couple times: The integer part holds the discriminant, and the ptr holds the data in both cases.
I'm looking at and trying to figure out why something like still doesn't get an ABI of while something like does get a ABI. Both types have a size of 16 and an alignment of 8. Furthermore, both store either or a pointer in the first 8 bytes and store a in the second 8 bytes. I used to get the layout of both types, and for I get: <details<summaryLayout (ABI is ScalarPair)</summary</detailsHowever, for I get: <details<summaryLayout (ABI is Aggregate)</summary</detailsThe only main difference between these types (other than being assigned a different ABI) is the of the variants. Specifically: - variant has offset variant has offset Gets ABI - variant has offset variant has offset Gets ABI - variant has offset variant has offset Gets ABI Does anyone know if it would be possible for to have a ABI? If so how would need to be changed to support such a thing? I figure it's somewhere in this loop but I'm not sure exactly how to make the change. EDIT: It seems like that loop is only used for the , but the is computed separately. So it seems like would need a more substantial change to use the . Specifically, we would need to: Allow the variants to (sometimes) avoid being when shifted after the niche. Allow the enum to use a even if some of the non-niche variants have data.

Discussion:
break; } }; <del> if let Some(pair) = common_prim { // This is pretty conservative. We could go fancier // by conflating things like i32 and u32, or even // realising that (u8, u8) could just cohabit with // u16 or even u32. if pair != (prim, offset) { </del> <ins> if let Some((old_prim, common_offset)) = common_prim { // All variants must be at the same offset if offset != common_offset { </ins> common_prim = None; break; } <ins> // This is pretty conservative. We could go fancier // by realising that (u8, u8) could just cohabit with // u16 or even u32. let new_prim = match (old_prim, prim) { // Allow all identical primitives. (x, y) if x == y => x, // Allow integers of the same size with differing signedness. // We arbitrarily choose the signedness of the first variant. (p @ Primitive::Int(x, _), Primitive::Int(y, _)) if x == y => p, // Allow integers mixed with pointers of the same layout. // We must represent this using a pointer, to avoid // roundtripping pointers through ptrtoint/inttoptr. (p @ Primitive::Pointer(_), i @ Primitive::Int(..)) | (i @ Primitive::Int(..), p @ Primitive::Pointer(_)) if p.size(dl) == i.size(dl) && p.align(dl) == i.align(dl) => { p } _ => { common_prim = None; break; } }; // We may be updating the primitive here, for example from int->ptr. common_prim = Some((new_prim, common_offset)); </ins> } else { common_prim = Some((prim, offset)); }

<ins> //@ compile-flags: -O #![crate_type = "lib"] #![feature(core_intrinsics)] // Tests that codegen works properly when enums like `Result<usize, Box<()>>` // are represented as `{ u64, ptr }`, i.e., for `Ok(123)`, `123` is stored // as a pointer. // CHECK-LABEL: @insert_int #[no_mangle] pub fn insert_int(x: usize) -> Result<usize, Box<()>> { // CHECK: start: // CHECK-NEXT: inttoptr i{{[0-9]+}} %x to ptr // CHECK-NEXT: insertvalue // CHECK-NEXT: ret { i{{[0-9]+}}, ptr } Ok(x) } // CHECK-LABEL: @insert_box #[no_mangle] pub fn insert_box(x: Box<()>) -> Result<usize, Box<()>> { // CHECK: start: // CHECK-NEXT: insertvalue { i{{[0-9]+}}, ptr } // CHECK-NEXT: ret Err(x) } // CHECK-LABEL: @extract_int // CHECK-NOT: nonnull // CHECK-SAME: (i{{[0-9]+}} {{[^,]+}} [[DISCRIMINANT:%[0-9]+]], ptr {{[^,]+}} [[PAYLOAD:%[0-9]+]]) #[no_mangle] pub unsafe fn extract_int(x: Result<usize, Box<()>>) -> usize { // CHECK: [[TEMP:%.+]] = ptrtoint ptr [[PAYLOAD]] to [[USIZE:i[0-9]+]] // CHECK: ret [[USIZE]] [[TEMP]] match x { Ok(v) => v, Err(_) => std::intrinsics::unreachable(), } } // CHECK-LABEL: @extract_box // CHECK-SAME: (i{{[0-9]+}} {{[^,]+}} [[DISCRIMINANT:%[0-9]+]], ptr {{[^,]+}} [[PAYLOAD:%[0-9]+]]) #[no_mangle] pub unsafe fn extract_box(x: Result<usize, Box<i32>>) -> Box<i32> { // CHECK: ret ptr [[PAYLOAD]] match x { Ok(_) => std::intrinsics::unreachable(), Err(e) => e, } } </ins>

#![crate_type = "lib"] #![feature(try_blocks)] <del> // These are now NOPs in LLVM 15, presumably thanks to nikic's change mentioned in // <https://github.com/rust-lang/rust/issues/85133#issuecomment-1072168354>. // Unfortunately, as of 2022-08-17 they're not yet nops for `u64`s nor `Option`. </del> use std::ops::ControlFlow::{self, Continue, Break}; <ins> use std::ptr::NonNull; // CHECK-LABEL: @option_nop_match_32 #[no_mangle] pub fn option_nop_match_32(x: Option<u32>) -> Option<u32> { // CHECK: start: // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: ret { i32, i32 } match x { Some(x) => Some(x), None => None, } } // CHECK-LABEL: @option_nop_traits_32 #[no_mangle] pub fn option_nop_traits_32(x: Option<u32>) -> Option<u32> { // CHECK: start: // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: ret { i32, i32 } try { x? } } </ins> // CHECK-LABEL: @result_nop_match_32 #[no_mangle] pub fn result_nop_match_32(x: Result<i32, u32>) -> Result<i32, u32> { <del> // CHECK: start // CHECK-NEXT: ret i64 %0 </del> <ins> // CHECK: start: // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: ret { i32, i32 } </ins> match x { Ok(x) => Ok(x), Err(x) => Err(x),

// CHECK-LABEL: @result_nop_traits_32 #[no_mangle] pub fn result_nop_traits_32(x: Result<i32, u32>) -> Result<i32, u32> { <del> // CHECK: start // CHECK-NEXT: ret i64 %0 </del> <ins> // CHECK: start: // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: ret { i32, i32 } try { x? } } // CHECK-LABEL: @result_nop_match_64 #[no_mangle] pub fn result_nop_match_64(x: Result<i64, u64>) -> Result<i64, u64> { // CHECK: start: // CHECK-NEXT: insertvalue { i64, i64 } // CHECK-NEXT: insertvalue { i64, i64 } // CHECK-NEXT: ret { i64, i64 } match x { Ok(x) => Ok(x), Err(x) => Err(x), } } // CHECK-LABEL: @result_nop_traits_64 #[no_mangle] pub fn result_nop_traits_64(x: Result<i64, u64>) -> Result<i64, u64> { // CHECK: start: // CHECK-NEXT: insertvalue { i64, i64 } // CHECK-NEXT: insertvalue { i64, i64 } // CHECK-NEXT: ret { i64, i64 } try { x? } } // CHECK-LABEL: @result_nop_match_ptr #[no_mangle] pub fn result_nop_match_ptr(x: Result<usize, Box<()>>) -> Result<usize, Box<()>> { // CHECK: start: // CHECK-NEXT: insertvalue { i{{[0-9]+}}, ptr } // CHECK-NEXT: insertvalue { i{{[0-9]+}}, ptr } // CHECK-NEXT: ret match x { Ok(x) => Ok(x), Err(x) => Err(x), } } // CHECK-LABEL: @result_nop_traits_ptr #[no_mangle] pub fn result_nop_traits_ptr(x: Result<u64, NonNull<()>>) -> Result<u64, NonNull<()>> { // CHECK: start: // CHECK-NEXT: insertvalue { i{{[0-9]+}}, ptr } // CHECK-NEXT: insertvalue { i{{[0-9]+}}, ptr } // CHECK-NEXT: ret </ins> try { x? }

// CHECK-LABEL: @control_flow_nop_match_32 #[no_mangle] pub fn control_flow_nop_match_32(x: ControlFlow<i32, u32>) -> ControlFlow<i32, u32> { <del> // CHECK: start // CHECK-NEXT: ret i64 %0 </del> <ins> // CHECK: start: // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: ret { i32, i32 } </ins> match x { Continue(x) => Continue(x), Break(x) => Break(x),

// CHECK-LABEL: @control_flow_nop_traits_32 #[no_mangle] pub fn control_flow_nop_traits_32(x: ControlFlow<i32, u32>) -> ControlFlow<i32, u32> { <del> // CHECK: start // CHECK-NEXT: ret i64 %0 </del> <ins> // CHECK: start: // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: insertvalue { i32, i32 } // CHECK-NEXT: ret { i32, i32 } </ins> try { x? }

<ins> //@ normalize-stderr-test "pref: Align([1-8] bytes)" -> "pref: $$PREF_ALIGN" //@ normalize-stderr-test "Int(I[0-9]+," -> "Int(I?," //@ normalize-stderr-test "valid_range: 0..=[0-9]+" -> "valid_range: $$VALID_RANGE" //! Enum layout tests related to scalar pairs with an int/ptr common primitive. #![feature(rustc_attrs)] #![feature(never_type)] #![crate_type = "lib"] #[rustc_layout(abi)] enum ScalarPairPointerWithInt { //~ERROR: abi: ScalarPair A(usize), B(Box<()>), } // Negative test--ensure that pointers are not commoned with integers // of a different size. (Assumes that no target has 8 bit pointers, which // feels pretty safe.) #[rustc_layout(abi)] enum NotScalarPairPointerWithSmallerInt { //~ERROR: abi: Aggregate A(u8), B(Box<()>), } </ins>

<ins> error: abi: ScalarPair(Initialized { value: Int(I?, false), valid_range: $VALID_RANGE }, Initialized { value: Pointer(AddressSpace(0)), valid_range: $VALID_RANGE }) --> $DIR/enum-scalar-pair-int-ptr.rs:12:1 | LL | enum ScalarPairPointerWithInt { | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ error: abi: Aggregate { sized: true } --> $DIR/enum-scalar-pair-int-ptr.rs:21:1 | LL | enum NotScalarPairPointerWithSmallerInt { | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ error: aborting due to 2 previous errors </ins>

A, B([u8; 15], !), // make sure there is space being reserved for this field. } <ins> #[rustc_layout(abi)] enum ScalarPairDifferingSign { //~ERROR: abi: ScalarPair A(u8), B(i8), } </ins>

LL | enum UninhabitedVariantSpace { | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ <del> error: aborting due to 2 previous errors </del> <ins> error: abi: ScalarPair(Initialized { value: Int(I8, false), valid_range: 0..=1 }, Initialized { value: Int(I8, false), valid_range: 0..=255 }) --> $DIR/enum.rs:21:1 | LL | enum ScalarPairDifferingSign { | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ error: aborting due to 3 previous errors </ins>