Repository: bytecodealliance/wasmtime
Pull request #5546 (2023-01-07 12:23:33)

cranelift: Optimize `select+icmp` into `{s,u}{min,max}`
👋 Hey,

This PR adds some egraph rules to optimize a `select + icmp` into the appropriate version of the `min`/`max` instructions.

We transform the following code:
```
function %select_sgt_to_smax(i32, i32) -> i32 {
block0(v0: i32, v1: i32):
    v2 = icmp sgt v0, v1
    v3 = select v2, v0, v1
    return v3
}
```

Into:

```
function %select_sgt_to_smax(i32, i32) -> i32 {
block0(v0: i32, v1: i32):
    v2 = smax v0, v1
    return v2
}
```

This is beneficial at least for the RISC-V backend, where we can emit the `max` instruction in these cases instead of branching.

I've been running this in fuzzgen for a while and it hasn't complained so far!

afonso360: > Perhaps we should take advantage of the egraph ability to represent equivalencies with a (simplify (icmp ty cc x y) (icmp ty (intcc_reverse cc) y x) rule.

Yeah, that sounds like a good idea, I'll try that!

> The same reasoning should work on vectors; I think copying each of these rules for vselect of icmp should work.

👍, I'll add those in this PR if that's OK with you. 

> Is there an equivalent floating-point rewrite? NaNs make my head hurt.

I'm not sure, 


> I'd love to see Sightglass benchmark figures for this PR. I think you can do /bench_x64 but I can't; it looks like my username isn't in the list in performance.yml.

I don't expect any benefits for x86 since I think right now we lower `min`/`max` as `cmp`+`cmov*` which is the same as the unoptimized form, but we can try it. Although we might see some benefits on the `vselect` version (maybe?).

/bench_x64
jlb6740: 
instantiation :: cycles :: benchmarks/bz2/benchmark.wasm

  No difference in performance.

  [160490 179150.32 293708] commit.so
  [155924 172967.04 189250] main.so

compilation :: cycles :: benchmarks/bz2/benchmark.wasm

  No difference in performance.

  [202562282 217920659.92 241944548] commit.so
  [199991838 212302882.08 235356036] main.so

instantiation :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  No difference in performance.

  [186500 203277.04 247336] commit.so
  [186216 207677.60 249312] main.so

compilation :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  No difference in performance.

  [293540152 337751307.12 374158094] commit.so
  [295516870 341031354.56 393946812] main.so

compilation :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  No difference in performance.

  [6965675788 7114549484.48 7213034856] commit.so
  [6955473648 7075012926.00 7165106004] main.so

instantiation :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  No difference in performance.

  [488512 504472.32 573802] commit.so
  [488122 501985.20 525900] main.so

execution :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  No difference in performance.

  [8327116 8476234.16 8683322] commit.so
  [8339612 8512672.24 8750996] main.so

execution :: cycles :: benchmarks/bz2/benchmark.wasm

  No difference in performance.

  [111521870 115012543.92 117344984] commit.so
  [111540296 114710730.08 122547780] main.so

execution :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  No difference in performance.

  [1024608234 1038058547.20 1067188634] commit.so
  [1027357456 1037305556.48 1048046760] main.so
afonso360: I tried adding the icmp reverse rule, and while it works and is correct it seems to be doing something weird. We now get a bunch of aliases after optimizing a normal `icmp eq`. I've left it as a separate commit so that its easier to investigate.

I'm worried that we might be unintentionally doing a bunch of stuff and worsening compile performance.

This is what we get now:
```
test optimize
set use_egraphs=true
target x86_64

function %icmp_eq(i32, i32) -> i8 {
block0(v0: i32, v1: i32):
    v2 = icmp eq v0, v1
    return v2
}

; check: block0(v0: i32, v1: i32):
; check:    v8 = icmp eq v0, v1
; check:    v9 -> v8
; check:    v10 -> v9
; check:    v11 -> v10
; check:    v12 -> v11
; check:    v13 -> v12
; check:    v14 -> v13
; check:    return v8 
```

---

> Is there an equivalent floating-point rewrite? NaNs make my head hurt.

So, I don't think there is an exact match for the full `fcmp` space, but the `f{min,max}_pseudo` instructions are an defined in terms of `fcmp {lt,gt} + select`, so we can at least get those.

---

I ran the Sightglass SIMD benchmarks locally and they measured no difference, but I'm not too confident in my benchmarking setup since its a linux vm inside a windows host, which doesn't seem ideal...

<details>
<summary>Sightglass SIMD benchmark results</summary>

```
     Running `target/debug/sightglass-cli benchmark --engine ./libwasmtime_main.so --engine ./libwasmtime_select.so -- benchmarks/blake3-simd/benchmark.wasm benchmarks/intgemm-simd/benchmark.wasm
`

compilation :: cycles :: benchmarks/blake3-simd/benchmark.wasm

  No difference in performance.

  [73345320 95091825.36 287267718] main.so
  [70722628 101732805.98 299985933] select.so

instantiation :: cycles :: benchmarks/intgemm-simd/benchmark.wasm

  No difference in performance.

  [168256 216817.60 382976] main.so
  [177600 220696.00 344096] select.so

instantiation :: cycles :: benchmarks/blake3-simd/benchmark.wasm

  No difference in performance.

  [133184 167869.12 416352] main.so
  [132128 165781.12 341184] select.so

compilation :: cycles :: benchmarks/intgemm-simd/benchmark.wasm

  No difference in performance.

  [709243689 815567491.90 1073223272] main.so
  [729598384 821242801.38 1027388168] select.so

execution :: cycles :: benchmarks/intgemm-simd/benchmark.wasm

  No difference in performance.

  [3855579532 3914533632.01 4071619781] main.so
  [3855359377 3917102361.59 4042283617] select.so

execution :: cycles :: benchmarks/blake3-simd/benchmark.wasm

  No difference in performance.

  [439936 513683.52 823392] main.so
  [439488 513457.28 862848] select.so
```
</details>


/bench_x64
jlb6740: 
instantiation :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  No difference in performance.

  [195254 206675.84 239848] commit.so
  [185986 200724.96 227526] main.so

instantiation :: cycles :: benchmarks/bz2/benchmark.wasm

  No difference in performance.

  [158588 177232.96 200728] commit.so
  [153838 181267.68 359084] main.so

compilation :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  No difference in performance.

  [310637436 342353446.56 390973658] commit.so
  [295484178 336498505.76 366404166] main.so

compilation :: cycles :: benchmarks/bz2/benchmark.wasm

  No difference in performance.

  [205285250 220044999.92 262015540] commit.so
  [202251150 217512094.00 237476954] main.so

execution :: cycles :: benchmarks/bz2/benchmark.wasm

  No difference in performance.

  [111173460 114115222.96 119495454] commit.so
  [111448476 114811907.92 115933528] main.so

compilation :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  No difference in performance.

  [6940653952 7096700981.20 7212066130] commit.so
  [6942954936 7124108036.08 7229159218] main.so

instantiation :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  No difference in performance.

  [478890 506487.60 662056] commit.so
  [485120 504785.28 608956] main.so

execution :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  No difference in performance.

  [1023014568 1034258955.84 1055892690] commit.so
  [1019706902 1030842327.04 1044519376] main.so

execution :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  No difference in performance.

  [8352392 8496849.04 8691182] commit.so
  [8360734 8482021.60 8626984] main.so
cfallin: It makes me quite happy to see the usage of the egraph framework to add a rewrite be so straightforward!

Regarding the benchmarking results: perhaps they are showing no effect because egraph-based optimization is still off by default? (I'm cooking up a PR to flip that, but it's not done yet.)
afonso360: > It makes me quite happy to see the usage of the egraph framework to add a rewrite be so straightforward!

It was also really intuitive to get to this solution which is awesome! 

> Regarding the benchmarking results: perhaps they are showing no effect because egraph-based optimization is still off by default? 

Oh! I completely forgot about that! 😄

I'll try to enable those settings in sightglass in the next run I make. But I'm not expecting great things here, x86 and aarch64 both lower to the same instructions, RISC-V does benefit from it since in can do `max`/`min` as a single instruction.

The `vselect` ones I think might make a difference? Unless existing toolchains already perform this optimization which is somewhat likely.
EdorianDark: Egraph based optimizations are now on by default, so I think an rebase should be enough to test if something changes.
bjorn3: Egraph optimizations are on by default if optimizations are on. Do benchmarks run with optimizations enabled?
afonso360: > Do benchmarks run with optimizations enabled?

I think optimizations are enabled by default, at least on the wasmtime cli that is the case. Although I'm not entirely sure about sightglass.

Rebased this on main and ran sightglass on AArch64, doesen't seem to have much of an impact. I wonder if the results would be any different for something like `cg_clif` that doesen't run on already optimized code.
Unfortunately I can't run this on RISCV which is where I would hope to see some positive effect.


<details>
<summary>Scalar Benchmarks AArch64</summary>

```
ubuntu@instance-20220805-0848:~/git/sightglass$ target/debug/sightglass-cli benchmark --engine /tmp/wasmtime_main.so --engine /tmp/wasmtime_select.so

compilation :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  Δ = 3282352.64 ± 1066807.74 (confidence = 99%)

  main.so is 1.02x to 1.04x faster than select.so!

  [111047763 118113547.11 123328519] main.so
  [115034752 121395899.75 128445997] select.so

execution :: cycles :: benchmarks/bz2/benchmark.wasm

  Δ = 6938.23 ± 1941.66 (confidence = 99%)

  select.so is 1.01x to 1.01x faster than main.so!

  [876198 884956.45 902123] main.so
  [869354 878018.22 901740] select.so

instantiation :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  No difference in performance.

  [2849 3260.40 12227] main.so
  [2804 3148.16 4268] select.so

compilation :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  No difference in performance.

  [4381352 5597680.00 6817168] main.so
  [4465781 5715523.09 7002568] select.so

instantiation :: cycles :: benchmarks/bz2/benchmark.wasm

  No difference in performance.

  [2361 2621.85 4044] main.so
  [2284 2575.55 3465] select.so

compilation :: cycles :: benchmarks/bz2/benchmark.wasm

  No difference in performance.

  [2440354 2586186.45 3163455] main.so
  [2460769 2621895.24 3155162] select.so

execution :: cycles :: benchmarks/pulldown-cmark/benchmark.wasm

  No difference in performance.

  [69303 75400.87 94495] main.so
  [70033 76346.64 97401] select.so

instantiation :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  No difference in performance.

  [5533 6360.26 18487] main.so
  [5558 6418.39 19011] select.so

execution :: cycles :: benchmarks/spidermonkey/benchmark.wasm

  No difference in performance.

  [8367754 8415382.47 8499654] main.so
  [8377518 8420745.48 8547947] select.so
```
</details>


<details>
<summary>SIMD Benchmarks AArch64</summary>

```
ubuntu@instance-20220805-0848:~/git/sightglass$ target/debug/sightglass-cli benchmark --engine /tmp/wasmtime_main.so --engine /tmp/wasmtime_select.so -- benchmarks/blake3-simd/benchmark.wasm benchmarks/intgemm-simd/benchmark.wasm

execution :: cycles :: benchmarks/intgemm-simd/benchmark.wasm

  Δ = 203985.92 ± 8663.93 (confidence = 99%)

  select.so is 1.00x to 1.00x faster than main.so!

  [45561021 45611295.11 45720621] main.so
  [45363259 45407309.19 45486165] select.so

execution :: cycles :: benchmarks/blake3-simd/benchmark.wasm

  No difference in performance.

  [3472 4728.82 13547] main.so
  [3477 4261.65 15734] select.so

instantiation :: cycles :: benchmarks/blake3-simd/benchmark.wasm

  No difference in performance.

  [1749 2124.65 3341] main.so
  [1724 2101.14 3474] select.so

compilation :: cycles :: benchmarks/intgemm-simd/benchmark.wasm

  No difference in performance.

  [5448821 5745023.23 6331575] main.so
  [5487095 5779248.67 6320724] select.so

compilation :: cycles :: benchmarks/blake3-simd/benchmark.wasm

  No difference in performance.

  [526706 615112.06 1567920] main.so
  [536207 613883.57 1149844] select.so

instantiation :: cycles :: benchmarks/intgemm-simd/benchmark.wasm

  No difference in performance.

  [2246 2495.36 3522] main.so
  [2126 2497.00 4032] select.so
```
</details>
uweigand:  



> Looks like s390x doesn't have lowerings for `umin`/`umax` on scalars, so when this PR rewrites this pattern to those instructions the s390x tests fail. cc: @uweigand

Well, we don't have any instructions for `umin`/`umax` on scalars, so the only possible implementation would be to turn it right back into (the equivalent of) `select` + `icmp` ...   Is this what we're supposed to be doing here?

uweigand: I've now opened https://github.com/bytecodealliance/wasmtime/pull/5762 to add support for these clif instructions on s390x.

Patch:
diff --git a/cranelift/codegen/src/isa/s390x/inst/emit.rs b/cranelift/codegen/src/isa/s390x/inst/emit.rs
index 23ea1e736..5a811e7a3 100644
--- a/cranelift/codegen/src/isa/s390x/inst/emit.rs
+++ b/cranelift/codegen/src/isa/s390x/inst/emit.rs
@@ -1568,7 +1568,7 @@ impl Inst {
 
                 let (opcode_rx, opcode_rxy) = match alu_op {
                     ALUOp::Add32 => (Some(0x5a), Some(0xe35a)),        // A(Y)
-                    ALUOp::Add32Ext16 => (Some(0x4a), Some(0xe34a)),   // AH(Y)
+                    ALUOp::Add32Ext16 => (Some(0x4a), Some(0xe37a)),   // AH(Y)
                     ALUOp::Add64 => (None, Some(0xe308)),              // AG
                     ALUOp::Add64Ext16 => (None, Some(0xe338)),         // AGH
                     ALUOp::Add64Ext32 => (None, Some(0xe318)),         // AGF
diff --git a/cranelift/codegen/src/isa/s390x/inst/mod.rs b/cranelift/codegen/src/isa/s390x/inst/mod.rs
index 3804a980c..d68e3317c 100644
--- a/cranelift/codegen/src/isa/s390x/inst/mod.rs
+++ b/cranelift/codegen/src/isa/s390x/inst/mod.rs
@@ -2359,11 +2359,16 @@ impl Inst {
                 let (rn, rn_fpr) = pretty_print_fpr(rn, allocs);
                 if opcode_fpr.is_some() && rd_fpr.is_some() && rn_fpr.is_some() {
                     format!(
-                        "{} {}, {}, {}",
+                        "{} {}, {}, {}{}",
                         opcode_fpr.unwrap(),
                         rd_fpr.unwrap(),
+                        mode,
                         rn_fpr.unwrap(),
-                        mode
+                        if opcode_fpr.unwrap().ends_with('a') {
+                            ", 0"
+                        } else {
+                            ""
+                        }
                     )
                 } else if opcode.starts_with('w') {
                     format!(
diff --git a/cranelift/codegen/src/isa/x64/inst.isle b/cranelift/codegen/src/isa/x64/inst.isle
index 213307cc4..cfc2afc61 100644
--- a/cranelift/codegen/src/isa/x64/inst.isle
+++ b/cranelift/codegen/src/isa/x64/inst.isle
@@ -139,7 +139,7 @@
        (MovImmM (size OperandSize)
 		(simm64 u64)
 		(dst SyntheticAmode))
-       
+
        ;; Integer stores: mov (b w l q) reg addr.
        (MovRM (size OperandSize) ;; 1, 2, 4, or 8
               (src Gpr)
@@ -1557,7 +1557,7 @@
 
 ;; Performs an xor operation of the two operands specified.
 (decl sse_xor (Type Xmm XmmMem) Xmm)
-(rule (sse_xor ty x y) (xmm_rm_r ty (sse_xor_op ty) x y))
+(rule (sse_xor ty x y) (xmm_rm_r (sse_xor_op ty) x y))
 
 ;; Generates a register value which has an all-ones pattern.
 ;;
@@ -2163,8 +2163,8 @@
          dst)))
 
 ;; Helper for creating `MInst.XmmRmR` instructions.
-(decl xmm_rm_r (Type SseOpcode Xmm XmmMem) Xmm)
-(rule (xmm_rm_r ty op src1 src2)
+(decl xmm_rm_r (SseOpcode Xmm XmmMem) Xmm)
+(rule (xmm_rm_r op src1 src2)
       (let ((dst WritableXmm (temp_writable_xmm))
             (_ Unit (emit (MInst.XmmRmR op src1 src2 dst))))
         dst))
@@ -2172,282 +2172,282 @@
 ;; Helper for creating `paddb` instructions.
 (decl x64_paddb (Xmm XmmMem) Xmm)
 (rule (x64_paddb src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Paddb) src1 src2))
+      (xmm_rm_r (SseOpcode.Paddb) src1 src2))
 
 ;; Helper for creating `paddw` instructions.
 (decl x64_paddw (Xmm XmmMem) Xmm)
 (rule (x64_paddw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Paddw) src1 src2))
+      (xmm_rm_r (SseOpcode.Paddw) src1 src2))
 
 ;; Helper for creating `paddd` instructions.
 (decl x64_paddd (Xmm XmmMem) Xmm)
 (rule (x64_paddd src1 src2)
-      (xmm_rm_r $I32X4 (SseOpcode.Paddd) src1 src2))
+      (xmm_rm_r (SseOpcode.Paddd) src1 src2))
 
 ;; Helper for creating `paddq` instructions.
 (decl x64_paddq (Xmm XmmMem) Xmm)
 (rule (x64_paddq src1 src2)
-      (xmm_rm_r $I64X2 (SseOpcode.Paddq) src1 src2))
+      (xmm_rm_r (SseOpcode.Paddq) src1 src2))
 
 ;; Helper for creating `paddsb` instructions.
 (decl x64_paddsb (Xmm XmmMem) Xmm)
 (rule (x64_paddsb src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Paddsb) src1 src2))
+      (xmm_rm_r (SseOpcode.Paddsb) src1 src2))
 
 ;; Helper for creating `paddsw` instructions.
 (decl x64_paddsw (Xmm XmmMem) Xmm)
 (rule (x64_paddsw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Paddsw) src1 src2))
+      (xmm_rm_r (SseOpcode.Paddsw) src1 src2))
 
 ;; Helper for creating `paddusb` instructions.
 (decl x64_paddusb (Xmm XmmMem) Xmm)
 (rule (x64_paddusb src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Paddusb) src1 src2))
+      (xmm_rm_r (SseOpcode.Paddusb) src1 src2))
 
 ;; Helper for creating `paddusw` instructions.
 (decl x64_paddusw (Xmm XmmMem) Xmm)
 (rule (x64_paddusw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Paddusw) src1 src2))
+      (xmm_rm_r (SseOpcode.Paddusw) src1 src2))
 
 ;; Helper for creating `psubb` instructions.
 (decl x64_psubb (Xmm XmmMem) Xmm)
 (rule (x64_psubb src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Psubb) src1 src2))
+      (xmm_rm_r (SseOpcode.Psubb) src1 src2))
 
 ;; Helper for creating `psubw` instructions.
 (decl x64_psubw (Xmm XmmMem) Xmm)
 (rule (x64_psubw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Psubw) src1 src2))
+      (xmm_rm_r (SseOpcode.Psubw) src1 src2))
 
 ;; Helper for creating `psubd` instructions.
 (decl x64_psubd (Xmm XmmMem) Xmm)
 (rule (x64_psubd src1 src2)
-      (xmm_rm_r $I32X4 (SseOpcode.Psubd) src1 src2))
+      (xmm_rm_r (SseOpcode.Psubd) src1 src2))
 
 ;; Helper for creating `psubq` instructions.
 (decl x64_psubq (Xmm XmmMem) Xmm)
 (rule (x64_psubq src1 src2)
-      (xmm_rm_r $I64X2 (SseOpcode.Psubq) src1 src2))
+      (xmm_rm_r (SseOpcode.Psubq) src1 src2))
 
 ;; Helper for creating `psubsb` instructions.
 (decl x64_psubsb (Xmm XmmMem) Xmm)
 (rule (x64_psubsb src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Psubsb) src1 src2))
+      (xmm_rm_r (SseOpcode.Psubsb) src1 src2))
 
 ;; Helper for creating `psubsw` instructions.
 (decl x64_psubsw (Xmm XmmMem) Xmm)
 (rule (x64_psubsw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Psubsw) src1 src2))
+      (xmm_rm_r (SseOpcode.Psubsw) src1 src2))
 
 ;; Helper for creating `psubusb` instructions.
 (decl x64_psubusb (Xmm XmmMem) Xmm)
 (rule (x64_psubusb src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Psubusb) src1 src2))
+      (xmm_rm_r (SseOpcode.Psubusb) src1 src2))
 
 ;; Helper for creating `psubusw` instructions.
 (decl x64_psubusw (Xmm XmmMem) Xmm)
 (rule (x64_psubusw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Psubusw) src1 src2))
+      (xmm_rm_r (SseOpcode.Psubusw) src1 src2))
 
 ;; Helper for creating `pavgb` instructions.
 (decl x64_pavgb (Xmm XmmMem) Xmm)
 (rule (x64_pavgb src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Pavgb) src1 src2))
+      (xmm_rm_r (SseOpcode.Pavgb) src1 src2))
 
 ;; Helper for creating `pavgw` instructions.
 (decl x64_pavgw (Xmm XmmMem) Xmm)
 (rule (x64_pavgw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Pavgw) src1 src2))
+      (xmm_rm_r (SseOpcode.Pavgw) src1 src2))
 
 ;; Helper for creating `pand` instructions.
 (decl x64_pand (Xmm XmmMem) Xmm)
 (rule (x64_pand src1 src2)
-      (xmm_rm_r $F32X4 (SseOpcode.Pand) src1 src2))
+      (xmm_rm_r (SseOpcode.Pand) src1 src2))
 
 ;; Helper for creating `andps` instructions.
 (decl x64_andps (Xmm XmmMem) Xmm)
 (rule (x64_andps src1 src2)
-      (xmm_rm_r $F32X4 (SseOpcode.Andps) src1 src2))
+      (xmm_rm_r (SseOpcode.Andps) src1 src2))
 
 ;; Helper for creating `andpd` instructions.
 (decl x64_andpd (Xmm XmmMem) Xmm)
 (rule (x64_andpd src1 src2)
-      (xmm_rm_r $F64X2 (SseOpcode.Andpd) src1 src2))
+      (xmm_rm_r (SseOpcode.Andpd) src1 src2))
 
 ;; Helper for creating `por` instructions.
 (decl x64_por (Xmm XmmMem) Xmm)
 (rule (x64_por src1 src2)
-      (xmm_rm_r $F32X4 (SseOpcode.Por) src1 src2))
+      (xmm_rm_r (SseOpcode.Por) src1 src2))
 
 ;; Helper for creating `orps` instructions.
 (decl x64_orps (Xmm XmmMem) Xmm)
 (rule (x64_orps src1 src2)
-      (xmm_rm_r $F32X4 (SseOpcode.Orps) src1 src2))
+      (xmm_rm_r (SseOpcode.Orps) src1 src2))
 
 ;; Helper for creating `orpd` instructions.
 (decl x64_orpd (Xmm XmmMem) Xmm)
 (rule (x64_orpd src1 src2)
-      (xmm_rm_r $F64X2 (SseOpcode.Orpd) src1 src2))
+      (xmm_rm_r (SseOpcode.Orpd) src1 src2))
 
 ;; Helper for creating `pxor` instructions.
 (decl x64_pxor (Xmm XmmMem) Xmm)
 (rule (x64_pxor src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Pxor) src1 src2))
+      (xmm_rm_r (SseOpcode.Pxor) src1 src2))
 
 ;; Helper for creating `xorps` instructions.
 (decl x64_xorps (Xmm XmmMem) Xmm)
 (rule (x64_xorps src1 src2)
-      (xmm_rm_r $F32X4 (SseOpcode.Xorps) src1 src2))
+      (xmm_rm_r (SseOpcode.Xorps) src1 src2))
 
 ;; Helper for creating `xorpd` instructions.
 (decl x64_xorpd (Xmm XmmMem) Xmm)
 (rule (x64_xorpd src1 src2)
-      (xmm_rm_r $F64X2 (SseOpcode.Xorpd) src1 src2))
+      (xmm_rm_r (SseOpcode.Xorpd) src1 src2))
 
 ;; Helper for creating `pmullw` instructions.
 (decl x64_pmullw (Xmm XmmMem) Xmm)
 (rule (x64_pmullw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Pmullw) src1 src2))
+      (xmm_rm_r (SseOpcode.Pmullw) src1 src2))
 
 ;; Helper for creating `pmulld` instructions.
 (decl x64_pmulld (Xmm XmmMem) Xmm)
 (rule (x64_pmulld src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Pmulld) src1 src2))
+      (xmm_rm_r (SseOpcode.Pmulld) src1 src2))
 
 ;; Helper for creating `pmulhw` instructions.
 (decl x64_pmulhw (Xmm XmmMem) Xmm)
 (rule (x64_pmulhw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Pmulhw) src1 src2))
+      (xmm_rm_r (SseOpcode.Pmulhw) src1 src2))
 
 ;; Helper for creating `pmulhrsw` instructions.
 (decl x64_pmulhrsw (Xmm XmmMem) Xmm)
 (rule (x64_pmulhrsw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Pmulhrsw) src1 src2))
+      (xmm_rm_r (SseOpcode.Pmulhrsw) src1 src2))
 
 ;; Helper for creating `pmulhuw` instructions.
 (decl x64_pmulhuw (Xmm XmmMem) Xmm)
 (rule (x64_pmulhuw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Pmulhuw) src1 src2))
+      (xmm_rm_r (SseOpcode.Pmulhuw) src1 src2))
 
 ;; Helper for creating `pmuldq` instructions.
 (decl x64_pmuldq (Xmm XmmMem) Xmm)
 (rule (x64_pmuldq src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Pmuldq) src1 src2))
+      (xmm_rm_r (SseOpcode.Pmuldq) src1 src2))
 
 ;; Helper for creating `pmuludq` instructions.
 (decl x64_pmuludq (Xmm XmmMem) Xmm)
 (rule (x64_pmuludq src1 src2)
-      (xmm_rm_r $I64X2 (SseOpcode.Pmuludq) src1 src2))
+      (xmm_rm_r (SseOpcode.Pmuludq) src1 src2))
 
 ;; Helper for creating `punpckhwd` instructions.
 (decl x64_punpckhwd (Xmm XmmMem) Xmm)
 (rule (x64_punpckhwd src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Punpckhwd) src1 src2))
+      (xmm_rm_r (SseOpcode.Punpckhwd) src1 src2))
 
 ;; Helper for creating `punpcklwd` instructions.
 (decl x64_punpcklwd (Xmm XmmMem) Xmm)
 (rule (x64_punpcklwd src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Punpcklwd) src1 src2))
+      (xmm_rm_r (SseOpcode.Punpcklwd) src1 src2))
 
 ;; Helper for creating `unpcklps` instructions.
 (decl x64_unpcklps (Xmm XmmMem) Xmm)
 (rule (x64_unpcklps src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Unpcklps) src1 src2))
+      (xmm_rm_r (SseOpcode.Unpcklps) src1 src2))
 
 ;; Helper for creating `andnps` instructions.
 (decl x64_andnps (Xmm XmmMem) Xmm)
 (rule (x64_andnps src1 src2)
-      (xmm_rm_r $F32X4 (SseOpcode.Andnps) src1 src2))
+      (xmm_rm_r (SseOpcode.Andnps) src1 src2))
 
 ;; Helper for creating `andnpd` instructions.
 (decl x64_andnpd (Xmm XmmMem) Xmm)
 (rule (x64_andnpd src1 src2)
-      (xmm_rm_r $F64X2 (SseOpcode.Andnpd) src1 src2))
+      (xmm_rm_r (SseOpcode.Andnpd) src1 src2))
 
 ;; Helper for creating `pandn` instructions.
 (decl x64_pandn (Xmm XmmMem) Xmm)
 (rule (x64_pandn src1 src2)
-      (xmm_rm_r $F64X2 (SseOpcode.Pandn) src1 src2))
+      (xmm_rm_r (SseOpcode.Pandn) src1 src2))
 
 ;; Helper for creating `addss` instructions.
 (decl x64_addss (Xmm XmmMem) Xmm)
 (rule (x64_addss src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Addss) src1 src2))
+      (xmm_rm_r (SseOpcode.Addss) src1 src2))
 
 ;; Helper for creating `addsd` instructions.
 (decl x64_addsd (Xmm XmmMem) Xmm)
 (rule (x64_addsd src1 src2)
-      (xmm_rm_r $F64 (SseOpcode.Addsd) src1 src2))
+      (xmm_rm_r (SseOpcode.Addsd) src1 src2))
 
 ;; Helper for creating `addps` instructions.
 (decl x64_addps (Xmm XmmMem) Xmm)
 (rule (x64_addps src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Addps) src1 src2))
+      (xmm_rm_r (SseOpcode.Addps) src1 src2))
 
 ;; Helper for creating `addpd` instructions.
 (decl x64_addpd (Xmm XmmMem) Xmm)
 (rule (x64_addpd src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Addpd) src1 src2))
+      (xmm_rm_r (SseOpcode.Addpd) src1 src2))
 
 ;; Helper for creating `subss` instructions.
 (decl x64_subss (Xmm XmmMem) Xmm)
 (rule (x64_subss src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Subss) src1 src2))
+      (xmm_rm_r (SseOpcode.Subss) src1 src2))
 
 ;; Helper for creating `subsd` instructions.
 (decl x64_subsd (Xmm XmmMem) Xmm)
 (rule (x64_subsd src1 src2)
-      (xmm_rm_r $F64 (SseOpcode.Subsd) src1 src2))
+      (xmm_rm_r (SseOpcode.Subsd) src1 src2))
 
 ;; Helper for creating `subps` instructions.
 (decl x64_subps (Xmm XmmMem) Xmm)
 (rule (x64_subps src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Subps) src1 src2))
+      (xmm_rm_r (SseOpcode.Subps) src1 src2))
 
 ;; Helper for creating `subpd` instructions.
 (decl x64_subpd (Xmm XmmMem) Xmm)
 (rule (x64_subpd src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Subpd) src1 src2))
+      (xmm_rm_r (SseOpcode.Subpd) src1 src2))
 
 ;; Helper for creating `mulss` instructions.
 (decl x64_mulss (Xmm XmmMem) Xmm)
 (rule (x64_mulss src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Mulss) src1 src2))
+      (xmm_rm_r (SseOpcode.Mulss) src1 src2))
 
 ;; Helper for creating `mulsd` instructions.
 (decl x64_mulsd (Xmm XmmMem) Xmm)
 (rule (x64_mulsd src1 src2)
-      (xmm_rm_r $F64 (SseOpcode.Mulsd) src1 src2))
+      (xmm_rm_r (SseOpcode.Mulsd) src1 src2))
 
 ;; Helper for creating `mulps` instructions.
 (decl x64_mulps (Xmm XmmMem) Xmm)
 (rule (x64_mulps src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Mulps) src1 src2))
+      (xmm_rm_r (SseOpcode.Mulps) src1 src2))
 
 ;; Helper for creating `mulpd` instructions.
 (decl x64_mulpd (Xmm XmmMem) Xmm)
 (rule (x64_mulpd src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Mulpd) src1 src2))
+      (xmm_rm_r (SseOpcode.Mulpd) src1 src2))
 
 ;; Helper for creating `divss` instructions.
 (decl x64_divss (Xmm XmmMem) Xmm)
 (rule (x64_divss src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Divss) src1 src2))
+      (xmm_rm_r (SseOpcode.Divss) src1 src2))
 
 ;; Helper for creating `divsd` instructions.
 (decl x64_divsd (Xmm XmmMem) Xmm)
 (rule (x64_divsd src1 src2)
-      (xmm_rm_r $F64 (SseOpcode.Divsd) src1 src2))
+      (xmm_rm_r (SseOpcode.Divsd) src1 src2))
 
 ;; Helper for creating `divps` instructions.
 (decl x64_divps (Xmm XmmMem) Xmm)
 (rule (x64_divps src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Divps) src1 src2))
+      (xmm_rm_r (SseOpcode.Divps) src1 src2))
 
 ;; Helper for creating `divpd` instructions.
 (decl x64_divpd (Xmm XmmMem) Xmm)
 (rule (x64_divpd src1 src2)
-      (xmm_rm_r $F32 (SseOpcode.Divpd) src1 src2))
+      (xmm_rm_r (SseOpcode.Divpd) src1 src2))
 
 (decl sse_blend_op (Type) SseOpcode)
 (rule 1 (sse_blend_op $F32X4) (SseOpcode.Blendvps))
@@ -2482,12 +2482,12 @@
 ;; Helper for creating `movsd` instructions.
 (decl x64_movsd_regmove (Xmm XmmMem) Xmm)
 (rule (x64_movsd_regmove src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Movsd) src1 src2))
+      (xmm_rm_r (SseOpcode.Movsd) src1 src2))
 
 ;; Helper for creating `movlhps` instructions.
 (decl x64_movlhps (Xmm XmmMem) Xmm)
 (rule (x64_movlhps src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Movlhps) src1 src2))
+      (xmm_rm_r (SseOpcode.Movlhps) src1 src2))
 
 ;; Helpers for creating `pmaxs*` instructions.
 (decl x64_pmaxs (Type Xmm XmmMem) Xmm)
@@ -2496,11 +2496,11 @@
 (rule (x64_pmaxs $I32X4 x y) (x64_pmaxsd x y))
 ;; No $I64X2 version (PMAXSQ) in SSE4.1.
 (decl x64_pmaxsb (Xmm XmmMem) Xmm)
-(rule (x64_pmaxsb src1 src2) (xmm_rm_r $I8X16 (SseOpcode.Pmaxsb) src1 src2))
+(rule (x64_pmaxsb src1 src2) (xmm_rm_r (SseOpcode.Pmaxsb) src1 src2))
 (decl x64_pmaxsw (Xmm XmmMem) Xmm)
-(rule (x64_pmaxsw src1 src2) (xmm_rm_r $I8X16 (SseOpcode.Pmaxsw) src1 src2))
+(rule (x64_pmaxsw src1 src2) (xmm_rm_r (SseOpcode.Pmaxsw) src1 src2))
 (decl x64_pmaxsd (Xmm XmmMem) Xmm)
-(rule (x64_pmaxsd src1 src2) (xmm_rm_r $I8X16 (SseOpcode.Pmaxsd) src1 src2))
+(rule (x64_pmaxsd src1 src2) (xmm_rm_r (SseOpcode.Pmaxsd) src1 src2))
 
 ;; Helpers for creating `pmins*` instructions.
 (decl x64_pmins (Type Xmm XmmMem) Xmm)
@@ -2509,11 +2509,11 @@
 (rule (x64_pmins $I32X4 x y) (x64_pminsd x y))
 ;; No $I64X2 version (PMINSQ) in SSE4.1.
 (decl x64_pminsb (Xmm XmmMem) Xmm)
-(rule (x64_pminsb src1 src2) (xmm_rm_r $I8X16 (SseOpcode.Pminsb) src1 src2))
+(rule (x64_pminsb src1 src2) (xmm_rm_r (SseOpcode.Pminsb) src1 src2))
 (decl x64_pminsw (Xmm XmmMem) Xmm)
-(rule (x64_pminsw src1 src2) (xmm_rm_r $I16X8 (SseOpcode.Pminsw) src1 src2))
+(rule (x64_pminsw src1 src2) (xmm_rm_r (SseOpcode.Pminsw) src1 src2))
 (decl x64_pminsd (Xmm XmmMem) Xmm)
-(rule (x64_pminsd src1 src2) (xmm_rm_r $I32X4 (SseOpcode.Pminsd) src1 src2))
+(rule (x64_pminsd src1 src2) (xmm_rm_r (SseOpcode.Pminsd) src1 src2))
 
 ;; Helpers for creating `pmaxu*` instructions.
 (decl x64_pmaxu (Type Xmm XmmMem) Xmm)
@@ -2522,11 +2522,11 @@
 (rule (x64_pmaxu $I32X4 x y) (x64_pmaxud x y))
 ;; No $I64X2 version (PMAXUQ) in SSE4.1.
 (decl x64_pmaxub (Xmm XmmMem) Xmm)
-(rule (x64_pmaxub src1 src2) (xmm_rm_r $I8X16 (SseOpcode.Pmaxub) src1 src2))
+(rule (x64_pmaxub src1 src2) (xmm_rm_r (SseOpcode.Pmaxub) src1 src2))
 (decl x64_pmaxuw (Xmm XmmMem) Xmm)
-(rule (x64_pmaxuw src1 src2) (xmm_rm_r $I8X16 (SseOpcode.Pmaxuw) src1 src2))
+(rule (x64_pmaxuw src1 src2) (xmm_rm_r (SseOpcode.Pmaxuw) src1 src2))
 (decl x64_pmaxud (Xmm XmmMem) Xmm)
-(rule (x64_pmaxud src1 src2) (xmm_rm_r $I8X16 (SseOpcode.Pmaxud) src1 src2))
+(rule (x64_pmaxud src1 src2) (xmm_rm_r (SseOpcode.Pmaxud) src1 src2))
 
 ;; Helper for creating `pminu*` instructions.
 (decl x64_pminu (Type Xmm XmmMem) Xmm)
@@ -2535,41 +2535,41 @@
 (rule (x64_pminu $I32X4 x y) (x64_pminud x y))
 ;; No $I64X2 version (PMINUQ) in SSE4.1.
 (decl x64_pminub (Xmm XmmMem) Xmm)
-(rule (x64_pminub src1 src2) (xmm_rm_r $I8X16 (SseOpcode.Pminub) src1 src2))
+(rule (x64_pminub src1 src2) (xmm_rm_r (SseOpcode.Pminub) src1 src2))
 (decl x64_pminuw (Xmm XmmMem) Xmm)
-(rule (x64_pminuw src1 src2) (xmm_rm_r $I8X16 (SseOpcode.Pminuw) src1 src2))
+(rule (x64_pminuw src1 src2) (xmm_rm_r (SseOpcode.Pminuw) src1 src2))
 (decl x64_pminud (Xmm XmmMem) Xmm)
-(rule (x64_pminud src1 src2) (xmm_rm_r $I8X16 (SseOpcode.Pminud) src1 src2))
+(rule (x64_pminud src1 src2) (xmm_rm_r (SseOpcode.Pminud) src1 src2))
 
 ;; Helper for creating `punpcklbw` instructions.
 (decl x64_punpcklbw (Xmm XmmMem) Xmm)
 (rule (x64_punpcklbw src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Punpcklbw) src1 src2))
+      (xmm_rm_r (SseOpcode.Punpcklbw) src1 src2))
 
 ;; Helper for creating `punpckhbw` instructions.
 (decl x64_punpckhbw (Xmm XmmMem) Xmm)
 (rule (x64_punpckhbw src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Punpckhbw) src1 src2))
+      (xmm_rm_r (SseOpcode.Punpckhbw) src1 src2))
 
 ;; Helper for creating `packsswb` instructions.
 (decl x64_packsswb (Xmm XmmMem) Xmm)
 (rule (x64_packsswb src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Packsswb) src1 src2))
+      (xmm_rm_r (SseOpcode.Packsswb) src1 src2))
 
 ;; Helper for creating `packssdw` instructions.
 (decl x64_packssdw (Xmm XmmMem) Xmm)
 (rule (x64_packssdw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Packssdw) src1 src2))
+      (xmm_rm_r (SseOpcode.Packssdw) src1 src2))
 
 ;; Helper for creating `packuswb` instructions.
 (decl x64_packuswb (Xmm XmmMem) Xmm)
 (rule (x64_packuswb src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Packuswb) src1 src2))
+      (xmm_rm_r (SseOpcode.Packuswb) src1 src2))
 
 ;; Helper for creating `packusdw` instructions.
 (decl x64_packusdw (Xmm XmmMem) Xmm)
 (rule (x64_packusdw src1 src2)
-      (xmm_rm_r $I16X8 (SseOpcode.Packusdw) src1 src2))
+      (xmm_rm_r (SseOpcode.Packusdw) src1 src2))
 
 ;; Helper for creating `MInst.XmmRmRImm` instructions.
 (decl xmm_rm_r_imm (SseOpcode Reg RegMem u8 OperandSize) Xmm)
@@ -2682,7 +2682,7 @@
 
 (decl x64_pmaddubsw (Xmm XmmMem) Xmm)
 (rule (x64_pmaddubsw src1 src2)
-      (xmm_rm_r $I8X16 (SseOpcode.Pmaddubsw) src1 src2))
+      (xmm_rm_r (SseOpcode.Pmaddubsw) src1 src2))
 
 ;; Helper for creating `insertps` instructions.
 (decl x64_insertps (Xmm XmmMem u8) Xmm)
@@ -3074,37 +3074,27 @@
 ;; Helper for creating `minpd` instructions.
 (decl x64_minpd (Xmm Xmm) Xmm)
 (rule (x64_minpd x y)
-      (let ((dst WritableXmm (temp_writable_xmm))
-            (_ Unit (emit (MInst.XmmRmR (SseOpcode.Minpd) x y dst))))
-        dst))
+      (xmm_rm_r (SseOpcode.Minpd) x y))
 
 ;; Helper for creating `maxss` instructions.
 (decl x64_maxss (Xmm Xmm) Xmm)
 (rule (x64_maxss x y)
-      (let ((dst WritableXmm (temp_writable_xmm))
-            (_ Unit (emit (MInst.XmmRmR (SseOpcode.Maxss) x y dst))))
-        dst))
+      (xmm_rm_r (SseOpcode.Maxss) x y))
 
 ;; Helper for creating `maxsd` instructions.
 (decl x64_maxsd (Xmm Xmm) Xmm)
 (rule (x64_maxsd x y)
-      (let ((dst WritableXmm (temp_writable_xmm))
-            (_ Unit (emit (MInst.XmmRmR (SseOpcode.Maxsd) x y dst))))
-        dst))
+      (xmm_rm_r (SseOpcode.Maxsd) x y))
 
 ;; Helper for creating `maxps` instructions.
 (decl x64_maxps (Xmm Xmm) Xmm)
 (rule (x64_maxps x y)
-      (let ((dst WritableXmm (temp_writable_xmm))
-            (_ Unit (emit (MInst.XmmRmR (SseOpcode.Maxps) x y dst))))
-        dst))
+      (xmm_rm_r (SseOpcode.Maxps) x y))
 
 ;; Helper for creating `maxpd` instructions.
 (decl x64_maxpd (Xmm Xmm) Xmm)
 (rule (x64_maxpd x y)
-      (let ((dst WritableXmm (temp_writable_xmm))
-            (_ Unit (emit (MInst.XmmRmR (SseOpcode.Maxpd) x y dst))))
-        dst))
+      (xmm_rm_r (SseOpcode.Maxpd) x y))
 
 
 ;; Helper for creating `MInst.XmmRmRVex` instructions.
@@ -3289,13 +3279,13 @@
 (rule (x64_pcmpeq $I64X2 x y) (x64_pcmpeqq x y))
 
 (decl x64_pcmpeqb (Xmm XmmMem) Xmm)
-(rule (x64_pcmpeqb x y) (xmm_rm_r $I8X16 (SseOpcode.Pcmpeqb) x y))
+(rule (x64_pcmpeqb x y) (xmm_rm_r (SseOpcode.Pcmpeqb) x y))
 (decl x64_pcmpeqw (Xmm XmmMem) Xmm)
-(rule (x64_pcmpeqw x y) (xmm_rm_r $I16X8 (SseOpcode.Pcmpeqw) x y))
+(rule (x64_pcmpeqw x y) (xmm_rm_r (SseOpcode.Pcmpeqw) x y))
 (decl x64_pcmpeqd (Xmm XmmMem) Xmm)
-(rule (x64_pcmpeqd x y) (xmm_rm_r $I32X4 (SseOpcode.Pcmpeqd) x y))
+(rule (x64_pcmpeqd x y) (xmm_rm_r (SseOpcode.Pcmpeqd) x y))
 (decl x64_pcmpeqq (Xmm XmmMem) Xmm)
-(rule (x64_pcmpeqq x y) (xmm_rm_r $I64X2 (SseOpcode.Pcmpeqq) x y))
+(rule (x64_pcmpeqq x y) (xmm_rm_r (SseOpcode.Pcmpeqq) x y))
 
 ;; Helpers for creating `pcmpgt*` instructions.
 (decl x64_pcmpgt (Type Xmm XmmMem) Xmm)
@@ -3305,13 +3295,13 @@
 (rule (x64_pcmpgt $I64X2 x y) (x64_pcmpgtq x y))
 
 (decl x64_pcmpgtb (Xmm XmmMem) Xmm)
-(rule (x64_pcmpgtb x y) (xmm_rm_r $I8X16 (SseOpcode.Pcmpgtb) x y))
+(rule (x64_pcmpgtb x y) (xmm_rm_r (SseOpcode.Pcmpgtb) x y))
 (decl x64_pcmpgtw (Xmm XmmMem) Xmm)
-(rule (x64_pcmpgtw x y) (xmm_rm_r $I16X8 (SseOpcode.Pcmpgtw) x y))
+(rule (x64_pcmpgtw x y) (xmm_rm_r (SseOpcode.Pcmpgtw) x y))
 (decl x64_pcmpgtd (Xmm XmmMem) Xmm)
-(rule (x64_pcmpgtd x y) (xmm_rm_r $I32X4 (SseOpcode.Pcmpgtd) x y))
+(rule (x64_pcmpgtd x y) (xmm_rm_r (SseOpcode.Pcmpgtd) x y))
 (decl x64_pcmpgtq (Xmm XmmMem) Xmm)
-(rule (x64_pcmpgtq x y) (xmm_rm_r $I64X2 (SseOpcode.Pcmpgtq) x y))
+(rule (x64_pcmpgtq x y) (xmm_rm_r (SseOpcode.Pcmpgtq) x y))
 
 ;; Helpers for read-modify-write ALU form (AluRM).
 (decl alu_rm (Type AluRmiROpcode Amode Gpr) SideEffectNoResult)
diff --git a/cranelift/codegen/src/isa/x64/inst/mod.rs b/cranelift/codegen/src/isa/x64/inst/mod.rs
index 518b544cf..6716ec21b 100644
--- a/cranelift/codegen/src/isa/x64/inst/mod.rs
+++ b/cranelift/codegen/src/isa/x64/inst/mod.rs
@@ -762,9 +762,9 @@ impl PrettyPrint for Inst {
                 format!(
                     "{} {}, {}, {}",
                     ljustify2(op.to_string(), String::new()),
-                    dst,
-                    src1,
                     src2,
+                    src1,
+                    dst,
                 )
             }
             Inst::UnaryRmR { src, dst, op, size } => {
diff --git a/cranelift/codegen/src/opts/algebraic.isle b/cranelift/codegen/src/opts/algebraic.isle
index 8193f7945..c25fb5a1c 100644
--- a/cranelift/codegen/src/opts/algebraic.isle
+++ b/cranelift/codegen/src/opts/algebraic.isle
@@ -345,3 +345,131 @@
              (uextend $I64 x @ (value_type $I32))
              (iconst _ (u64_from_imm64 0))))
       (iconst ty (imm64 1)))
+
+
+;; Transform select-of-icmp into {u,s}{min,max} instructions where possible.
+(rule (simplify
+       (select ty (icmp _ (IntCC.SignedGreaterThan) x y) x y))
+      (smax ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.SignedGreaterThanOrEqual) x y) x y))
+      (smax ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.UnsignedGreaterThan) x y) x y))
+      (umax ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.UnsignedGreaterThanOrEqual) x y) x y))
+      (umax ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.SignedLessThan) x y) x y))
+      (smin ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.SignedLessThanOrEqual) x y) x y))
+      (smin ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.UnsignedLessThan) x y) x y))
+      (umin ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.UnsignedLessThanOrEqual) x y) x y))
+      (umin ty x y))
+
+
+;; These are the same rules as above, but when the operands for select are swapped
+(rule (simplify
+       (select ty (icmp _ (IntCC.SignedLessThan) x y) y x))
+      (smax ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.SignedLessThanOrEqual) x y) y x))
+      (smax ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.UnsignedLessThan) x y) y x))
+      (umax ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.UnsignedLessThanOrEqual) x y) y x))
+      (umax ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.SignedGreaterThan) x y) y x))
+      (smin ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.SignedGreaterThanOrEqual) x y) y x))
+      (smin ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.UnsignedGreaterThan) x y) y x))
+      (umin ty x y))
+(rule (simplify
+       (select ty (icmp _ (IntCC.UnsignedGreaterThanOrEqual) x y) y x))
+      (umin ty x y))
+
+;; Transform vselect-of-icmp into {u,s}{min,max} instructions where possible.
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.SignedGreaterThan) x y) x y))
+      (smax ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.SignedGreaterThanOrEqual) x y) x y))
+      (smax ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.UnsignedGreaterThan) x y) x y))
+      (umax ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.UnsignedGreaterThanOrEqual) x y) x y))
+      (umax ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.SignedLessThan) x y) x y))
+      (smin ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.SignedLessThanOrEqual) x y) x y))
+      (smin ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.UnsignedLessThan) x y) x y))
+      (umin ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.UnsignedLessThanOrEqual) x y) x y))
+      (umin ty x y))
+
+;; These are the same rules as above, but when the operands for select are swapped
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.SignedLessThan) x y) y x))
+      (smax ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.SignedLessThanOrEqual) x y) y x))
+      (smax ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.UnsignedLessThan) x y) y x))
+      (umax ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.UnsignedLessThanOrEqual) x y) y x))
+      (umax ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.SignedGreaterThan) x y) y x))
+      (smin ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.SignedGreaterThanOrEqual) x y) y x))
+      (smin ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.UnsignedGreaterThan) x y) y x))
+      (umin ty x y))
+(rule (simplify
+       (vselect ty (icmp _ (IntCC.UnsignedGreaterThanOrEqual) x y) y x))
+      (umin ty x y))
+
+;; For floats convert fcmp lt into pseudo_min and gt into pseudo_max
+;;
+;; fmax_pseudo docs state:
+;; The behaviour for this operations is defined as  fmax_pseudo(a, b) = (a < b) ? b : a, and the behaviour for zero
+;; or NaN inputs follows from the behaviour of < with such inputs.
+;;
+;; That is exactly the operation that we match here!
+(rule (simplify
+       (select ty (fcmp _ (FloatCC.LessThan) x y) x y))
+      (fmin_pseudo ty x y))
+(rule (simplify
+       (select ty (fcmp _ (FloatCC.GreaterThan) x y) x y))
+      (fmax_pseudo ty x y))
+
+;; Do the same for vectors
+(rule (simplify
+       (vselect ty (fcmp _ (FloatCC.LessThan) x y) x y))
+      (fmin_pseudo ty x y))
+(rule (simplify
+       (vselect ty (fcmp _ (FloatCC.GreaterThan) x y) x y))
+      (fmax_pseudo ty x y))


Test patch:
diff --git a/cranelift/codegen/src/isa/s390x/inst/emit_tests.rs b/cranelift/codegen/src/isa/s390x/inst/emit_tests.rs
index e2cb0e053..943762540 100644
--- a/cranelift/codegen/src/isa/s390x/inst/emit_tests.rs
+++ b/cranelift/codegen/src/isa/s390x/inst/emit_tests.rs
@@ -587,7 +587,7 @@ fn test_s390x_binemit() {
                 flags: MemFlags::trusted(),
             },
         },
-        "E3102000004A",
+        "E3102000007A",
         "ahy %r1, 0(%r2)",
     ));
     insns.push((
@@ -7999,7 +7999,7 @@ fn test_s390x_binemit() {
             rn: vr(12),
         },
         "B344008C",
-        "ledbra %f8, %f12, 0",
+        "ledbra %f8, 0, %f12, 0",
     ));
     insns.push((
         Inst::FpuRound {
@@ -8029,7 +8029,7 @@ fn test_s390x_binemit() {
             rn: vr(12),
         },
         "B357708C",
-        "fiebr %f8, %f12, 7",
+        "fiebr %f8, 7, %f12",
     ));
     insns.push((
         Inst::FpuRound {
@@ -8039,7 +8039,7 @@ fn test_s390x_binemit() {
             rn: vr(12),
         },
         "B35F708C",
-        "fidbr %f8, %f12, 7",
+        "fidbr %f8, 7, %f12",
     ));
     insns.push((
         Inst::FpuRound {
@@ -8049,7 +8049,7 @@ fn test_s390x_binemit() {
             rn: vr(12),
         },
         "B357608C",
-        "fiebr %f8, %f12, 6",
+        "fiebr %f8, 6, %f12",
     ));
     insns.push((
         Inst::FpuRound {
@@ -8059,7 +8059,7 @@ fn test_s390x_binemit() {
             rn: vr(12),
         },
         "B35F608C",
-        "fidbr %f8, %f12, 6",
+        "fidbr %f8, 6, %f12",
     ));
     insns.push((
         Inst::FpuRound {
@@ -8069,7 +8069,7 @@ fn test_s390x_binemit() {
             rn: vr(12),
         },
         "B357508C",
-        "fiebr %f8, %f12, 5",
+        "fiebr %f8, 5, %f12",
     ));
     insns.push((
         Inst::FpuRound {
@@ -8079,7 +8079,7 @@ fn test_s390x_binemit() {
             rn: vr(12),
         },
         "B35F508C",
-        "fidbr %f8, %f12, 5",
+        "fidbr %f8, 5, %f12",
     ));
     insns.push((
         Inst::FpuRound {
@@ -8089,7 +8089,7 @@ fn test_s390x_binemit() {
             rn: vr(12),
         },
         "B357408C",
-        "fiebr %f8, %f12, 4",
+        "fiebr %f8, 4, %f12",
     ));
     insns.push((
         Inst::FpuRound {
@@ -8099,7 +8099,7 @@ fn test_s390x_binemit() {
             rn: vr(12),
         },
         "B35F408C",
-        "fidbr %f8, %f12, 4",
+        "fidbr %f8, 4, %f12",
     ));
     insns.push((
         Inst::FpuRound {
diff --git a/cranelift/filetests/filetests/egraph/licm.clif b/cranelift/filetests/filetests/egraph/licm.clif
index 7e1339200..7d44f53fe 100644
--- a/cranelift/filetests/filetests/egraph/licm.clif
+++ b/cranelift/filetests/filetests/egraph/licm.clif
@@ -31,8 +31,8 @@ block2(v9: i32):
 ; check:      v8 = iadd v2, v3
 ; check:      brif v6, block2, block1(v8)
 
+
 ; check:  block2:
 ; check:      v10 = iconst.i32 1
 ; check:      v4 = iadd.i32 v1, v10
 ; check:      return v4
-
diff --git a/cranelift/filetests/filetests/egraph/select.clif b/cranelift/filetests/filetests/egraph/select.clif
new file mode 100644
index 000000000..12096ce8f
--- /dev/null
+++ b/cranelift/filetests/filetests/egraph/select.clif
@@ -0,0 +1,155 @@
+test optimize
+set opt_level=speed
+set use_egraphs=true
+target x86_64
+target aarch64
+target s390x
+target riscv64
+
+function %select_sgt_to_smax(i32, i32) -> i32 {
+block0(v0: i32, v1: i32):
+    v2 = icmp sgt v0, v1
+    v3 = select v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32, v1: i32):
+; check:    v4 = smax v0, v1
+; check:    return v4
+
+
+; This tests an inverted select, where the operands are swapped.
+function %select_sgt_to_smax_inverse(i32, i32) -> i32 {
+block0(v0: i32, v1: i32):
+    v2 = icmp sgt v0, v1
+    v3 = select v2, v1, v0
+    return v3
+}
+
+; check: block0(v0: i32, v1: i32):
+; check:    v4 = smin v0, v1
+; check:    return v4
+
+
+function %select_sge_to_smax(i32, i32) -> i32 {
+block0(v0: i32, v1: i32):
+    v2 = icmp sge v0, v1
+    v3 = select v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32, v1: i32):
+; check:    v4 = smax v0, v1
+; check:    return v4
+
+
+function %select_ugt_to_umax(i32, i32) -> i32 {
+block0(v0: i32, v1: i32):
+    v2 = icmp ugt v0, v1
+    v3 = select v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32, v1: i32):
+; check:    v4 = umax v0, v1
+; check:    return v4
+
+
+function %select_uge_to_umax(i32, i32) -> i32 {
+block0(v0: i32, v1: i32):
+    v2 = icmp uge v0, v1
+    v3 = select v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32, v1: i32):
+; check:    v4 = umax v0, v1
+; check:    return v4
+
+
+
+function %select_slt_to_smin(i32, i32) -> i32 {
+block0(v0: i32, v1: i32):
+    v2 = icmp slt v0, v1
+    v3 = select v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32, v1: i32):
+; check:    v4 = smin v0, v1
+; check:    return v4
+
+
+function %select_sle_to_smin(i32, i32) -> i32 {
+block0(v0: i32, v1: i32):
+    v2 = icmp sle v0, v1
+    v3 = select v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32, v1: i32):
+; check:    v4 = smin v0, v1
+; check:    return v4
+
+
+function %select_ult_to_umin(i32, i32) -> i32 {
+block0(v0: i32, v1: i32):
+    v2 = icmp ult v0, v1
+    v3 = select v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32, v1: i32):
+; check:    v4 = umin v0, v1
+; check:    return v4
+
+
+function %select_ule_to_umin(i32, i32) -> i32 {
+block0(v0: i32, v1: i32):
+    v2 = icmp ule v0, v1
+    v3 = select v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32, v1: i32):
+; check:    v4 = umin v0, v1
+; check:    return v4
+
+
+
+function %select_with_different_regs_does_not_optimize(i32, i32, i32, i32) -> i32 {
+block0(v0: i32, v1: i32, v2: i32, v3: i32):
+    v4 = icmp ule v0, v1
+    v5 = select v4, v2, v3
+    return v5
+}
+
+; check: block0(v0: i32, v1: i32, v2: i32, v3: i32):
+; check:    v4 = icmp ule v0, v1
+; check:    v5 = select v4, v2, v3
+; check:    return v5
+
+
+
+
+function %select_fcmp_gt_to_fmax_pseudo(f32, f32) -> f32 {
+block0(v0: f32, v1: f32):
+    v2 = fcmp gt v0, v1
+    v3 = select v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: f32, v1: f32):
+; check:    v4 = fmax_pseudo v0, v1
+; check:    return v4
+
+function %select_fcmp_lt_to_fmin_pseudo(f32, f32) -> f32 {
+block0(v0: f32, v1: f32):
+    v2 = fcmp lt v0, v1
+    v3 = select v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: f32, v1: f32):
+; check:    v4 = fmin_pseudo v0, v1
+; check:    return v4
diff --git a/cranelift/filetests/filetests/egraph/vselect.clif b/cranelift/filetests/filetests/egraph/vselect.clif
new file mode 100644
index 000000000..805f7b61c
--- /dev/null
+++ b/cranelift/filetests/filetests/egraph/vselect.clif
@@ -0,0 +1,154 @@
+test optimize
+set opt_level=speed
+set use_egraphs=true
+target x86_64
+target aarch64
+target s390x
+
+function %vselect_sgt_to_smax(i32x4, i32x4) -> i32x4 {
+block0(v0: i32x4, v1: i32x4):
+    v2 = icmp sgt v0, v1
+    v3 = vselect v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32x4, v1: i32x4):
+; check:    v4 = smax v0, v1
+; check:    return v4
+
+
+; This tests an inverted vselect, where the operands are swapped.
+function %vselect_sgt_to_smax(i32x4, i32x4) -> i32x4 {
+block0(v0: i32x4, v1: i32x4):
+    v2 = icmp sgt v0, v1
+    v3 = vselect v2, v1, v0
+    return v3
+}
+
+; check: block0(v0: i32x4, v1: i32x4):
+; check:    v4 = smin v0, v1
+; check:    return v4
+
+
+
+function %vselect_sge_to_smax(i32x4, i32x4) -> i32x4 {
+block0(v0: i32x4, v1: i32x4):
+    v2 = icmp sge v0, v1
+    v3 = vselect v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32x4, v1: i32x4):
+; check:    v4 = smax v0, v1
+; check:    return v4
+
+
+function %vselect_ugt_to_umax(i32x4, i32x4) -> i32x4 {
+block0(v0: i32x4, v1: i32x4):
+    v2 = icmp ugt v0, v1
+    v3 = vselect v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32x4, v1: i32x4):
+; check:    v4 = umax v0, v1
+; check:    return v4
+
+
+function %vselect_uge_to_umax(i32x4, i32x4) -> i32x4 {
+block0(v0: i32x4, v1: i32x4):
+    v2 = icmp uge v0, v1
+    v3 = vselect v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32x4, v1: i32x4):
+; check:    v4 = umax v0, v1
+; check:    return v4
+
+
+
+function %vselect_slt_to_smin(i32x4, i32x4) -> i32x4 {
+block0(v0: i32x4, v1: i32x4):
+    v2 = icmp slt v0, v1
+    v3 = vselect v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32x4, v1: i32x4):
+; check:    v4 = smin v0, v1
+; check:    return v4
+
+
+function %vselect_sle_to_smin(i32x4, i32x4) -> i32x4 {
+block0(v0: i32x4, v1: i32x4):
+    v2 = icmp sle v0, v1
+    v3 = vselect v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32x4, v1: i32x4):
+; check:    v4 = smin v0, v1
+; check:    return v4
+
+
+function %vselect_ult_to_umin(i32x4, i32x4) -> i32x4 {
+block0(v0: i32x4, v1: i32x4):
+    v2 = icmp ult v0, v1
+    v3 = vselect v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32x4, v1: i32x4):
+; check:    v4 = umin v0, v1
+; check:    return v4
+
+
+function %vselect_ule_to_umin(i32x4, i32x4) -> i32x4 {
+block0(v0: i32x4, v1: i32x4):
+    v2 = icmp ule v0, v1
+    v3 = vselect v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: i32x4, v1: i32x4):
+; check:    v4 = umin v0, v1
+; check:    return v4
+
+
+
+function %vselect_with_different_regs_does_not_optimize(i32x4, i32x4, i32x4, i32x4) -> i32x4 {
+block0(v0: i32x4, v1: i32x4, v2: i32x4, v3: i32x4):
+    v4 = icmp ule v0, v1
+    v5 = vselect v4, v2, v3
+    return v5
+}
+
+; check: block0(v0: i32x4, v1: i32x4, v2: i32x4, v3: i32x4):
+; check:    v4 = icmp ule v0, v1
+; check:    v5 = vselect v4, v2, v3
+; check:    return v5
+
+
+
+function %vselect_fcmp_gt_to_fmax_pseudo(f32x4, f32x4) -> f32x4 {
+block0(v0: f32x4, v1: f32x4):
+    v2 = fcmp gt v0, v1
+    v3 = vselect v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: f32x4, v1: f32x4):
+; check:    v4 = fmax_pseudo v0, v1
+; check:    return v4
+
+function %vselect_fcmp_lt_to_fmin_pseudo(f32x4, f32x4) -> f32x4 {
+block0(v0: f32x4, v1: f32x4):
+    v2 = fcmp lt v0, v1
+    v3 = vselect v2, v0, v1
+    return v3
+}
+
+; check: block0(v0: f32x4, v1: f32x4):
+; check:    v4 = fmin_pseudo v0, v1
+; check:    return v4
diff --git a/cranelift/filetests/filetests/isa/s390x/floating-point-arch13.clif b/cranelift/filetests/filetests/isa/s390x/floating-point-arch13.clif
index dadca7c76..9bddd0413 100644
--- a/cranelift/filetests/filetests/isa/s390x/floating-point-arch13.clif
+++ b/cranelift/filetests/filetests/isa/s390x/floating-point-arch13.clif
@@ -386,7 +386,7 @@ block0(v0: i64):
 ; block0:
 ;   ldgr %f2, %r2
 ;   wcdlgb %f4, %f2, 0, 3
-;   ledbra %f0, %f4, 4
+;   ledbra %f0, 4, %f4, 0
 ;   br %r14
 
 function %fcvt_from_sint_i64_f32(i64) -> f32 {
@@ -398,7 +398,7 @@ block0(v0: i64):
 ; block0:
 ;   ldgr %f2, %r2
 ;   wcdgb %f4, %f2, 0, 3
-;   ledbra %f0, %f4, 4
+;   ledbra %f0, 4, %f4, 0
 ;   br %r14
 
 function %fcvt_from_uint_i8_f64(i8) -> f64 {
diff --git a/cranelift/filetests/filetests/isa/s390x/floating-point.clif b/cranelift/filetests/filetests/isa/s390x/floating-point.clif
index 4a03009a5..eee92024b 100644
--- a/cranelift/filetests/filetests/isa/s390x/floating-point.clif
+++ b/cranelift/filetests/filetests/isa/s390x/floating-point.clif
@@ -285,7 +285,7 @@ block0(v0: f64):
 }
 
 ; block0:
-;   ledbra %f0, %f0, 0
+;   ledbra %f0, 0, %f0, 0
 ;   br %r14
 
 function %ceil_f32(f32) -> f32 {
@@ -295,7 +295,7 @@ block0(v0: f32):
 }
 
 ; block0:
-;   fiebr %f0, %f0, 6
+;   fiebr %f0, 6, %f0
 ;   br %r14
 
 function %ceil_f64(f64) -> f64 {
@@ -305,7 +305,7 @@ block0(v0: f64):
 }
 
 ; block0:
-;   fidbr %f0, %f0, 6
+;   fidbr %f0, 6, %f0
 ;   br %r14
 
 function %floor_f32(f32) -> f32 {
@@ -315,7 +315,7 @@ block0(v0: f32):
 }
 
 ; block0:
-;   fiebr %f0, %f0, 7
+;   fiebr %f0, 7, %f0
 ;   br %r14
 
 function %floor_f64(f64) -> f64 {
@@ -325,7 +325,7 @@ block0(v0: f64):
 }
 
 ; block0:
-;   fidbr %f0, %f0, 7
+;   fidbr %f0, 7, %f0
 ;   br %r14
 
 function %trunc_f32(f32) -> f32 {
@@ -335,7 +335,7 @@ block0(v0: f32):
 }
 
 ; block0:
-;   fiebr %f0, %f0, 5
+;   fiebr %f0, 5, %f0
 ;   br %r14
 
 function %trunc_f64(f64) -> f64 {
@@ -345,7 +345,7 @@ block0(v0: f64):
 }
 
 ; block0:
-;   fidbr %f0, %f0, 5
+;   fidbr %f0, 5, %f0
 ;   br %r14
 
 function %nearest_f32(f32) -> f32 {
@@ -355,7 +355,7 @@ block0(v0: f32):
 }
 
 ; block0:
-;   fiebr %f0, %f0, 4
+;   fiebr %f0, 4, %f0
 ;   br %r14
 
 function %nearest_f64(f64) -> f64 {
@@ -365,7 +365,7 @@ block0(v0: f64):
 }
 
 ; block0:
-;   fidbr %f0, %f0, 4
+;   fidbr %f0, 4, %f0
 ;   br %r14
 
 function %fma_f32(f32, f32, f32) -> f32 {
@@ -732,7 +732,7 @@ block0(v0: i8):
 ;   llgcr %r4, %r2
 ;   ldgr %f4, %r4
 ;   wcdlgb %f6, %f4, 0, 3
-;   ledbra %f0, %f6, 4
+;   ledbra %f0, 4, %f6, 0
 ;   br %r14
 
 function %fcvt_from_sint_i8_f32(i8) -> f32 {
@@ -745,7 +745,7 @@ block0(v0: i8):
 ;   lgbr %r4, %r2
 ;   ldgr %f4, %r4
 ;   wcdgb %f6, %f4, 0, 3
-;   ledbra %f0, %f6, 4
+;   ledbra %f0, 4, %f6, 0
 ;   br %r14
 
 function %fcvt_from_uint_i16_f32(i16) -> f32 {
@@ -758,7 +758,7 @@ block0(v0: i16):
 ;   llghr %r4, %r2
 ;   ldgr %f4, %r4
 ;   wcdlgb %f6, %f4, 0, 3
-;   ledbra %f0, %f6, 4
+;   ledbra %f0, 4, %f6, 0
 ;   br %r14
 
 function %fcvt_from_sint_i16_f32(i16) -> f32 {
@@ -771,7 +771,7 @@ block0(v0: i16):
 ;   lghr %r4, %r2
 ;   ldgr %f4, %r4
 ;   wcdgb %f6, %f4, 0, 3
-;   ledbra %f0, %f6, 4
+;   ledbra %f0, 4, %f6, 0
 ;   br %r14
 
 function %fcvt_from_uint_i32_f32(i32) -> f32 {
@@ -784,7 +784,7 @@ block0(v0: i32):
 ;   llgfr %r4, %r2
 ;   ldgr %f4, %r4
 ;   wcdlgb %f6, %f4, 0, 3
-;   ledbra %f0, %f6, 4
+;   ledbra %f0, 4, %f6, 0
 ;   br %r14
 
 function %fcvt_from_sint_i32_f32(i32) -> f32 {
@@ -797,7 +797,7 @@ block0(v0: i32):
 ;   lgfr %r4, %r2
 ;   ldgr %f4, %r4
 ;   wcdgb %f6, %f4, 0, 3
-;   ledbra %f0, %f6, 4
+;   ledbra %f0, 4, %f6, 0
 ;   br %r14
 
 function %fcvt_from_uint_i64_f32(i64) -> f32 {
@@ -809,7 +809,7 @@ block0(v0: i64):
 ; block0:
 ;   ldgr %f2, %r2
 ;   wcdlgb %f4, %f2, 0, 3
-;   ledbra %f0, %f4, 4
+;   ledbra %f0, 4, %f4, 0
 ;   br %r14
 
 function %fcvt_from_sint_i64_f32(i64) -> f32 {
@@ -821,7 +821,7 @@ block0(v0: i64):
 ; block0:
 ;   ldgr %f2, %r2
 ;   wcdgb %f4, %f2, 0, 3
-;   ledbra %f0, %f4, 4
+;   ledbra %f0, 4, %f4, 0
 ;   br %r14
 
 function %fcvt_from_uint_i8_f64(i8) -> f64 {
diff --git a/cranelift/filetests/filetests/isa/x64/band_not_bmi1.clif b/cranelift/filetests/filetests/isa/x64/band_not_bmi1.clif
index 29101eaff..96dc33ef3 100644
--- a/cranelift/filetests/filetests/isa/x64/band_not_bmi1.clif
+++ b/cranelift/filetests/filetests/isa/x64/band_not_bmi1.clif
@@ -11,12 +11,11 @@ block0(v0: i8, v1: i8):
 ;   pushq   %rbp
 ;   movq    %rsp, %rbp
 ; block0:
-;   andn    %eax, %esi, %edi
+;   andn    %edi, %esi, %eax
 ;   movq    %rbp, %rsp
 ;   popq    %rbp
 ;   ret
 
-
 function %reversed_operands(i8, i8) -> i8 {
 block0(v0: i8, v1: i8):
     v2 = bnot v0
@@ -27,7 +26,7 @@ block0(v0: i8, v1: i8):
 ;   pushq   %rbp
 ;   movq    %rsp, %rbp
 ; block0:
-;   andn    %eax, %edi, %esi
+;   andn    %esi, %edi, %eax
 ;   movq    %rbp, %rsp
 ;   popq    %rbp
 ;   ret
diff --git a/cranelift/filetests/filetests/wasm/i32-not-x64.wat b/cranelift/filetests/filetests/wasm/i32-not-x64.wat
index 45d3798e7..e4eb54957 100644
--- a/cranelift/filetests/filetests/wasm/i32-not-x64.wat
+++ b/cranelift/filetests/filetests/wasm/i32-not-x64.wat
@@ -40,7 +40,7 @@
 ;; block0:
 ;;   jmp     label1
 ;; block1:
-;;   andn    %eax, %esi, %edi
+;;   andn    %edi, %esi, %eax
 ;;   movq    %rbp, %rsp
 ;;   popq    %rbp
 ;;   ret