Repository: bytecodealliance/wasmtime
Pull request #5795 (2023-02-16 00:12:44)

x64: Add rudimentary support for some AVX instructions
I was poking around Spidermonkey's wasm backend and saw that the various assembler functions used are all `v*`-prefixed which look like they're intended for use with AVX instructions. I looked at Cranelift and it currently doesn't have support for many AVX-based instructions, so I figured I'd take a crack at it!

The support added here is a bit of a mishmash when viewed alone, but my general goal was to take a single instruction from the SIMD proposal for WebAssembly and migrate all of its component instructions to AVX. I, by random chance, picked a pretty complicated instruction of `f32x4.min`. This wasm instruction is implemented on x64 with 4 unique SSE instructions and ended up being a pretty good candidate.

Further digging about AVX-vs-SSE shows that there should be two major benefits to using AVX over SSE:

* Primarily AVX instructions largely use a three-operand form where two input registers are operated with and an output register is also specified. This is in contrast to SSE's predominant one-register-is-input-but-also-output pattern. This should help free up the register allocator a bit and additionally remove the need for movement between registers.

* As #4767 notes the memory-based operations of VEX-encoded instructions (aka AVX instructions) do not have strict alignment requirements which means we would be able to sink loads and stores into individual instructions instead of having separate instructions.

So I set out on my journey to implement the instructions used by `f32x4.min`. The first few were fairly easy. The machinst backends are already of the shape "take these inputs and compute the output" where the x86 requirement of a register being both input and output is postprocessed in. This means that the `inst.isle` creation helpers for SSE instructions were already of the correct form to use AVX. I chose to add new `rule` branches for the instruction creation helpers, for example `x64_andnps`. The new `rule` conditionally only runs if AVX is enabled and emits an AVX instruction instead of an SSE instruction for achieving the same goal. This means that no lowerings of clif instructions were modified, instead just new instructions are being generated.

The VEX encoding was previously not heavily used in Cranelift. The only current user are the FMA-style instructions that Cranelift has at this time. These FMA instructions have one extra operand than `vandnps`, for example, so I split the existing `XmmRmRVex` into a few more variants to fit the shape of the instructions that needed generating for `f32x4.min`. This was accompanied then with more AVX opcode definitions, more emission support, etc.

Upon implementing all of this it turned out that the test suite was failing on my machine due to the memory-operand encodings of VEX instructions not being supported. I didn't explicitly add those in myself but some preexisting RIP-relative addressing was leaking into the new instructions with existing tests. I opted to go ahead and fill out the memory addressing modes of VEX encoding to get the tests passing again.

All-in-all this PR adds new instructions to the x64 backend for a number of AVX instructions, updates 5 existing instruction producers to use AVX instructions conditionally, implements VEX memory operands, and adds some simple tests for the new output of `f32x4.min`. The existing runtest for `f32x.min` caught a few intermediate bugs along the way and I additionally added a plain `target x86_64` to that runtest to ensure that it executes with and without AVX to test the various lowerings. I'll also note that this, and future support, should be well-fuzzed through Wasmtime's fuzzing which may explicitly disable AVX support despite the machine having access to AVX, so non-AVX lowerings should be well-tested into the future.

It's also worth mentioning that I am not an AVX or VEX or x64 expert. Implementing the memory operand part for VEX was the hardest part of this PR and while I think it should be good someone else should definitely double-check me. Additionally I haven't added many instructions to the x64 backend yet so I may have missed obvious places to tests or such, so am happy to follow-up with anything to be more thorough if necessary.

Finally I should note that this is just the tip of the iceberg when it comes to AVX. My hope is to get some of the idioms sorted out to make it easier for future PRs to add one-off instruction lowerings or such.

<!--

Please ensure that the following steps are all taken care of before submitting
the PR.

- [ ] This has been discussed in issue #..., or if not, please tell us why
  here.
- [ ] A short description of what this does, why it is needed; if the
  description becomes long, the matter should probably be discussed in an issue
  first.
- [ ] This PR contains test cases, if meaningful.
- [ ] A reviewer from the core maintainer team has been assigned for this PR.
  If you don't know who could review this, please indicate so. The list of
  suggested reviewers on the right can help you.

Please ensure all communication adheres to the [code of
conduct](https://github.com/bytecodealliance/wasmtime/blob/master/CODE_OF_CONDUCT.md).
-->


alexcrichton: Oh I should note that the total encoding size of `f32x4.min` was reduced from 45 to 35 bytes with this change. (less `movdqa` necessary)
github-actions[bot]: #### Subscribe to Label Action

cc @cfallin, @fitzgen

<details>
This issue or pull request has been labeled: "cranelift", "cranelift:area:x64", "isle"

Thus the following users have been cc'd because of the following labels:

* cfallin: isle
* fitzgen: isle

To subscribe or unsubscribe from this label, edit the <code>.github/subscribe-to-label.json</code> configuration file.

[Learn more.](https://github.com/bytecodealliance/subscribe-to-label-action)
</details>

Patch:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 6bdcd18b4..07008be3f 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -54,6 +54,8 @@ jobs:
   # incompatible licences
   cargo_deny:
     name: Cargo deny
+    needs: determine
+    if: needs.determine.outputs.audit
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
@@ -76,6 +78,8 @@ jobs:
   # Ensure dependencies are vetted. See https://mozilla.github.io/cargo-vet/
   cargo_vet:
     name: Cargo vet
+    needs: determine
+    if: needs.determine.outputs.audit
     runs-on: ubuntu-latest
     env:
       CARGO_VET_VERSION: 0.3.1
@@ -116,6 +120,7 @@ jobs:
       test-matrix: ${{ steps.calculate.outputs.test-matrix }}
       test-capi: ${{ steps.calculate.outputs.test-capi }}
       build-fuzz: ${{ steps.calculate.outputs.build-fuzz }}
+      audit: ${{ steps.calculate.outputs.audit }}
     steps:
     - uses: actions/checkout@v3
     - id: calculate
@@ -143,6 +148,9 @@ jobs:
           if grep -q fuzz names.log; then
             echo build-fuzz=true >> $GITHUB_OUTPUT
           fi
+          if grep -q Cargo.lock names.log; then
+            echo audit=true >> $GITHUB_OUTPUT
+          fi
         fi
         matrix="$(node ./ci/build-test-matrix.js ./commits.log ./names.log $run_full)"
         echo "test-matrix={\"include\":$(echo $matrix)}" >> $GITHUB_OUTPUT
@@ -152,6 +160,7 @@ jobs:
             echo run-full=true >> $GITHUB_OUTPUT
             echo test-capi=true >> $GITHUB_OUTPUT
             echo build-fuzz=true >> $GITHUB_OUTPUT
+            echo audit=true >> $GITHUB_OUTPUT
         fi
 
   # Build all documentation of Wasmtime, including the C API documentation,
@@ -439,7 +448,7 @@ jobs:
     - run: |
         sudo apt-get update && sudo apt-get install -y gdb lldb llvm
         cargo test test_debug_dwarf -- --ignored --test-threads 1
-      if: matrix.os == 'ubuntu-latest' && matrix.target == ''
+      if: matrix.os == 'ubuntu-latest' && matrix.target == ''&& needs.determine.outputs.run-full
       env:
         RUST_BACKTRACE: 1
 
@@ -670,6 +679,7 @@ jobs:
       - bench
       - meta_deterministic_check
       - verify-publish
+      - determine
     if: always()
     steps:
     - name: Dump needs context
diff --git a/.github/workflows/triage.yml b/.github/workflows/triage.yml
index 4da009f8b..c7bde9931 100644
--- a/.github/workflows/triage.yml
+++ b/.github/workflows/triage.yml
@@ -4,11 +4,11 @@ on:
   issues:
     types: ["labeled"]
   schedule:
-    # Run pull request triage every 5 minutes. Ideally, this would be on
+    # Run pull request triage once an hour. Ideally, this would be on
     # "labeled" types of pull request events, but that doesn't work if the pull
     # request is from another fork. For example, see
     # https://github.com/actions/labeler/issues/12
-    - cron: '*/5 * * * *'
+    - cron: '42 * * * *'
 
 concurrency:
   group: issue-triage
diff --git a/cranelift/codegen/src/egraph.rs b/cranelift/codegen/src/egraph.rs
index 02bfeb013..2cbe412e3 100644
--- a/cranelift/codegen/src/egraph.rs
+++ b/cranelift/codegen/src/egraph.rs
@@ -7,7 +7,7 @@ use crate::dominator_tree::DominatorTree;
 use crate::egraph::domtree::DomTreeWithChildren;
 use crate::egraph::elaborate::Elaborator;
 use crate::fx::FxHashSet;
-use crate::inst_predicates::{is_mergeable_for_egraph, is_pure_for_egraph};
+use crate::inst_predicates::is_pure_for_egraph;
 use crate::ir::{
     DataFlowGraph, Function, Inst, InstructionData, Type, Value, ValueDef, ValueListPool,
 };
@@ -285,41 +285,10 @@ where
     fn optimize_skeleton_inst(&mut self, inst: Inst) -> bool {
         self.stats.skeleton_inst += 1;
 
-        // First, can we try to deduplicate? We need to keep some copy
-        // of the instruction around because it's side-effecting, but
-        // we may be able to reuse an earlier instance of it.
-        if is_mergeable_for_egraph(self.func, inst) {
-            let result = self.func.dfg.inst_results(inst)[0];
-            trace!(" -> mergeable side-effecting op {}", inst);
-            let inst = NewOrExistingInst::Existing(inst);
-            let gvn_context = GVNContext {
-                union_find: self.eclasses,
-                value_lists: &self.func.dfg.value_lists,
-            };
-
-            // Does this instruction already exist? If so, add entries to
-            // the value-map to rewrite uses of its results to the results
-            // of the original (existing) instruction. If not, optimize
-            // the new instruction.
-            let key = inst.get_inst_key(&self.func.dfg);
-            if let Some(&orig_result) = self.gvn_map.get(&key, &gvn_context) {
-                // Hit in GVN map -- reuse value.
-                self.value_to_opt_value[result] = orig_result;
-                self.eclasses.union(orig_result, result);
-                trace!(" -> merges result {} to {}", result, orig_result);
-                true
-            } else {
-                // Otherwise, insert it into the value-map.
-                self.value_to_opt_value[result] = result;
-                self.gvn_map.insert(key, result, &gvn_context);
-                trace!(" -> inserts as new (no GVN)");
-                false
-            }
-        }
-        // Otherwise, if a load or store, process it with the alias
-        // analysis to see if we can optimize it (rewrite in terms of
-        // an earlier load or stored value).
-        else if let Some(new_result) =
+        // If a load or store, process it with the alias analysis to see
+        // if we can optimize it (rewrite in terms of an earlier load or
+        // stored value).
+        if let Some(new_result) =
             self.alias_analysis
                 .process_inst(self.func, self.alias_analysis_state, inst)
         {
diff --git a/cranelift/codegen/src/inst_predicates.rs b/cranelift/codegen/src/inst_predicates.rs
index f82f41daa..9b284e8b8 100644
--- a/cranelift/codegen/src/inst_predicates.rs
+++ b/cranelift/codegen/src/inst_predicates.rs
@@ -73,25 +73,6 @@ pub fn is_pure_for_egraph(func: &Function, inst: Inst) -> bool {
     has_one_result && (is_readonly_load || (!op.can_load() && !trivially_has_side_effects(op)))
 }
 
-/// Can the given instruction be merged into another copy of itself?
-/// These instructions may have side-effects, but as long as we retain
-/// the first instance of the instruction, the second and further
-/// instances are redundant if they would produce the same trap or
-/// result.
-pub fn is_mergeable_for_egraph(func: &Function, inst: Inst) -> bool {
-    let op = func.dfg.insts[inst].opcode();
-    // We can only merge one-result operators due to the way that GVN
-    // is structured in the egraph implementation.
-    let has_one_result = func.dfg.inst_results(inst).len() == 1;
-    has_one_result
-        // Loads/stores are handled by alias analysis and not
-        // otherwise mergeable.
-        && !op.can_load()
-        && !op.can_store()
-        // Can only have idempotent side-effects.
-        && (!has_side_effect(func, inst) || op.side_effects_idempotent())
-}
-
 /// Does the given instruction have any side-effect as per [has_side_effect], or else is a load,
 /// but not the get_pinned_reg opcode?
 pub fn has_lowering_side_effect(func: &Function, inst: Inst) -> bool {
diff --git a/cranelift/codegen/src/isa/x64/encoding/rex.rs b/cranelift/codegen/src/isa/x64/encoding/rex.rs
index 41ae596eb..f48fea830 100644
--- a/cranelift/codegen/src/isa/x64/encoding/rex.rs
+++ b/cranelift/codegen/src/isa/x64/encoding/rex.rs
@@ -312,18 +312,50 @@ pub(crate) fn emit_std_enc_mem(
 
     prefixes.emit(sink);
 
+    // After prefixes, first emit the REX byte depending on the kind of
+    // addressing mode that's being used.
     match *mem_e {
-        Amode::ImmReg { simm32, base, .. } => {
-            // First, the REX byte.
+        Amode::ImmReg { base, .. } => {
             let enc_e = int_reg_enc(base);
             rex.emit_two_op(sink, enc_g, enc_e);
+        }
 
-            // Now the opcode(s).  These include any other prefixes the caller
-            // hands to us.
-            while num_opcodes > 0 {
-                num_opcodes -= 1;
-                sink.put1(((opcodes >> (num_opcodes << 3)) & 0xFF) as u8);
-            }
+        Amode::ImmRegRegShift {
+            base: reg_base,
+            index: reg_index,
+            ..
+        } => {
+            let enc_base = int_reg_enc(*reg_base);
+            let enc_index = int_reg_enc(*reg_index);
+            rex.emit_three_op(sink, enc_g, enc_index, enc_base);
+        }
+
+        Amode::RipRelative { .. } => {
+            // note REX.B = 0.
+            rex.emit_two_op(sink, enc_g, 0);
+        }
+    }
+
+    // Now the opcode(s).  These include any other prefixes the caller
+    // hands to us.
+    while num_opcodes > 0 {
+        num_opcodes -= 1;
+        sink.put1(((opcodes >> (num_opcodes << 3)) & 0xFF) as u8);
+    }
+
+    // And finally encode the mod/rm bytes and all further information.
+    emit_modrm_sib_disp(sink, enc_g, mem_e, bytes_at_end)
+}
+
+pub(crate) fn emit_modrm_sib_disp(
+    sink: &mut MachBuffer<Inst>,
+    enc_g: u8,
+    mem_e: &Amode,
+    bytes_at_end: u8,
+) {
+    match *mem_e {
+        Amode::ImmReg { simm32, base, .. } => {
+            let enc_e = int_reg_enc(base);
 
             // Now the mod/rm and associated immediates.  This is
             // significantly complicated due to the multiple special cases.
@@ -377,15 +409,6 @@ pub(crate) fn emit_std_enc_mem(
             let enc_base = int_reg_enc(*reg_base);
             let enc_index = int_reg_enc(*reg_index);
 
-            // The rex byte.
-            rex.emit_three_op(sink, enc_g, enc_index, enc_base);
-
-            // All other prefixes and opcodes.
-            while num_opcodes > 0 {
-                num_opcodes -= 1;
-                sink.put1(((opcodes >> (num_opcodes << 3)) & 0xFF) as u8);
-            }
-
             // modrm, SIB, immediates.
             if low8_will_sign_extend_to_32(simm32) && enc_index != regs::ENC_RSP {
                 sink.put1(encode_modrm(1, enc_g & 7, 4));
@@ -401,16 +424,6 @@ pub(crate) fn emit_std_enc_mem(
         }
 
         Amode::RipRelative { ref target } => {
-            // First, the REX byte, with REX.B = 0.
-            rex.emit_two_op(sink, enc_g, 0);
-
-            // Now the opcode(s).  These include any other prefixes the caller
-            // hands to us.
-            while num_opcodes > 0 {
-                num_opcodes -= 1;
-                sink.put1(((opcodes >> (num_opcodes << 3)) & 0xFF) as u8);
-            }
-
             // RIP-relative is mod=00, rm=101.
             sink.put1(encode_modrm(0, enc_g & 7, 0b101));
 
diff --git a/cranelift/codegen/src/isa/x64/encoding/vex.rs b/cranelift/codegen/src/isa/x64/encoding/vex.rs
index 3aa9bb8d5..df2c92169 100644
--- a/cranelift/codegen/src/isa/x64/encoding/vex.rs
+++ b/cranelift/codegen/src/isa/x64/encoding/vex.rs
@@ -4,7 +4,10 @@
 use super::evex::Register;
 use super::rex::{LegacyPrefixes, OpcodeMap};
 use super::ByteSink;
-use crate::isa::x64::encoding::rex::encode_modrm;
+use crate::isa::x64::args::Amode;
+use crate::isa::x64::encoding::rex;
+use crate::isa::x64::inst::Inst;
+use crate::machinst::MachBuffer;
 
 /// Constructs a VEX-encoded instruction using a builder pattern. This approach makes it visually
 /// easier to transform something the manual's syntax, `VEX.128.66.0F 73 /7 ib` to code:
@@ -16,11 +19,29 @@ pub struct VexInstruction {
     opcode: u8,
     w: bool,
     reg: u8,
-    rm: Register,
+    rm: RegisterOrAmode,
     vvvv: Option<Register>,
     imm: Option<u8>,
 }
 
+#[allow(missing_docs)]
+pub enum RegisterOrAmode {
+    Register(Register),
+    Amode(Amode),
+}
+
+impl From<u8> for RegisterOrAmode {
+    fn from(reg: u8) -> Self {
+        RegisterOrAmode::Register(reg.into())
+    }
+}
+
+impl From<Amode> for RegisterOrAmode {
+    fn from(amode: Amode) -> Self {
+        RegisterOrAmode::Amode(amode)
+    }
+}
+
 impl Default for VexInstruction {
     fn default() -> Self {
         Self {
@@ -30,7 +51,7 @@ impl Default for VexInstruction {
             opcode: 0x00,
             w: false,
             reg: 0x00,
-            rm: Register::default(),
+            rm: RegisterOrAmode::Register(Register::default()),
             vvvv: None,
             imm: None,
         }
@@ -105,12 +126,12 @@ impl VexInstruction {
         self
     }
 
-    /// Set the register to use for the `rm` bits; many instructions use this as the "read from
-    /// register/memory" operand. Currently this does not support memory addressing (TODO).Setting
-    /// this affects both the ModRM byte (`rm` section) and the VEX prefix (the extension bits for
-    /// register encodings > 8).
+    /// Set the register to use for the `rm` bits; many instructions use this
+    /// as the "read from register/memory" operand. Setting this affects both
+    /// the ModRM byte (`rm` section) and the VEX prefix (the extension bits
+    /// for register encodings > 8).
     #[inline(always)]
-    pub fn rm(mut self, reg: impl Into<Register>) -> Self {
+    pub fn rm(mut self, reg: impl Into<RegisterOrAmode>) -> Self {
         self.rm = reg.into();
         self
     }
@@ -150,15 +171,33 @@ impl VexInstruction {
     /// The X bit in encoded format (inverted).
     #[inline(always)]
     fn x_bit(&self) -> u8 {
-        // TODO
-        (!0) & 1
+        let reg = match &self.rm {
+            RegisterOrAmode::Register(_) => 0,
+            RegisterOrAmode::Amode(Amode::ImmReg { .. }) => 0,
+            RegisterOrAmode::Amode(Amode::ImmRegRegShift { index, .. }) => {
+                index.to_real_reg().unwrap().hw_enc()
+            }
+            RegisterOrAmode::Amode(Amode::RipRelative { .. }) => 0,
+        };
+
+        !(reg >> 3) & 1
     }
 
     /// The B bit in encoded format (inverted).
     #[inline(always)]
     fn b_bit(&self) -> u8 {
-        let rm: u8 = self.rm.into();
-        (!(rm >> 3)) & 1
+        let reg = match &self.rm {
+            RegisterOrAmode::Register(r) => (*r).into(),
+            RegisterOrAmode::Amode(Amode::ImmReg { base, .. }) => {
+                base.to_real_reg().unwrap().hw_enc()
+            }
+            RegisterOrAmode::Amode(Amode::ImmRegRegShift { base, .. }) => {
+                base.to_real_reg().unwrap().hw_enc()
+            }
+            RegisterOrAmode::Amode(Amode::RipRelative { .. }) => 0,
+        };
+
+        !(reg >> 3) & 1
     }
 
     /// Is the 2 byte prefix available for this instruction?
@@ -176,6 +215,7 @@ impl VexInstruction {
         // encoded by the three-byte form of VEX
         !(self.map == OpcodeMap::_0F3A || self.map == OpcodeMap::_0F38)
     }
+
     /// The last byte of the 2byte and 3byte prefixes is mostly the same, share the common
     /// encoding logic here.
     #[inline(always)]
@@ -225,8 +265,8 @@ impl VexInstruction {
         sink.put1(last_byte);
     }
 
-    /// Emit the VEX-encoded instruction to the code sink:
-    pub fn encode<CS: ByteSink + ?Sized>(&self, sink: &mut CS) {
+    /// Emit the VEX-encoded instruction to the provided buffer.
+    pub fn encode(&self, sink: &mut MachBuffer<Inst>) {
         // 2/3 byte prefix
         if self.use_2byte_prefix() {
             self.encode_2byte_prefix(sink);
@@ -237,13 +277,21 @@ impl VexInstruction {
         // 1 Byte Opcode
         sink.put1(self.opcode);
 
-        // 1 ModRM Byte
-        // Not all instructions use Reg as a reg, some use it as an extension of the opcode.
-        let rm: u8 = self.rm.into();
-        sink.put1(encode_modrm(3, self.reg & 7, rm & 7));
-
-        // TODO: 0/1 byte SIB
-        // TODO: 0/1/2/4 bytes DISP
+        match &self.rm {
+            // Not all instructions use Reg as a reg, some use it as an extension
+            // of the opcode.
+            RegisterOrAmode::Register(reg) => {
+                let rm: u8 = (*reg).into();
+                sink.put1(rex::encode_modrm(3, self.reg & 7, rm & 7));
+            }
+            // For address-based modes reuse the logic from the `rex` module
+            // for the modrm and trailing bytes since VEX uses the same
+            // encoding.
+            RegisterOrAmode::Amode(amode) => {
+                let bytes_at_end = if self.imm.is_some() { 1 } else { 0 };
+                rex::emit_modrm_sib_disp(sink, self.reg & 7, amode, bytes_at_end);
+            }
+        }
 
         // Optional 1 Byte imm
         if let Some(imm) = self.imm {
@@ -278,8 +326,9 @@ impl Default for VexVectorLength {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::isa::x64::inst::args::Gpr;
     use crate::isa::x64::inst::regs;
-    use std::vec::Vec;
+    use crate::opts::MemFlags;
 
     #[test]
     fn vpslldq() {
@@ -288,7 +337,7 @@ mod tests {
 
         let dst = regs::xmm1().to_real_reg().unwrap().hw_enc();
         let src = regs::xmm2().to_real_reg().unwrap().hw_enc();
-        let mut sink0 = Vec::new();
+        let mut sink = MachBuffer::new();
 
         VexInstruction::new()
             .length(VexVectorLength::V128)
@@ -299,9 +348,10 @@ mod tests {
             .vvvv(dst)
             .rm(src)
             .imm(0x17)
-            .encode(&mut sink0);
+            .encode(&mut sink);
 
-        assert_eq!(sink0, vec![0xc5, 0xf1, 0x73, 0xfa, 0x17]);
+        let bytes = sink.finish().data;
+        assert_eq!(bytes.as_slice(), [0xc5, 0xf1, 0x73, 0xfa, 0x17]);
     }
 
     #[test]
@@ -314,7 +364,7 @@ mod tests {
         let a = regs::xmm2().to_real_reg().unwrap().hw_enc();
         let b = regs::xmm3().to_real_reg().unwrap().hw_enc();
         let c = regs::xmm4().to_real_reg().unwrap().hw_enc();
-        let mut sink0 = Vec::new();
+        let mut sink = MachBuffer::new();
 
         VexInstruction::new()
             .length(VexVectorLength::V128)
@@ -326,9 +376,10 @@ mod tests {
             .vvvv(a)
             .rm(b)
             .imm_reg(c)
-            .encode(&mut sink0);
+            .encode(&mut sink);
 
-        assert_eq!(sink0, vec![0xc4, 0xe3, 0x69, 0x4b, 0xcb, 0x40]);
+        let bytes = sink.finish().data;
+        assert_eq!(bytes.as_slice(), [0xc4, 0xe3, 0x69, 0x4b, 0xcb, 0x40]);
     }
 
     #[test]
@@ -339,7 +390,7 @@ mod tests {
         let dst = regs::xmm10().to_real_reg().unwrap().hw_enc();
         let a = regs::xmm11().to_real_reg().unwrap().hw_enc();
         let b = regs::xmm12().to_real_reg().unwrap().hw_enc();
-        let mut sink0 = Vec::new();
+        let mut sink = MachBuffer::new();
 
         VexInstruction::new()
             .length(VexVectorLength::V256)
@@ -350,8 +401,91 @@ mod tests {
             .vvvv(a)
             .rm(b)
             .imm(4)
-            .encode(&mut sink0);
+            .encode(&mut sink);
+
+        let bytes = sink.finish().data;
+        assert_eq!(bytes.as_slice(), [0xc4, 0x41, 0x24, 0xc2, 0xd4, 0x04]);
+    }
+
+    #[test]
+    fn vandnps() {
+        // VEX.128.0F 55 /r
+        // VANDNPS xmm0, xmm1, xmm2
+
+        let dst = regs::xmm2().to_real_reg().unwrap().hw_enc();
+        let src1 = regs::xmm1().to_real_reg().unwrap().hw_enc();
+        let src2 = regs::xmm0().to_real_reg().unwrap().hw_enc();
+        let mut sink = MachBuffer::new();
+
+        VexInstruction::new()
+            .length(VexVectorLength::V128)
+            .prefix(LegacyPrefixes::None)
+            .map(OpcodeMap::_0F)
+            .opcode(0x55)
+            .reg(dst)
+            .vvvv(src1)
+            .rm(src2)
+            .encode(&mut sink);
+
+        let bytes = sink.finish().data;
+        assert_eq!(bytes.as_slice(), [0xc5, 0xf0, 0x55, 0xd0]);
+    }
+
+    #[test]
+    fn vandnps_mem() {
+        // VEX.128.0F 55 /r
+        // VANDNPS 10(%r13), xmm1, xmm2
+
+        let dst = regs::xmm2().to_real_reg().unwrap().hw_enc();
+        let src1 = regs::xmm1().to_real_reg().unwrap().hw_enc();
+        let src2 = Amode::ImmReg {
+            base: regs::r13(),
+            flags: MemFlags::trusted(),
+            simm32: 10,
+        };
+        let mut sink = MachBuffer::new();
+
+        VexInstruction::new()
+            .length(VexVectorLength::V128)
+            .prefix(LegacyPrefixes::None)
+            .map(OpcodeMap::_0F)
+            .opcode(0x55)
+            .reg(dst)
+            .vvvv(src1)
+            .rm(src2)
+            .encode(&mut sink);
+
+        let bytes = sink.finish().data;
+        assert_eq!(bytes.as_slice(), [0xc4, 0xc1, 0x70, 0x55, 0x55, 0x0a]);
+    }
+
+    #[test]
+    fn vandnps_more_mem() {
+        // VEX.128.0F 55 /r
+        // VANDNPS 100(%rax,%r13,4), xmm1, xmm2
+
+        let dst = regs::xmm2().to_real_reg().unwrap().hw_enc();
+        let src1 = regs::xmm1().to_real_reg().unwrap().hw_enc();
+        let src2 = Amode::ImmRegRegShift {
+            base: Gpr::new(regs::rax()).unwrap(),
+            index: Gpr::new(regs::r13()).unwrap(),
+            flags: MemFlags::trusted(),
+            simm32: 100,
+            shift: 2,
+        };
+        let mut sink = MachBuffer::new();
+
+        VexInstruction::new()
+            .length(VexVectorLength::V128)
+            .prefix(LegacyPrefixes::None)
+            .map(OpcodeMap::_0F)
+            .opcode(0x55)
+            .reg(dst)
+            .vvvv(src1)
+            .rm(src2)
+            .encode(&mut sink);
 
-        assert_eq!(sink0, vec![0xc4, 0x41, 0x24, 0xc2, 0xd4, 0x04]);
+        let bytes = sink.finish().data;
+        assert_eq!(bytes.as_slice(), [0xc4, 0xa1, 0x70, 0x55, 0x54, 0xa8, 100]);
     }
 }
diff --git a/cranelift/codegen/src/isa/x64/inst.isle b/cranelift/codegen/src/isa/x64/inst.isle
index cfc2afc61..f0f3c70d0 100644
--- a/cranelift/codegen/src/isa/x64/inst.isle
+++ b/cranelift/codegen/src/isa/x64/inst.isle
@@ -227,8 +227,24 @@
          (mask Xmm)
          (dst WritableXmm))
 
-       ;; XMM (scalar or vector) binary op that relies on the VEX prefix.
-       (XmmRmRVex (op AvxOpcode)
+       ;; XMM (scalar or vector) binary op that relies on the VEX prefix and
+       ;; has two inputs.
+       (XmmRmiRVex (op AvxOpcode)
+                   (src1 Xmm)
+                   (src2 XmmMemImm)
+                   (dst WritableXmm))
+
+       ;; XMM (scalar or vector) ternary op that relies on the VEX prefix and
+       ;; has two dynamic inputs plus one immediate input.
+       (XmmRmRImmVex (op AvxOpcode)
+                     (src1 Xmm)
+                     (src2 XmmMem)
+                     (dst WritableXmm)
+                     (imm u8))
+
+       ;; XMM (scalar or vector) ternary op that relies on the VEX prefix and
+       ;; has three dynamic inputs.
+       (XmmRmRVex3 (op AvxOpcode)
                    (src1 Xmm)
                    (src2 Xmm)
                    (src3 XmmMem)
@@ -1132,11 +1148,16 @@
 (decl cc_nz_or_z (CC) CC)
 (extern extractor cc_nz_or_z cc_nz_or_z)
 
-(type AvxOpcode extern
+(type AvxOpcode
       (enum Vfmadd213ss
             Vfmadd213sd
             Vfmadd213ps
-            Vfmadd213pd))
+            Vfmadd213pd
+            Vminps
+            Vorps
+            Vandnps
+            Vcmpps
+            Vpsrld))
 
 (type Avx512Opcode extern
       (enum Vcvtudq2ps
@@ -1226,6 +1247,10 @@
 (decl xmm_to_xmm_mem_imm (Xmm) XmmMemImm)
 (extern constructor xmm_to_xmm_mem_imm xmm_to_xmm_mem_imm)
 
+;; Convert an `XmmMem` into an `XmmMemImm`.
+(decl xmm_mem_to_xmm_mem_imm (XmmMem) XmmMemImm)
+(extern constructor xmm_mem_to_xmm_mem_imm xmm_mem_to_xmm_mem_imm)
+
 ;; Allocate a new temporary GPR register.
 (decl temp_writable_gpr () WritableGpr)
 (extern constructor temp_writable_gpr temp_writable_gpr)
@@ -1438,6 +1463,9 @@
 (decl use_sse41 (bool) Type)
 (extern extractor infallible use_sse41 use_sse41)
 
+(decl pure has_avx () bool)
+(extern constructor has_avx has_avx)
+
 ;;;; Helpers for Merging and Sinking Immediates/Loads  ;;;;;;;;;;;;;;;;;;;;;;;;;
 
 ;; Extract a constant `Imm8Reg.Imm8` from a value operand.
@@ -1455,6 +1483,10 @@
 (decl shift_mask (Type) u32)
 (extern constructor shift_mask shift_mask)
 
+;; Mask a constant with the type's shift mask
+(decl shift_amount_masked (Type Imm64) u32)
+(extern constructor shift_amount_masked shift_amount_masked)
+
 ;; Extract a constant `GprMemImm.Imm` from a value operand.
 (decl simm32_from_value (GprMemImm) Value)
 (extern extractor simm32_from_value simm32_from_value)
@@ -2281,8 +2313,11 @@
 
 ;; Helper for creating `orps` instructions.
 (decl x64_orps (Xmm XmmMem) Xmm)
-(rule (x64_orps src1 src2)
+(rule 0 (x64_orps src1 src2)
       (xmm_rm_r (SseOpcode.Orps) src1 src2))
+(rule 1 (x64_orps src1 src2)
+      (if-let $true (has_avx))
+      (xmm_rmir_vex (AvxOpcode.Vorps) src1 src2))
 
 ;; Helper for creating `orpd` instructions.
 (decl x64_orpd (Xmm XmmMem) Xmm)
@@ -2356,8 +2391,11 @@
 
 ;; Helper for creating `andnps` instructions.
 (decl x64_andnps (Xmm XmmMem) Xmm)
-(rule (x64_andnps src1 src2)
+(rule 0 (x64_andnps src1 src2)
       (xmm_rm_r (SseOpcode.Andnps) src1 src2))
+(rule 1 (x64_andnps src1 src2)
+      (if-let $true (has_avx))
+      (xmm_rmir_vex (AvxOpcode.Vandnps) src1 src2))
 
 ;; Helper for creating `andnpd` instructions.
 (decl x64_andnpd (Xmm XmmMem) Xmm)
@@ -2598,12 +2636,18 @@
 (rule (x64_cmpp $F64X2 x y imm) (x64_cmppd x y imm))
 
 (decl x64_cmpps (Xmm XmmMem FcmpImm) Xmm)
-(rule (x64_cmpps src1 src2 imm)
+(rule 0 (x64_cmpps src1 src2 imm)
       (xmm_rm_r_imm (SseOpcode.Cmpps)
                     src1
                     src2
                     (encode_fcmp_imm imm)
                     (OperandSize.Size32)))
+(rule 1 (x64_cmpps src1 src2 imm)
+      (if-let $true (has_avx))
+      (xmm_rmr_imm_vex (AvxOpcode.Vcmpps)
+                       src1
+                       src2
+                       (encode_fcmp_imm imm)))
 
 ;; Note that `Size32` is intentional despite this being used for 64-bit
 ;; operations, since this presumably induces the correct encoding of the
@@ -2854,8 +2898,11 @@
 
 ;; Helper for creating `psrld` instructions.
 (decl x64_psrld (Xmm XmmMemImm) Xmm)
-(rule (x64_psrld src1 src2)
+(rule 0 (x64_psrld src1 src2)
       (xmm_rmi_xmm (SseOpcode.Psrld) src1 src2))
+(rule 1 (x64_psrld src1 src2)
+      (if-let $true (has_avx))
+      (xmm_rmir_vex (AvxOpcode.Vpsrld) src1 src2))
 
 ;; Helper for creating `psrlq` instructions.
 (decl x64_psrlq (Xmm XmmMemImm) Xmm)
@@ -3066,10 +3113,11 @@
 
 ;; Helper for creating `minps` instructions.
 (decl x64_minps (Xmm Xmm) Xmm)
-(rule (x64_minps x y)
-      (let ((dst WritableXmm (temp_writable_xmm))
-            (_ Unit (emit (MInst.XmmRmR (SseOpcode.Minps) x y dst))))
-        dst))
+(rule 0 (x64_minps x y)
+      (xmm_rm_r (SseOpcode.Minps) x y))
+(rule 1 (x64_minps x y)
+      (if-let $true (has_avx))
+      (xmm_rmir_vex (AvxOpcode.Vminps) x y))
 
 ;; Helper for creating `minpd` instructions.
 (decl x64_minpd (Xmm Xmm) Xmm)
@@ -3097,15 +3145,25 @@
       (xmm_rm_r (SseOpcode.Maxpd) x y))
 
 
-;; Helper for creating `MInst.XmmRmRVex` instructions.
-(decl xmm_rmr_vex (AvxOpcode Xmm Xmm XmmMem) Xmm)
-(rule (xmm_rmr_vex op src1 src2 src3)
+;; Helper for creating `MInst.XmmRmiRVex` instructions.
+(decl xmm_rmir_vex (AvxOpcode Xmm XmmMemImm) Xmm)
+(rule (xmm_rmir_vex op src1 src2)
       (let ((dst WritableXmm (temp_writable_xmm))
-            (_ Unit (emit (MInst.XmmRmRVex op
-                                           src1
-                                           src2
-                                           src3
-                                           dst))))
+            (_ Unit (emit (MInst.XmmRmiRVex op src1 src2 dst))))
+        dst))
+
+;; Helper for creating `MInst.XmmRmRImmVex` instructions.
+(decl xmm_rmr_imm_vex (AvxOpcode Xmm XmmMem u8) Xmm)
+(rule (xmm_rmr_imm_vex op src1 src2 imm)
+      (let ((dst WritableXmm (temp_writable_xmm))
+            (_ Unit (emit (MInst.XmmRmRImmVex op src1 src2 dst imm))))
+        dst))
+
+;; Helper for creating `MInst.XmmRmRVex3` instructions.
+(decl xmm_rmr_vex3 (AvxOpcode Xmm Xmm XmmMem) Xmm)
+(rule (xmm_rmr_vex3 op src1 src2 src3)
+      (let ((dst WritableXmm (temp_writable_xmm))
+            (_ Unit (emit (MInst.XmmRmRVex3 op src1 src2 src3 dst))))
         dst))
 
 ;; Helper for creating `vfmadd213ss` instructions.
@@ -3113,28 +3171,28 @@
 ; but we don't support VEX memory encodings yet
 (decl x64_vfmadd213ss (Xmm Xmm Xmm) Xmm)
 (rule (x64_vfmadd213ss x y z)
-      (xmm_rmr_vex (AvxOpcode.Vfmadd213ss) x y z))
+      (xmm_rmr_vex3 (AvxOpcode.Vfmadd213ss) x y z))
 
 ;; Helper for creating `vfmadd213sd` instructions.
 ; TODO: This should have the (Xmm Xmm XmmMem) signature
 ; but we don't support VEX memory encodings yet
 (decl x64_vfmadd213sd (Xmm Xmm Xmm) Xmm)
 (rule (x64_vfmadd213sd x y z)
-      (xmm_rmr_vex (AvxOpcode.Vfmadd213sd) x y z))
+      (xmm_rmr_vex3 (AvxOpcode.Vfmadd213sd) x y z))
 
 ;; Helper for creating `vfmadd213ps` instructions.
 ; TODO: This should have the (Xmm Xmm XmmMem) signature
 ; but we don't support VEX memory encodings yet
 (decl x64_vfmadd213ps (Xmm Xmm Xmm) Xmm)
 (rule (x64_vfmadd213ps x y z)
-      (xmm_rmr_vex (AvxOpcode.Vfmadd213ps) x y z))
+      (xmm_rmr_vex3 (AvxOpcode.Vfmadd213ps) x y z))
 
 ;; Helper for creating `vfmadd213pd` instructions.
 ; TODO: This should have the (Xmm Xmm XmmMem) signature
 ; but we don't support VEX memory encodings yet
 (decl x64_vfmadd213pd (Xmm Xmm Xmm) Xmm)
 (rule (x64_vfmadd213pd x y z)
-      (xmm_rmr_vex (AvxOpcode.Vfmadd213pd) x y z))
+      (xmm_rmr_vex3 (AvxOpcode.Vfmadd213pd) x y z))
 
 
 ;; Helper for creating `sqrtss` instructions.
@@ -3832,6 +3890,7 @@
 (convert RegMemImm XmmMemImm mov_rmi_to_xmm)
 (convert Xmm XmmMem xmm_to_xmm_mem)
 (convert Xmm XmmMemImm xmm_to_xmm_mem_imm)
+(convert XmmMem XmmMemImm xmm_mem_to_xmm_mem_imm)
 (convert XmmMem RegMem xmm_mem_to_reg_mem)
 (convert WritableXmm Xmm writable_xmm_to_xmm)
 (convert WritableXmm WritableReg writable_xmm_to_reg)
diff --git a/cranelift/codegen/src/isa/x64/inst/args.rs b/cranelift/codegen/src/isa/x64/inst/args.rs
index 76787bfb6..c4de650e4 100644
--- a/cranelift/codegen/src/isa/x64/inst/args.rs
+++ b/cranelift/codegen/src/isa/x64/inst/args.rs
@@ -165,6 +165,12 @@ macro_rules! newtype_of_reg {
             }
         }
 
+        impl From<$newtype_reg_mem> for $newtype_reg_mem_imm {
+            fn from(r: $newtype_reg_mem) -> Self {
+                $newtype_reg_mem_imm(r.0.into())
+            }
+        }
+
         impl $newtype_reg_mem_imm {
             /// Construct this newtype from the given `RegMemImm`, or return
             /// `None` if the `RegMemImm` is not a valid instance of this
@@ -631,6 +637,15 @@ impl RegMemImm {
     }
 }
 
+impl From<RegMem> for RegMemImm {
+    fn from(rm: RegMem) -> RegMemImm {
+        match rm {
+            RegMem::Reg { reg } => RegMemImm::Reg { reg },
+            RegMem::Mem { addr } => RegMemImm::Mem { addr },
+        }
+    }
+}
+
 impl PrettyPrint for RegMemImm {
     fn pretty_print(&self, size: u8, allocs: &mut AllocationConsumer<'_>) -> String {
         match self {
@@ -730,6 +745,12 @@ impl RegMem {
     }
 }
 
+impl From<Reg> for RegMem {
+    fn from(reg: Reg) -> RegMem {
+        RegMem::Reg { reg }
+    }
+}
+
 impl From<Writable<Reg>> for RegMem {
     fn from(r: Writable<Reg>) -> Self {
         RegMem::reg(r.to_reg())
@@ -884,6 +905,7 @@ pub(crate) enum InstructionSet {
     #[allow(dead_code)] // never constructed (yet).
     BMI2,
     FMA,
+    AVX,
     AVX512BITALG,
     AVX512DQ,
     AVX512F,
@@ -1477,14 +1499,7 @@ impl fmt::Display for SseOpcode {
     }
 }
 
-#[derive(Clone, PartialEq)]
-#[allow(missing_docs)]
-pub enum AvxOpcode {
-    Vfmadd213ss,
-    Vfmadd213sd,
-    Vfmadd213ps,
-    Vfmadd213pd,
-}
+pub use crate::isa::x64::lower::isle::generated_code::AvxOpcode;
 
 impl AvxOpcode {
     /// Which `InstructionSet`s support the opcode?
@@ -1494,25 +1509,20 @@ impl AvxOpcode {
             | AvxOpcode::Vfmadd213sd
             | AvxOpcode::Vfmadd213ps
             | AvxOpcode::Vfmadd213pd => smallvec![InstructionSet::FMA],
+            AvxOpcode::Vminps
+            | AvxOpcode::Vorps
+            | AvxOpcode::Vandnps
+            | AvxOpcode::Vcmpps
+            | AvxOpcode::Vpsrld => {
+                smallvec![InstructionSet::AVX]
+            }
         }
     }
 }
 
-impl fmt::Debug for AvxOpcode {
-    fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
-        let name = match self {
-            AvxOpcode::Vfmadd213ss => "vfmadd213ss",
-            AvxOpcode::Vfmadd213sd => "vfmadd213sd",
-            AvxOpcode::Vfmadd213ps => "vfmadd213ps",
-            AvxOpcode::Vfmadd213pd => "vfmadd213pd",
-        };
-        write!(fmt, "{}", name)
-    }
-}
-
 impl fmt::Display for AvxOpcode {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        fmt::Debug::fmt(self, f)
+        format!("{self:?}").to_lowercase().fmt(f)
     }
 }
 
diff --git a/cranelift/codegen/src/isa/x64/inst/emit.rs b/cranelift/codegen/src/isa/x64/inst/emit.rs
index 830565eff..9d984edab 100644
--- a/cranelift/codegen/src/isa/x64/inst/emit.rs
+++ b/cranelift/codegen/src/isa/x64/inst/emit.rs
@@ -8,7 +8,7 @@ use crate::isa::x64::encoding::rex::{
     low8_will_sign_extend_to_32, low8_will_sign_extend_to_64, reg_enc, LegacyPrefixes, OpcodeMap,
     RexFlags,
 };
-use crate::isa::x64::encoding::vex::{VexInstruction, VexVectorLength};
+use crate::isa::x64::encoding::vex::{RegisterOrAmode, VexInstruction, VexVectorLength};
 use crate::isa::x64::inst::args::*;
 use crate::isa::x64::inst::*;
 use crate::machinst::{inst_common, MachBuffer, MachInstEmit, MachLabel, Reg, Writable};
@@ -121,6 +121,7 @@ pub(crate) fn emit(
             InstructionSet::BMI1 => info.isa_flags.use_bmi1(),
             InstructionSet::BMI2 => info.isa_flags.has_bmi2(),
             InstructionSet::FMA => info.isa_flags.has_fma(),
+            InstructionSet::AVX => info.isa_flags.has_avx(),
             InstructionSet::AVX512BITALG => info.isa_flags.has_avx512bitalg(),
             InstructionSet::AVX512DQ => info.isa_flags.has_avx512dq(),
             InstructionSet::AVX512F => info.isa_flags.has_avx512f(),
@@ -1991,7 +1992,94 @@ pub(crate) fn emit(
             }
         }
 
-        Inst::XmmRmRVex {
+        Inst::XmmRmiRVex {
+            op,
+            src1,
+            src2,
+            dst,
+        } => {
+            let dst = allocs.next(dst.to_reg().to_reg());
+            let src1 = allocs.next(src1.to_reg());
+            let src2 = src2.clone().to_reg_mem_imm().with_allocs(allocs);
+
+            let src2 = match src2 {
+                // For opcodes where one of the operands is an immediate the
+                // encoding is a bit different, notably the usage of
+                // `opcode_ext`, so handle that specially here.
+                RegMemImm::Imm { simm32 } => {
+                    let (opcode, opcode_ext, prefix) = match op {
+                        AvxOpcode::Vpsrld => (0x72, 2, LegacyPrefixes::_66),
+                        _ => panic!("unexpected avx opcode with immediate {op:?}"),
+                    };
+                    VexInstruction::new()
+                        .length(VexVectorLength::V128)
+                        .prefix(prefix)
+                        .map(OpcodeMap::_0F)
+                        .opcode(opcode)
+                        .opcode_ext(opcode_ext)
+                        .vvvv(dst.to_real_reg().unwrap().hw_enc())
+                        .prefix(LegacyPrefixes::_66)
+                        .rm(src1.to_real_reg().unwrap().hw_enc())
+                        .imm(simm32.try_into().unwrap())
+                        .encode(sink);
+                    return;
+                }
+                RegMemImm::Reg { reg } => {
+                    RegisterOrAmode::Register(reg.to_real_reg().unwrap().hw_enc().into())
+                }
+                RegMemImm::Mem { addr } => RegisterOrAmode::Amode(addr.finalize(state, sink)),
+            };
+            let (prefix, opcode) = match op {
+                AvxOpcode::Vminps => (LegacyPrefixes::None, 0x5D),
+                AvxOpcode::Vandnps => (LegacyPrefixes::None, 0x55),
+                AvxOpcode::Vorps => (LegacyPrefixes::None, 0x56),
+                AvxOpcode::Vpsrld => (LegacyPrefixes::_66, 0xD2),
+                _ => panic!("unexpected rmir vex opcode {op:?}"),
+            };
+            VexInstruction::new()
+                .length(VexVectorLength::V128)
+                .prefix(prefix)
+                .opcode(opcode)
+                .map(OpcodeMap::_0F)
+                .reg(dst.to_real_reg().unwrap().hw_enc())
+                .vvvv(src1.to_real_reg().unwrap().hw_enc())
+                .rm(src2)
+                .encode(sink);
+        }
+
+        Inst::XmmRmRImmVex {
+            op,
+            src1,
+            src2,
+            dst,
+            imm,
+        } => {
+            let dst = allocs.next(dst.to_reg().to_reg());
+            let src1 = allocs.next(src1.to_reg());
+            let src2 = src2.clone().to_reg_mem().with_allocs(allocs);
+
+            let (w, opcode) = match op {
+                AvxOpcode::Vcmpps => (false, 0xC2),
+                _ => unreachable!(),
+            };
+
+            match src2 {
+                RegMem::Reg { reg: src } => VexInstruction::new()
+                    .length(VexVectorLength::V128)
+                    .prefix(LegacyPrefixes::None)
+                    .map(OpcodeMap::_0F)
+                    .w(w)
+                    .opcode(opcode)
+                    .reg(dst.to_real_reg().unwrap().hw_enc())
+                    .rm(src.to_real_reg().unwrap().hw_enc())
+                    .vvvv(src1.to_real_reg().unwrap().hw_enc())
+                    .imm(*imm)
+                    .encode(sink),
+                _ => todo!(),
+            };
+        }
+
+        Inst::XmmRmRVex3 {
             op,
             src1,
             src2,
@@ -2009,6 +2097,7 @@ pub(crate) fn emit(
                 AvxOpcode::Vfmadd213sd => (true, 0xA9),
                 AvxOpcode::Vfmadd213ps => (false, 0xA8),
                 AvxOpcode::Vfmadd213pd => (true, 0xA8),
+                _ => unreachable!(),
             };
 
             match src3 {
diff --git a/cranelift/codegen/src/isa/x64/inst/mod.rs b/cranelift/codegen/src/isa/x64/inst/mod.rs
index 6716ec21b..a1056b859 100644
--- a/cranelift/codegen/src/isa/x64/inst/mod.rs
+++ b/cranelift/codegen/src/isa/x64/inst/mod.rs
@@ -142,7 +142,9 @@ impl Inst {
             | Inst::XmmRmREvex { op, .. }
             | Inst::XmmRmREvex3 { op, .. } => op.available_from(),
 
-            Inst::XmmRmRVex { op, .. } => op.available_from(),
+            Inst::XmmRmiRVex { op, .. }
+            | Inst::XmmRmRVex3 { op, .. }
+            | Inst::XmmRmRImmVex { op, .. } => op.available_from(),
         }
     }
 }
@@ -303,11 +305,11 @@ impl Inst {
     }
 
     #[cfg(test)]
-    pub(crate) fn xmm_rm_r_vex(op: AvxOpcode, src3: RegMem, src2: Reg, dst: Writable<Reg>) -> Self {
+    pub(crate) fn xmm_rmr_vex3(op: AvxOpcode, src3: RegMem, src2: Reg, dst: Writable<Reg>) -> Self {
         src3.assert_regclass_is(RegClass::Float);
         debug_assert!(src2.class() == RegClass::Float);
         debug_assert!(dst.to_reg().class() == RegClass::Float);
-        Inst::XmmRmRVex {
+        Inst::XmmRmRVex3 {
             op,
             src3: XmmMem::new(src3).unwrap(),
             src2: Xmm::new(src2).unwrap(),
@@ -988,7 +990,42 @@ impl PrettyPrint for Inst {
                 )
             }
 
-            Inst::XmmRmRVex {
+            Inst::XmmRmiRVex {
+                op,
+                src1,
+                src2,
+                dst,
+                ..
+            } => {
+                let dst = pretty_print_reg(dst.to_reg().to_reg(), 8, allocs);
+                let src1 = pretty_print_reg(src1.to_reg(), 8, allocs);
+                let src2 = src2.pretty_print(8, allocs);
+
+                format!("{} {}, {}, {}", ljustify(op.to_string()), src1, src2, dst)
+            }
+
+            Inst::XmmRmRImmVex {
+                op,
+                src1,
+                src2,
+                dst,
+                imm,
+                ..
+            } => {
+                let dst = pretty_print_reg(dst.to_reg().to_reg(), 8, allocs);
+                let src1 = pretty_print_reg(src1.to_reg(), 8, allocs);
+                let src2 = src2.pretty_print(8, allocs);
+
+                format!(
+                    "{} ${imm} {}, {}, {}",
+                    ljustify(op.to_string()),
+                    src1,
+                    src2,
+                    dst
+                )
+            }
+
+            Inst::XmmRmRVex3 {
                 op,
                 src1,
                 src2,
@@ -1892,7 +1929,21 @@ fn x64_get_operands<F: Fn(VReg) -> VReg>(inst: &Inst, collector: &mut OperandCol
             collector.reg_reuse_def(dst.to_writable_reg(), 0);
             src2.get_operands(collector);
         }
-        Inst::XmmRmRVex {
+        Inst::XmmRmiRVex {
+            src1, src2, dst, ..
+        } => {
+            collector.reg_def(dst.to_writable_reg());
+            collector.reg_use(src1.to_reg());
+            src2.get_operands(collector);
+        }
+        Inst::XmmRmRImmVex {
+            src1, src2, dst, ..
+        } => {
+            collector.reg_def(dst.to_writable_reg());
+            collector.reg_use(src1.to_reg());
+            src2.get_operands(collector);
+        }
+        Inst::XmmRmRVex3 {
             op,
             src1,
             src2,
diff --git a/cranelift/codegen/src/isa/x64/lower.isle b/cranelift/codegen/src/isa/x64/lower.isle
index e8fd01f84..1275ab2ad 100644
--- a/cranelift/codegen/src/isa/x64/lower.isle
+++ b/cranelift/codegen/src/isa/x64/lower.isle
@@ -473,7 +473,7 @@
 (rule (lower (has_type ty @ $I8X16 (ishl src amt)))
       (let (
             ;; Mask the amount to ensure wrapping behaviour
-            (masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty))))
+            (masked_amt RegMemImm (mask_xmm_shift ty amt))
             ;; Shift `src` using 16x8. Unfortunately, a 16x8 shift will only be
             ;; correct for half of the lanes; the others must be fixed up with
             ;; the mask below.
@@ -515,16 +515,13 @@
 ;; 16x8, 32x4, and 64x2 shifts can each use a single instruction, once the shift amount is masked.
 
 (rule (lower (has_type ty @ $I16X8 (ishl src amt)))
-      (let ((masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty)))))
-        (x64_psllw src (mov_rmi_to_xmm masked_amt))))
+      (x64_psllw src (mov_rmi_to_xmm (mask_xmm_shift ty amt))))
 
 (rule (lower (has_type ty @ $I32X4 (ishl src amt)))
-      (let ((masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty)))))
-        (x64_pslld src (mov_rmi_to_xmm masked_amt))))
+      (x64_pslld src (mov_rmi_to_xmm (mask_xmm_shift ty amt))))
 
 (rule (lower (has_type ty @ $I64X2 (ishl src amt)))
-      (let ((masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty)))))
-        (x64_psllq src (mov_rmi_to_xmm masked_amt))))
+      (x64_psllq src (mov_rmi_to_xmm (mask_xmm_shift ty amt))))
 
 ;;;; Rules for `ushr` ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
@@ -580,7 +577,7 @@
 (rule (lower (has_type ty @ $I8X16 (ushr src amt)))
       (let (
             ;; Mask the amount to ensure wrapping behaviour
-            (masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty))))
+            (masked_amt RegMemImm (mask_xmm_shift ty amt))
             ;; Shift `src` using 16x8. Unfortunately, a 16x8 shift will only be
             ;; correct for half of the lanes; the others must be fixed up with
             ;; the mask below.
@@ -625,16 +622,19 @@
 ;; 16x8, 32x4, and 64x2 shifts can each use a single instruction, once the shift amount is masked.
 
 (rule (lower (has_type ty @ $I16X8 (ushr src amt)))
-      (let ((masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty)))))
-        (x64_psrlw src (mov_rmi_to_xmm masked_amt))))
+      (x64_psrlw src (mov_rmi_to_xmm (mask_xmm_shift ty amt))))
 
 (rule (lower (has_type ty @ $I32X4 (ushr src amt)))
-      (let ((masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty)))))
-        (x64_psrld src (mov_rmi_to_xmm masked_amt))))
+      (x64_psrld src (mov_rmi_to_xmm (mask_xmm_shift ty amt))))
 
 (rule (lower (has_type ty @ $I64X2 (ushr src amt)))
-      (let ((masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty)))))
-        (x64_psrlq src (mov_rmi_to_xmm masked_amt))))
+      (x64_psrlq src (mov_rmi_to_xmm (mask_xmm_shift ty amt))))
+
+(decl mask_xmm_shift (Type Value) RegMemImm)
+(rule (mask_xmm_shift ty amt)
+      (gpr_to_reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty)))))
+(rule 1 (mask_xmm_shift ty (iconst n))
+      (RegMemImm.Imm (shift_amount_masked ty n)))
 
 ;;;; Rules for `sshr` ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
@@ -701,7 +701,7 @@
 (rule (lower (has_type ty @ $I8X16 (sshr src amt @ (value_type amt_ty))))
       (let ((src_ Xmm (put_in_xmm src))
             ;; Mask the amount to ensure wrapping behaviour
-            (masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty))))
+            (masked_amt RegMemImm (mask_xmm_shift ty amt))
             ;; In order for `packsswb` later to only use the high byte of each
             ;; 16x8 lane, we shift right an extra 8 bits, relying on `psraw` to
             ;; fill in the upper bits appropriately.
@@ -728,12 +728,10 @@
 ;; that if the shift amount is in a register, it is in an XMM register.
 
 (rule (lower (has_type ty @ $I16X8 (sshr src amt)))
-      (let ((masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty)))))
-        (x64_psraw src (mov_rmi_to_xmm masked_amt))))
+      (x64_psraw src (mov_rmi_to_xmm (mask_xmm_shift ty amt))))
 
 (rule (lower (has_type ty @ $I32X4 (sshr src amt)))
-      (let ((masked_amt Reg (x64_and $I64 amt (RegMemImm.Imm (shift_mask ty)))))
-        (x64_psrad src (mov_rmi_to_xmm masked_amt))))
+      (x64_psrad src (mov_rmi_to_xmm (mask_xmm_shift ty amt))))
 
 ;; The `sshr.i64x2` CLIF instruction has no single x86 instruction in the older
 ;; feature sets. Newer ones like AVX512VL + AVX512F include `vpsraq`, a 128-bit
diff --git a/cranelift/codegen/src/isa/x64/lower/isle.rs b/cranelift/codegen/src/isa/x64/lower/isle.rs
index 7d97d761e..8adbf9446 100644
--- a/cranelift/codegen/src/isa/x64/lower/isle.rs
+++ b/cranelift/codegen/src/isa/x64/lower/isle.rs
@@ -185,6 +185,11 @@ impl Context for IsleContext<'_, '_, MInst, X64Backend> {
         imm.encode()
     }
 
+    #[inline]
+    fn has_avx(&mut self) -> bool {
+        self.backend.x64_flags.has_avx()
+    }
+
     #[inline]
     fn avx512vl_enabled(&mut self, _: Type) -> bool {
         self.backend.x64_flags.use_avx512vl_simd()
@@ -259,6 +264,10 @@ impl Context for IsleContext<'_, '_, MInst, X64Backend> {
         ty.lane_bits() - 1
     }
 
+    fn shift_amount_masked(&mut self, ty: Type, val: Imm64) -> u32 {
+        (val.bits() as u32) & self.shift_mask(ty)
+    }
+
     #[inline]
     fn simm32_from_value(&mut self, val: Value) -> Option<GprMemImm> {
         let inst = self.lower_ctx.dfg().value_def(val).inst()?;
@@ -435,6 +444,11 @@ impl Context for IsleContext<'_, '_, MInst, X64Backend> {
         r.into()
     }
 
+    #[inline]
+    fn xmm_mem_to_xmm_mem_imm(&mut self, r: &XmmMem) -> XmmMemImm {
+        r.clone().into()
+    }
+
     #[inline]
     fn temp_writable_gpr(&mut self) -> WritableGpr {
         Writable::from_reg(Gpr::new(self.temp_writable_reg(I64).to_reg()).unwrap())
diff --git a/cranelift/codegen/src/machinst/lower.rs b/cranelift/codegen/src/machinst/lower.rs
index 5115c5b01..c199aa15a 100644
--- a/cranelift/codegen/src/machinst/lower.rs
+++ b/cranelift/codegen/src/machinst/lower.rs
@@ -10,8 +10,8 @@ use crate::fx::{FxHashMap, FxHashSet};
 use crate::inst_predicates::{has_lowering_side_effect, is_constant_64bit};
 use crate::ir::{
     ArgumentPurpose, Block, Constant, ConstantData, DataFlowGraph, ExternalName, Function,
-    GlobalValue, GlobalValueData, Immediate, Inst, InstructionData, MemFlags, Opcode, RelSourceLoc,
-    Type, Value, ValueDef, ValueLabelAssignments, ValueLabelStart,
+    GlobalValue, GlobalValueData, Immediate, Inst, InstructionData, MemFlags, RelSourceLoc, Type,
+    Value, ValueDef, ValueLabelAssignments, ValueLabelStart,
 };
 use crate::machinst::{
     writable_value_regs, BlockIndex, BlockLoweringOrder, Callee, LoweredBlock, MachLabel, Reg,
@@ -898,39 +898,29 @@ impl<'func, I: VCodeInst> Lower<'func, I> {
         bindex: BlockIndex,
         // Original CLIF block:
         block: Block,
-        branches: &SmallVec<[Inst; 2]>,
-        targets: &SmallVec<[MachLabel; 2]>,
+        branch: Inst,
+        targets: &[MachLabel],
     ) -> CodegenResult<()> {
         trace!(
-            "lower_clif_branches: block {} branches {:?} targets {:?}",
+            "lower_clif_branches: block {} branch {:?} targets {:?}",
             block,
-            branches,
+            branch,
             targets,
         );
-        // A block should end with at most two branches. The first may be a
-        // conditional branch; a conditional branch can be followed only by an
-        // unconditional branch or fallthrough. Otherwise, if only one branch,
-        // it may be an unconditional branch, a fallthrough, a return, or a
-        // trap. These conditions are verified by `is_block_basic()` during the
-        // verifier pass.
-        assert!(branches.len() <= 2);
-        if branches.len() == 2 {
-            assert!(self.data(branches[1]).opcode() == Opcode::Jump);
-        }
         // When considering code-motion opportunities, consider the current
-        // program point to be the first branch.
-        self.cur_inst = Some(branches[0]);
-        // Lower the first branch in ISLE.  This will automatically handle
-        // the second branch (if any) by emitting a two-way conditional branch.
+        // program point to be this branch.
+        self.cur_inst = Some(branch);
+
+        // Lower the branch in ISLE.
         backend
-            .lower_branch(self, branches[0], targets)
+            .lower_branch(self, branch, targets)
             .unwrap_or_else(|| {
                 panic!(
                     "should be implemented in ISLE: branch = `{}`",
-                    self.f.dfg.display_inst(branches[0]),
+                    self.f.dfg.display_inst(branch),
                 )
             });
-        let loc = self.srcloc(branches[0]);
+        let loc = self.srcloc(branch);
         self.finish_ir_inst(loc);
         // Add block param outputs for current block.
         self.lower_branch_blockparam_args(bindex);
@@ -967,26 +957,20 @@ impl<'func, I: VCodeInst> Lower<'func, I> {
         &self,
         bindex: BlockIndex,
         _bb: Block,
-        branches: &mut SmallVec<[Inst; 2]>,
         targets: &mut SmallVec<[MachLabel; 2]>,
-    ) {
-        branches.clear();
+    ) -> Option<Inst> {
         targets.clear();
         let mut last_inst = None;
         for &(inst, succ) in self.vcode.block_order().succ_indices(bindex) {
-            // Avoid duplicates: this ensures a br_table is only inserted once.
-            if last_inst != Some(inst) {
-                branches.push(inst);
-            } else {
-                debug_assert!(
-                    self.f.dfg.insts[inst].opcode() == Opcode::BrTable
-                        || self.f.dfg.insts[inst].opcode() == Opcode::Brif
-                );
-                debug_assert!(branches.len() == 1);
-            }
+            // Basic blocks may end in a single branch instruction, but those instructions may have
+            // multiple destinations. As such, all `inst` values in `succ_indices` must be the
+            // same, or this basic block would have multiple branch instructions present.
+            debug_assert!(last_inst.map_or(true, |prev| prev == inst));
             last_inst = Some(inst);
             targets.push(MachLabel::from_block(succ));
         }
+
+        last_inst
     }
 
     /// Lower the function.
@@ -1010,7 +994,6 @@ impl<'func, I: VCodeInst> Lower<'func, I> {
         self.vcode.set_entry(BlockIndex::new(0));
 
         // Reused vectors for branch lowering.
-        let mut branches: SmallVec<[Inst; 2]> = SmallVec::new();
         let mut targets: SmallVec<[MachLabel; 2]> = SmallVec::new();
 
         // get a copy of the lowered order; we hold this separately because we
@@ -1032,10 +1015,9 @@ impl<'func, I: VCodeInst> Lower<'func, I> {
 
             // End branches.
             if let Some(bb) = lb.orig_block() {
-                self.collect_branches_and_targets(bindex, bb, &mut branches, &mut targets);
-                if branches.len() > 0 {
-                    self.lower_clif_branches(backend, bindex, bb, &branches, &targets)?;
-                    self.finish_ir_inst(self.srcloc(branches[0]));
+                if let Some(branch) = self.collect_branches_and_targets(bindex, bb, &mut targets) {
+                    self.lower_clif_branches(backend, bindex, bb, branch, &targets)?;
+                    self.finish_ir_inst(self.srcloc(branch));
                 }
             } else {
                 // If no orig block, this must be a pure edge block;


Test patch:
diff --git a/ci/build-test-matrix.js b/ci/build-test-matrix.js
index c02752c5d..d0dd51b93 100644
--- a/ci/build-test-matrix.js
+++ b/ci/build-test-matrix.js
@@ -36,7 +36,8 @@ const array = [
   {
     "os": "ubuntu-latest",
     "name": "Test Linux x86_64",
-    "filter": "linux-x64"
+    "filter": "linux-x64",
+    "isa": "x64"
   },
   {
     "os": "macos-latest",
diff --git a/cranelift/codegen/src/isa/x64/inst/emit_tests.rs b/cranelift/codegen/src/isa/x64/inst/emit_tests.rs
index 3b9250329..f63eb3e8e 100644
--- a/cranelift/codegen/src/isa/x64/inst/emit_tests.rs
+++ b/cranelift/codegen/src/isa/x64/inst/emit_tests.rs
@@ -3964,25 +3964,25 @@ fn test_x64_emit() {
     // XMM FMA
 
     insns.push((
-        Inst::xmm_rm_r_vex(AvxOpcode::Vfmadd213ss, RegMem::reg(xmm2), xmm1, w_xmm0),
+        Inst::xmm_rmr_vex3(AvxOpcode::Vfmadd213ss, RegMem::reg(xmm2), xmm1, w_xmm0),
         "C4E271A9C2",
         "vfmadd213ss %xmm0, %xmm1, %xmm2, %xmm0",
     ));
 
     insns.push((
-        Inst::xmm_rm_r_vex(AvxOpcode::Vfmadd213sd, RegMem::reg(xmm5), xmm4, w_xmm3),
+        Inst::xmm_rmr_vex3(AvxOpcode::Vfmadd213sd, RegMem::reg(xmm5), xmm4, w_xmm3),
         "C4E2D9A9DD",
         "vfmadd213sd %xmm3, %xmm4, %xmm5, %xmm3",
     ));
 
     insns.push((
-        Inst::xmm_rm_r_vex(AvxOpcode::Vfmadd213ps, RegMem::reg(xmm2), xmm1, w_xmm0),
+        Inst::xmm_rmr_vex3(AvxOpcode::Vfmadd213ps, RegMem::reg(xmm2), xmm1, w_xmm0),
         "C4E271A8C2",
         "vfmadd213ps %xmm0, %xmm1, %xmm2, %xmm0",
     ));
 
     insns.push((
-        Inst::xmm_rm_r_vex(AvxOpcode::Vfmadd213pd, RegMem::reg(xmm5), xmm4, w_xmm3),
+        Inst::xmm_rmr_vex3(AvxOpcode::Vfmadd213pd, RegMem::reg(xmm5), xmm4, w_xmm3),
         "C4E2D9A8DD",
         "vfmadd213pd %xmm3, %xmm4, %xmm5, %xmm3",
     ));
diff --git a/cranelift/filetests/filetests/isa/x64/simd-bitselect.clif b/cranelift/filetests/filetests/isa/x64/simd-bitselect.clif
index ea1f1f913..ce4e8d0e8 100644
--- a/cranelift/filetests/filetests/isa/x64/simd-bitselect.clif
+++ b/cranelift/filetests/filetests/isa/x64/simd-bitselect.clif
@@ -1,6 +1,6 @@
 test compile precise-output
 set enable_simd
-target x86_64 skylake
+target x86_64
 
 function %mask_from_icmp(i8x16, i8x16) -> i8x16 {
 block0(v0: i8x16, v1: i8x16):
diff --git a/cranelift/filetests/filetests/isa/x64/simd-bitwise-avx.clif b/cranelift/filetests/filetests/isa/x64/simd-bitwise-avx.clif
new file mode 100644
index 000000000..eeb096b3e
--- /dev/null
+++ b/cranelift/filetests/filetests/isa/x64/simd-bitwise-avx.clif
@@ -0,0 +1,228 @@
+test compile precise-output
+set enable_simd
+target x86_64 has_avx
+
+function %mask_from_icmp(f32x4, f32x4) -> f32x4 {
+block0(v0: f32x4, v1: f32x4):
+    v2 = fmin v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   vminps  %xmm0, %xmm1, %xmm3
+;   vminps  %xmm1, %xmm0, %xmm5
+;   vorps   %xmm3, %xmm5, %xmm7
+;   vcmpps  $3 %xmm7, %xmm5, %xmm9
+;   vorps   %xmm7, %xmm9, %xmm11
+;   vpsrld  %xmm9, $10, %xmm13
+;   vandnps %xmm13, %xmm11, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   vminps %xmm1, %xmm0, %xmm3
+;   vminps %xmm0, %xmm1, %xmm5
+;   vorps %xmm5, %xmm3, %xmm7
+;   vcmpunordps %xmm5, %xmm7, %xmm9
+;   vorps %xmm9, %xmm7, %xmm11
+;   vpsrld $0xa, %xmm9, %xmm13
+;   vandnps %xmm11, %xmm13, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %or_from_memory(f32x4, i64) -> f32x4 {
+block0(v0: f32x4, v1: i64):
+    v2 = load.f32x4 notrap aligned v1
+    v3 = bor v0, v2
+    return v3
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   movups  0(%rdi), %xmm4
+;   vorps   %xmm0, %xmm4, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   movups (%rdi), %xmm4
+;   vorps %xmm4, %xmm0, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %copysign_from_memory(i64) -> f32 {
+block0(v0: i64):
+    v1 = f32const 0.0
+    v2 = load.f32 notrap aligned v0
+    v3 = fcopysign v1, v2
+    return v3
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   movss   0(%rdi), %xmm7
+;   movl    $-2147483648, %ecx
+;   movd    %ecx, %xmm8
+;   vandnps %xmm8, const(0), %xmm9
+;   andps   %xmm8, %xmm7, %xmm8
+;   vorps   %xmm9, %xmm8, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   movss (%rdi), %xmm7
+;   movl $0x80000000, %ecx
+;   movd %ecx, %xmm8
+;   vandnps 0x16(%rip), %xmm8, %xmm9
+;   andps %xmm7, %xmm8
+;   vorps %xmm8, %xmm9, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+
+function %bor_f32x4(f32x4, f32x4) -> f32x4 {
+block0(v0: f32x4, v1: f32x4):
+    v2 = bor v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   vorps   %xmm0, %xmm1, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   vorps %xmm1, %xmm0, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %band_not_f32x4(f32x4, f32x4) -> f32x4 {
+block0(v0: f32x4, v1: f32x4):
+    v2 = band_not v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   vandnps %xmm1, %xmm0, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   vandnps %xmm0, %xmm1, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %i32x4_shr(i32x4, i32) -> i32x4 {
+block0(v0: i32x4, v1: i32):
+    v2 = ushr v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   movq    %rdi, %rcx
+;   andq    %rcx, $31, %rcx
+;   movd    %ecx, %xmm5
+;   vpsrld  %xmm0, %xmm5, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   movq %rdi, %rcx
+;   andq $0x1f, %rcx
+;   movd %ecx, %xmm5
+;   vpsrld %xmm5, %xmm0, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %f32x4_abs(f32x4) -> f32x4 {
+block0(v0: f32x4):
+    v1 = fabs v0
+    return v1
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   pcmpeqd %xmm2, %xmm2, %xmm2
+;   vpsrld  %xmm2, $1, %xmm4
+;   andps   %xmm0, %xmm4, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   pcmpeqd %xmm2, %xmm2
+;   vpsrld $1, %xmm2, %xmm4
+;   andps %xmm4, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
diff --git a/cranelift/filetests/filetests/isa/x64/simd-bitwise-compile.clif b/cranelift/filetests/filetests/isa/x64/simd-bitwise-compile.clif
index 056f25601..24950d773 100644
--- a/cranelift/filetests/filetests/isa/x64/simd-bitwise-compile.clif
+++ b/cranelift/filetests/filetests/isa/x64/simd-bitwise-compile.clif
@@ -1,6 +1,6 @@
 test compile precise-output
 set enable_simd
-target x86_64 skylake
+target x86_64
 
 function %band_f32x4(f32x4, f32x4) -> f32x4 {
 block0(v0: f32x4, v1: f32x4):
@@ -360,6 +360,117 @@ block0(v0: i32):
 ;   addb %al, (%rax)
 ;   addb %al, (%rax)
 
+function %ishl_i8x16_imm(i8x16) -> i8x16 {
+block0(v0: i8x16):
+    v1 = iconst.i32 124
+    v2 = ishl v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   psllw   %xmm0, $4, %xmm0
+;   movdqu  const(0), %xmm4
+;   pand    %xmm0, %xmm4, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   psllw $4, %xmm0
+;   movdqu 0xf(%rip), %xmm4
+;   pand %xmm4, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+
+function %ishl_i16x8_imm(i16x8) -> i16x8 {
+block0(v0: i16x8):
+    v1 = iconst.i32 1
+    v2 = ishl v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   psllw   %xmm0, $1, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   psllw $1, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %ishl_i32x4_imm(i32x4) -> i32x4 {
+block0(v0: i32x4):
+    v1 = iconst.i32 100
+    v2 = ishl v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   pslld   %xmm0, $4, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   pslld $4, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %ishl_i64x2_imm(i64x2) -> i64x2 {
+block0(v0: i64x2):
+    v1 = iconst.i32 100
+    v2 = ishl v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   psllq   %xmm0, $36, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   psllq $0x24, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
 function %ushr_i8x16_imm() -> i8x16 {
 block0:
     v0 = iconst.i32 1
@@ -373,14 +484,9 @@ block0:
 ;   movq    %rsp, %rbp
 ; block0:
 ;   movdqu  const(1), %xmm0
-;   movl    $1, %r9d
-;   andq    %r9, $7, %r9
-;   movd    %r9d, %xmm5
-;   psrlw   %xmm0, %xmm5, %xmm0
-;   lea     const(0), %rsi
-;   shlq    $4, %r9, %r9
-;   movdqu  0(%rsi,%r9,1), %xmm13
-;   pand    %xmm0, %xmm13, %xmm0
+;   psrlw   %xmm0, $1, %xmm0
+;   movdqu  const(0), %xmm3
+;   pand    %xmm0, %xmm3, %xmm0
 ;   movq    %rbp, %rsp
 ;   popq    %rbp
 ;   ret
@@ -390,21 +496,109 @@ block0:
 ;   pushq %rbp
 ;   movq %rsp, %rbp
 ; block1: ; offset 0x4
-;   movdqu 0xb4(%rip), %xmm0
-;   movl $1, %r9d
-;   andq $7, %r9
-;   movd %r9d, %xmm5
-;   psrlw %xmm5, %xmm0
-;   leaq 0x1a(%rip), %rsi
-;   shlq $4, %r9
-;   movdqu (%rsi, %r9), %xmm13
-;   pand %xmm13, %xmm0
+;   movdqu 0x34(%rip), %xmm0
+;   psrlw $1, %xmm0
+;   movdqu 0x17(%rip), %xmm3
+;   pand %xmm3, %xmm0
 ;   movq %rbp, %rsp
 ;   popq %rbp
 ;   retq
 ;   addb %al, (%rax)
 ;   addb %al, (%rax)
-;   addb %bh, %bh
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   addb %al, (%rax)
+;   jg 0xb1
+;   jg 0xb3
+;   jg 0xb5
+;   jg 0xb7
+;   jg 0xb9
+;   jg 0xbb
+;   jg 0xbd
+;   jg 0xbf
+;   addb %al, (%rcx)
+;   addb (%rbx), %al
+;   addb $5, %al
+
+function %ushr_i16x8_imm(i16x8) -> i16x8 {
+block0(v0: i16x8):
+    v1 = iconst.i32 1
+    v2 = ushr v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   psrlw   %xmm0, $1, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   psrlw $1, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %ushr_i32x4_imm(i32x4) -> i32x4 {
+block0(v0: i32x4):
+    v1 = iconst.i32 100
+    v2 = ushr v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   psrld   %xmm0, $4, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   psrld $4, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %ushr_i64x2_imm(i64x2) -> i64x2 {
+block0(v0: i64x2):
+    v1 = iconst.i32 100
+    v2 = ushr v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   psrlq   %xmm0, $36, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   psrlq $0x24, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
 
 function %sshr_i8x16(i32) -> i8x16 {
 block0(v0: i32):
@@ -465,19 +659,15 @@ block0(v0: i8x16, v1: i32):
 ;   pushq   %rbp
 ;   movq    %rsp, %rbp
 ; block0:
-;   movl    $3, %r10d
-;   andq    %r10, $7, %r10
-;   movdqa  %xmm0, %xmm13
-;   punpcklbw %xmm13, %xmm0, %xmm13
-;   movdqa  %xmm13, %xmm12
-;   movdqa  %xmm0, %xmm13
-;   punpckhbw %xmm13, %xmm0, %xmm13
-;   addl    %r10d, $8, %r10d
-;   movd    %r10d, %xmm14
-;   movdqa  %xmm12, %xmm0
-;   psraw   %xmm0, %xmm14, %xmm0
-;   psraw   %xmm13, %xmm14, %xmm13
-;   packsswb %xmm0, %xmm13, %xmm0
+;   movdqa  %xmm0, %xmm7
+;   punpcklbw %xmm7, %xmm0, %xmm7
+;   movdqa  %xmm7, %xmm8
+;   movdqa  %xmm0, %xmm7
+;   punpckhbw %xmm7, %xmm0, %xmm7
+;   movdqa  %xmm8, %xmm0
+;   psraw   %xmm0, $11, %xmm0
+;   psraw   %xmm7, $11, %xmm7
+;   packsswb %xmm0, %xmm7, %xmm0
 ;   movq    %rbp, %rsp
 ;   popq    %rbp
 ;   ret
@@ -487,19 +677,104 @@ block0(v0: i8x16, v1: i32):
 ;   pushq %rbp
 ;   movq %rsp, %rbp
 ; block1: ; offset 0x4
-;   movl $3, %r10d
-;   andq $7, %r10
-;   movdqa %xmm0, %xmm13
-;   punpcklbw %xmm0, %xmm13
-;   movdqa %xmm13, %xmm12
-;   movdqa %xmm0, %xmm13
-;   punpckhbw %xmm0, %xmm13
-;   addl $8, %r10d
-;   movd %r10d, %xmm14
-;   movdqa %xmm12, %xmm0
-;   psraw %xmm14, %xmm0
-;   psraw %xmm14, %xmm13
-;   packsswb %xmm13, %xmm0
+;   movdqa %xmm0, %xmm7
+;   punpcklbw %xmm0, %xmm7
+;   movdqa %xmm7, %xmm8
+;   movdqa %xmm0, %xmm7
+;   punpckhbw %xmm0, %xmm7
+;   movdqa %xmm8, %xmm0
+;   psraw $0xb, %xmm0
+;   psraw $0xb, %xmm7
+;   packsswb %xmm7, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %sshr_i16x8_imm(i16x8) -> i16x8 {
+block0(v0: i16x8):
+    v1 = iconst.i32 1
+    v2 = sshr v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   psraw   %xmm0, $1, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   psraw $1, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %sshr_i32x4_imm(i32x4) -> i32x4 {
+block0(v0: i32x4):
+    v1 = iconst.i32 100
+    v2 = sshr v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   psrad   %xmm0, $4, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   psrad $4, %xmm0
+;   movq %rbp, %rsp
+;   popq %rbp
+;   retq
+
+function %sshr_i64x2_imm(i64x2) -> i64x2 {
+block0(v0: i64x2):
+    v1 = iconst.i32 100
+    v2 = sshr v0, v1
+    return v2
+}
+
+; VCode:
+;   pushq   %rbp
+;   movq    %rsp, %rbp
+; block0:
+;   pextrd.w $0, %xmm0, %rdx
+;   pextrd.w $1, %xmm0, %r9
+;   sarq    $36, %rdx, %rdx
+;   sarq    $36, %r9, %r9
+;   uninit  %xmm0
+;   pinsrd.w $0, %xmm0, %rdx, %xmm0
+;   pinsrd.w $1, %xmm0, %r9, %xmm0
+;   movq    %rbp, %rsp
+;   popq    %rbp
+;   ret
+; 
+; Disassembled:
+; block0: ; offset 0x0
+;   pushq %rbp
+;   movq %rsp, %rbp
+; block1: ; offset 0x4
+;   pextrq $0, %xmm0, %rdx
+;   pextrq $1, %xmm0, %r9
+;   sarq $0x24, %rdx
+;   sarq $0x24, %r9
+;   pinsrq $0, %rdx, %xmm0
+;   pinsrq $1, %r9, %xmm0
 ;   movq %rbp, %rsp
 ;   popq %rbp
 ;   retq
diff --git a/cranelift/filetests/filetests/runtests/simd-arithmetic.clif b/cranelift/filetests/filetests/runtests/simd-arithmetic.clif
index 4bc1ac828..ec1e4ad01 100644
--- a/cranelift/filetests/filetests/runtests/simd-arithmetic.clif
+++ b/cranelift/filetests/filetests/runtests/simd-arithmetic.clif
@@ -4,6 +4,7 @@ test run
 target aarch64
 target s390x
 set enable_simd
+target x86_64
 target x86_64 skylake
 
 function %iadd_i32x4(i32x4, i32x4) -> i32x4 {
diff --git a/cranelift/filetests/filetests/wasm/duplicate-loads-dynamic-memory-egraph.wat b/cranelift/filetests/filetests/wasm/duplicate-loads-dynamic-memory-egraph.wat
deleted file mode 100644
index b6d4e88dc..000000000
--- a/cranelift/filetests/filetests/wasm/duplicate-loads-dynamic-memory-egraph.wat
+++ /dev/null
@@ -1,92 +0,0 @@
-;;! target = "x86_64"
-;;!
-;;! optimize = true
-;;!
-;;! settings = [
-;;!   "enable_heap_access_spectre_mitigation=true",
-;;!   "opt_level=speed_and_size",
-;;!   "use_egraphs=true"
-;;! ]
-;;!
-;;! [globals.vmctx]
-;;! type = "i64"
-;;! vmctx = true
-;;!
-;;! [globals.heap_base]
-;;! type = "i64"
-;;! load = { base = "vmctx", offset = 0 }
-;;!
-;;! [globals.heap_bound]
-;;! type = "i64"
-;;! load = { base = "vmctx", offset = 8 }
-;;!
-;;! [[heaps]]
-;;! base = "heap_base"
-;;! min_size = 0
-;;! offset_guard_size = 0xffffffff
-;;! index_type = "i32"
-;;! style = { kind = "dynamic", bound = "heap_bound" }
-
-(module
-  (memory (export "memory") 0)
-  (func (export "load-without-offset") (param i32) (result i32 i32)
-    local.get 0
-    i32.load
-    local.get 0
-    i32.load
-  )
-  (func (export "load-with-offset") (param i32) (result i32 i32)
-    local.get 0
-    i32.load offset=1234
-    local.get 0
-    i32.load offset=1234
-  )
-)
-
-;; function u0:0(i32, i64 vmctx) -> i32, i32 fast {
-;;     gv0 = vmctx
-;;     gv1 = load.i64 notrap aligned gv0+8
-;;     gv2 = load.i64 notrap aligned gv0
-;;
-;;                                 block0(v0: i32, v1: i64):
-;; @0057                               v4 = uextend.i64 v0
-;; @0057                               v5 = iconst.i64 4
-;; @0057                               v6 = uadd_overflow_trap v4, v5, heap_oob  ; v5 = 4
-;; @0057                               v7 = load.i64 notrap aligned v1+8
-;; @0057                               v8 = load.i64 notrap aligned v1
-;; @0057                               v11 = icmp ugt v6, v7
-;; @0057                               v10 = iconst.i64 0
-;; @0057                               v9 = iadd v8, v4
-;; @0057                               v12 = select_spectre_guard v11, v10, v9  ; v10 = 0
-;; @0057                               v13 = load.i32 little heap v12
-;;                                     v2 -> v13
-;; @005f                               jump block1
-;;
-;;                                 block1:
-;; @005f                               return v13, v13
-;; }
-;;
-;; function u0:1(i32, i64 vmctx) -> i32, i32 fast {
-;;     gv0 = vmctx
-;;     gv1 = load.i64 notrap aligned gv0+8
-;;     gv2 = load.i64 notrap aligned gv0
-;;
-;;                                 block0(v0: i32, v1: i64):
-;; @0064                               v4 = uextend.i64 v0
-;; @0064                               v5 = iconst.i64 1238
-;; @0064                               v6 = uadd_overflow_trap v4, v5, heap_oob  ; v5 = 1238
-;; @0064                               v7 = load.i64 notrap aligned v1+8
-;; @0064                               v8 = load.i64 notrap aligned v1
-;; @0064                               v12 = icmp ugt v6, v7
-;; @0064                               v11 = iconst.i64 0
-;; @0064                               v9 = iadd v8, v4
-;;                                     v26 = iconst.i64 1234
-;; @0064                               v10 = iadd v9, v26  ; v26 = 1234
-;; @0064                               v13 = select_spectre_guard v12, v11, v10  ; v11 = 0
-;; @0064                               v14 = load.i32 little heap v13
-;;                                     v2 -> v14
-;; @006e                               jump block1
-;;
-;;                                 block1:
-;; @006e                               return v14, v14
-;; }