Repository: iree-org/iree

Issue #12444: Intermittent failures in `mobilenetv3_fake_weights.mlir`
Created: 2023-03-01T20:13:46Z
This test started failing intermittently after the most recent integrate (#12419).
Sample failure: https://github.com/openxla/iree/actions/runs/4307115731/jobs/7511785781#step:4:8565
Error:
FAILED: tests/e2e/models/check_cuda_cuda_mobilenetv3_fake_weights.mlir_module.vmfb /work/full-build-dir/tests/e2e/models/check_cuda_cuda_mobilenetv3_fake_weights.mlir_module.vmfb 
cd /work/full-build-dir/tests/e2e/models && /work/full-build-dir/tools/iree-compile --output-format=vm-bytecode --mlir-print-op-on-diagnostic=false --iree-hal-target-backends=cuda --iree-input-type=mhlo /work/tests/e2e/models/mobilenetv3_fake_weights.mlir -o check_cuda_cuda_mobilenetv3_fake_weights.mlir_module.vmfb --iree-hal-executable-object-search-path=\"/work/full-build-dir\" --iree-llvm-embedded-linker-path=\"/work/full-build-dir/llvm-project/bin/lld\" --iree-llvm-wasm-linker-path=\"/work/full-build-dir/llvm-project/bin/lld\"
/work/tests/e2e/models/mobilenetv3_fake_weights.mlir:933:12: error: cannot replace an op with another op producing a different number of results while tracking handles
    %695 = "mhlo.reduce"(%694, %266) ( {
           ^
/work/tests/e2e/models/mobilenetv3_fake_weights.mlir:938:12: error: 'hal.executable.variant' op Listener failed.
    %696 = mhlo.divide %695, %260 : tensor<1x288xf32>
           ^
/work/tests/e2e/models/mobilenetv3_fake_weights.mlir:938:12: error: Failed to performCanonicalization after transform
    %696 = mhlo.divide %695, %260 : tensor<1x288xf32>
           ^
/work/tests/e2e/models/mobilenetv3_fake_weights.mlir:938:12: error: transform dialect interpreter failed
    %696 = mhlo.divide %695, %260 : tensor<1x288xf32>
           ^
/work/tests/e2e/models/mobilenetv3_fake_weights.mlir:938:12: error: failed to run translation of source executable to target executable for backend #hal.executable.target<"cuda", "cuda-nvptx-fb", {target_arch = "sm_35"}>
    %696 = mhlo.divide %695, %260 : tensor<1x288xf32>
           ^
/work/tests/e2e/models/mobilenetv3_fake_weights.mlir:938:12: error: failed to serialize executables
    %696 = mhlo.divide %695, %260 : tensor<1x288xf32>
           ^

Duplicate issue #12463: Non-deterministic Listening / Rewrite interactions introduced recently.
Created: 2023-03-02T10:42:47Z
What happened?
@matthias-springer has been doing a recent push on deep upstream cleanups of the Rewriter infra.
This currently triggers some non-determinism that we need to address with high-pri (for reference, the triggering issue was #12445).
This is likely related to still missing listener plumbing through the Rewriter APIs and/or patterns that have not yet been cleaned up to properly use rewriter methods. This is also likely a symptom of "MLIR deleted and reallocated and operation in the same pointer", as we had seen in the past.
Here are the ongoing mitigations:

@matthias-springer has a draft PR to help detect patterns that do not properly use the rewriter API at runtime (link) [https://reviews.llvm.org/D144552].
we also sync'ed today and I suggested a bigger hammer: make all IR modification APIs on Operation/Value/Block private and expose a very few select friends. This should turn all bad usages into compiler failures and I expect will also let us clean up a lot of things on the IREE side. I don't imagine this can land upstream given past experience with such changes but I believe as a draft PR to get a static version of the above, it makes sense.
on the pure transform-dialect side, we had a cleanup procedure that deleted the tracking of handles that went out of scope, this was retired not too long ago but we could think of reactivating to provide an additional safety net (@ftynse).
on our usage, I have some ongoing cleanups to remove blanket canonicalizations and hoisting patterns now that we can control things more finely. I am prioritizing those on my end to reduce the amount of such manipulations that happen automatically under the hood.

We'll post links as available during the investigation.
Steps to reproduce your issue

Go to '...'
Click on '....'
Scroll down to '....'
See error

What component(s) does this issue relate to?
No response
Version information
No response
Additional context
No response

Closing comment (2023-03-02T10:59:09Z): Duplicate of #12444