Change:
Symptom: The test failed with the following messages: Previous instances: (2018-11-08T05:44:13Z) (2018-11-11T03:44:11Z) This might be caused by a mis-optimization like .
Added to the list
I suspect on i686-musl is the same as this issue. Perhaps we should run the collectiontest in miri. The stacktrace doesn't seem meaningful.
https://travis-
We've asked for help on internals:
I've managed to reproduce this locally, using the same Android emulator used by CI. I'm working on creating a self-contained script to make it easy to test this locally.
I believe I've obtained a backtrace from the emulator: Interestingly enough, it appears that the process actually hangs in the emulator, rather than dying. This allowed me to install gdb and connect the process after the test had failed.
I strongly suspect that this is related to the test in . However, I'm not certain, as I'm having difficulty narrowing things down to the offending test.
Some additional results: I the following script as a test (it contains some logic from ): I then manually uploaded it to the emulator, and invoked it in a loop with the following Bash script (run from ): After running this for about 15-20 minutes, I stopped the loop. For reasons I don't yet understand, several of the spawned processes has segfaulted, wthout stopping the loop. I then uploaded a static GDB binary from to the emulator (this was the only way I could manage to get GDB to work). I managed to obtain the following backtrace from my test program: From what I can see, the panic hook appears to be executing, and then somehow jumping back into the body of the loop. EDIT: I believe that this is actually showing the normal invocation of my test function. Note that this was the only thread running. I'm not sure if this is related to the original issue, but it suggets that something weird is going on with the panicking threads. My current hypothesis is that there's some sort of concurrency and/or codegen issue with the the logic (maybe the compiler intrinsic?).
Further evidence for panicking being involved: Another run crashed with this log (I wasn't able to obtain a backtrace): The test immediately following is . This test is marked with ], so it seems that simply pancking while a test is running may trigger the issue.
Thanks We discussed this in the infra team meeting today and decided that we would disable the shouldpanic tests on Android. This'll basically be a gate on each of those tests. Note that android is a tier-2 platform so we (the Rust team) ourselves don't plan to invest time in fixing it, but if someone wants to attempt to narrow this down further we'll gladly review patches.
I'm temporarily assigning this to myself -- I plan to add that cfg soon. Of course, if someone wants to take that on and save me some time, I'd be happy :)
I'm going to use the functionality in instead of a cfg-gate across all of std/test/alloc -- that seems quite a bit cleaner.
Posted and unassigning myself. That PR won't fix all of the problems here so it doesn't fix this bug (it doesn't properly inform rustdoc to skip should_panic tests) so it should not close this issue.
While this happened rarely in Azure Pipelines (at least in the recent months), I noticed it happens a lot on GitHub Actions. We should find a workaround before we start gating on GHA.
Do you still have instructions for reproducing this bug locally? I will look into this issue.
Unfortunately, I don't think that I do (I really should have documented what I previously did...) I'll see if I can reconstruct what I did, and post instructions.
I ran the tests in the container in a loop overnight, and couldn't reproduce the issue. Could you link to some of the failed Github Actions jobs?
It turns out that I was accidentally running the test with my local , which had debug assertions enabled. With the that gets built by the CI scripts, I was able to reproduce the issue. Run from this branch. This script runs the tests in a loop until a crash occurs.
I was finally able to obtain an android tombstone file: This was quite difficult to obtain - the emulator appears to get force-killed when a test fails (the outer Docker contain shuts down with the emulator still running), which caused the tombstone written to to get lost. I needed to modify the tool to not exit when a binary failed, which allowed me to attach to the running Docker container:
I came across the following code: I don't know if it's related, but it's something to keep in mind.
API level 9 means Android 2.3-2.3.3 released 9 years ago which seems overly conservative. From what I could find was by API level 16 which is Android 4.1.x released almost 8 years ago. Bumping API level locally and enabling seems worth giving a try. Rust should probably consider defaulting to some newer API as well.
It might be worth switching it to plain temporarily to see if this fixes the crash (even though it is technically wrong to ignore the alignment).
Unlike x86 on ARM alignment is very important. Normal load and store instructions can't access unaligned memory.
Sure but the default malloc alignment is enough for that.
I think it's going to be extremely difficult to debug this further in the android vm. I'm currently trying to reproduce this without android, using a combination of and built for the target. However, it's possible that there's something Android-specific going on here (e.g. the emulator, the QEMU arm backend, or some code.
So, here's what I got so far. I've narrowed it down to this command line, which runs only 35 tests: Crashes are rare, but when it does crash it seems to always be in (confirmed by disassembling the address in the tombstone): The segmentation fault is misleading, that's just what Android's libc uses for instead of SIGABRT. The abort message is: From the looks of it, there is no problem in our code: this is purely a bug in Android's dlmalloc. My guess is that there is some integer overflow issue in dlmalloc's internal size calculations, but dependent on some internal malloc state (hence the non-determinism).
Looking at the memory dumps from the tombstone, we can see that the bug is somewhere in : Note the chunk prefix of which says "this chunk has a length of 2GB + 8". This is clearly impossible since the address of the block (0xb8caf518) is in the second half of the address space. Note that this bug is not an issue for modern Android versions since those use jemalloc instead of dlmalloc for the default libc allocator. In light of this I think we should just disable the tests on Android.
Thanks for investigating this :heart:
EDIT: I misunderstood how is being used. The function is in fact called by in certain cases, so analysis seems correct. How do you know it's caused by ? The tombstone file I captured contained the following backtrace for one of the threads (not the crashing one): It looks like Rust isn't actually calling into from - instead, it's falling back to freeing and re-allocating memory. If is getting called, then it's either being done by the standard library making an explicit call to an function (which I think never happens for ), or it's being done by libc.

Discussion:
#[test] #[cfg_attr(miri, ignore)] // Miri does not support signalling OOM <ins> #[cfg_attr(target_os = "android", ignore)] // Android used in CI has a broken dlmalloc </ins> fn test_try_reserve() { // These are the interesting cases: // * exactly isize::MAX should never trigger a CapacityOverflow (can be OOM)

#[test] #[cfg_attr(miri, ignore)] // Miri does not support signalling OOM <ins> #[cfg_attr(target_os = "android", ignore)] // Android used in CI has a broken dlmalloc </ins> fn test_try_reserve_exact() { // This is exactly the same as test_try_reserve with the method changed. // See that test for comments.