Code change:
a programming language that I am working on, and the VM is written in Rust. Up until Rust nightly 2018-08-17, everything works fine. Starting with the nightly from the 17th, I'm observing various crashes and different program behaviour. For example: On Windows it will either fail with a , or (more on this in a moment). On Linux . Note that the funny segfault output is because the command is started with Ruby, and Ruby installs its own segmentation fault handler. Locally it will usually fail with the same runtime error as observed in Windows above, but sometimes it will segfault. Sometimes it will panic because certain operations are performed using NULL pointers where this is not expected. The last nightly that did not suffer from these problems was Rust 2018-08-16. Stable Rust also works fine. When the segmentation faults happen, they are usually in different places. For example, for one segmentation fault the backtrace is as follows: 0x00007ffff7e12763 in intmalloc () from 0x00007ffff7e13ada in malloc () from 0x0000555555568e6b in alloc::alloc::alloc (layout=...) at <libinko::chunk::Chunk<T::new (capacity=3) at libinko::register::Register::new (amount=3) at libinko::executioncontext::ExecutionContext::fromblock (block=0x7fffdc0971e0, returnregister=Some = {...}) at libinko::vm::machine::Machine::run (self=<optimized out, process=<optimized out) at 0x0000555555568b7f in libinko::vm::machine::Machine::runwitherrorhandling (self=0x55555567d8c0, process=0x7ffff75fcbd0) at 0x00005555555c88a4 in libinko::vm::machine::Machine::startprimarythreads::{{closure}} (process=...) at <libinko::pool::PoolInner<T::process (self=<optimized out, index=0, closure=0x7ffff75fcc60) at 0x00005555555b739d in <libinko::pool::Pool<T::run::{{closure}} () at std::syscommon::backtrace::rustbeginshortbacktrace (f=...) at 0x00005555555cb0dc in std::thread::Builder::spawn::{{closure}}::{{closure}} () at <std::panic::AssertUnwindSafe<Fas core::ops::function::FnOnce<()::callonce (self=..., args=<optimized out) at std::panicking::try::docall (data=<optimized out) at 0x0000555555618a3a in rustmaybecatchpanic () at 0x00005555555ba39b in std::panicking::try (f=...) at std::panic::catchunwind (f=...) at std::thread::Builder::spawn::{{closure}} () at <F as alloc::boxed::FnBox<A::callbox (self=0x55555567db50, args=<optimized out) at 0x00005555556090db in LTdynLTCOutputRu20u20aGTasLTGT::callonce::h904fcd0dbdc71d4f () at std::syscommon::thread::startthread () at 0x00005555555f83b6 in std::sys::unix::thread::Thread::new::threadstart () at 0x00007ffff7f73a9d in startthread () from 0x00007ffff7e89a43 in clone () from While for another segfault the backtrace is instead: libinko::runtimepanic::displaypanic (process=0x7f, message="ObjectValue::asblock() called on a non block object") at 0x0000555555568baa in libinko::vm::machine::Machine::panic (self=0x55555567d8c0, process=0x7ffff71f8bd0, message="") at libinko::vm::machine::Machine::runwitherrorhandling (self=0x55555567d8c0, process=0x7ffff71f8bd0) at 0x00005555555c88a4 in libinko::vm::machine::Machine::startprimarythreads::{{closure}} (process=...) at <libinko::pool::PoolInner<T::process (self=<optimized out, index=4, closure=0x7ffff71f8c60) at 0x00005555555b739d in <libinko::pool::Pool<T::run::{{closure}} () at std::syscommon::backtrace::rustbeginshortbacktrace (f=...) at 0x00005555555cb0dc in std::thread::Builder::spawn::{{closure}}::{{closure}} () at <std::panic::AssertUnwindSafe<Fas core::ops::function::FnOnce<()::callonce (self=..., args=<optimized out) at std::panicking::try::docall (data=<optimized out) at 0x0000555555618a3a in rustmaybecatchpanic () at 0x00005555555ba39b in std::panicking::try (f=...) at std::panic::catchunwind (f=...) at std::thread::Builder::spawn::{{closure}} () at <F as alloc::boxed::FnBox<A::callbox (self=0x55555567e5d0, args=<optimized out) at 0x00005555556090db in LTdynLTCOutputRu20u20aGTasLTGT::callonce::h904fcd0dbdc71d4f () at std::syscommon::thread::startthread () at 0x00005555555f83b6 in std::sys::unix::thread::Thread::new::threadstart () at 0x00007ffff7f73a9d in startthread () from 0x00007ffff7e89a43 in clone () from And a third segfault: 0x0000555555568d84 in libinko::vm::machine::Machine::run (self=<optimized out, process=<optimized out) at 0x0000555555568b7f in libinko::vm::machine::Machine::runwitherrorhandling (self=0x55555567d8c0, process=0x7ffff6cf3bd0) at 0x00005555555c88a4 in libinko::vm::machine::Machine::startprimarythreads::{{closure}} (process=...) at <libinko::pool::PoolInner<T::process (self=<optimized out, index=10, closure=0x7ffff6cf3c60) at 0x00005555555b739d in <libinko::pool::Pool<T::run::{{closure}} () at std::syscommon::backtrace::rustbeginshortbacktrace (f=...) at 0x00005555555cb0dc in std::thread::Builder::spawn::{{closure}}::{{closure}} () at <std::panic::AssertUnwindSafe<Fas core::ops::function::FnOnce<()::callonce (self=..., args=<optimized out) at std::panicking::try::docall (data=<optimized out) at 0x0000555555618a3a in rustmaybecatchpanic () at 0x00005555555ba39b in std::panicking::try (f=...) at std::panic::catchunwind (f=...) at std::thread::Builder::spawn::{{closure}} () at <F as alloc::boxed::FnBox<A::callbox (self=0x55555567f590, args=<optimized out) at 0x00005555556090db in LTdynLTCOutputRu20u20aGTasLTGT::callonce::h904fcd0dbdc71d4f () at std::syscommon::thread::startthread () at 0x00005555555f83b6 in std::sys::unix::thread::Thread::new::threadstart () at 0x00007ffff7f73a9d in startthread () from 0x00007ffff7e89a43 in clone () from In case of the last segfault, it seems certain local variables that are used are NULL pointers, when this should be impossible. Debugging this in GDB proves to be quite difficult, as a variety of variables are reported as <optimized outeven when debugging symbols are included. For example, for the last backtrace the output of "info locals" is: The VM test suite passes, even when running . I'm wondering if perhaps code is optimised in the wrong way, and this is somehow not triggered in the test suite (certainly possible, code coverage is not 100%). Reproducing this is a bit weird. If we leave the code as-is, the segmentation faults rarely occur, instead the VM panics with the following: Stack trace (the most recent call comes last): 0: "", line 324, in "<block" 1: "", line 281, in "<lambda" 2: "", line 220, in "run" Process 1 panicked: ObjectValue::as_block() called on a non block object However, if we apply the following patch things will start to segfault really quick: Note that the last command requires Ruby 2.3 or newer. This will run the test suite of the standard library, which is where all the crashes happen rather frequently (probably because they run much more than the VM's own test suite).
There's a lot of questionable unsafe code in that project. For example, should really be marked as (same goes for . It's very likely that the cause of the issue is preexisting UB due to misuse of .
Simply marking functions as to my knowledge doesn't influence the compiler's code generation. Many of the unsafe operations are safe in practise, as they will only fail if (for example) the input bytecode is invalid. Tagging them as does nothing but make the code significantly more verbose. The various operations in the bytecode parser are there to convert sequences of bytes into particular types. To the best of my knowledge there is no stable way of converting a into say a , requiring you to use instead. The closest is but this: nightly the platform's endianness, instead of allowing you to specify which to use.
Yes, but marking a function as requires all call sites to opt into that unsafety, and the functions I mentioned all assume that their arguments are valid (and cause UB if not). This additional constraint on the arguments is expressed by marking the function and documenting the safety constraints. Check out the crate.
This wouldn't achieve much, as a lot would have to be marked unsafe as VMs and GCs inherently perform many operations Rust may consider unsafe. There are a few cases where I want to get rid of more risky code (e.g. the borrowing of you highlighted), but all of this feels rather offtopic since it's very project specific. I avoided it in the past because it also just used , so it didn't feel like much of an improvement. It appears this is no longer the case, so I'll take another look.
To better understand what is going on I dumped the MIR output using and (2018-08-18). After replacing a bunch of SHA hashes with placeholders (to reduce diff noise), I ended up with the exact same MIR before and after. Next I'll try LLVM IR.
It appears the LLVM IR uses a different order for things when using the different nightly versions. This generates a lot of diff noise, so I'm not sure how useful this information will be. I tried to clean it up a bit by replacing various SHA hashes with placeholders, but even then there is a lot of noise.
The ASM output was a bit easier to clean up, but there are still lots of differences. To be exact, reports: Again this is probably because of ordering. The new version has 99 lines more, but I'm really not sure if that's expected or not.
Using I found that nightly-2018-08-19 is the first version that introduces the regression, while nightly-2018-08-18 works fine. The commits included into the 18th nightly can be found at (if I got things right at least).
This hints at , which can change the behaviour of any latent UB, or , since your project uses in a few places.
I don't fully understand the first mentioned PR. What would I need to do to verify if that has any impact? For the second one, I'll see if I can somehow get my code to use a version of from before that PR, to see if it affects anything.
A custom version of probably isn't going to work. For example, I'm using Rayon which only implements its traits for the default :<
I ended up replacing all my uses of with to test if the VecDeque changes are to blame. I used LinkedList because its API is closest to VecDeque, making it easier to test things. When using LinkedList, my code runs fine. This suggests that the changes in may have somehow broken things. cc
The first PR changes details about the generated code that is passed to LLVM. In fact, it just removes unneeded information. The easiest way to check if it's at fault is to manually build the compiler from with the PR reverted. If the problem stops happening, it's at fault (or more likely, UB in your code is). It would also be helpful if you could reduce the code as much as possible so that we can get a few hundred line example that shows the issue.
Specifically, the patch I applied is:
I narrowed this down to the following file being broken by the changes: If I switch back to for this file, my code breaks. There are two possibilities here: was a bug before in my code that only somehow didn't get triggered before VecDeque changes themselves break things The Mailbox data structure has multiple threads accessing it concurrently. This is synchronised using a mutex of type (provided by parking_lot). The structure is broken up into essentially two parts: a synchronised external part, and an unsynchronised internal part. The internal part is only used by a single thread, while multiple threads can compete for the external part. For this reason the mutex is separate, so we don't need to synchronise both. Edit: I forgot, this is synchronised earlier on, preventing concurrent writes. I'll see if I can refactor this code a bit so it's safer, that way we can at least rule out my own code being the problem (or not).
Upon closer inspection, the VecDeque changes specifically affect , and I happen to use that in my Mailbox structure. Specifically, I have the following: I don't remember why this is using since is also a , probably I wrote this when not paying attention. Regardless, changing this to doesn't make any difference regarding the errors. Next I changed this code to the following: This is based on the old implementation of , which was: This then produces output like: This is a bit odd considering the implementation is the same as before, but at least now part of the program is running.
There was another place where I relied on . After also replacing that with the pattern my code now runs fine again, both on nightly and stable Rust. This suggests the failure is definitely due to the changes made to . Now to see if I can get this in a standalone example.
Good news: Thanks to the great investigation of and I think I found the reason for the segfault: the new calls which unsoundly creates mutable slices to uninitialized memory. I will try to send a fixing PR tomorrow.
Aha, thanks! I was looking at the code to see how I might be able to trigger this problem, but I hadn't made my way to just yet. Looking at the code, I'm not sure exactly how to trigger this reliably.
That could be it, nice find!
The problem AFAIK is that this function is triggering Undefined Behavior, and so can cause the program to segfault randomly, and so it may be difficult to reproduce it reliably.
I can verify that this is because of my PR. Minimal example that segfaults for me: And the reason for the segfault is apparent when run with simpler types This will print and continue in such a loop forever, printing the buffer over and over again. This happens because of line hitting an edge case when that causes head to be placed outside of the buffer. This can be fixed by changing that line to I think that is the only thing that is causing these segfaults.

Related discussion:
text: } } <ins> #[test] fn issue_53529() { use boxed::Box; let mut dst = VecDeque::new(); dst.push_front(Box::new(1)); dst.push_front(Box::new(2)); assert_eq!(*dst.pop_back().unwrap(), 1); let mut src = VecDeque::new(); src.push_front(Box::new(2)); dst.append(&mut src); for a in dst { assert_eq!(*a, 2); } } </ins> }