Change:
We're to update Fuchsia's toolchain to the latest nightly with builds failing due to this ICE: <details</detailsWe think that this error is occurring on our builders due to many concurrent instances of exhausting which limits the number of pipe pages a user is allowed to have: It seems that the failure started with which updated the cargo subtree, causing the jobserver dependency for rustc to update to . That release contains by which on Linux attempts to reserve necessary space in the pipe for , returning an for client creation if that reservation fails. When accessing the jobserver for internal concurrency control, by default attempts to read the path to a pipe . If no jobserver has been configured, it via where the new call was . This fallback is likely never exercised in builds driven by or because they both provide a jobserver to their subprocesses. On Fuchsia, we invoke using , which , so every instance of ends up creating its own pipe. I think based on reading the code that each "fallback pipe" needs the default 2 pages, so we'd be able to run 512 s if no other processes for the user were creating pipes. In practice our build machines are quite large and they also run some other processes which create pipes, so we seem to be able to hit this limit pretty often. We're considering workarounds including a static limit on concurrent invocations in our build or implementing the jobserver protocol ourselves somehow, but it does seem like it should be possible to invoke concurrently many times without make's protocol. Both of these solutions would come with significant downsides, though. Is it possible for to work without access to a jobserver client? Could the fallback case be made to use an in-process stub instead of creating an actual jobserver client? Alternatively, it seems that even though the pipe resizing is failing that the pipe creation itself has not been previously failing in our build. Perhaps we could find a way for to ignore the error from the resizing since it only ever asks for space to schedule tasks in the fallback configuration? These are the options that came up in a brief internal discussion but other ideas would be welcome! cc
The small crater run we just did ran into a large number of errors due to this. What do you think about temporarily reverting the jobserver change until we have a better handle on how to approach this issue?
Oh dear, thanks for cc'ing me on this! For some background, the motivation for this change to the crate was where on Linux a single-page-backed pipe can quickly deadlock based on . This means that previously all the pipes on your system were theoretically susceptible to deadlocking. Now that being said the deadlocking happens when a token is recycled 4096 times at minimum, and for rustc tokens correspond basically to units of work on codegen units and you're unlikely to ever get close to 4096 codegen units. This means that for rustc's use case specifically it was highly unlikely to actually hit this deadlock in practice. Now as for solving this bug, I think there's a few possible routes: For rustc's own personal use case, it could choose to ignore this situation and simply forge ahead and ignore the failure. For rustc's own personal use case, there's no need for process inheritance of jobserver fds, which means that a special in-process variant of the jobserver can be created for rustc if it ends up creating one. Rustc could opt to not use the jobserver at all for your use case. If each rustc generates its own jobserver it's likely you probably don't care much about limiting parallelism. You could update to the latest-and-greatest Linux kernel which should have a patch that fixes this behavior (the default "limited" pipe size is now 2 pages instead of 1 page) You could increase the pipe limits on your build machine so pipes more frequently get the 16-page default backing buffer. We could just back out the changes (as mentions) and not really ever put them back in. This is fixed in a future version of Linux and the problem seems very rare otherwise today. If this is coming up enough then it's not worth the hypothetical protection against deadlock it gives. I had most of this comment written up before I saw the crater report, but I think that for now I'll revert this and we can just deal with it later if we really need to.
this isn't quite right. the pipe fix was backported back to 4.4 (the earliest kernel still supported upstream), so you need only upgrade to the latest kernel on your branch, which you should do anyways (see gkh "All users must upgrade").
this is probably a good idea. pipes are much slower than shm futexes; the only reason to use pipes as semaphores is to maintain compatibility with GNU make or with platforms that don't support shared semaphores.
Yeah if it's showing up in crater then my assumption about being limited to use outside of cargo might have been incorrect. Appreciate the revert, I also filed as a possible follow-up that might remove this scaling factor entirely. EDIT: jinx!

Discussion:
~~~~ {.xfail-test} pub fn compress(src: &[u8]) -> ~[u8] { #[fixed_stack_segment]; #[inline(never)]; unsafe { let srclen = src.len() as size_t; let psrc = vec::raw::to_ptr(src);

~~~~ {.xfail-test} pub fn uncompress(src: &[u8]) -> Option<~[u8]> { #[fixed_stack_segment]; #[inline(never)]; unsafe { let srclen = src.len() as size_t; let psrc = vec::raw::to_ptr(src);

unsafe fn snappy_max_compressed_length(source_length: size_t) -> size_t { #[fixed_stack_segment]; #[inline(never)]; return snappy_max_compressed_length(source_length); #[link_args = "-lsnappy"] extern { fn snappy_max_compressed_length(source_length: size_t) -> size_t;

2. The call occurs inside of an `extern fn`; 3. The call occurs within a stack closure created by some other safe fn. All of these conditions ensure that you are running on a large stack <del> segmented. However, they are sometimes too strict. If your application </del> <ins> segment. However, they are sometimes too strict. If your application </ins> will be making many calls into C, it is often beneficial to promote the `#[fixed_stack_segment]` attribute higher up the call chain.  For example, the Rust compiler actually labels main itself as requiring a

pub fn new(value: T) -> Unique<T> { #[fixed_stack_segment]; #[inline(never)]; unsafe { let ptr = malloc(std::sys::size_of::<T>() as size_t) as *mut T; assert!(!ptr::is_null(ptr));

fn drop(&mut self) { #[fixed_stack_segment]; #[inline(never)]; unsafe { let x = intrinsics::init(); // dummy value to swap in // moving the object out is needed to call the destructor