jiebi/CodeConvo huggingface dataset:
If glibc were to guarantee that all addresses are 32-byte aligned, there is no way in which we could exploit that information in the implementation of the system allocator, because the platform only requires 16-byte alignment, and it is valid to pick any allocator at run-time that satisfies that.
So there a document somewhere saying "for x86-64 bit, 16-byte alignment is guaranteed"? But that document contains an exception for small sizes, making s behavior legal?
Yes, there are documents containing the rules of what's legal, at least for Linux x8664. Those document allow an exception for small sizes. One API that uses those is , which requires the alignment to be at least , which is 8 on x86_64, and which allows you to allocate bytes, with a smaller alignment. I still have no idea what jemalloc has to do with any of this - your original question is about , and that has nothing to do with jemalloc. But yes, jemalloc exploits this behavior, to be able to satisfy 1 byte request, and return addresses that are 1 byte aligned. Otherwise each 1 byte request would consume 16 bytes of memory within a memory page.
A document that could be relevant is the C standard. Unfortunately, as mentioned in the commit message for , it is rather tautological on this topic: The standard does not give any numeric value. Like many other aspects of C, it looks like the actual value of is implementation-defined.
When using jemalloc, becomes jemalloc. That's how happened. Do you have a reference for something that concretely calls out an exception for small types? The C standard only does this very indirectly, as you showed.
That was a bug in our implementation of system that applies to pretty much all allocators, not only jemalloc. says (emphasis mine): Since the alignment of an object is a power of two that is always smaller than or equal to the object size, rounding down the alignment of an allocation to the previous power of two always produces an allocation properly aligned for every type that can fit it. That holds for all allocations, so there is no exception for that. That's the rule. Since that would mean that large allocations must be unreasonably aligned, the standard does provide an exception for allocations of size larger than than .
These only need to be aligned at an boundary. If you believe this is incorrect, please show a counter-example for which this doesn't work.
AFAIK it is a normal way of using to make it override the symbol? Don't we still do that in rustc, or when using "jemallocator"? I don't say it is incorrect, I say it is awfully indirect. It's like reading tea leaves. I am looking for a clear definite statement. That doesn't seem to exist though. :( There clearly is some kind of "non-continuity" where all allocations are aligned MINALIGN except for small ones. Or vice versa, all allocations are aligned "size rounded down to power of 2" except for big ones. You are convincingly deriving that non-continuity from standard wording. I don't say you are wrong, but this feels like interpreting ancient scripture. A clear spec looks different. Also, an object of size 2 with alignment requirement 4 is "fundamental" according to your definition, and yet rounding down the size to the next power of 2 does not give the right alignment. So your rule does have exceptions, namely types marked .
Can you show such an object in C and/or Rust ? It's probably the most common way of switching allocators. What does is it requires to be linked to your binary, allowing (and making use of) jemalloc-specific APIs. If you do this, when using directly (not ), you can obviously rely on assumptions that only hold for . But cannot do that because, then it wouldn't work with other allocators.
The goal is not to rely on assumptions that only hold for . The goal is to make sure that we do not make assumptions that don't hold for . Because that's what we used to do, before got fixed. We were assuming something that was true for system but not for : that all allocations, no matter the size, are aligned. To make sure we don't do that again, I want to be sure that all assumptions we are making hold for . That's why I'd like jemalloc to spell out what we can assume.
Ah I forgot about the stride vs. size thing. So such an object does not exist.
But such a exists, and it is something we want to support in our allocator API (we even have a test for that). And we do have objects with align size, namely all ZST.
Yeah, I agree. The only cases for which the "round to the previous power of two approach" doesn't work is for ZSTs, and types (or allocations) with align size. Since those just cannot happen in C, the C standard doesn't mention them as exceptions. That is, we can only call when that holds, and if it that doesn't hold, we need to call a different API that supports the align size case.
I've seen this quite a lot recently Example logs: Example Travis runs: https://travis- https://travis- https://travis- https://travis- I'm opening a tracking issue so we can collect some more logs and hopefully draw conclusions from them at some point. Until then I'm not really sure how we'd deal with this...
Is there a way to collect the coredump from the segfault so we could attempt to track down the reason behind the segfault? Perhaps we could at least pass to clang so we could try to reproduce locally?
your guess is as good as mine!
If you set , the core dump will end up in .
https://travis- is the first job where we got a stack trace: I wouldn't necessarily call that... illuminating
I wonder if there would be a way to print what the files we're linking are? Maybe that would help since maybe the linker segfaults on an improperly formatted file or something like that; knowing what the files are (names and lengths) may help. I think passing to clang would be good enough, at least as a start.
PRs are always welcome! I don't have any magical tricks up my sleeves to implement tricks like that unfortunately.
Next successful stack trace: https://travis-
another failure of this kind https://travis-
These look like assertion failures: Now the question is how we recover the assert message. Any Mac person knows of a way?
good thinking, I totally missed that before! Looks to be: (that's a disassembly of the executable on my system, which looks to be the same). I believe and sure enough there's in the source
Well the pthreads explains why it's nondeterministic at least...
Random attempt to help this:
Looks like did the trick, we haven't seen this in ~2 weeks, so closing.
The suggestion that rustc gives is incorrect and seems to not take into account what characters are in the format argument. I tried this code: I expected to see this happen: Instead, this happened: Using beta.2/nightly : <!-- TRIAGEBOTSTART --<!-- TRIAGEBOTASSIGNSTART --<!-- TRIAGEBOTASSIGNDATASTARTTRIAGEBOTASSIGNDATAEND --<!-- TRIAGEBOTASSIGNEND --<!-- TRIAGEBOTEND --
label -C-bug +A-diagnostics +T-compiler
claim
Alternate title: "Rust is literally C++" As of nightly-2020-10-29, rustc fails to parse the following code. The previous nightly and all past stable releases since 1.0.0 are not affected. The relevant commit range is
Mentioning because "Tweak invalid  header and body parsing" sounds extremely relevant.
Assigning and removing as in the prioritization working group.
searched nightlies: from nightly-2020-10-27 to nightly-2020-10-29 regressed nightly: nightly-2020-10-29 searched commits: from to regressed commit: <details<summarybisected with <a href=''cargo-bisect-rustc</av0.6.0</summaryHost triple: x86_64-apple-darwin Reproduce with: </detailsYeah, seems likely that is the culprit.
Yep, this was me :-/ I'll have a PR up in the next hour.
Currently (something after 0.5) rustc (IIUIC) is using rpaths to point libs and binaries to the appropriate locations. This is problematic when rustc and the other libs/bins are package e.g. for Fedora. The rpath usage should be made optional - can be used instead - and LDLIBRARYPATH.
Why is this problematic for packaging? Is it because the rpath values end up with the wrong values? Each crate's rpath has several values, one of which is supposed to be the final installation location (controlled with ). Can we just provide more control over which locations are used during rpathing?
In general Fedora recommends to not use rpaths [0], because the paths are hardcoded in the binary. Fedora preferres to point the dynamic linker to the appropriate paths [1]. But yes, we are more flexible about internal libraries - which the case is here. So yes, it would also help us if we could controll the rpath path (which goes a bit into the direction if issue ) [0] [1]
The following rpath is used in e.g. the cargo binary: The rpath referres to rpm's buildroot, where the lib won't stay (it will be instaled to ) Maybe the problem could adressed if the rpath would respect - but this could lead to problems at build time. Maybe LDLIBRARYPATH can be used to point to the correct dir's at build time, and the rpath can be used to point to the correct dir's at runtime. Just my 2ct - but you ar ethe expert :)
Thanks for those links. We do generate rpaths with the expectation that some of them will be invalidated by moving the binaries around. It's possible we could consider removing rpath, but it sure is nice to not think about dynamic loading when running binaries out of my build directory. We could easily add a flag to the configure script, to turn off the rpath for all the rust-provided libraries, with the assumption that package managers would modify during install. If we did that I feel like we would still want the installed rustc to generate rpaths with user-built libraries - otherwise the experience would be different depending on the rust installation. We could also just remove rpath completely - it doesn't work on windows anyway.
Another thing to consider is that, if we just used rustpkg for all builds, and let rustpkg install all libraries at a known location, then we might be able to conveniently not use rpath while still doing the right thing at load time. Can user directories like be put into ?
Yes - I can fully understand how easy running rust in the build directory is when using rpaths :) For the build directory use case it should be possible to create wrappers for rustc and friends which use to point to the correct dirs. You mention thet some libs might reside in , could you explain the overall concept you are following with this? In general I'd expect libraries to appear in /lib, /usr/lib some and some private /usr/lib/myapp. There are also some xdg specs that specify and - but let usfirst see what you say about ~/.rustpkg. Oh, I don't think that ~/.rustpkg would be expanded correctly if put into
Just to summarize my thoughts: If rpath is used, the rpath path beeing used should only use directories below (to point to non-standard dirs containing libs relevant for a rust component). The rpath path should not point to any build-time dir. To include libraries at buildtime or when using the binaries in the build directory I'd suggest using wrappers (isn't autotools doing it like this?) which set appropriately.
Any follow up thoughts?
Our package manager will by default not be installing libraries to a system location and will instead put them in the user's directory, . This doesn't seem out of the ordinary to me; consider cabal, etc. The behavior you recommend requires some deep changes to how Rust works, so need to be considered carefully. Does the current behavior prevent packaging in Fedora, and is there a minimal change we can make to unblock packaging? You'll probably be interested in this subject.
Thanks for the rustpkg explanation. Do I understand that rustpkg will create binaries and libs with rpaths pointing to the user's specific rustpkg dir ? My feeling is that this user specific rpaths should be configurable at build time. The use case is obviously when the binary or lib is going to be installed systemwide. The only rpath exception is the rpath pointing to the system wide package specific libdir (e.g. ).
I think that I'll spend a bit more time with looking at and and might get a better feeling for the problem. Btw.: I noted that the libdir isn't configurabel. Fedora for example keeps x86_64 libs in (/usr)?/lib64, and i686 in (/usr)/lib. IIUIC this is currently not possible with rust as the libs will be installed to /usr/lib/arch-specific-path
The point about libdir is well taken and I updated . I am not entirely sure how rustpkg does it's build now or how and intend for it to work, but based on the current language design I would expect that, yes, binaries installed to would have rpaths pointing there. Currently there are several rpaths configured for each linked library so that they hopefully fall back to something useful when the libraries are moved around during install or during the bootstrapping process, something like: The relative path to the library The absolute path to where the library was at build time The system installation path Under this scheme I would expect that would, when linking against libraries in , create rpaths to .
Yes, there are several rpaths used (seen in and ) and I'd say - even with looking at rust itself, without the packaging glasses on - that it should be configurable what rpaths are used. Maybe it should be an option of . As said, system wide binaries are the best example for binaries where you don't want to many rpaths - including no rpaths pointing to some user dir.
Please don't tag TJC :(
Sorry, TJC! Maybe it's time to give in and start working on Rust :) Seriously though, sorry. I'm sure it's a real PITA.
mentioned that this is a security issue, which I didn't realize. Debian also says .
By the way, it's possible to strip this with but it would be much nicer to not require a third party tool.
I generally wonder why you don't add a user-specific to to point to the user specific rustpkg dir instead of using rpath. That makes it much easier to cope with - and that would also be in par with (at least) the Fedora guidelines. And I suppose with Debian's too, as it's the common way. (See 3rd party java binaries. They modify the to point to some hidden dir in the userdir.)
We used to.
Users kept forgetting and/or getting annoyed having to adjust it when running from a build dir. Rpaths made it possible to run foo/bar/rustc no matter which rustc you chose.
right. IIUIC - I always feel like a noob when it comes to autotools - autotools are generating wrappers for binaries which are created during a build process and moves the original binaries to hiddend irectories. The wrappers are then modifying the environment (setting env vars etc) to achieve the same goal (so e.g. that rustc can be run from the dev directory without messing with the env vars yourself). I wonder if this could help.
Okay, it's actually not autools which creates these wrappers but libtool:
Sorry, I didn't mean to imply a disinterest in fixing this better, just noting how it came to be. Also strongly disinterested in using libtool. Would prefer going back to telling users to set env vars.
How about defaulting to static linking (when it works)? Dynamic linking is valuable, but mostly for global installs handled by a package manager. Of course, if we end up with a bunch of tools using that would be wasteful. I think RPATH will just lead to confusion, and it's an easy security hole to leave open.
I would be very happy to get static linking working well enough to make it a default in many such cases. But we ought to do the right thing when linking dynamically too. Is rpath-using- considered acceptable? I.e. is it just the absolute paths that are problematic?
The absolute paths are the ones that can easily be a security issue, so isn't that bad. I would still rather avoid it for a globally installed package but it's easy enough to strip for people who don't need it and won't be an issue if you don't know about it (like the absolute path could be). Using the rpath could be a flag, but the absolute path should never be there.
Nominating for milestone 2: backwards compatible. (The choice of whether one needs to put a setting in or use some configure flag for certain systems, etc, seems like a backwards compatibility issue. Though maybe I misunderstand and these are issues only for people developing itself, not for clients of ?) It also could well be that this is more appropriately assocated with milestone 1: well-defined.
I suspect and have a more concrete notion about this.
I don't have anything much to add to this despite it coming up for triage today. I think it has to be fixed still, I'm still happy to do it via whatever combination of static linking by default and use of is sufficient for packagers, and I haven't seen any clear movement on it in the meantime.
accepted for backwards-compatible milestone
I'm getting worried that we won't resolve this for 1.0. I agree this is important to fix. Here's my current suggestion: Modify the build system to once again set LDLIBRARYPATH/PATH to find the appropriate dynamic libraries. Add a flag to rustc to disable rpath Add a configure flag to disable rpathing the rust build itself. Packagers can use this to do whatever they want. With this logic the rust package itself could avoid rpath where distros have their own way of dealing with library paths. The rustc installed by these distros would still be rpathing user's crates for the sake of convenience. Is this an acceptable compromise or do distros want us to completely disable rpath? I'm afraid that doing so would provide an inconsistent developer experience depending on how rustc is acquired; if we need to do that then we instead must find a way to completely eliminate rpath.
If we start to just statically link everything by default then rpath becomes much less important and we can just say it's up to users to figure it out if they opt into dynamic linking.
I think we should completely remove absolute rpaths. It's okay to have $ORIGIN-based ones but it may not be a good default. I'm not sure what the answer is to that, but agree that static linking basically resolves this.
It doesn't appear that this is fully working. There are still rpaths in the binaries when configured with even though the flag for rustc does work. /usr/bin/rustc: RPATH=$ORIGIN/../lib:/build/rust-git/src/rust/x8664-unknown-linux-gnu/stage1/lib/rustlib/x8664-unknown-linux-gnu/lib:/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib
What steps did you follow to having the rpath show up? I thought I double checked on linux that the configure flag worked, but I may have overlooked something.
<!-- [31mICE[0m: Rustc '' 'thread 'rustc' panicked at 'assertion failed: bpos.tou32() = () + as u32'', 'thread 'rustc' panicked at 'assertion failed: bpos.tou32() = () + as u32'' File: --snippet: Version information Command: <!-- Include a backtrace in the code block by setting in your environment. E.g. . --<details<summary<strongProgram output</strong</summary<p</p</details<!-- query stack: [typeck] type-checking [analysis] running analysis passes on this crate --
cc
<!-- Thank you for filing a regression report! A regression is something that changed between versions of Rust but was not supposed to. Please provide a short summary of the regression, along with any information you feel is relevant to replicate it. --I tried this code: on stable 1.78.0, this checks in 0.02 seconds. on beta 1.79.0-beta.3, this hangs somewhere in , I killed it after 5 minutes of no progress.
cc
Sharing the stack would be useful
:thinking: that doesn't really seem to be it
You need to switch to like thread 2 or 3 if you're doing this in gdb
it's not particularly deep, but it looks downstream of so I guess this must be it
Minimal:
Same error, with the last github version.
Also getting this, I'm running the Yosemite GM 1. Not sure if it's related.
I also getting this on MacOs
Still getting this make error, like many others did. Anyone want to fix this?
I have no idea how to fix this, other than just installing xetex or however it's spelled
The configure(or make) did find LuaTeX.
Sadly I don't have time to work on this right now. A simple fix would be to disable pdf generation unless specified in configure, or to require xetex.
I don't think that LaTeX should be a dependency to build a programming language like Rust.
it seems reasonable to have the option to not build PDF docs if you don't want to
I think I'm getting the same error, which is strange because I believe I was able to a couple days ago. In the log, the complaint is that is not found. As suggested above, is included with MacPorts' package and installing that allowed compilation of docs to go through.
will build without building the PDF docs
Error happened while working on updating closure capture analysis in rustc which is part of rustctypeck. The highlighted line is what changed between the build that broke and the build before it. <!-- If you're using the stable version of the compiler, you should also check if the bug also exists in the beta or nightly versions. --Version reported in build log and similarly one or two more Complete build log: Build log with : Build directory: (Heads up, this is 29G tar, 39G extracted)
What was the exact build command that caused the ICE? If this is with , then I also see this issue every now and then. Locally built rustc all have the "dev" version string, which means it doesn't have to rebuild everything just because you do , but it also means it cannot detect that the ABI changed, so it might try to load old files with the new rustc and of course that will fail. I am not sure if much can be done about this.
Some potential duplicates: , , ,
I do . This happened to me again last night: The assertion is fired in the not the same line but the same file, I haven't gotten around looking at the query code to see what failed. Only thing I worked on since the last x.py clean (maybe 10-15ish builds in, though I could be wrong) was change closure/generator pretty printing (in rustcmiddle/src/ty/print/pretty), haven't rebased since. Though I can see that rustc builds are themselves are more prone incr compile errors, also didn't think had anything to do with the build process, otherthan maybe some version strings.
Cc I mentioned because one way to avoid this problem might be to embed the git commit in the version string used for rebuild detection -- but that would be rather annoying in many cases.
You can set ignore-git to false in and that'll embed git commit hashes. But I think that must only patch over bugs - it can't be necessary. I doubt it will fully fix this bug. I suspect the problem is that Cargo doesn't delete incremental state on dependency changes - which means that rustc must be prepared, currently, to see incremental state serialized by a different rustc... ideally, handling that in a way that doesn't just throw everything out, but even just not panicking would probably be enough.