jiebi/CodeConvo huggingface dataset:
Code change:
opt local-rust 0 "use an installed rustc rather than downloading a snapshot" opt pax-flags 0 "apply PaX flags to rustc binaries (required for GRSecurity/PaX-patched kernels)" opt inject-std-version 1 "inject the current compiler version of libstd into programs" <ins> opt rpath 1 "build rpaths into rustc itself" </ins> valopt prefix "/usr/local" "set installation prefix" valopt local-rust-root "/usr/local" "set prefix for local rust binary" valopt llvm-root "" "set LLVM root"

Related discussion:
text: Currently (something after 0.5) rustc (IIUIC) is using rpaths to point libs and binaries to the appropriate locations. This is problematic when rustc and the other libs/bins are package e.g. for Fedora. The rpath usage should be made optional - can be used instead - and LDLIBRARYPATH.
Why is this problematic for packaging? Is it because the rpath values end up with the wrong values? Each crate's rpath has several values, one of which is supposed to be the final installation location (controlled with ). Can we just provide more control over which locations are used during rpathing?
In general Fedora recommends to not use rpaths [0], because the paths are hardcoded in the binary. Fedora preferres to point the dynamic linker to the appropriate paths [1]. But yes, we are more flexible about internal libraries - which the case is here. So yes, it would also help us if we could controll the rpath path (which goes a bit into the direction if issue ) [0] [1]
The following rpath is used in e.g. the cargo binary: The rpath referres to rpm's buildroot, where the lib won't stay (it will be instaled to ) Maybe the problem could adressed if the rpath would respect - but this could lead to problems at build time. Maybe LDLIBRARYPATH can be used to point to the correct dir's at build time, and the rpath can be used to point to the correct dir's at runtime. Just my 2ct - but you ar ethe expert :)
Thanks for those links. We do generate rpaths with the expectation that some of them will be invalidated by moving the binaries around. It's possible we could consider removing rpath, but it sure is nice to not think about dynamic loading when running binaries out of my build directory. We could easily add a flag to the configure script, to turn off the rpath for all the rust-provided libraries, with the assumption that package managers would modify during install. If we did that I feel like we would still want the installed rustc to generate rpaths with user-built libraries - otherwise the experience would be different depending on the rust installation. We could also just remove rpath completely - it doesn't work on windows anyway.
Code change:
opt local-rust 0 "use an installed rustc rather than downloading a snapshot" opt pax-flags 0 "apply PaX flags to rustc binaries (required for GRSecurity/PaX-patched kernels)" opt inject-std-version 1 "inject the current compiler version of libstd into programs" <ins> opt rpath 1 "build rpaths into rustc itself" </ins> valopt prefix "/usr/local" "set installation prefix" valopt local-rust-root "/usr/local" "set prefix for local rust binary" valopt llvm-root "" "set LLVM root"

Related discussion:
text: Another thing to consider is that, if we just used rustpkg for all builds, and let rustpkg install all libraries at a known location, then we might be able to conveniently not use rpath while still doing the right thing at load time. Can user directories like be put into ?
Yes - I can fully understand how easy running rust in the build directory is when using rpaths :) For the build directory use case it should be possible to create wrappers for rustc and friends which use to point to the correct dirs. You mention thet some libs might reside in , could you explain the overall concept you are following with this? In general I'd expect libraries to appear in /lib, /usr/lib some and some private /usr/lib/myapp. There are also some xdg specs that specify and - but let usfirst see what you say about ~/.rustpkg. Oh, I don't think that ~/.rustpkg would be expanded correctly if put into
Just to summarize my thoughts: If rpath is used, the rpath path beeing used should only use directories below (to point to non-standard dirs containing libs relevant for a rust component). The rpath path should not point to any build-time dir. To include libraries at buildtime or when using the binaries in the build directory I'd suggest using wrappers (isn't autotools doing it like this?) which set appropriately.
Any follow up thoughts?
Our package manager will by default not be installing libraries to a system location and will instead put them in the user's directory, . This doesn't seem out of the ordinary to me; consider cabal, etc. The behavior you recommend requires some deep changes to how Rust works, so need to be considered carefully. Does the current behavior prevent packaging in Fedora, and is there a minimal change we can make to unblock packaging? You'll probably be interested in this subject.
Thanks for the rustpkg explanation. Do I understand that rustpkg will create binaries and libs with rpaths pointing to the user's specific rustpkg dir ? My feeling is that this user specific rpaths should be configurable at build time. The use case is obviously when the binary or lib is going to be installed systemwide. The only rpath exception is the rpath pointing to the system wide package specific libdir (e.g. ).
Code change:
opt local-rust 0 "use an installed rustc rather than downloading a snapshot" opt pax-flags 0 "apply PaX flags to rustc binaries (required for GRSecurity/PaX-patched kernels)" opt inject-std-version 1 "inject the current compiler version of libstd into programs" <ins> opt rpath 1 "build rpaths into rustc itself" </ins> valopt prefix "/usr/local" "set installation prefix" valopt local-rust-root "/usr/local" "set prefix for local rust binary" valopt llvm-root "" "set LLVM root"

Related discussion:
text: I think that I'll spend a bit more time with looking at and and might get a better feeling for the problem. Btw.: I noted that the libdir isn't configurabel. Fedora for example keeps x86_64 libs in (/usr)?/lib64, and i686 in (/usr)/lib. IIUIC this is currently not possible with rust as the libs will be installed to /usr/lib/arch-specific-path
The point about libdir is well taken and I updated . I am not entirely sure how rustpkg does it's build now or how and intend for it to work, but based on the current language design I would expect that, yes, binaries installed to would have rpaths pointing there. Currently there are several rpaths configured for each linked library so that they hopefully fall back to something useful when the libraries are moved around during install or during the bootstrapping process, something like: The relative path to the library The absolute path to where the library was at build time The system installation path Under this scheme I would expect that would, when linking against libraries in , create rpaths to .
Yes, there are several rpaths used (seen in and ) and I'd say - even with looking at rust itself, without the packaging glasses on - that it should be configurable what rpaths are used. Maybe it should be an option of . As said, system wide binaries are the best example for binaries where you don't want to many rpaths - including no rpaths pointing to some user dir.
Please don't tag TJC :(
Sorry, TJC! Maybe it's time to give in and start working on Rust :) Seriously though, sorry. I'm sure it's a real PITA.
mentioned that this is a security issue, which I didn't realize. Debian also says .
By the way, it's possible to strip this with but it would be much nicer to not require a third party tool.
I generally wonder why you don't add a user-specific to to point to the user specific rustpkg dir instead of using rpath. That makes it much easier to cope with - and that would also be in par with (at least) the Fedora guidelines. And I suppose with Debian's too, as it's the common way. (See 3rd party java binaries. They modify the to point to some hidden dir in the userdir.)
We used to.
Code change:
opt local-rust 0 "use an installed rustc rather than downloading a snapshot" opt pax-flags 0 "apply PaX flags to rustc binaries (required for GRSecurity/PaX-patched kernels)" opt inject-std-version 1 "inject the current compiler version of libstd into programs" <ins> opt rpath 1 "build rpaths into rustc itself" </ins> valopt prefix "/usr/local" "set installation prefix" valopt local-rust-root "/usr/local" "set prefix for local rust binary" valopt llvm-root "" "set LLVM root"

Related discussion:
text: Users kept forgetting and/or getting annoyed having to adjust it when running from a build dir. Rpaths made it possible to run foo/bar/rustc no matter which rustc you chose.
right. IIUIC - I always feel like a noob when it comes to autotools - autotools are generating wrappers for binaries which are created during a build process and moves the original binaries to hiddend irectories. The wrappers are then modifying the environment (setting env vars etc) to achieve the same goal (so e.g. that rustc can be run from the dev directory without messing with the env vars yourself). I wonder if this could help.
Okay, it's actually not autools which creates these wrappers but libtool:
Sorry, I didn't mean to imply a disinterest in fixing this better, just noting how it came to be. Also strongly disinterested in using libtool. Would prefer going back to telling users to set env vars.
How about defaulting to static linking (when it works)? Dynamic linking is valuable, but mostly for global installs handled by a package manager. Of course, if we end up with a bunch of tools using that would be wasteful. I think RPATH will just lead to confusion, and it's an easy security hole to leave open.
I would be very happy to get static linking working well enough to make it a default in many such cases. But we ought to do the right thing when linking dynamically too. Is rpath-using- considered acceptable? I.e. is it just the absolute paths that are problematic?
The absolute paths are the ones that can easily be a security issue, so isn't that bad. I would still rather avoid it for a globally installed package but it's easy enough to strip for people who don't need it and won't be an issue if you don't know about it (like the absolute path could be). Using the rpath could be a flag, but the absolute path should never be there.
Nominating for milestone 2: backwards compatible. (The choice of whether one needs to put a setting in or use some configure flag for certain systems, etc, seems like a backwards compatibility issue. Though maybe I misunderstand and these are issues only for people developing itself, not for clients of ?) It also could well be that this is more appropriately assocated with milestone 1: well-defined.
Code change:
opt local-rust 0 "use an installed rustc rather than downloading a snapshot" opt pax-flags 0 "apply PaX flags to rustc binaries (required for GRSecurity/PaX-patched kernels)" opt inject-std-version 1 "inject the current compiler version of libstd into programs" <ins> opt rpath 1 "build rpaths into rustc itself" </ins> valopt prefix "/usr/local" "set installation prefix" valopt local-rust-root "/usr/local" "set prefix for local rust binary" valopt llvm-root "" "set LLVM root"

Related discussion:
text: I suspect and have a more concrete notion about this.
I don't have anything much to add to this despite it coming up for triage today. I think it has to be fixed still, I'm still happy to do it via whatever combination of static linking by default and use of is sufficient for packagers, and I haven't seen any clear movement on it in the meantime.
accepted for backwards-compatible milestone
I'm getting worried that we won't resolve this for 1.0. I agree this is important to fix. Here's my current suggestion: Modify the build system to once again set LDLIBRARYPATH/PATH to find the appropriate dynamic libraries. Add a flag to rustc to disable rpath Add a configure flag to disable rpathing the rust build itself. Packagers can use this to do whatever they want. With this logic the rust package itself could avoid rpath where distros have their own way of dealing with library paths. The rustc installed by these distros would still be rpathing user's crates for the sake of convenience. Is this an acceptable compromise or do distros want us to completely disable rpath? I'm afraid that doing so would provide an inconsistent developer experience depending on how rustc is acquired; if we need to do that then we instead must find a way to completely eliminate rpath.
If we start to just statically link everything by default then rpath becomes much less important and we can just say it's up to users to figure it out if they opt into dynamic linking.
I think we should completely remove absolute rpaths. It's okay to have $ORIGIN-based ones but it may not be a good default. I'm not sure what the answer is to that, but agree that static linking basically resolves this.
It doesn't appear that this is fully working. There are still rpaths in the binaries when configured with even though the flag for rustc does work. /usr/bin/rustc: RPATH=$ORIGIN/../lib:/build/rust-git/src/rust/x8664-unknown-linux-gnu/stage1/lib/rustlib/x8664-unknown-linux-gnu/lib:/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib
What steps did you follow to having the rpath show up? I thought I double checked on linux that the configure flag worked, but I may have overlooked something.
Code change:
# Make `RefCell` store additional debugging information, which is printed out when # a borrow error occurs debug_refcell = [] <ins> [lints.rust.unexpected_cfgs] level = "warn" # x.py uses beta cargo, so `check-cfg` entries do not yet take effect # for rust-lang/rust. But for users of `-Zbuild-std` it does. # The unused warning is waiting for rust-lang/cargo#13925 to reach beta. check-cfg = [ 'cfg(bootstrap)', 'cfg(no_fp_fmt_parse)', 'cfg(stdarch_intel_sde)', # This matches `EXTRA_CHECK_CFGS` in `src/bootstrap/src/lib.rs`. 'cfg(feature, values(any()))', ] </ins>

Related discussion:
text: With extra verbose flag (), one would see a lots of warnings about check-cfg when building -Zbuild-std. But it doesn't need to be that way. Can we doodfooding check-cfg flag to and to resolve them? Step to reproduce: I expected to see this happen: No warnings about " unexpected " Some are: label +F-check-cfg +-Zbuild-std cc
Code change:
/// Metadata encoding version. /// NB: increment this if you change the format of metadata such that <del> /// the rustc version can't be found to compare with `RUSTC_VERSION`. pub const METADATA_VERSION: u8 = 3; </del> <ins> /// the rustc version can't be found to compare with `rustc_version()`. pub const METADATA_VERSION: u8 = 4; </ins> /// Metadata header which includes `METADATA_VERSION`. /// To get older versions of rustc to ignore this metadata, /// there are 4 zero bytes at the start, which are treated /// as a length of 0 by old compilers. /// <del> /// This header is followed by the position of the `CrateRoot`. </del> <ins> /// This header is followed by the position of the `CrateRoot`, /// which is encoded as a 32-bit big-endian unsigned integer, /// and further followed by the rustc version string. </ins> pub const METADATA_HEADER: &'static [u8; 12] = &[0, 0, 0, 0, b'r', b'u', b's', b't', 0, 0, 0, METADATA_VERSION];

Related discussion:
text: thread 'rustc' panicked at 'index out of bounds: the len is but the index is ', ../src/libserialize kind of expected, after switching toolchain from 1.15 back to 1.14. cargo clean && cargo build "repaired it"
Do you have some more information on how to reproduce this?
I hit this compiling hyper with 1.13 stable.
Which version of hyper is that? I can't reproduce with latest .
It was with master. I tried and now it works. So rustc/cargo were using libs compiled when I tried this with 1.15nightly?
Sounds a bit like cargo was re-using libraries that it should have re-compiled because the compiler has changed. cc
Sorry if I did not save the full backtrace - It was my first issue report. I switched away from 1.15 nightly back to 1.14 nightly in order to solve a serde macro issue. Cargo cleaning and rebuilding fixed the issue. I will make sure to run again and capture the output if it happens again. On 11/16/2016 4:54 PM, Michael Woerister wrote:
Sounds suspiciously like a Cargo bug! If there's a reproduction I'd be more than willing to dig in.
I believe I just reproduced this in Cargo itself: When executing with the last few lines are: So what's happening here is: is populating the entire crate cache with a bunch of libraries on the nightly compiler. switching compilers, Cargo decides to recompile everything. doesn't delete everything as it recompiles, rather incrementally builds up a new crate cache compiler, given to that directory, will read crates in that directory. compiler panics when reading a crate compiled by a future compiler. So I take back what I said earlier about this likely being a Cargo bug, it seems like this may be a bug in the compiler trying to read stale metadata? I thought we had like 10 checks in place to ensure we didn't do that, but maybe it's still leaking through?
I'm going to tag this as a stable-to-nightly regression as this appears to have been introduced in nightly. If I build cargo with beta and then switch to beta, the compiler doesn't ICE.
The panic on stable happens in we check version information. Did we change how we encode version information?
Code change:
/// Metadata encoding version. /// NB: increment this if you change the format of metadata such that <del> /// the rustc version can't be found to compare with `RUSTC_VERSION`. pub const METADATA_VERSION: u8 = 3; </del> <ins> /// the rustc version can't be found to compare with `rustc_version()`. pub const METADATA_VERSION: u8 = 4; </ins> /// Metadata header which includes `METADATA_VERSION`. /// To get older versions of rustc to ignore this metadata, /// there are 4 zero bytes at the start, which are treated /// as a length of 0 by old compilers. /// <del> /// This header is followed by the position of the `CrateRoot`. </del> <ins> /// This header is followed by the position of the `CrateRoot`, /// which is encoded as a 32-bit big-endian unsigned integer, /// and further followed by the rustc version string. </ins> pub const METADATA_HEADER: &'static [u8; 12] = &[0, 0, 0, 0, b'r', b'u', b's', b't', 0, 0, 0, METADATA_VERSION];

Related discussion:
text: cc
I incremented the header version number though... It should be impossible for older rustc to get that far.
on it looks the same on . Perhaps we just need a PR to bump that?
Oh it looks like my changes have reached stable. I wonder if perhaps the is in a precarious spot and/or has been moved or the serialization otherwise changed. This requires further investigation.
Why is this even calling leb128? Are we using variable-length integers in the metadata header?
I assumed leb128 was the crate being built, not one used by the compiler.
Sure enough. Now every change to requires us to update the LLVM version number, because checking the rustc version number requires parsing . I think we should go back to JSON version guard.
Ahh that's what it was. We can put it at the start then, before/after the absolute address. I'd forgotten that the whole struct would have to be decoded.
What was the PR that modified the versioning scheme? I gather that something has changed about it. Can somebody explain more clearly what is going wrong here?
I mistakenly put the version string inside a larger , somehow thinking that it being at the start helped, but if the whole (that is, ) can't be decoded (e.g. because its definition changed) then you don't get to see the version string, whereas before it was separate. The fix would be to separate it again, put it back at the start and leave it there. Or just encode/decode it before instead of as part of it. Either will work as long as it can't be affected by changes to the metadata schema. The PR that introduced the new schema was , although breakage didn't happen until changed later, for procedural macros and/or the name resolution changes ( would know).
removed the field from for macro modularization.
I believe I have a fix ready. Does this affect beta too now, or just nightly?
AFAIK it's just nightly, not beta or stable
Code change:
pub struct BufWriter<W: Write> { inner: Option<W>, buf: Vec<u8>, <ins> // #30888: If the inner writer panics in a call to write, we don't want to // write the buffered data a second time in BufWriter's destructor. This // flag tells the Drop impl if it should skip the flush. panicked: bool, </ins> } /// An error returned by `into_inner` which combines an error that

Related discussion:
text: There is a panic safety issue in : after a call panics, the impl of calls again, which means the buffer contents are potentially written twice. This may cause an application to overwrite parts of a file that it did not mean to overwrite (in a DB engine written in Rust, this could cause unrecoverable data corruption!). Demonstration: The expected output of the demo program is , the actual output is: More generally, we need a story for panic safety in Rust. My takeaway from the related discussions (e.g. , , the trait) was that only code and impls should have to worry about panic safety. The demo app contains none of these, so I'd consider this a bug in . (otherwise all implementations would need to provide the strong exception safety guarantee?) Solution: could use temporarily mark the buffer as empty during the calls; so that the impl doesn't do anything after a panic. However, this doesn't help if the panic occurs during a call...
Note that Rust is normally referring to "memory safety" when talking about safety, so the issue here isn't in conflict with the related discussions. It's definitely a bug in , but I want to point out how specific it is here: the only reason you have odd behaviour is because you have a implementation that will panic after successfully writing the data it is given. I'm all for making more robust, but it's not just "if the call panics", it's "if the call panics after successfully writing data". Panicking before a successful write (or if the write fails), would not have the same issue.
If your impl is panicking in the middle of a write, it seems to me like you're in data corruption land with or without a second write. This is not quite the case - and thread locals expand that surface area. What's specifically the expected behavior here? Is the implementation specifically the thing that needs tweaking, or should the totally poison itself? For reference, Java's class has no special handling for exceptions thrown from the inner : , so adjusting alone seems like a reasonable route to take.
Without the , a panic in the middle of a write (without ) only corrupts the part of the file that the application meant to overwrite. In a DB engine context, that means the resulting corruption can be repaired when the transaction is rolled back.