Change:
Tracking issue for rust-lang/rfcs. Unresolved questions: [x] Does assigning directly to a union field trigger a drop of the previous contents? , , , ) Answered by [ ] Under what conditions can you implement for a union? For example, what if some variants are of non-Copy type? All variants? [ ] What interaction is there between unions and enum layout optimizations? () Open issues of high import: [x] -- MIR-based unsafety checker sometimes accepts unsafe accesses to union fields in presence of uninhabited fields
I may have missed it in the discussion on the RFC, but am I correct in thinking that destructors of union variants are never run? Would the destructor for the run in this example?
My current understanding is that will run the destructor but would not. That is, assigning to a lvalue will cause a drop like always, but assigning to a lvalue will not.
So an assignment to a variant is like an assertion that the field was previously "valid"?
For types, yeah, that's my understanding. If they weren't previously valid you need to use the constructor form or . From a quick grep, it doesn't seem like the RFC is explicit about this detail, though. I see it as an instantiation of the general rule that writing to a lvalue causes a destructor call.
Should a &mut union with Drop variants be a lint? On Friday, 8 April 2016, Scott Olson wrote:
On April 8, 2016 3:36:22 PM PDT, Scott Olson wrote: I should have covered that case explicitly. I think both behaviors are defensible, but I think it'd be far less surprising to never implicitly drop a field. The RFC already recommends a lint for union fields with types that implement Drop. I don't think assigning to a field implies that field was previously valid.
Yeah, that approach seems a bit less dangerous to me as well.
Not dropping when assigning to a union field would make act differently from , because you can't make the latter case not drop. I think my approach is less surprising. It's not a new problem, either; programmers already have to deal with other situations where is UB if is uninitialized and .
I personally don't plan to use Drop types with unions at all. So I'll defer entirely to people who have worked with analogous unsafe code on the semantics of doing so.
I also don't intend to use Drop types in unions so either way doesn't matter to me as long as it is consistent.
I don't intend to use mutable references to unions, and probably just "weirdly-tagged" ones with Into<EnumOn Friday, 8 April 2016, Peter Atashian wrote:
Seems like this is a good issue to raise up as an unresolved question. I'm not sure yet which approach I prefer.
As much as I find it awkward for assigning to a union field of a type with Drop to require previous validity of that field, the reference case mentioned seems almost unavoidable. I think this might just be a gotcha associated with code that intentionally disables the lint for putting a type with Drop in a union. (And a short explanation of it should be in the explanatory text for that lint.)
And I'd like to reiterate that programmers must already generally know that means to write safe code. Not dropping union fields would be one more exception to learn, not one less. (NB: the drop doesn't happen when is statically known to already be uninitialized, like .) But I support having a default warning against variants in unions that people have to since this is a fairly non-obvious detail.
this is not true for and maybe only sometimes true for but it is certainly true for . This uncertainty is what made me hesitant about it. For example, this compiles: (though trying to print later fails, but I consider that a bug)
That example is new to me. I guess I would have considered it a bug that that example compiles, given my previous experience. But I'm not sure I see the relevance of that example. Why is what I said not true for and only sometimes for ? Say, if had a type with a destructor, surely that destructor is called:
Maybe just lint against that kind of write?
My point is only that does not always run the destructor; it uses some knowledge about whether the target is known to be initialized. On Tue, Apr 12, 2016 at 04:10:39PM -0700, Scott Olson wrote:
It runs the destructor if the drop flag is set. But I think that kind of write is confusing anyway, so why not just forbid it? You can always do .
I already mentioned that: But I didn't account for dynamic checking of drop flags, so this is definitely more complicated than I considered.
Drop flags are only semi-dynamic - after zeroing drop is gone, they are a part of codegen. I say we forbid that kind of write because it does more confusion than good.
Should types even be allowed in unions? If I'm understanding things correctly, the main reason to have unions in Rust is to interface with C code that has unions, and C doesn't even have destructors. For all other purposes, it seems that it's better to just use an in Rust code.
There is a valid use case for using a union to implement a type which inhibits drop.
As well as invoking such code manually via or similar.
To me dropping a field value while writing to it is definitely wrong because the previous option type is undefined. Would it be possible to prohibit field setters but require full union replacement? In this case if the union implements Drop full union drop would be called for the value replaced as expected.
I don't think it makes sense to prohibit field setters; most uses of unions should have no problem using those, and fields without a Drop implementation will likely remain the common case. Unions with fields that implement Drop will produce a warning by default, making it even less likely to hit this case accidentally.
For the sake of discussion, I intend to expose mutable references to fields in unions and put arbitrary (possibly ) types into them. Basically, I would like to use unions to write custom space-efficient enums. For example,
I'd like to propose a concrete answer to the question currently listed as unresolved here. To avoid unnecessarily complex semantics, assigning to a union field should act like assigning to a struct field, which means dropping the old contents. It's easy enough to avoid this if you know about it, by assigning to the whole union instead. This is still slightly surprising behavior, but having a union field that implements at all will produce a warning, and the text of that warning can explicitly mention this as a caveat. Would it make sense to provide an RFC pull request amending RFC1444 to document this behavior?
Since is away on vacation, I'll answer: I think it's great form to file an amendment RFC for resolving questions like this. We'd often fast-track such RFC PRs when appropriate.
Thanks. I've filed the new RFC PR with these RFC1444, to resolve this issue.
<sub<sub( you can check-off that unresolved question now.)</sub</sub
I have some preliminary implementation in Status: Implemented (modulo bugs), PR submitted ().
Awesome! Looks great so far.
I'm not quite sure how to treat unions with non- fields in move checker. Suppose is an initialized value of and now we move out of one of the fields: 1) This is simple, is also put into uninitialized state. Sanity check: should behave exactly like + field alias. 2) Supposedly should still be initialized, because is simply a and doesn't change in any way. 2) This is the strangest case; supposedly should also be put into uninitialized state despite being . values can be uninitialized (e.g. ), but changing their state from initialized to uninitialized is something new, AFAIK.
I know this is completely irrelevant to FFI needs :) The good news is that it's not a blocker, I'm going to implement whatever behavior is simpler and submit PR this weekend.
my instinct is that unions are a "bit-bucket", essentially. You are responsible for tracking whether the data is initialized or not and what it's true type is. This is very similar to the referent of a raw pointer. This is why we can't drop the data for you, and also why any access to the fields is unsafe (even if, say, there is only one variant). By these rules, I would expect unions to implement if copy is implemented for them. Unlike structs/enums, however, there would be no internal sanity checks: you can always implement copy for a union type if you like.
Let me give some examples to clarify: This union is affine, because has not been implemented. This union type is copy, because has been implemented. Note that if were a struct, it would be an error to implement because of the type of the field . Huh, I guess I'm not actually answering your question though, now that I re-read it. =)
OK, so, I realize I was not answering your question at all. So let me try again. Following the "bit bucket" principle, I would still expect that we can move out from a union at will. But of course another option would be to treat it like we treat a , and require you to use to move out. EDIT: I'm not actually entire sure why we would prohibit such moves. It might have had to do w/ moving drop -- or perhaps just because it's easy to make a mistake and it seems better to make "moves out" more explicit? I am having trouble remembering the history here.
Ha, I, on the contrary, would like to give as many guarantees about union's content as we can for such a dangerous construct. The interpretation is that union is an enum for which we don't know the discriminant, i.e. we can guarantee that at any moment of time at least one of union's variants has valid value (unless unsafe code is involved). All the borrow/move rules in the current implementation support this guarantee, simultaneously this is the most conservative interpretation, that allows us to go either the "safe" way (for example, allowing safe access to unions with equally typed fields, this can be ) or the "bit bucket" way in the future, when more experience with Rust unions is gathered. Actually, I'd like to make it even more conservative as described in
Note that unsafe code is always involved, when working with a union, since every access to a field is unsafe. The way I think of it is, I think, similar. Basically a union is like an enum but it can be in more than one variant simultaneously. The set of valid variants is not known to the compiler at any point, though sometimes we can figure out that the set is empty (i.e., the enum is uninitialized). So I see any use of as basically an implicit (and unsafe) assertion that the set of valid variants currently includes . This seems compatible with how the borrow-checker integration works; if you borrow field and then try to use , you are getting an error because you are basically saying that the data is simultaneously and (and it is borrowed). (In contrast, with a regular enum, it is not possible to inhabit more than one variant at a time, and you ). Anyway, the point is, when we "move" from one field of a union, the question at hand I guess is whether we can deduce that this implies that interpreting the value as the other variants is no longer valid. I think it'd be not so hard to argue either way, though. I consider this a grey zone. The danger of being conservative is that we might well rule out unsafe code that would otherwise make sense and be valid. But I'm ok with starting out tighter and deciding whether to loosen later. We should discuss the matter of what conditions are needed to implement on a union -- also, we should make sure we have a complete list of these grey areas listed above to make sure we address and document before stabilization!
One argument against the "more than one variant" interpretation is how unions behave in constant expressions - for these unions we always know the single active variant and also can't access inactive variants because transmuting at compile time is generally bad (unless we are trying to turn the compiler into some kind of partial target emulator). My interpretation is that at runtime inactive variants are still inactive but can be accessed if they are layout compatible with the union's active variant (more restrictive definition) or rather with union's fragment assignment history (more vague, but more useful). I'm going to amend the union RFC in some not-so-remote future! The "enum" interpretation has pretty fun consequences.
This is one of the goals of my project. Miri can already do transmutes and various raw pointer shenanigans. It would be a small amount of work to make Miri handle unions (nothing new on the raw memory handling side). And is pushing to replace rustc constant evaluation with a version of Miri.
How to best support the use of unions in constants is an interesting question, but I see no problem with restricting constant expressions to a subset of runtime behavior (this is what we always do, anyhow). That is, just because we may not be able to fully support some particular transmute at compilation time doesn't mean it's illegal at runtime. Hmm, I'm trying to think how this is different from saying that the union belongs to all of those variants simultaneously. I don't really see a difference yet. :) I feel like this interpretation has odd interactions with moves in general. For example, if the data is "really" an X, and you interpret it as a Y, but Y is affine, then is it still an X? Regardless, I think it's fine that having a move of any field consume the entire union can be seen as consistent with any of these interpretations. For example, in the "set of variants" approach, the idea is just that moving the value deinitializes all existing variants (and of course the variant you used must be one of the valid set). In your version, it would seem to "transmute" into that variant (and consume the original). Such confidence! You're going to try ;) Care to shed a few more details on what concrete changes you have in mind?
More detailed description of the implementation (i.e. better documentation), some small extensions (like empty unions and in union patterns), two main (contradicting) alternatives of union evolution - more unsafe and less restrictive "scratch space" interpretation and more safe and more restrictive "enum with unknown discriminant" interpretation - and their consequences for move/initialization checker, impls, ty of field access, etc.
It would also be useful to define when accessing an inactive union field is UB, e.g. but this is an infinitely tricky area.
Sounds likely, cross-field semantics are basically a pointer cast right? (() as *u8) On Thursday, 1 September 2016, Vadim Petrochenkov wrote:
Isn't field access always unsafe? On Thursday, 1 September 2016, Vadim Petrochenkov wrote:
It can be made safe sometimes, e.g. assignment to trivially-destructible union fields is safe. any access to fields of a (i.e. all fields have the same type) is safe in "enum with unknown discriminant" interpretation.
It seems better not to apply special conditions to this, from a user point of view. Just call it unsafe, always.
Currently testing out the support for unions in the latest rustc from git. Everything I've tried works perfectly.
I ran into an interesting case in the dead field checker. Try the following code: You'll get this error: Looks like the dead_code checker didn't notice the initialization. (I already filed PR about the use of "struct field", changing it to just "field".)
Unions cannot currently contain dynamically sized fields, but the RFC doesn't specify this behavior either way: Output:
Contextual keyword does not work outside the module/crate root context: Seems like a pretty nasty consistency wart.
Are you using some older version of rustc by accident? I've just checked your example on playpen and it works (modulo "empty union" errors). There's also a run-pass test checking for this specific situation -
ah, I used , but it seems like it might have reverted to or something. Never mind me, then.
I think unions will eventually need to support safe fields, the evil twins of unsafe fields from . cc
I do think it would make sense to have a way of declaring "safe unions", based on various criteria. For instance, a union containing exclusively Copy non-Drop fields, all with the same size, seems safe; no matter how you access the fields, you might get unexpected data, but you can't encounter a memory safety problem or undefined behavior.
You would also need to make sure there are no "holes" in the types from which you could read uninitialized data. As I like to say: "Uninitialized data is either unpredictable random data or your SSH private key, whichever is worse."
Isn't Copy and non-Drop? Put that in a "safe" union with , and you've got a rogue reference generator. So the rules will have to be a bit more strict than that.
I realize this is just an off-the-cuff example, not a serious proposal, but here are some examples to illustrate how tricky this is: and have the same size, but most values are invalid for and ignoring this triggers UB and have the same size and are for all and (as long as both or neither are ) transmuting between / and is currently only possible in unsafe code. I believe such transmutes are always safe but this does enlarge the safe language so it may be controverial. Violating privacy (e.g. punning between and ) is a big no no as wel, since privacy might be used to uphold invariants relevant to safety.
When I was writing that, I'd meant to include a note about not having any internal padding, and somehow forgot to do so. Thanks for catching that. I was trying to define "plain old data", as in something that contains zero pointers. You're right, the definition would need to exclude references. Probably easier to whitelist a set of allowed types and combinations of those types. Good point; the same issue applies to enums. I'd forgotten that. Agreed on both points; this seems acceptable to allow. If you don't know the internals of the type, you can't know if the internals meet the requirements (such as no internal padding). So you could exclude this by requiring that all components be plain-old-data, recursively, and that you have sufficient visibility to verify that.
Floating point numbers have signalling NaN which is a trap representation which results in UB.
Does Rust run on any platforms that don't support disabling floating-point traps? (That doesn't change the UB issue, but in theory we could make that problem go away.)
I think I've come around to this interpretation -- well, not exactly this. I still think of it as there is some set of legal variants which is determined at the point where you store, as I always did. I think of storing a value into a union as a bit like putting it into a "quantum state" -- it could now potentially be transmuted into one of many legal interpretations. But I agree that when you move out from one of these variants, you have "forced" it into just one of those, and consumed the value. Therefore, you shouldn't be able to use the enum again (if that type is not ). So , basically.
Question about : as recently pointed out to me, the current spec states that if a union is not , it is illegal to store with field and read with field . Presumably this is because we are not required to start all fields at the same offset. I can see some utility in this: for example, a sanitizer might implement unions by storing them like a normal enum (or even a struct...?) and checking that you use the same variant that you put in. But it seems like a kind of footgun, and also one of those repr guarantees we would never be able to actually change in practice, because too many people will be relying on it in the wild. Thoughts?
The worst part is variant/field fragments, which are directly accessible for unions. Consider this code: All fields are , no ownership involved and move checker is happy, but the partial assignment to the inactive field turns the union into state with valid variants. I haven't thought how to deal with it yet. Make such assignments UB? Change the interpretation? Something else?
This would be my assumption, yes. When you assigned with , the variant was not in the set of valid variants, and hence later using (whether to read or assign) is invalid.
I think the appropriate wording here is that 1) Reading from fields that are not "layout compatible" (this is vague) with previously written fields/field-fragments is UB 2) For unions users know what layouts are (from ABI docs) so they can discern between UB and non-UB 3) For union layouts are unspecified so users can't say what is UB and what is not, but WE (rustc/libstd + their tests) have this sacred knowledge, so we can separate the wheat from the chaff and use in non-UB fashion. 4) After size/stride and field reordering questions are decided upon I'd expect struct and union layouts to be set in stone and specified, so users will know the layouts as well and will be able to use unions as freely as and the problem will go away.
In the discussion of the union RFC, people mentioned wanting to have native Rust code that uses unions to build compact data structures.
Is there anything stopping from people using ? If not, then I don't see the need to provide any sort of guarantees for , just leave it as "here be dragons". Probably would be best to have a lint that is warn by default for unions that aren't .
It seems reasonable to me that doesn't guarantee any particular layout or overlap. I would just suggest that should not in practice break people's assumptions about the memory usage of a union ("no larger than the largest member").
That isn’t really a valid question to ask. First of all, optimiser itself can rely on the UB-ness of trap representations and rewrite the program in unexpected ways. Moreover, Rust does not really support altering FP environment either. Adding a lint or some such which inspects the program flow and throws a complaint at the user if read is done from a field when the enum was provably written in some other field would help with this¹. MIR-based lint would make a short work of that. If a CFG does not allow making any conclusions on legality of the union field load and the user makes a mistake, the undefined behaviour is the best we can specify without having specified the Rust repr itself IMO. ¹: Especially effective if people begin using union as a poor man’s transmute for some reason. I disagree. It might make a lot sense to extend stuff to the size of a machine word on some architectures, for example.
One issue that may want to be considered before stabilization is Looks like with the most recent version of LLDB debugging unions may not work :(
Does it work with GDB?
From what I can tell, yes. The Linux bots seem to be running the test just fine.
Then that means Rust provides all the right debugging information, and LLDB just has a bug here. I don't think a bug in one of multiple debuggers, not present in another, should block stabilizing this. LLDB just needs fixing.
It would be cool to see if we could get this feature into FCP for the 1.17 cycle (that's the March 16th beta). Can anyone give a summary of the outstanding questions and the current situation of the feature so we can see if we can reach consensus and resolve everything?
My plans are Wait for the upcoming release (Feb 3). Propose stabilization of unions with fields. This will cover all FFI needs - FFI libraries will be able to use unions on stable. "POD" unions are used for decades in C/C++ and well understood (modulo type based aliasing, but Rust doesn't have it), there are also no known blockers. Write until Feb 3. It will describe the current implementation of unions and outline future directions. The future of unions with non- fields will be decided in the process of discussing this RFC. Note, that exposing something like  or from the standard library doesn't require stabilizing unions. STATUS UPDATE (Feb 4): I'm writing the RFC, but I'm having writer's block after each sentence, as usual, so there's a chance I'll finish it the next weekend (Feb 11-12) and not this weekend (Feb 4-5). STATUS UPDATE (Feb 11): The text is 95% ready, I'll submit it tomorrow.
that seems like a very reasonable course of action.
That sounds reasonable to me. I also reviewed your unions 1.2 proposal, and provided some comments; overall, it looks good to me.
I was thinking that, while in the meeting we talked about keeping the check-lists up to date, I would actually like to see that -- for each of these points -- we make an affirmative decision (i.e., ideally with This would probably suggest a distinct issue (or even amendment RFC). We could do this over time, but until then I don't feel like we've "settled" the answers to the open questions definitively. Along those lines, extracting and summarizing the relevant conversation into an amendment RFC or even just an issue we can link to from here seems like an excellent step to helping ensure that everyone is on the same page -- and something anyone who is interested can do, of course, not just members or shepherds.
So, I have submitted the "Unions 1.2" RFC - Now I'd like to propose stabilization of a conservative subset of union - all fields of the union should be , the number of fields should be non-zero and the union should not implement . (I'm not sure the last requirement is viable though, because it may be easily circumvent by wrapping the union into a struct and implementing for that struct.) Such unions cover all the needs of FFI libraries, which are supposed to be the primary consumer of this language feature. The text of "Unions 1.2" RFC doesn't really tell anything new about FFI-style unions, except that it explicitly confirms that type punning is permitted. EDIT: "Unions 1.2" RFC is also going to make assignments to fields safe (see , ), this affects FFI-style unions as well. This text also provides documentation necessary for the stabilization. The "Overview" section can be copy-pasted into the book and "Detailed design" into the reference.
ping
Does something like this really need to be as part of the language? It took me about 20 minutes to knock up an implementation of a union using a little and . I feel like it wouldn't be difficult for someone to write a macro which can generate something like that except ensuring that the internal array is the size of the largest type. Then instead of my completely generic (and hideously unsafe) they could add a trait bound to limit the types you can get and set. You might even add specific getter and setter methods if you want named fields. I'm thinking they might write something like this: My point is, this is something you could quite feasibly do as part of a library instead of adding extra syntax and complexity to an already fairly complex language. Plus with proc macros it's already quite possible, even if that hasn't fully hit stable yet.
We don't have constant yet though.
It's not just enough to have a array, you also need to get the alignment correct. I in fact already use macros to handle unions but due to the lack of constant and I have to manually allocate the correct space, plus because there is no usable ident concatenation in declarative macros I have to manually specify the names for both the getters and setters. Even just initializing a union is hard at the moment because I have to first initialize it with some default value and then set the value to the variant I want (or add another set of methods to construct the union which is even more verbosity in the definition of the union). It's overall a lot more work and prone to error and uglier than native support for unions. Maybe you should read the RFC and the discussion that went with it so you can understand why this feature is so important.
And the same for alignment.
I imagine ident concatenation shouldn't be too difficult now exists. It allows you to do operations on the AST passed in, so you could take two [Idents][ident], extract their string representation ( implements ), then create a new which is the concatenation of the two using . The RFC mentions a lot about how existing macro implementations are cumbersome to use, however with the recent creation of crates like and , it's now a lot easier to do proc macros. I feel like that'd go a long way towards improving ergonomics and make things less error prone. For example, you could have a which just zero's the union's internal buffer, then a , where has a trait bound ensuring you can only initialize with the correct types. In terms of alignment and size, are you able to use the module from the standard library (i.e. [std::mem::alignof()][alignof] and friends)? I guess everything I'm proposing would depend on being able to use those at macro expansion time to figure out the size and alignment required. 99.9% of the times unions are used it's done with primitive types anyway, so I feel like you'd be able to write a helper function which takes a type's name and returns its alignment or size (possibly asking the compiler, although that's more of an implementation detail). I admit, built in pattern matching would be very nice, but most of the time any unions you use in FFI would get wrapped in a thin abstraction layer anyway. So you might be able to get away with a couple if/else statements or using a helper function. [ident]: [alignof]:
That is not going to work in any cross compilation context.
All of these discussions and many more were had in the history of . To summarize the responses to your specific concerns, in addition to those already mentioned: you'd have to reimplement the padding and alignment rules of every target platform/compiler, and use awkward syntax throughout your FFI code (which has in fact done extensively for Windows bindings and can vouch for the awkwardness of). Also, proc macros currently only work for deriving; you can't extend syntax elsewhere. Also: Not true at all. C code extensively uses a "struct of unions of structs" pattern, where most of the union fields consist of different struct types.
fcp merge per comment I have nothing to add, just triggering the bot
Team member has proposed to merge this. The next step is review by the rest of the tagged teams: [x] [x] [x] [x] [x] [x] No concerns currently listed. Once these reviewers reach consensus, this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up! See for info about what commands tagged team members can give me.
PSA: I'm going to update the "Unions 1.2" RFC with one more change affecting FFI-style unions - I'll move safe assignments to trivially-destructible fields of unions from "Future directions" to the RFC proper. Why: Assignments to trivially-destructible union fields are unconditionally safe, regardless of interpretation of unions. It will remove roughly a half of union-related blocks. It will be harder to do later due to potential large number of warnings/errors in stable code.
Do you mean "trivially destructible union fields", or "unions with entirely trivially destructible fields"? Are you proposing that all the unsafe behavior occurs on read, where you choose a interpretation? For instance, having a union containing an enum and other fields, where the union value contains an invalid discriminant? At a high level that seems plausible. It does allow some things I'd consider unsafe, but Rust in general doesn't, such as bypassing destructors or leaking memory. At a low level, I'd hesitate to consider that sound.
I feel ok about stabilizing that subset. I don't know my opinion about this Unions 1.2 RFC yet as I haven't had time to read it! I'm not sure what I think about allowing safe access to fields in some cases. I feel like our efforts to make a "minimal" notion of what is unsafe (just dereferencing pointers) was a mistake, in retrospect, and we should have declared a wider swath of things unsafe (e.g., a lot of casts), since they interact in complex ways with LLVM. I feel like this may be the case here too. Put another way, I might rather pull back the rules about in concert with more progress on unsafe code guidelines.
"trivially destructible fields", I've tweaked the wording. Are you proposing that all the unsafe behavior occurs on read, where you choose a interpretation? Yes. The write alone can't cause anything dangerous without a subsequent read. EDIT: I feel like our efforts to make a "minimal" notion of what is unsafe (just dereferencing pointers) was a mistake, in retrospect Oh. Safe writes are completely in line with the current approach to unsafety, but if you are going to change it, then I should probably wait.
I don't feel great about stabilising this subset. Usually when we stabilise a subset it is a syntactic subset or at least fairly obvious subset. This subset feels a bit complex to me. If there is so much undecided about the feature that we aren't ready to stabilise the current implementation, then I'd rather leave the whole thing unstable for a while longer.
Well, the subset is rather obvious - "FFI unions", or "C unions", or "pre-C++11 unions" - even if it's not syntactic. My initial goal was to stabilize this subset ASAP (this cycle, ideally) so it could be used in libraries like . There's nothing especially dubious about the remaining subset and its implementation, it's just not urgent and need to wait for unclear amount of time until the process for "Unions 1.2" RFC completes. My expectations would be to stabilize the remaining parts in 1, 2 or 3 cycles after stabilization of the initial subset.
I think I have an ultimate argument for safe field assignments. Unsafe field assignment is equivalent to safe full union assignment except that the safe version is paradoxically less safe because it will overwrite 's bytes outside of with undefs, while field assignment have guarantee about leaving them intact.
The extreme of that is , right?
Equivalence between the two snippets is only true if field in question is "trivially destructible", no? In that sense making assignments like these safe but only in some cases seems extremely inconsistent to me. On Feb 22, 2017 14:50, "Vadim Petrochenkov" <:
The extreme of that is sizeofval(&value) == 0, right? Yep. I don't understand why one additional simple rule eliminating large portion of false positives is extremely inconsistent. "only some cases" cover all FFI unions in particular. I think "extremely inconsistent" is a large overestimation. Not as large as "only variables can be assigned? horrible inconsistency!", but still going in this direction.
please consider such a case: Now since adding trait implementations is generally not a breaking change, Mr. Somebody Somewhere figures out it may be a good idea to add following implementation and release a 1.1.0 (semver compatible with 1.0.0 AFAIK) version of the crate. Suddenly Mr. Somebody Else’s crate does not compile anymore: And therefore sometimes allowing safe assignments to union fields is not as trivial as only locals being mutation-able. As I’ve written down this example I’ve became unsure which stance I should take wrt this, honestly. On one hand, I’d like to preserve the property that adding implementations is generally not a breaking change (ignoring potential cases of XID). On another hand, changing any field of union from trivially destructible to non-trivially destructible is obviously a semver incompatible change that’s extremely easy to overlook and the proposed rule would make these incompatibilities more visible (as long as the assignment is not in an unsafe block already).
This is a good argument, I didn't thought about compatibility. The problem seems solvable though. To avoid compatibility issues do the same thing as coherence does - avoid negative reasoning. I.e. replace "trivially-destructible" == "no components implement " with closest positive approximation - "implements ". type cannot un-implement backward-compatibly, and types still represent majority of "trivially-destructible" types, especially in context of FFI unions.
Implementing is already not backwards-compatible and this has nothing to do with the union feature:
Which in turn sounds like implementing Drop drops implicit Copy. And Copy can be depended on. , jethrogb <:
I wanted to mention this issue, but didn't because it has few rather special preconditions - the struct implementing should have a public field and that field should not be . The union case affects all structs unconditionally.
I consider that maybe an argument that creating a union ought to be unsafe :)
what is the development and nightly-user-experience path for stabilizing a subset? Do we add a new feature gate first for the subset, so that people can get concrete experience using the subset before it becomes stabilized?
What I assumed is just to stop requiring for this subset, no new features or other bureaucracy. FFI-style unions are supposed to be the most often used kind of unions, so a new feature would mean guaranteed breakage just before stabilization, which, I presume, would be annoying.
I'd just like to note that because alignment and packing attributes haven't been implemented yet (nevermind stabilized), I don't really have a strong need for this to be stabilized yet.
Packing? If you mean then it's supported on unions right now (unlike attributes).
. Packing other than 1 is needed quite a bit in winapi. It's not that I need these things supported on unions specifically, I just don't want to jump to a new major version to increase my minimum Rust requirement unless I can get all of those things at the same time.
To clarify the state of play a bit here: The current FCP proposal is for pure- unions only. That's the case with, as far as I'm aware, basically no outstanding questions other than "Should we stabilize?" The discussion since the motion to FCP has all been about new RFC. and from discussion on IRC, I suspect that you are both ready to check off your boxes, but I'll leave that to you ;-)
:bell: This is now entering its final comment period, as per the . :bell:
It seems to me that this would benefit from an idea I floated recently. () While not proposed with unions in mind, the trait as explained (subject to bikeshedding) would allow any union composed of only types to be used without any unsafety. It codifies the property that any bit pattern in memory is equally valid, so you can solve the initialization issues by mandating memory be zeroed out, and it ensures reads cannot invoke UB. In context of FFI, being as defined is also a de facto requirement for such code being sound in many cases, while cases where non- types are useful are rare and difficult to set up safely.
Factual existence of a named trait aside, it might be prudent to split this feature in two. With unqualified enforcing the requirements and allowing use without unsafety, and with relaxed requirements for content and more headache for users. That would allow stabilizing either without barring the way towards adding the other in the future.
This seems sufficiently orthogonal to unions. I would estimate accepting into Rust in the near future as not very likely + making more accesses to union fields safe is more or less backward compatible (not entirely due to lints), so I wouldn't delay unions due to it.
I'm not suggesting to delay unions in waiting for any closure on my proposal, but rather to consider the existence of such possible restrictions on its own right. Making more accesses to union fields safe in the future may face hindrances because it creates more inconsistency in the language. In particular, making it so that safety of a field access varies from field to field seems ugly. Hence my suggestion of making the declaration , so that unconditionally safe-to-use version can be introduced later without adding new keywords. It is worth noting that completely safe-to-use version is enough for most use cases. Edit: Tried to clarify what I mean to say.
Another place where this possibility may inform current and future design is initialization. For unconditionally safe union, it is necessary that the entire span of memory reserved for it is zeroed. Even with current version, ensuring this would cut down on potential UB scenarios and make using unions easier.
The final comment period is now complete.
Now that the FCP to merge is complete, what's the next step here? Would be nice to stabilize this for 1.19.
It'd be awesome if someone could add more detail to that message (). It could make it easier for someone unfamiliar with the process to jump in and move things along.
The path is clear to get this into 1.19. Anybody on the hook for it? cc
Is the enum layout optimization guaranteed to apply through a ? For example, should not represent as a null pointer. should not read uninitialized memory. (used in ) has hacks to deal with this.
This is currently marked as an unresolved question in . In the current implementation this program prints , i.e. the optimization is not performed. cc
This is unlikely to make 1.19.
This is unlikely to make 1.19. The is merged.
With untagged unions now shipping in 1.19 (in part from ) - is there anything remaining on this issue or should we close?
There is still a whole world of unions with non- fields! The progress is mostly blocked on the clarification/documentation RFC ().
Has there been any progress on unions with non- fields since August? The Unions 1.2 RFC seems stalled (I'm guessing due to the impl period?)
Allowing types in unions – if only for unions of a single type – would make it easy to implement rust union ManuallyDrop<T: ?Sized{ value: T }
You only really need to require that at most one type can be unsized.
I am looking at some Rust code using s, and have no idea whether it invokes undefined behavior or not. only mentions: But I can't find a definition of "layout compatible" neither in nor in . Looking through the RFCs I haven't been able to find a definition of "layout compatible" either, only hand-waved examples of what should and shouldn't work in the (not merged). The only seems to allow transmuting an union to its variants as long as it does not invoke undefined behavior but I can't find anywhere in the RFC when that is/isn't the case. Are the precise rules that tell me whether a piece of code using unions has defined behavior written down somewhere (and what the defined behavior is)?
To a first approximation: you may not access padding, may not access an enum that contains an invalid discriminant, may not access a bool that contains a value other than true or false, may not access an invalid or signaling floating-point value, and a few other things like those. If you point to specific code involving unions, we could look at it and tell you if it's doing anything undefined.
Actually the latest consensus is that reading arbitrary floats is fine (). I would add one more requirement: both the source and target union variants are and so are all their fields (and recursively) if they are structs.
I stand corrected, thank you.
So I guess that the rules are not written anywhere? I am looking at how to use with its new interface. Unless we stabilize some extras with it one will need to use unions to do type punning with some of the simd types, like this:
AFAIK writing one union field and reading another is very much like using , with the same restrictions. That those restrictions are still a bit nebulous is not union-specific. For that matter, the function you linked could just use . Although the union version is arguable nicer, at least once the transmute that's currently there is removed: It could be just .
I've filled a clippy issue to add that lint: I guess that the rules I am looking for are when does transmute invoke undefined behavior then (so I'll go look for those). I think it would have helped me if the language reference about unions would have specified the rules for unions in terms of transmute (even if the rules for transmute aren't 100% clear yet) instead of just mention "layout compatibility" and leaving it at that. The leap from "layout compatibility" to "if transmute doesn't invoke undefined behavior, then the types are layout compatible and can be accessed via type punning" was not obvious to me.
To be clear, transmute[copy] is not "more primitive" than unions. In fact transmutecopy is literally just pointer casts plus . additionally needs (deprecated) or (a union) or something like that, and is implemented as intrinsic for efficency, but it too boils down to a type-punning memcpy. The main reason I drew the connection to transmute is because it's older and historically over-emphasized and therefore we currently have more write-ups and folklore knowledge that focuses on transmute specifically. The real underlying concept, which dictates what's valid and what not (and which a spec would describe), is how values are stored in memory as bytes and which byte sequences are UB to read as which types.
Correction: transmute doesn't actually need uninitialized storage (via intrinsics, or unions, or otherwise). Efficency aside, you can do something like this (untested, may contain embarassing typos): The only "magical" part of transmute is that it can constrain the type parameters to be of equal size at compile time.
The reference and and Unions 1.2 RFC are intentionally vague on this matter because the rules for transmuting in general are not settled. The intent was "for unions see third-party ABI specifications, for unions layout compatibility is mostly unspecified (unless it is)".
Is it too late to revisit the drop check semantics of unions with drop fields? The original issue is that adding caused Josephine to become unsound, because it (rather naughtily) relied on perma-borrowed values not having their backing store reclaimed without first running their destructor. A stripped-down example is at The idea is that there's a type , with a method whose safety relies on the invariant "after calling , if the memory backing the pin is ever reclaimed, then the pin's destructor must have been run". This invariant was maintained by Rust until was , and used by The invariant would be restored if the drop checker considered unions with drop fields to have a Drop impl. This is a breaking change, but I doubt any code in the wild relies on the current semantics. IRC conversation: Josephine issue: cc:
Destructors are not guaranteed to run. Rust doesn't guarantee that. It tries, but, for instance,  was made into a safe function. Unions are unsafe largely because you can't know what field of the union is valid. The union can't have an automatic impl; if you wanted such an impl, you'd need to write it manually, taking into account whatever means you have to know whether the union field with a impl is valid. One clarification here: I don't believe we should ever allow unions with fields by default without at least a warn-by-default lint, if not an error-by-default lint. shouldn't disappear as part of the stabilization process. EDIT: oops, didn't mean to hit "close and comment".
yes, Rust doesn't guarantee that destructors will run, but it happened (prior to 1.19) to maintain the invariant that perma-borrowed values would only have their memory reclaimed if the destructor ran. This is even true in the presence of , since you can't call it on a perma-borrowed value. This is what Joephine rather naughtily was relying on, but isn't true any more because of how the drop checker treats . It would be fine if were considered to be an unsafe annotation, this wouldn't be a drastic change, AFAICT, it would just require to check for .
I'm still trying to understand the thing... so, if I follow the example correctly, the reason this "works" is that forces the borrow to last for as long as the lifetime annotated in the type, and that lifetime is moreover invariant. That's an interesting observation! I was not aware of this trick. My gut feeling is that this works "by accident", i.e. safe Rust doesn't happen to provide a way to prevent the destructor from running but that doesn't make this part of the "contract". Notably, not list leaks.
IMO it doesn't matter if it works by accident or on purpose. There was no way to avoid running with this trick prior to existing (which requires unsafe code to be implemented), and now we can't rely on that anymore. The addition of basically killed that neat behaviour of Rust and saying that it shouldn't have been relied on in the first place sounds like circular reasoning to me. If didn't allow calling , would there be any other way to make calling unsound? I don't think so.
I don't think we can commit to preserving every guarantee that rustc accidentally happens to provide right now. We have no idea what these guarantees may be, so we'd be stabilizing a pig in a poke (okay I hope this idiom makes sense... it's what the dictionary tells me matches my native language idiom, which would literally translate to "the cat in the bag" ;) -- what I want to say is we'd have no clue what we would be stabilizing). Also, this is a two-edged sword -- every additional guarantee we decide to provide is something unsafe code has to take care of. So discovering a new guarantee may just as well break existing unsafe code (sitting silently somewhere on without being aware) as enable new unsafe code (the latter was the case here). For example, it is very conceivable that lexical lifetimes enable unsafe code that gets broken by non-lexical lifetimes. Currently all lifetimes are well-nested, maybe there is a way for unsafe code to exploit this? Only with non-lexical lifetimes there can be lifetimes that overlap, but neither is included in the other. Does this make NLL a breaking change? I hope not! With code, there would be. So by declaring this trick sound, you are declaring some unsafe code that would be sound if we decide is okay.
What we described is a very ergonomic way to integrate Rust with GCs. What I'm trying to say is that it sounds wrong to me to tell us that this was just an accident that it worked and that we should forget about it, when I can't find of any use case for not constraining unions with fields as described by here, and when this really is the only wart breaking Josephine. I'll be happy to forget about it if someone can show it was unsound even without .
I see no indication of this trick ever being "designed", so I think it is quite fair to call it an accident. I should have made clearer that this part is just my personal gut feeling. I think it could also be a reasonable point of action to declare this a "happy accident" and actually make it a guarantee -- if we are reasonably confident that indeed all the other code respects this guarantee, and that providing this guarantee is more important than the use case. This is a trade-off, similar to leakpocalypse, where we can't eat our cake and have it, too (we can't have both with its current API and the -based scoped threads; we can't have both and ) so we have to make a decision either way. That said, I find it hard to express the actual guarantee provided here in a precise way, which makes me personally lean more towards the " is fine" side of things.
Fair enough, I heartily agree with that. Note that if we consider what described as undefined behaviour in the end, that can bring back a -based scoped thread API.
As far as I understand Alan’s proposal is not to remove , only to make dropck assume that it (and other unions with fields) has a destructor. (That destructors happens to do nothing, but its mere existence affects what programs dropck accepts or rejects.)
Not sure if this qualifies, but here's my first attempt: A silly implementation of something like that works in pre- Rust. (Yeah I probably have to do some more work to get the alignment right, but that could be done as well by sacrificing some bytes.) Playground showing this breaks : This is what I meant by two-edged sword above -- as far as I can see, my respects all the rules we have put out. So, we have two pieces of incompatible unsafe code -- and . Who's "right"? I'd say relies on guarantees we have never made and hence it's "wrong" here, but this is a judgment call, not a proof.
Now that's interesting. In some versions of our pinning stuff, takes a , but it wouldn't be unreasonable for your to have a impl, right?
Here is a playground that shows that (unsurprisingly) still breaks with a -taking method.
Yeah, I just the API I needed for this example. The obvious should work just fine. Ah, I had missed that; sorry about that. Adding the following to my example keeps it working though: Only if I remove the Rust rejects it. So, at the very least, we'd have to come up with some rule that the above code violates -- just saying "there exists some code we want to be sound that this is incompatible with" is a bad call because it makes it pretty much impossible to look at some code and check if it is sound. I think what makes me most uneasy about this "accidental guarantee" is that I don't see a single good reason that this works. The way things are wired up in Rust makes this hold together, but dropck has been not to prevent leaks but to avoid unsound references to dead data (a common problem in destructors). The reasoning for to work is not based on "here is some mechanism in the Rust compiler, or some type system guarantee, that pretty clearly says perma-borrowed data cannot be leaked" -- it is rather based on "we've tried hard and we have not been able to leak perma-borrowed data, so we think that's okay". Relying on this for soundness makes me pretty nervous. EDIT: The fact that dropck is involved makes me even more nervous because this part of the compiler has a history of nasty soundness bugs. The reason this works seems to be that perma-borrows are at odds with safe . This really seems to be "reasoning based on exhaustive case analysis of what one can do with perma-borrowed data". Now, to be fair, one could say similar things about interior mutability -- it happens to be the case that permitting modifications through shared references actually works safely in some cases, if we pick the right API. However, making this work actually required explicit support in the compiler () because it clashes with optimizations, and there is unsafe code that would be sound without interior mutability but is not sound with interior mutability. Another difference is that interior mutability was a design goal from the start (or from very early on -- this is way before my time in the Rust community), which is not the case for "perma-borrowed doesn't get leaked". And finally, for interior mutability, I think there's a pretty good story about "sharing makes mutation dangerous, but not impossible, and shared references' API just says you don't get mutability in general but doesn't exclude permitting more operations for specific types", resulting in a coherent overall picture. Of course, I've spent thinking about shared references, so maybe there is an equally coherent picture for the issue at hand that I'm just not aware of.
Time zones are fun, I only just got up! There seem to be two issues here (invariants in general, and dropck in particular), so I'll put them in separate comments...
yes, this is an issue about the invariants being maintained by unsafe Rust. For any version of Rust+std, there's more than one choice of invariant which is maintained using rely-guarantee reasoning. And indeed there may be two libraries and , which chose incompatible and , such that Rust+ is safe and Rust+ is safe, but Rust++ is unsafe. In this case, is and is , and it's pretty clear that is going to win since it's now in , which has much stronger backward compatibility constraints than Josephine. Interestingly, the guidelines at are written as "It is the programmer's responsibility when writing unsafe code that it is not possible to let safe code exhibit these behaviors: ..." that is, it's a contextual property (for all safe contexts C, C[P] can't go wrong) and so is dependent on version (since v1.20 of Rust+std has more safe contexts than v1.18). In particular, I'd claim that pinning actually did satisfy this constraint for Rust before 1.20, since there was no safe context C s.t. C[Pinning] goes wrong. However, this is just barracks-room lawyering, I think everyone agrees that there's a problem with this contextual defn, hence all the discussions about unsafe code guidelines. If nothing else, I think that pinning has shown an interesting example of accidental invariants going wrong.
The particular thing that untagged unions (and hence ) did was in the interaction with the drop checker, in particular acts like its defn is: and then you can have a conversation about whether this is allowed or not :) Indeed, this conversation is happening over in the thread starting at your code shows an interesting corner case, where the run-time type for is , but it's compile-time type is . Which type counts as far as is concerned?
Ah, interesting. I agree this is clearly not sufficient -- this would make the original scoped threads sound. To be meaningful, this has to (at least) specify the set of unsafe code that the safe code is permitted to call. Personally, I feel a better way of specifying this is to give the invariants that are to be maintained. But I'm clearly biased here, because the methodology I use for proving things about Rust requires such an invariant. ;) I'm a little surprised that page doesn't contain some sort of disclaimer of being preliminary; we're not really certain yet what exactly the limit will be -- as this discussion shows. We require unsafe code to at least do what that document says, but we probably have to require more. For example, the limits of undefined behavior and what unsafe code can do are not the same. See a recent discussion on that topic: Duplicating a for does not lead to any undefined behavior directly, and yet is clearly considered illegal for unsafe code to do. The reason is that other unsafe code may rely on its ownership discipline being respected, and duplicating things violates that discipline.
Oh, that certainly. And I wonder what we can do to avoid this in the future? Maybe put some big warning onto saying "just because an invariant happens to hold in rustc+libstd, doesn't mean unsafe code can rely on it; instead here's some invariants that you can rely on"?
yes, I don't think anyone's in love with the contextual defn of "correctness", primarily because it's brittle wrt the observational power of contexts. I'd be a lot happier with a semantic defn in terms of invariants. The only thing I'd ask for is please can we give ourselves some wiggle room and define two invariants for rely-guarantee reasoning (code can rely on R and should guarantee G, where G implies R). That way there's some room for strengthening R and weakening G. If we just have one invariant (i.e. R=G) we're stuck never being able to change them!
Constant checking currently doesn't special-case union fields: (cc The above code produces the miri evaluation error "calling non-const fn - ". Changing it to force the type of to be observed by the const-checker: results in "destructors cannot be evaluated at compile-time". Relevant implementation code is here (specifically the call):
Better analysis of had revealed that the semantics for when destructors run when assigning to subfields of unions are insufficiently ready for stabilization. See that issue for details.
Is it realistic to just entirely rule out types in unions, and implement separately (as a lang-item)? From what I can tell, seems to be the biggest motivation for in unions, but that's a very special case. Absent of a positive "no drop" trait, we could then say a union is well-formed if every field is either or of the form . That would entirely side-step all the complications around dropping when assigning union fields (where it seems every possible solution will be full of surprising footguns), and the is a clear marker to programmers that they have to handle themselves here. (The check could be smarter, e.g. it could traverse through product types and through nominal types that are declared in the same crate. Of course, having a positive way to say "this type will never implement " would be nicer.) The checklist in the first post does not mention unsized unions, nor does the RFC---but still , restricted to single-variant unions. This is closely related to the interaction with layout optimizations because it presupposes (once thin-pointer DSTs enter the picture) that a single-variant union has to be "valid" in some sense (it may be dropped, but it can't be any odd bit pattern either). This conflicts with the way unions are sometimes used in C, which is an an "extension point" (IIRC was the one to mention this at the all-hands): A header file may declare 3 variants for a union, but this is considered forward-compatible with adding more variants later (as long as that does not increase the size of the union). The user of the library promises to not touch the union data if the tag (sitting somewhere else) indicates that they do not know the current variant. Crucially, if you only know a single variant, that doesn't mean that there only is a single variant!
This predicate already exists, but is conservative in generics due to there being no trait to bound on. You can access it via (which uses an intrinsic which implements).
will take forwards compatibility into account, or will it happily look into other creates to determine if their types implement ? The goal here is to have a check that will never break from semver-compatible changes, where e.g. adding an to a struct with no type or lifetime parameters and only private fields is semver compatible.
This conflicts with the way unions are sometimes used in C, which is an an "extension point" (IIRC was the one to mention this at the all-hands): A header file may declare 3 variants for a union, but this is considered forward-compatible with adding more variants later (as long as that does not increase the size of the union). The user of the library promises to not touch the union data if the tag (sitting somewhere else) indicates that they do not know the current variant. Crucially, if you only know a single variant, that doesn't mean that there only is a single variant! That's a very specific case. It only affects C-style unions (so there are no destructors and everything is , exactly the subset available on stable) generated from C headers. We can easily add a or field to such unions and keep benefitting from safer "enum" model by default. A FFI union being an "extension point" is something that needs to be well documented anyway.
On April 17, 2018 10:08:54 AM PDT, Vadim Petrochenkov <: We can easily add a or field to such unions and keep benefitting from our safer "enum" model by default. I've seen people talk about treating unions like enums for which we just don't know the discriminant, but to the best of my knowledge, I don't know of any actual model or treatment of them as such. In the original discussion, even non-FFI unions wanted the "multiple variants valid at a time" model, including the motivating use cases for wanting non-FFI unions at all. Adding a variant to a union shouldn't change anything, and unions should not be required to do so to get the semantics they expect. Unions should continue being a bag of bits, with Rust having no idea what they might contain at any given time until unsafe code accesses it. FFI union being an "extension point" is something that need to be well documented anyway. We should certainly document the semantics as precisely as we can.
No, it behaves like s do, exposing all internal details.
There's currently some discussion around "active fields" and drop in unions over at
This is exactly how I would expect unions to work. They are an advanced feature to squeeze out extra performance and interact with C code, where there are no such things as destructors. For me, if you want to drop the contents of a union, you should ~have to cast/transmute (maybe it can't be transmute because it might be bigger with some unused bits at the end for another variant) it to the type you want to drop~ take pointers to the fields that need dropping and use , or use the field syntax to extract the value. If I knew nothing about unions this is how I would expect them to work: I'm going to post this then edit it so I don't loose my working. EDIT way to drop fields.
(If it’s only to let it drop there is no need to extract the value in the sense of moving it, you can take a pointer to one of the fields and use .)
Related: For constants I'm currently arguing that at least one field of a union inside a constant needs to be correct: (if you have a ZST field that's always true)
Please note that edits are not reflected in email notifications. If you’re going to make significant changes to your comment, consider making a new comment instead or in addition.
(and everyone else) I'd be interested in your feedback for I've seen the implementation but not seen the argument. ;)
Well... argument by "not checking at all seemed weird". I'll be happy to implement any scheme we come up with in the const checker, but my intuition always was that one variant of a union needs to be fully correct. Otherwise, unions are just a pretty way to specify a type with a specific size and alignment and some compiler generated convenience for transmuting between a fixed set of types.
I think unions are "bags of uninterpreted bits" with some convenient way to access them. I see nothing weird at all about not checking them. AFAIK there are actually some use-cases mentioned at the Berlin all-hands where the first half of the union matches one field and the second half matches some other field.
I think unions are "bags of uninterpreted bits" with some convenient way to access them. I see nothing weird at all about not checking them. I always though this interpretation goes against the spirit of the language somewhat. In other places we use static analysis to prevent footguns, check that uninitialized or borrowed values are not accessed, but for unions that analysis is suddenly disabled, please shoot.
I see that as exactly the purpose of . I mean we also have raw pointers where all analysis is disabled. Unions provide full control over data layout, just like raw pointers provide full control over memory access. Both go at the cost of safety. Also, this makes simple. I think being simple is important, and even more important when unsafe code is involved (which will always be the case with unions). We should only accept extra complexity here if it provide tangible benefits.
We don't think we have to pay that cost for unions since the bag-of-bits model doesn't give any new opportunities compared to enum-with-unknown-variant model.
The property in question here is at least as much of a burden for unsafe code to uphold as it is a safeguard. There's no static analysis which can prevent all mistakes that could break this property since we do want to use unions for unsafe type punning<sup1</sup, so "enum with unknown variant" really means code handling unions has to be super careful with how it writes to the union or risk instant UB, without really reducing the unsafety involved in reading from the union, since reading already requires knowing (through channels the compiler doesn't understand) that the bits are valid for the variant you're reading. We can only actually warn users about a union that isn't valid for any of its variants is when running under miri, not in the vast majority of cases where it happens at runtime. <sup1</supFor example, assuming tuples are repr(C) for simplicity, allows you to construct something that's invalid just by field assignments.
union Foo { a: (bool, u8), b: (u8, bool) } Hey, that's :) And it's valid under the RFC 1897's model (at least one of "leaf" fragments -1, -1, -2, -2 is valid after any partial assignments). code handling unions has to be super careful with how it writes to the union or risk instant UB That's the point of RFC 1897's model, static checking ensures that no safe operation (like assignment or partial assignment) can turn the union into invalid state, so you don't need to be super careful all the time and don't get instant UB. Only union-unrelated unsafe operations like writes through wild pointers can make a union invalid. On the other hand, without move checking, union can be put into invalid state very easily.
You can automatically recognize some kinds of writes as not violating the extra invariants imposed on unions, but it's still extra invariants that need to be upheld by writers. Since reading is still unsafe and requires manually ensuring that the bits will be valid for the variant that's read, this doesn't actually help readers, it just makes writers' lifes harder. Neither "bag of bits" nor "enum with unknown variant" helps solve the hard problem of unions: how to ensure it actually stores the kind of data you want to read.
How would the fancier type-checking affect Dropping? If you create a union then pass it to C, which takes ownership, will rust try to free the data, perhaps causing a double-free? Or would you always implement yourself? edit it would be way cool if unions were like "enums where the variant is checked statically at compile time", if I've understood the suggestion edit 2 could unions start off as a bag of bits and then later allow safe access whilst being backwards-compatible?
If we decide we want this to be valid, I think should update miri's checks to reflect that -- with merged, it would be rejected by miri. The part I do not understand is what this buys us. We get extra complexity, in terms of implementation (static analysis) and usage (user still needs to be aware of the exact rules). This extra complexity adds up to fact that when unions are used, we are already in an unsafe context so things are naturally more complex. I think we should have a clear motivation for why this extra complexity is worth it. I do not consider "it violates the spirit of the language somewhat" to be a clear motivation. The one thing I can think of is layout optimizations. In a "bag of bits" model, a union has no niche, ever. However, I feel that is better addresses by giving the programmer more manual control over the niche, which would .
I think I am missing something fundamental here. I agree with that the hard problem with unions is making sure that the union currently stores the data that the program wants to read. But AFAIK this problem cannot be solved “locally” by static analysis. We would at least nead whole program analysis, and even then it would still be a hard problem to solve. So... is there a solution for this problem on the table? Or, what does the exact solutions being proposed actually buy us? Say I get an union from C, without analyzing the whole Rust and C program, what can the proposed static analyses actually guarantee for readers?
I think the only guarantee we'd get is what wrote above Your proposal does not protect against bad reads either, I don't think that's possible. Also, I imagined some very basic "initialized" tracking along the lines of "writing to any field initializes the union". We'd need something anyway when is allowed. For better or worse, we have to decide when and where to insert automatic drop calls for a union. Those rules should be as simple as at all possible because this is extra code that we are inserting into existing subtle unsafe code. For unions that do implement , I also imagined a restriction similar to that does not allow writing to a field unless the data structure is already initialized. No. Once we say it's a bag of bits, there could be unsafe code assuming that's allowed.
I think there's value in the bare-minimum move checking to see if a union is initialized. The original RFC explicitly specified that initializing or assigning to any union field makes the whole union initialized. Beyond that, though, rustc should not try to infer anything about the value in a union that the user doesn't explicitly specify; a union may contain any value at all, including a value that isn't valid for any of its fields. One use case for that, for instance: consider a C-style tagged union that's explicitly extensible with more tags in the future. C and Rust code reading that union must not assume it knows every possible field type.
Perhaps I should start from the other direction. Should this code work 1) for unions 2) for non-unions? For me the answer is obvious "no" in both cases, because this is a whole class of errors that Rust can and want to prevent, regardless of "union"-ness of . This means move checker should have some kind of scheme in accordance to which it implements that support. Given that move checker (and borrow checker) generally work in per-field fashion, the simplest scheme for unions would be "same rules as for structs + (de)initialization/borrow of a field also (de)initializes/borrows its sibling fields". This simple rule covers all the static checking. Then, the enum model is simply a consequence of the static checking described above + one more condition. If 1) initialization checking is enabled and 2) unsafe code doesn't write arbitrary invalid bytes into the area belonging to the union, then one of unions "leaf" fields is automatically valid. This is dynamic uncheckable (at least for unions with 1 fields and outside of const-evaluator) guarantee, but it's targeted at people reading code first of all. This case from , for example One use case for that, for instance: consider a C-style tagged union that's explicitly extensible with more tags in the future. C and Rust code reading that union must not assume it knows every possible field type. would be much clearer for people reading code if the union explicitly had an extra field for "possible future extensions". Of course, we can keep the basic static initialization checking, but reject the second condition and allow writing arbitrary possibly invalid data to the union through some unsafe "third party" means without it being instant UB. Then we wouldn't have that dynamic people-targeted guarantee anymore, I just think that would be a net loss.
Agreed, this level of checking for uninitialized values seems reasonable, and quite feasible. Agreed so far, assuming I understand the rules for structs. That additional condition isn't valid for unions. That's not how C unions work, nor how Rust unions were specified to work. (And I'd question whether it'd be clearer, or simply whether it matches a different set of expectations.) Changing this would make Rust unions no longer fit for some of the purposes for which they were designed and proposed. Those 'unsafe "third party" means' include "getting a union from FFI", which is a completely valid use case. Here's a concrete example: That's completely valid code that people can and will write using unions.
Fine for me. Woah. The struct rules make sense because they are all based on the fact that different fields are disjoint. You can't just invalidate that basic assumption and still use the same rules. The fact that you need an addendum to the rules show that. I would never expect unions to be checked similar to structs. If anything, one might expect them to be checked similar to enums -- but of course that cannot work, because enums can only be accessed via match. I think it is extremely desirable for the basic validity assumptions to be dynamically checkable (given type information). Then we can check them during CTFE in miri, we can even check them during "full" miri runs (e.g. of a test suite), we can eventually have some kind of sanitizer or maybe a mode where Rust emits in critical places to check the validity invariants. I think the experience with C's uncheckable rules gives ample evidence that these are problematic. Usually, the first step to actually understand and clarify what the rules are is to find a dynamically checkable way to express them. Even for concurrency memory models, "dynamically checkable" variants (operational semantics explaining everything in terms of step-by-step execution of a virtual machine) are showing up and seem to be the only way to solve long-standing open problems of the axiomatic models that were previously used ("ouf of thin air problem" is a keyword here). I can hardly overstate how important I think it is to have dynamically checkable rules. I think we should aim to have 0 uncheckable cases of UB. (We're not there yet, but it's the goal we should have.) That is the only responsible way to have UB in your language, everything else is a case of compiler/language authors making their life easier at the expense of everyone who has to live with the consequences. (I am currently working on dynamically checkable rules for aliasing and raw pointer accesses.) Even if that would be the only problem, as far as I am concerned "not dynamically checkable" is sufficient grounds to not use this approach. That said, I see no fundamental reason why this should not be checkable: For every byte in the union, go over all variants to see which values are allowed for that byte in this variant, and take the union (heh ;) ) of all of those sets. A sequence of bytes is valid for a union if every byte is valid according to this definition. This is, however, quite hard to actually implement a check for -- by far the most complex basic type validity invariant we would have in Rust. That is a direct consequence of the fact that this validity rule is somewhat tricky to describe, which is why I don't like it. What does that guarantee buy us? Where does it actually help? Right now, all I see is that everyone has to work hard and be careful to uphold it. I don't see the benefit we, the people, get out of that. The model proposed by allows those usecases, by adding a field to the union. However, I don't think that should be necessary. Conceivably, binding generators could add such a field.
This is dynamic uncheckable (at least for unions with 1 fields and outside of const-evaluator) guarantee I think it is extremely desirable for the basic validity assumptions to be dynamically checkable A clarification: I meant uncheckable in "by default"/"in release mode", of course it can be checkable in "slow mode" with some extra instrumentation, but you already wrote about this better than I could.
Yes, I understood that that was the proposal. They could, but they'd have to systematically add it to every single union. I have yet to see an argument for why it makes sense to break primary use cases of unions in favor of some unspecified use case that depends on limiting what bit patterns they can contain.
primary use cases of unions It's not obvious to me at all why this is the primary use case. It may be true for unions if you assume that all uses of unions for tagged unions / "Rust enum emulation" in FFI assume extensibility (which is not true), but from what I've seen, uses of unions (drop control, intialization control, transmutes) do not expect "unexpected variants" suddenly appearing in them.
I didn't say "break the primary use case", I said "break primary use cases". FFI is one of the primary use cases of unions.
There's certainly an attractive obviousness to a statement that "the possible values of a union are the union of the possible values of all its possible variants"...
True. However, that's not the proposal -- we all agree that the following should be legal: Actually I think it is a bug that this even requires . So, the union has to be taken bytewise, at least. Also, I don't think "attractive obviousness" on its own is a sufficiently good reason. Any invariant we decide on is a significant burden for unsafe code authors, we should have concrete advantages that we get in turn.
Actually I think it is a bug that this even requires unsafe. I don't know about the new MIR-based unsafety-checker implementation, but in the old HIR-based one it was certainly a checker limitation/simplification - only expressions of the form were analyzed for possible "field assignment" unsafety opt-out, everything else was conservatively treated as generic "field access" that's unsafe for unions.
Answering the comment in So the idea is that compiler still doesn't know anything about the 's contract and can't e.g. do layout optimizations. Ok, this position is understood. This means that internally, inside of 's module, implementation of module can, for example, temporarily write "unexpected values" into it, if it doesn't leak them to users, and compiler will be okay with them. I'm not sure though how exactly the part of s contract about absence of unexpected values is related to field privacy. First of all, regardless of fields being private or public, unexpected values cannot be written directly through those fields. You need something like a raw pointer, or code on the other side of FFI to do it, and it can be done without any field access, just by having a pointer to the whole union. So we need to approach this from some other direction than access to a field being restricted. As I interpret you comment, the approach is to say that a private field (in union or a struct, doesn't matter) implies an arbitrary invariant unknown to user, so any operations changing that field (directly or through wild pointers, doesn't matter) result in UB because they can potentially break that unspecified invariant. This means that if a union has a single private field, then its implementer (but not compiler) can assume that no third party will write an unexpected value into that union. That's a "default union documentation clause" for the user in some sense: (Default) If a union has a private field you can't write garbage into it. Otherwise, you can write garbage into a union unless its docs explicitly prohibit it. If some union wants to prohibit unexpected values while still providing access to its expected fields (e.g. when those fields have no their own invariants), then it still can do it through documentation, that's why the "unless" in the second clause is necessary. Does this describe you position accurately? How scenarios like this are treated?
No, that is not what I meant. There are multiple invariants. I do not know how many we will need, but there will be at least two (and I don't have great names for them): The "Layout-level invariant" (or "syntactic invariant") of a type is completely defined by the syntactic shape of the type. These are things like " is non-NULL and aligned", " is or ", " cannot exist". On this level, is the same as -- both allow any value (or maybe any initialized value, but that distinction is for another discussion). We are, eventually, going to have a document spelling out these invariants for all types, by structural recursion: The layout-level invariant of a struct is that all its fields have their invariant maintained, etc. Visibility does not play a role here. Violating the layout-level invariant is instantaneous UB. This is a statement we can make because we have defined this invariant in very simple terms, and we make it part of the definition of the language itself. We can then exploit this UB (and we already do), e.g. to perform enum layout optimizations. The "Custom type-level invariant" (or "semantic invariant") of a type is picked by whoever implements the type. The compiler cannot know this invariant as we do not have a language to express it, and the same goes for the language definition. We cannot make violating this invariant UB, as we cannot even say what that invariant is! The fact that it is even possible to have custom invariants is a feature of any useful type system: Abstraction. I wrote more about this in a . The connection between the custom, semantic invariant and UB is that we declare that unsafe code may rely on its semantic invariants being preserved by foreign code. That makes it incorrect to just go ahead any put random stuff into a 's size field. Note that I said incorrect (I sometimes use the term unsound) -- but not undefined behavior! Another example to demonstrate this difference (really, the same example) is the . Creating a dangling well-aligned non-null is never immediate UB, but it is still incorrect/unsound because one may write unsafe code which relies on this not to happen. It would be nice to align these two concepts, but I do not think it is practical. First of all, for some types (function pointers, dyn traits), the definition of the custom, semantic invariant actually uses the definition of UB in the language. This definition would be circular if we wanted to say that it is UB to ever violate the custom, semantic invariant. Secondly, I'd prefer if the definition of our language, and whether a certain execution trace exhibits UB, was a decidable property. Semantic, custom invariants are frequently not decidable. Essentially, when a type chooses its custom invariant, it has to make sure that anything that safe code can do preserves the invariant. After all, the promise is that just using this type's safe API can never lead to UB. This is applies to both structs and unions. One of the things safe code can do is access public fields, which is where this connection comes from. For example, a public field of a struct cannot have a custom invariant that is different from the custom invariant of the field type: After all, any safe user could write arbitrary data into that field, or read form the field and expect "good" data. A struct where all fields are public can be safely constructed, placing further restrictions on the field. A union with a public field... well that's somewhat interesting. Reading union fields is unsafe anyway, so nothing changes there. Writing union fields is safe, so a union with a public field has to be able to handle arbitrary data which satisfies that field's type's custom invariant being put into the field. I doubt this will be very useful... So, to recap, when you choose a custom invariant, it is your responsibility to make sure that foreign safe code cannot break this invariant (and you have tools like private fields to help you achieve this). It is the responsibility of foreign unafe* code to not violate your invariant when that code does something safe code could not do. Correct. (panic-safety is a concern here but you are probably aware). This is just like, in , I can safely do and there is no UB. In terms of the syntactic layout invariant, can do anything (assuming the function call ABI and signature matches). In terms of the semantic custom invariant, that's not visible in the code -- whoever wrote this module had an invariant in mind, they should document it next to their union definition and then make sure that the FFI function returns a value which satisfies the invariant.
I finally about whether and when must be initialized, and the two kinds of invariants I mentioned above.
Is there anything left to track here that’s not already covered by , or should we close?
E0658 still points here:
This currently plays terribly with atomics, since they do not implement . Does anyone know a workaround?
When is implemented, you’ll be able to use in a union. Until then, the only work-around is to use Nightly (or not use and find some alternative).
With that implemented, you shouldn't even need ; after all rustc knows that does not implement .
Assigning myself to switch the tracking issue to the new one.

Discussion:
/// Allows exhaustive pattern matching on types that contain uninhabited types. (active, exhaustive_patterns, "1.13.0", Some(51085), None), <del> /// Allows untagged unions `union U { ... }`. (active, untagged_unions, "1.13.0", Some(32836), None), </del> <ins> /// Allows `union`s to implement `Drop`. Moreover, `union`s may now include fields /// that don't implement `Copy` as long as they don't have any drop glue. /// This is checked recursively. On encountering type variable where no progress can be made, /// `T: Copy` is used as a substitute for "no drop glue". /// /// NOTE: A limited form of `union U { ... }` was accepted in 1.19.0. (active, untagged_unions, "1.13.0", Some(55149), None), </ins> /// Allows `#[link(..., cfg(..))]`. (active, link_cfg, "1.14.0", Some(37406), None),

/// Allows the definition of `const extern fn` and `const unsafe extern fn`. (active, const_extern_fn, "1.40.0", Some(64926), None), <del> // Allows the use of raw-dylibs (RFC 2627). </del> <ins> /// Allows the use of raw-dylibs (RFC 2627). </ins> (active, raw_dylib, "1.40.0", Some(58713), None), <del> /// Enable accurate caller location reporting during panic (RFC 2091). </del> <ins> /// Allows `#[track_caller]` to be used which provides /// accurate caller location reporting during panic (RFC 2091). </ins> (active, track_caller, "1.40.0", Some(47809), None), <del> /// Non-object safe trait objects safe to use but cannot be created in safe rust </del> <ins> /// Allows making `dyn Trait` well-formed even if `Trait` is not object safe. /// In that case, `dyn Trait: Trait` does not hold. Moreover, coercions and /// casts in safe Rust to `dyn Trait` for such a `Trait` is also forbidden. </ins> (active, object_safe_for_dispatch, "1.40.0", Some(43561), None), // -------------------------------------------------------------------------

LL | | } | |_^ | <del> = note: for more information, see https://github.com/rust-lang/rust/issues/32836 </del> <ins> = note: for more information, see https://github.com/rust-lang/rust/issues/55149 </ins> = help: add `#![feature(untagged_unions)]` to the crate attributes to enable error[E0658]: unions with non-`Copy` fields are unstable

LL | | } | |_^ | <del> = note: for more information, see https://github.com/rust-lang/rust/issues/32836 </del> <ins> = note: for more information, see https://github.com/rust-lang/rust/issues/55149 </ins> = help: add `#![feature(untagged_unions)]` to the crate attributes to enable error[E0658]: unions with `Drop` implementations are unstable

LL | | } | |_^ | <del> = note: for more information, see https://github.com/rust-lang/rust/issues/32836 </del> <ins> = note: for more information, see https://github.com/rust-lang/rust/issues/55149 </ins> = help: add `#![feature(untagged_unions)]` to the crate attributes to enable error[E0740]: unions may not contain fields that need dropping