Change:
Unicode is released on a yearly basis, so we update the data files we used accordingly after each Unicode release in the Rust project. (Keep in mind that new dependencies might be over time.) Tracking issues are used to record the overall progress of implementation. They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions. A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature. Instead, open a dedicated issue for the specific matter and add the relevant feature gate label. Goal: Unicode 15.0 (year 2022) [x] Updating : [x] Verify whether each crate is Unicode version dependent and add support for the new Unicode version in compiler dependency libraries [x] Bump dependencies and ensure updated version is used in [x] [x] [x] (Decide whether it's a valid identifier) Current: 0.2.2 (Unicode 13) Goal: 0.2.4 (Unicode 15) [x] (Preprocess identifiers for equality) Current: 0.1.13 (Unicode 9) Goal: 0.1.22 (Unicode 15) [x] (Decide whether lints against unwanted usages should be triggered) Current: 0.0.5 (Unicode 13) Goal: 0.1.0 (Unicode 15) [x] (Used by for script detection) Current: 0.5.3 (Unicode 13) Goal: 0.5.5 (Unicode 15) [x] (used by , and many others) Current: 0.1.8 (Unicode 13) Goal: 0.1.10 (Unicode 15) [x] (used by ) Current: 0.1.0 (Unicode 15) Goal: 0.1.0 (Unicode 15) [x] Removed: [x] Removed: [x] Removed: [x] Removed: [x] Removed:[x] Verify whether each crate is Unicode version dependent and advocate support for the new Unicode version in dependency libraries [x] Bump dependencies and ensure updated version is used in [x] [x] [x] [x] (used by then then [, , , , , , , ]) Previously: 0.3.4 (Unicode 10) Goal: =0.3.10 (Unicode 15) [x] (used by ) Previously: 1.9.0 (Unicode 14) Goal: =1.10.0 (Unicode 15) [x] (used by ) Mentioned above in compiler diagnostics section [x] Removed: [x] (used by , then [, , ]) Current: 2.6.0 (Unclear, No release in 3 years) Goal: =2.7.0 (Unicode 15) [x] (in-tree maintainence tool): Unicode version independent [x] : Unicode version independent (used by tool) [x] : Unicode version independent (used by , then ) [x] , , , : Not really Unicode version independent but we only use Unicode version independent part. They're outdated, current: 0.9.1 (CLDR 37, Spring 2020, ~= Unicode 13), would be nice if a new release is used.
is , which is updated with the libcore update (or more specifically, this year it needed no update). is handled by
I happened to notice this in the UCD spec It seems that as of 15.0.0 have to start parsing some of the comments to look for decls now. This is pretty annoying and will likely lead to some bugs (likely minor). I'm certain I've written code that would be buggy under this. (Hell, I'm not sure I've ever seen code that handled these, and I've read a good number of UCD wrangler scripts -- usually it's just to say "assign the obvious default value to anything that doesn't get specified", as in the line above) It looks like the only files in UCD that take advantage of this (e.g. use it several times to define a full range as having the obvious default case) are , , and (note: this is just the UCD, and I haven't checked stuff like the UTS39 security data, though). This doesn't include anything we ship in the stdlib AFAIK, but almost certainly impacts some stuff in the compiler (in probably minor ways, if it's limited to the above). I suspect the argument is folks we were always doing this wrong if they ignored it though? Oof. CC (since I suspect you know more details here, and even if not, I believe is yours), (since I noticed it while adding something to ucd-generate, presumably there are a few things in that might be impacted if you update its tables)
This seems like another reason for us to potentially switch over to ICU4X (maintained by Unicode), potentially with unicode-rs (or somewhere) maintaining crates of data output. Unfortunately it's unlikely will move to something heavier than the unicode-rs crates which have always stayed zero-dep sicne they're at the bottom of many deptrees. I don't have time to fixup unicode-width but would be happy to accept a patch.
I think rustc doing that is probably very good. It's hard for me to imagine anybody fighting you on it. For the tables... presumably you mean inside the table generator (definitely we couldn't add a dep to libcore on icu4x). We could use it to query the data we build the tables with, instead of ucd-parse. That doesn't sound like a problem eventually. (I'll look into it soonish since I'm messing around with Unicode table generation again). For the ecosystem at large, I dunno. I suspect it will be a tough sell, due to compile time, binary size, and overall heavyweight dependency trees already being in the most common complaints.
Yeah for the core unicode stuff it'd probably be best done as a table generation. Worth noting though: you could still use the efficient code point trie impl that ICU4X uses for this, and not take on too many dependencies. Note that ICU4X's model also involves running a data generation tool; so it might not be an improvement over ucd-parse for us. Eventually, ICU4X will have a copy of various slices of data on a CDN, and then it would be an improvement over ucd-parse. The only difference right now would be that ICU4X is much more likely to update quickly to the latest unicode stuff over ucdparse, but that may not be a huge deal since ucdparse doesn't need much work. I wasn't really thinking about the ecosystem at large. I'm fine continuing to maintain unicode-rs, I just don't have that much time to devote to it. Overall rustc and other users tend to make PRs when they need them and that works out fine. FWIW, ICU4X is very much optimized for binary size.. Perhaps not compile times, but it does tend to produce small binary sizes (and it's shipping on embedded devices already). I'm not really pushing on us moving to ICU4X yet (I think is investigating a bit though!), I think the status quo is manageable, but it's definitely an option on the table!
I don't think I have any strong opinions. It seems like could be updated to handle these declarations without too much trouble though? And then things would be fine? Or is there something else I'm missing? Also worth pointing out that the whole stuff is something I built for lack of a better alternative at the time. My goal was to get rid of an ever increasing pile of bespoke Python scripts. But I don't have much vision beyond that, other than keeping it "simplistic" and framing it as a "kitchen sink" of sorts. I've also updated 's README to link to ICU4X (wording changes very welcome):
Oh yeah to be clear, no shade on ucd-parse and ucd-generate! They're great tools and I don't necessarily mean to say ICU4X is much better or anything. Rustc already needs ICU4X for list formatting and was investigating using it for more things like the XID and security properties (which rely on those manual python scripts).
I can't make heads or tails of the declarations. From what I can tell, they're mainly being used to apply property values to... unassigned codepoints? Just looking at the diff between Unicode 14 and 15 for , this text was removed from 14 and replaced with declarations in 15: So it seems to me like parsers of UCD that ignore in Unicode 15 are probably still just as wrong as they were with Unicode 14? The Unicode docs do a really terrible job of even explaining what is supposed to mean and why it even exists at all. The only explanation of what they actually are seems to be the first sentence of the section you linked: Most of the rest of that section is documenting the format. The key thing that's missing from the docs is what actually is and why it exists. Like... why does there need tobe a mechanism for defining default values for ranges of codepoints when the files themselves are already defining values for ranges of codepoints? Something is missing from my conceptual understanding because I don't understand why the declarations aren't just converted into normal lines in the data files.
I have some guesses as to what the answer is but I don't want to say something incorrect, so I've just asked these questions to the Unicode people.
I'd like to hear back from contacts in case I'm mistaken (or missing some detail), but I'm confident I know what they're for, and have thoughts on them in the context of a low level library like . These exist because several of the UCD files do already require special handling for missing values, but it's not usable in a programmatic manner and needs hard-coding. I believe the goal is that for properties that have lines, you can now determine the property values for all codepoints using just the data programmatically readable from the UCD, without reading the documentation of that property. Here are most of the interesting cases I've found for this (some of which are more useful than others), which should give you a good idea of why it exists and why with very complex defaults, like . I believe this is intended to replace things like . Presumably this list in 's source should have have been getting updated to synchronize with the comments in 's header on each Unicode version update, but that comment didn't really indicate that it was going to change, so that was not happening. Now, these are in lines, which allows that to happen without manual work. I don't know if it changed before now (some documentation hints that it did), but I'd weakly recommend to the Unicode folks that when it changes, the and files are also changed to contain tests that fail if the updates have not happened. That may already happen, but that's really the only many people know if their parsing code is wrong for something like this. with a single default value. In most of these, the default is listed in that property's documentation (or prose in comments in the file) and is kind of the "obvious fallback" (often , if that's a value for that property). Now you can read line to determine that the default for is , that defaults to , or that defaults to , and so on. An odd case here is that the empty string is spelled , but given that that's only used for a property whose value is a codepoint (and not an enumeration), it's unsurprising that it has a special case. , boolean properties which default to and only contain UCD lines when they're . There are a few of these in the normalization properties, and it is probably only useful for code that wants to handle things super generically, but that's something. that default to the codepoint itself, such as is the codepoint itself. Previously, you just had to know this (it's kind of obvious if you think about it, though), but now a line can be used. that default to the value of another property. For example, the has . This codifies that the default value for the property is the value that codepoint has for the property. There are some annoying things about this: a user of a library like (but definitely not for something high level like ), I might want to distinguish between "this is a value explicitly from the file" and "this is the default from ". This is generally true for several of these, but especially true for a few cases like , where I may want to be handling values entirely separately. Obviously a high-level library for Unicode properties would absolutely not expose this distinction, but for something which is explicitly for parsing the UCD files, I suspect including values should be optional (even if it's done by default). now, the property value will often use a different property alias than is used elsewhere in the file. For example, a line for EAW uses , whereas the file uses . Similarly, uses / in but and in the file. Most of these use the "long" form of the property value in the line, and the "short" version in the file itself. I can see the logic here, but it's annoying, as to handle this fully programmatically, you need to read . 's incomplete and completing it feel like it might be a bit annoying. For example, still has "All code points not explicitly listed for Case_Folding have the value C for the status field, and the code point itself for the mapping field" which is not reflected in a line. Given that this is not a single property, it will probably require a special case for the lines. 's also not clear which of these are "stable" values you can rely on, but I presume the answer is "technically none of them, but in practice it's probably safe to assume something like on a derived property is unlikely to change". That said, overall it's probably a good thing, but the release notes for this definitely ignores many of the reasons why this might be a bit of a pain in the ass to handle.
TLDR: they make the files more maintainable and easier to read comments for
Some more background from Asmus, including some explicit steps on how to handle them. They brought up an interesting point that the order of parsing them matters a lot.

Discussion:
[[package]] name = "unicase" <del> version = "2.6.0" </del> <ins> version = "2.7.0" </ins> source = "registry+https://github.com/rust-lang/crates.io-index" <del> checksum = "50f37be617794602aabbeee0be4f259dc1778fabe05e2d67ee8f79326d5cb4f6" </del> <ins> checksum = "f7d2d4dafb69621809a81864c9c1b864479e1235c0dd4e199924b9742439ed89" </ins> dependencies = [ "version_check", ]