Change:
<!-- Thank you for creating a tracking issue! Tracking issues are for tracking a feature from implementation to stabilization. Make sure to include the relevant RFC for the feature if it has one. If the new feature is small, it may be fine to skip the RFC process. In that case, you can use use in your initial implementation PR. The reviewer will ask you to open a tracking issue if they agree your feature can be without an RFC. --Feature gate: This is a tracking issue for the function `. It returns the byte position of the next character, or the length of the underlying string if there are no more characters. This is useful for getting ranges over strings you're iterating over. <!-- Include a short description of the feature. --<!-- For most library features, it'd be useful to include a summarized version of the public API. (E.g. just the public function signatures without their doc comments or implementation.) --<!-- For larger features, more steps might be involved. If the feature is changed later, please add those PRs here as well. --- [X] Implementation: [ ] Final commenting period (FCP) [ ] Stabilization PR <!-- Once the feature has gone through a few release cycles and there are no unresolved questions left, the feature might be ready for stabilization. If this feature didn't go through the RFC process, a final commenting period (FCP) is always needed before stabilization. This works as follows: A library team member can kick off the stabilization process, at which point the rfcbot will ask all the team members to verify they agree with stabilization. Once enough members agree and there are no concerns, the final commenting period begins: this issue will be marked as such and will be listed in the next This Week in Rust newsletter. If no blocking concerns are raised in that period of 10 days, a stabilzation PR can be opened by anyone. --<!-- Include any open questions that need to be answered before the feature can be stabilised. If multiple (unrelated) big questions come up, it can be a good idea to open a separate issue for each, to make it easier to keep track of the discussions. It's useful to link any relevant discussions and conclusions (whether on GitHub, Zulip, or the internals forum) here. --- Bad API
Is there anything holding this back?
I just realized I desperately wanted this today. As it stands it's rather unpleasant assembling a range of characters in a string because you can't easily get the byte offset of the next character to be the end of the range, so then constructing a slice when what you have now is the last character is unpleasant.
I realized I sort of forgot about this, sorry about that, is there anything I need to do to move this along in stabilization?
No idea but I'd really love to see it stabilized.
Agreed, this would be very useful
Yeah, it'd be really nice to have this finally released in the stable builds. Any progress on this?
Maybe you can ask around on the Rust dev fora about how to progress this?
As I learned a while back on u.r.l.o, there's also that can be used to construct a slice representing (or ending at) the current char.
merge
Team member has proposed to merge this. The next step is review by the rest of the tagged team members: [x] [x] [x] [x] [x] [ ] Concerns: resolved by resolved by Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up! See for info about what commands tagged team members can give me.
In my experience, additional methods on iterator types tend to be hard/unergonomic to use. In this case it would be better to return the offset as part of a tuple in the iterator's item type. concern bad API
Can someone show a small but real world program where this method is used? Also I think that suggestion would require building a new iterator type just for this. Probably an method, if the use case makes sense, is the less-bad option.
I disagree, methods on iterators are very difficult to use ergonomically. Consider : it provides useful functionality, but is such a pain to use that I actively avoid it wherever I can. In that specific case the only reason we didn't go with the approach of returning a new iterator that yields is that the lifetime could not be expressed due to the lack of GATs. If character offsets are really needed then they should be returned through a proper iterator interface that is compatible with all the existing iterator combinators.
I have only a big real world program where I needed it, not a small one. I had a hand-written lexical analyzer that needed it. However, in the intervening years, I replaced the lexical analyzer with one generated by re2c now that it produces rust code, so I don't have an example handy any more.
You seem to have gotten a bit confused here - your suggestion is precisely the iterator that already exists. Its whole deal is returning the offset as part of a tuple in its item type. This proposal is to add a method to to access the current position (which it always knows) before advancing, or when at the end. The describes a few of the problems I just ran into while trying to use in a lexer. Here's an excerpt of something I was just working on, followed by how it looks with . Realistically, though, the cleanest approach while this method is unstable is to write a near copy of with this method , rather than working around the edge cases with peeking and manual tracking.
Thanks, this gives a better idea of how this is useful in practice. I believe this API is good, but I don't like the naming: isn't very clear (what offset?). Perhaps or would be clearer?
Indeed, this baffled me a bit at first as well. It's supposed to be the index that you get next (with the difference that it retuns one-past-the-end when would return ). Furthermore, "offset" is a word alien to the current public API; the type and method names use "indices" and the docs additionally talk about "position". If "offset" is just a synonym to "index" or "position" then one of those words should be used instead, preferably "index" because that's what the API already uses. So or ?
Personally, I always think of it as "the current position". I've written near copies of CharIndices in the past and this field would always be named something like or . But it would likely make sense to use "index" as mentioned because of the type name and the phrase "byte index" used in docs. I'm ambivalent on attaching to the name. It's accurate to think of it simply as the current position of the iterator. The reason returns this value (if not at the end) is because the next char always starts at the current position (but extends beyond it, if multibyte). In either case it's one of those things that is really straightforward if you go-to-definition into the stdlib source and a bit wordier to explain in prose...
Another possible name is or for analogy with the iterator.
It's still valid at the end, though, which makes that analogy misleading.
The offset of the iterator into the string it iterates over. 'next offset' or 'next char offset' wouldn't make sense for the offset at the end (when is empty), when there is no next item. Calling this or could work, although then it's less clear we're talking about a byte offset rather than counting chars (like .chars().position()). (But perhaps that ship has sailed by calling this type instead of .)
Thinking about this some more, it seems that what you really want here is a iterator that returns a for each character. Would that work better for your use case? As I've said before, I'm not really a fan of methods on iterators since they don't work well with typical use cases for them.
I recently ran into nearly the same use case as . (I currently use and use to do pointer math to get the offset.) In my use case, I don't just want , because I also use the offset for error reporting to know where we are in the input, without having to store the location separately when the iterator already knows anyway. And without an offset/position/index method, I'd still have to special case the final location where the iterator returns None but I still needs it offset in the input (at the end).
I use Chars::as_str() quite often to get the unconsumed part of the string back for later use. I don't think I've ever had any issues with its API.
Some similar code only stores the start index for each token, since you can reconstruct ranges later, so ranges aren't necessary. Ranges also are not sufficient because they can't give you the iterator's position when it's at the end, for an Eof token. (It would unhelpfully give you that value slightly too early, with the last char, before you know you're about to hit Eof.) On a more general note, I find that additional methods on iterators often make the difference between getting frustrated and doing things by hand instead, or feeling like I'm using a beautiful API that's considered the gaps and edge cases. I'm suspicious of iterators that only support - they're often missing something important. From another angle if I just "follow the data" this method exposes something CharIndices must know, and it comes down to whether it needlessly obscures it from me or not.
I agree. While I think "consider a new iterator" is a good idea in general, if there's a niche use case that can be served by adding a natural method to an existing iterator type, then that seems like a win to me.
Yes, it makes sense from the implementer's point of view, but IMO not the user's. And API naming should of course be user-centric. The fact that uses the current state is just an implementation detail. From the user's POV the state change happens when is called, not before.
It'd be an issue if we have a method that modifies the state to advance the iterator and also have a method that does not modify the state and just returns the current position.
I agree that this method is hard to use "in the usual places where you'd use an iterator". This isn't really a problem: this method it's not meant to be used in such cases; it's meant to be used in logic that's dealing specifically with the type (typically code that's parsing a string). This type already has an method, which exposes also exposes part of the the underlying data. is also "hard to use" in the usual contexts where you'd use a generic iterator. And it also isn't intended to be used in such places. From what I understand, the current major holdback is the naming. Perhaps is unambiguous enough?
Delaying a useful API for long periods because of difficulty finding the right name seems unfortunate. Perhaps someone could just post a poll with a few suggestions and be done with it?
Thinking back on this I agree that is not a great name. Since already uses "Indices" as the term, maybe is a decent name? I would think that using as a prefix/suffix might imply mutation, so if we're going to add something extra maybe or might be reasonable? Though as the method already returns the remaining string and not the whole string, I think it's fairly clear what it does even without any extra attachments.
One can ask what color to paint the bikeshed endlessly. (This doesn't mean having a good name isn't important, just that the conversation may not terminate, having an okayish name and having the thing available is better than waiting forever for perfection, and someone is going to have to make a decision.)
A function with the following signature would be useful to me in the code I'm currently writing: However, if the implementation of this function would be no better than , then it seems fine to return the offset by itself, in which case seems like a fine function name. Currently I am using which is just extremely wasteful and unergonomic.
A work-around for finding the current byte index is to subtract from the length of the original string. This idea has the following advantages: It avoids calling . It avoids parsing the UTF8 bytes twice. It avoids a special case for the end of the string. It's clearer what you mean when iterating through a substring. It even works with a (as opposed to a ). The main disadvantage is that you have to separately record the length of the original string. PS I agree with calling it .
I do think this needs a better name, something that makes it clear it's the index of the next character. concern better-name But I do think we should have it: reviewed
IMO is fine as is, and I don't think a better name has been proposed so far in this issue. I agree with the following 3 comments on why putting into the name would not be better: Namely: The offset it returns can be conceived as the current offset of the iterator. The iterator has already been mutated as part of advancing past the character that it most recently returned. The offset of the character already returned is the offset of the previous character from the point of view of the iterator. The caller already has the offset of the previous character because they received it in the iterator's most recent Item. That item is no longer what the iterator refers to, so I don't see ambiguity about whether the iterator's offset is still that previous location.There is not necessarily any next character in the iterator. Offset doesn't return the offset of a next character, it returns the current position of the iterator.Having one that advances the iterator and returns tuple of offset and char, and a different that returns offset without char, but does not advance the iterator, is misleading. The difference in mental model between "previous"/"current" vs "previous"/"next" is similar to something we debated previously in For linked list cursor, it did make sense to me that the cursor points in between linked list nodes, so "before"/"after" was a better way to design the API than "before"/"current". But notice that this is meaningfully different than the char offsets use case! We redesigned linked list cursors from pointing at a node, to pointing at the boundary between two nodes. But offsets already refer to the boundary between two chars! The char range 0..1 refers to a range from the boundary before the initial byte, to the boundary between the initial byte and the next one. When a char indices iterator is at a particular char boundary, there is no "previous"/"next" offset. There is a previous/next char, and there is a current offset.
David's reasoning above resolves my concern. resolve bad API
Alright, I'm convinced by the description that it's the offset of the current character after having called next. I think that's going to need some clear documentation showing a loop and calling attention to how after the iterator is already pointing after the returned character, but as a name it works. resolved better-name
:bell: This is now entering its final comment period, as per the . :bell:
The final comment period, with a disposition to merge, as per the , is now complete. As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed. This will be merged soon.
Hooray!
Excellent

Discussion:
#![cfg_attr(bootstrap, feature(offset_of_nested))] #![feature(array_ptr_get)] #![feature(asm_experimental_arch)] <del> #![feature(char_indices_offset)] </del> #![feature(const_align_of_val)] #![feature(const_align_of_val_raw)] #![feature(const_align_offset)]

/// Returns the byte position of the next character, or the length /// of the underlying string if there are no more characters. /// <ins> /// This means that, when the iterator has not been fully consumed, /// the returned value will match the index that will be returned /// by the next call to [`next()`](Self::next). /// </ins> /// # Examples /// /// ``` <del> /// #![feature(char_indices_offset)] </del> /// let mut chars = "a楽".char_indices(); /// <ins> /// // `next()` has not been called yet, so `offset()` returns the byte /// // index of the first character of the string, which is always 0. </ins> /// assert_eq!(chars.offset(), 0); <ins> /// // As expected, the first call to `next()` also returns 0 as index. </ins> /// assert_eq!(chars.next(), Some((0, 'a'))); /// <ins> /// // `next()` has been called once, so `offset()` returns the byte index /// // of the second character ... </ins> /// assert_eq!(chars.offset(), 1); <ins> /// // ... which matches the index returned by the next call to `next()`. </ins> /// assert_eq!(chars.next(), Some((1, '楽'))); /// <ins> /// // Once the iterator has been consumed, `offset()` returns the length /// // in bytes of the string. </ins> /// assert_eq!(chars.offset(), 4); /// assert_eq!(chars.next(), None); /// ``` #[inline] #[must_use] <del> #[unstable(feature = "char_indices_offset", issue = "83871")] </del> <ins> #[stable(feature = "char_indices_offset", since = "CURRENT_RUSTC_VERSION")] </ins> pub fn offset(&self) -> usize { self.front_offset }