Code change:
The following code causes an ICE: As far as I can tell: Any number of characters can be inserted between and and the error will still happen. can be replaced with any unicode character and the error will still happen. No escape sequences other than cause the error. The error is reproducible on the stable or nightly compiler at outputs the following on my machine: <details<summary<strongBacktrace</strong</summary<p</p</details<!-- TRIAGEBOTSTART --<!-- TRIAGEBOTASSIGNSTART --<!-- TRIAGEBOTASSIGNDATASTARTTRIAGEBOTASSIGNDATAEND --<!-- TRIAGEBOTASSIGNEND --<!-- TRIAGEBOTEND --
Godbolt says this regressed in 1.29.0.
pre-triage: this doesn't seem to be very important but would be of course nice to fix. Tagging it as .
claim Time to try an easy issue
I tried a few things and I will document what I found, so hopefully this can jumpstart some ideas: I removed the assert that was initiating the panic, and I see: Notice the OBOE on the positioning of the So, I tried different escape sequences, and got mildly confusing results Here we have an off by two error in the other direction Now, I tried the unicode escape that should (assuming I didn't misremember something) be equivalent to r: And that seems to work just fine. So, I suspect we somehow interpret escape sequences as the character they represent, but not other times. If this causes us to be offcut in the middle of a UTF-8 multibyte sequence, then we ICE, but there are cases where the will end up in the wrong spot even if we don't ICE. EDIT: a few more observations Turns out the OBOE can be seen with just old fashioned ASCII (no multi-byte utf-8 characters needed): In addition, springboarding off what found, going from 1.28.0 to.1.29.0 the ` in the error messages got a lot more specific, e.g. in 1.28.0 we see:
After a bunch of debug prints, and counting bytes, I am now convinced that the code in the function that is panicing, is not the problem. The inputs to that function are faulty. In particular the is OBO, potentially putting it in the middle of a UTF-8 sequence. I chased down the source of the bad data, and I think it is from in . The code appears to be computing the difference between how many bytes it takes in the source to represent an escape sequence vs how many the interpreted value takes. There are a number of match statements that seem to leave out. Adding them seems to fix the original problem. In addition the code for dealing with doesn't seem to take into account the varying number bytes the UTF-8 value will take. This seems to line up with what I was seeing. Also, I suspect the escape might need some tweaking for code points = 0x80 I forked and branched with what I have done so far:
unclaim I've been too distracted, and it seems has done most of the bug chasing
claim I'd like to take this up if this is still open.
It is open, go ahead :)
, FYI what is in my branch: seems to solve the r issue just fine, but does not address the u{} issue at all. My quick dirty attempts at that failed. You are welcome to leverage off of my stuff if you are taking this over. This has just been a fun distraction when I have time, but I can't reliably dedicate time to it. Also, I have not addressed unit tests at all yet. Also, I am beginning to suspect there may be a larger (yet subtle) underlying confusion somewhere else in the code about bytes vs characters. The findskips function I just updated has comments talking about characters, but data derived from what it generates is later fed into (in ) where we are dealing with bytes. So, perhaps there is a simplification if we can always deal with either just one of bytes or chars (and avoid any conversions). Or, on the other hand there may be a lot of comments that could use clarification.
I have also found that this off by n error will occur with any unicode character whose display width is non-standardized and non one. I have not yet checked the source to try to fix the issue, but I have worked with in my own code, and as it is a compiler dependency is probably the source of this problem. The largest issue with this is as far as I could find there is no standardization for the display width of these characters.
Hi, are you still working on this issue
release-assignment

Related discussion:
text: (' ' | 'n' | 't', _) if eat_ws => { skips.push(pos); } <del> ('', Some((next_pos, 'n' | 't' | '0' | '' | ''' | '"'))) => { </del> <ins> ('', Some((next_pos, 'n' | 't' | 'r' | '0' | '' | ''' | '"'))) => { </ins> skips.push(*next_pos); let _ = s.next(); }