Repository: dankamongmen/notcurses (language: python)

Issue:
notcurses-tester crashes in alacritty+sixel
```c
/home/dank/src/dankamongmen/notcurses/src/tests/sixel.cpp:141: FATAL ERROR: test case CRASHED: SIGSEGV - Segmentation violation signal

===============================================================================
/home/dank/src/dankamongmen/notcurses/src/tests/sixel.cpp:141:
TEST CASE:  Sixels

DEEPEST SUBCASE STACK REACHED (DIFFERENT FROM THE CURRENT ONE):
  SixelRoundtrip

==================================================================
```
mark plane transitions with hpa
running `whiteout` just now, i had an idea...it really doesn't matter if *within a plane* we have an incorrect idea of glyph widths (outside of pathological cases like on the last line, when we could cause accidental scrolling). the problem is on plane transitions within a row, like the flyer in `whiteout`'s top left. but if we threw an `hpa` in on any plane transition, that can't happen. it's a good chunk of extra output, though -- maybe we could only use it following certain character classes?

this could largely overcome the problem of unpredictable character widths, though, so it's worth thinking about. taken to the extreme, this would involve a `cup` before every glyph. obviously we're not doing that.
mark plane transitions with hpa
running `whiteout` just now, i had an idea...it really doesn't matter if *within a plane* we have an incorrect idea of glyph widths (outside of pathological cases like on the last line, when we could cause accidental scrolling). the problem is on plane transitions within a row, like the flyer in `whiteout`'s top left. but if we threw an `hpa` in on any plane transition, that can't happen. it's a good chunk of extra output, though -- maybe we could only use it following certain character classes?

this could largely overcome the problem of unpredictable character widths, though, so it's worth thinking about. taken to the extreme, this would involve a `cup` before every glyph. obviously we're not doing that.
Improved semantics for RtL text
The FAQ says:

> Notcurses doesn't honor the BiDi state machine, and in fact forces left-to-right with BiDi codes.

Why? I mean, sure, BiDi takes some effort, but why categorically declare non-support? As opposed to "this is a low-priority fix" or such language?
notcurses-tester crashes in alacritty+sixel
```c
/home/dank/src/dankamongmen/notcurses/src/tests/sixel.cpp:141: FATAL ERROR: test case CRASHED: SIGSEGV - Segmentation violation signal

===============================================================================
/home/dank/src/dankamongmen/notcurses/src/tests/sixel.cpp:141:
TEST CASE:  Sixels

DEEPEST SUBCASE STACK REACHED (DIFFERENT FROM THE CURRENT ONE):
  SixelRoundtrip

==================================================================
```
notcurses-tester crashes in alacritty+sixel
```c
/home/dank/src/dankamongmen/notcurses/src/tests/sixel.cpp:141: FATAL ERROR: test case CRASHED: SIGSEGV - Segmentation violation signal

===============================================================================
/home/dank/src/dankamongmen/notcurses/src/tests/sixel.cpp:141:
TEST CASE:  Sixels

DEEPEST SUBCASE STACK REACHED (DIFFERENT FROM THE CURRENT ONE):
  SixelRoundtrip

==================================================================
```
mark plane transitions with hpa
running `whiteout` just now, i had an idea...it really doesn't matter if *within a plane* we have an incorrect idea of glyph widths (outside of pathological cases like on the last line, when we could cause accidental scrolling). the problem is on plane transitions within a row, like the flyer in `whiteout`'s top left. but if we threw an `hpa` in on any plane transition, that can't happen. it's a good chunk of extra output, though -- maybe we could only use it following certain character classes?

this could largely overcome the problem of unpredictable character widths, though, so it's worth thinking about. taken to the extreme, this would involve a `cup` before every glyph. obviously we're not doing that.
mark plane transitions with hpa
running `whiteout` just now, i had an idea...it really doesn't matter if *within a plane* we have an incorrect idea of glyph widths (outside of pathological cases like on the last line, when we could cause accidental scrolling). the problem is on plane transitions within a row, like the flyer in `whiteout`'s top left. but if we threw an `hpa` in on any plane transition, that can't happen. it's a good chunk of extra output, though -- maybe we could only use it following certain character classes?

this could largely overcome the problem of unpredictable character widths, though, so it's worth thinking about. taken to the extreme, this would involve a `cup` before every glyph. obviously we're not doing that.


Discussion / hints:

mintty, it ought be noted, has a mode where it alerts us whenever a glyph wider than a single column is emitted. unfortunately, i don't see how it can be effectively used, as we'd essentially need to emit a character and then wait (how long?) to see whether it was wide or not.
let's finally try this out and see what happens.
we can get a quick preview by simply commenting out the `return 0;` in `goto_location()`, let's see...
in kitty, *always* dropping an hpa:
* resolves all mess in `uniblock`
* resolves the overeat in `mojibake`
* completely fixes `whiteout`
* has negligible effect on total `notcurses-demo` runtime
succeeding rather beyond my wildest dreams. like, this at first glance fixes all outstanding width problems, with no significant cost. and that's the naive version, that issues hpa every move--we could cut it down to plane transitions, as speced here.

Alacritty has some small boundary problems in `uniblock` resolved.

XTerm gets much crappier whenever we're clearing sixel. here's another place where we'd really like DECERA to apply to sixel =[. but with the plane-transition change mentioned above, this goes away.
so i think this solves a very real problem, and solves it elegantly and completely. i'm honestly surprised by just how much this improves the situation -- literally every alignment problem i was aware of has been resolved in one fell swoop.

i think it's sensible to add a stat for this, so we can track the number of "unnecessary" `hpa`s being inserted. it might furthermore be reasonable to only insert the `hpa` when we've seen a possibly-wide character, though then we're back to guessing what terminals are going to do.

XTerm's `[xray]` does get substantially crappier, despite the change to only perform on a plane transition -- perhaps we fucked this up? hence the value of the proposed stat:

before:
```c
             runtime│ frames│output(B)│    FPS│%r│%a│%w│TheoFPS║                
══╤════════╤════════╪═══════╪═════════╪═══════╪══╪══╪══╪═══════╣                
 1│   intro│   5.95s│    298│   3.72Mi│   50.1│ 0│ 0│46│ 105.79║                
 2│    xray│  28.97s│    486│  23.93Mi│   16.8│ 0│ 0│67│  24.60║                
 3│   eagle│  13.77s│    368│   7.03Mi│   26.7│ 0│ 0│81│  32.62║                
══╧════════╧════════╪═══════╪═════════╪═══════╧══╧══╧══╧═══════╝                
              48.69s│   1152│  34.68Mi│   
1155 renders, 228.28ms (69.72µs min, 197.65µs avg, 5.40ms max)
1155 rasters, 47.51ms (20.53µs min, 41.14µs avg, 104.79µs max)
1155 writes, 33.58s (20.55µs min, 29.07ms avg, 160.14ms max)
34.69MiB (96B min, 30.75KiB avg, 503.88KiB max) 0 inputs
0 failed renders, 0 failed rasters, 0 refreshes, 0 input errors
RGB emits:elides: def 1180:810404 fg 208921:318784 bg 218348:1120940
Cell emits:elides: 1339288:6592488 (83.11%) 99.85% 60.41% 83.70%
Bitmap emits:elides: 14:761 (98.19%) 25.45MiB (73.37%) SuM: 0 (0.00%)
```

after:

```c
             runtime│ frames│output(B)│    FPS│%r│%a│%w│TheoFPS║                
══╤════════╤════════╪═══════╪═════════╪═══════╪══╪══╪══╪═══════╣                
 1│   intro│   6.72s│    280│   3.56Mi│   41.7│ 0│ 0│53│  76.74║                
 2│    xray│  84.63s│    486│  28.30Mi│    5.7│ 0│ 0│99│   5.76║                
 3│   eagle│  14.70s│    368│   7.93Mi│   25.0│ 0│ 0│83│  29.93║                
══╧════════╧════════╪═══════╪═════════╪═══════╧══╧══╧══╧═══════╝                
             106.05s│   1134│  39.79Mi│                                         
1137 renders, 239.77ms (91.40µs min, 210.88µs avg, 3.08ms max)
1137 rasters, 48.44ms (21.35µs min, 42.60µs avg, 132.83µs max)
1137 writes, 100.10s (20.68µs min, 88.03ms avg, 293.69ms max)
39.83MiB (96B min, 35.87KiB avg, 506.29KiB max) 0 inputs
0 failed renders, 0 failed rasters, 0 refreshes, 0 input errors
RGB emits:elides: def 1180:810404 fg 197140:303059 bg 211390:1100392
Cell emits:elides: 1311782:6496162 (83.20%) 99.85% 60.59% 83.89%
Bitmap emits:elides: 14:748 (98.16%) 25.45MiB (63.90%) SuM: 0 (0.00%)
```

so let's add the stat, and see what we can do about `[xray]` with sixel, but this is happening, and it's happening for 2.4.9.
maaaaaaaaaaaaaaaaaaaaaaaaaaaybe we even make this a per-terminal thing, since some terminals have a single width for all characters?
looks like we're definitely releasing too many, which would explain continued sixel slowdowns:

`49G🚱^[[51G✊🏿69^[[55G🔬^[[57G🧬^[[59G🏴`
> looks like we're definitely releasing too many, which would explain continued sixel slowdowns:
> 
> `49G🚱^[[51G✊🏿69^[[55G🔬^[[57G🧬^[[59G🏴`

this of course means we must verify continued effectiveness once we eliminate the unnecessary `hpa`s.
i'm so fuckin' excited about this! a major, major improvement.
mintty, it ought be noted, has a mode where it alerts us whenever a glyph wider than a single column is emitted. unfortunately, i don't see how it can be effectively used, as we'd essentially need to emit a character and then wait (how long?) to see whether it was wide or not.
let's finally try this out and see what happens.
we can get a quick preview by simply commenting out the `return 0;` in `goto_location()`, let's see...
in kitty, *always* dropping an hpa:
* resolves all mess in `uniblock`
* resolves the overeat in `mojibake`
* completely fixes `whiteout`
* has negligible effect on total `notcurses-demo` runtime
succeeding rather beyond my wildest dreams. like, this at first glance fixes all outstanding width problems, with no significant cost. and that's the naive version, that issues hpa every move--we could cut it down to plane transitions, as speced here.

Alacritty has some small boundary problems in `uniblock` resolved.

XTerm gets much crappier whenever we're clearing sixel. here's another place where we'd really like DECERA to apply to sixel =[. but with the plane-transition change mentioned above, this goes away.
so i think this solves a very real problem, and solves it elegantly and completely. i'm honestly surprised by just how much this improves the situation -- literally every alignment problem i was aware of has been resolved in one fell swoop.

i think it's sensible to add a stat for this, so we can track the number of "unnecessary" `hpa`s being inserted. it might furthermore be reasonable to only insert the `hpa` when we've seen a possibly-wide character, though then we're back to guessing what terminals are going to do.

XTerm's `[xray]` does get substantially crappier, despite the change to only perform on a plane transition -- perhaps we fucked this up? hence the value of the proposed stat:

before:
```c
             runtime│ frames│output(B)│    FPS│%r│%a│%w│TheoFPS║                
══╤════════╤════════╪═══════╪═════════╪═══════╪══╪══╪══╪═══════╣                
 1│   intro│   5.95s│    298│   3.72Mi│   50.1│ 0│ 0│46│ 105.79║                
 2│    xray│  28.97s│    486│  23.93Mi│   16.8│ 0│ 0│67│  24.60║                
 3│   eagle│  13.77s│    368│   7.03Mi│   26.7│ 0│ 0│81│  32.62║                
══╧════════╧════════╪═══════╪═════════╪═══════╧══╧══╧══╧═══════╝                
              48.69s│   1152│  34.68Mi│   
1155 renders, 228.28ms (69.72µs min, 197.65µs avg, 5.40ms max)
1155 rasters, 47.51ms (20.53µs min, 41.14µs avg, 104.79µs max)
1155 writes, 33.58s (20.55µs min, 29.07ms avg, 160.14ms max)
34.69MiB (96B min, 30.75KiB avg, 503.88KiB max) 0 inputs
0 failed renders, 0 failed rasters, 0 refreshes, 0 input errors
RGB emits:elides: def 1180:810404 fg 208921:318784 bg 218348:1120940
Cell emits:elides: 1339288:6592488 (83.11%) 99.85% 60.41% 83.70%
Bitmap emits:elides: 14:761 (98.19%) 25.45MiB (73.37%) SuM: 0 (0.00%)
```

after:

```c
             runtime│ frames│output(B)│    FPS│%r│%a│%w│TheoFPS║                
══╤════════╤════════╪═══════╪═════════╪═══════╪══╪══╪══╪═══════╣                
 1│   intro│   6.72s│    280│   3.56Mi│   41.7│ 0│ 0│53│  76.74║                
 2│    xray│  84.63s│    486│  28.30Mi│    5.7│ 0│ 0│99│   5.76║                
 3│   eagle│  14.70s│    368│   7.93Mi│   25.0│ 0│ 0│83│  29.93║                
══╧════════╧════════╪═══════╪═════════╪═══════╧══╧══╧══╧═══════╝                
             106.05s│   1134│  39.79Mi│                                         
1137 renders, 239.77ms (91.40µs min, 210.88µs avg, 3.08ms max)
1137 rasters, 48.44ms (21.35µs min, 42.60µs avg, 132.83µs max)
1137 writes, 100.10s (20.68µs min, 88.03ms avg, 293.69ms max)
39.83MiB (96B min, 35.87KiB avg, 506.29KiB max) 0 inputs
0 failed renders, 0 failed rasters, 0 refreshes, 0 input errors
RGB emits:elides: def 1180:810404 fg 197140:303059 bg 211390:1100392
Cell emits:elides: 1311782:6496162 (83.20%) 99.85% 60.59% 83.89%
Bitmap emits:elides: 14:748 (98.16%) 25.45MiB (63.90%) SuM: 0 (0.00%)
```

so let's add the stat, and see what we can do about `[xray]` with sixel, but this is happening, and it's happening for 2.4.9.
maaaaaaaaaaaaaaaaaaaaaaaaaaaybe we even make this a per-terminal thing, since some terminals have a single width for all characters?
looks like we're definitely releasing too many, which would explain continued sixel slowdowns:

`49G🚱^[[51G✊🏿69^[[55G🔬^[[57G🧬^[[59G🏴`
> looks like we're definitely releasing too many, which would explain continued sixel slowdowns:
> 
> `49G🚱^[[51G✊🏿69^[[55G🔬^[[57G🧬^[[59G🏴`

this of course means we must verify continued effectiveness once we eliminate the unnecessary `hpa`s.
i'm so fuckin' excited about this! a major, major improvement.
> Why? I mean, sure, BiDi takes some effort, but why categorically declare non-support? As opposed to "this is a low-priority fix" or such language?

It's a great question! I'm absolutely open to revisiting this, but let me reference the thinking on it thus far:

#896
#1204 
#850

and i'll comment a bit further below.
Essentially, it comes down to the following fact: Notcurses is a multiplanar model, and shifting text directions seem very difficult to integrate with multiple planes.

Let's say we have the characters 0, 1, 2, ...9 (note i mean here *index* 0..9, **not** the arabic digits 0 through 9).

LTR: 0123456789
RTL: 9876543210

all well and good. no problems here. perhaps we've written these to a 10x2 plane, so we have:

```
/----------\
|0123456789|  plane P0, 10x2, offset 0,0
|9876543210|
\----------/
```

now we add a new plane. this is  4x2, and it has two copies of the LTR row ABCD


```
/----------\
|ABCD|  plane P1, 4x2, offset 2,0
|ABCD| 
\----------/
```

so you now have this chunk of ABCD siitting atop the P0, yielding

`98ABCD3210`

so in this case, ought we reverse ABCD? probably not, right? but then is 98....3210 correct? remember, the *sequence of input* was 0123456789; it just *appears* as 9876543210 because it's RTL. so it seems just as valid for ABCD to have knocked out 2345 rather than 7654. but in that case, it should surely be 9876ABCD10, right? except that's not where we *want* ABCD.

add to this the fact that some terminals do RTL, and some don't, and it's impossible to know whether they will (indeed, in at least some it's a configuration option). so even if we try emitting 0123456789, hoping it'll be displayed as 9876543210, there's no guarantee of that.

sooooooooooooooooooooooooo the only thing that made sense to me was: *never* allow the terminal to do RTL. we enforce this by inserting hard LTR markers after each EGC with a bidi classification. this way, we always know that 0123456789 provided to the terminal will show up as 0123456789...and that 9876543210 provided to the terminal will always show up as 9876543210.

so, to answer your question, we do it because we don't otherwise know what the terminal is going to do.

NOW, surely we don't want to just shit all over our Arabic, Hebrew, Rohingya, Maldivian and *shudder* Mongolian friends, right? Well, they can just *emit their characters in the order they want them to show up, assuming left-to-right motion*. i.e. if you have  ތާނަ, emit it as  ނަ   ތާ (no space), and it will show up exactly how you want it.

does that make sense? do you have a better suggestion? i only know a scant bit of Hebrew, so i'm not particularly well-informed, and would love good advice.
oh, and give my love to חֵיפָה and the boys (and girls) down in Unit 8200! 🇮🇱
oh hey shit, i read your "faster across the pcie bus" paper from back in 2017. i thought i recognized your name from somewhere!
My suggestion: Follow the same design pattern as the original RTL terminals, which is to provide APIs via which application developers can indicate their intent. Basically a way to say I'm writing RTL now, so move the cursor left after each character is output instead of right.

But you should never reorder characters that are already on screen, because you can't possibly know what they represent. Are two Hebrew characters next to each part of the same word, or are they from two separate windows with unrelated content that just happened to have lined up? Maybe they're just part of some kind of "ASCII art"?

The only person that knows what the characters represent is the app dev, so they should be the one that gets to decide what order they are output. Second guessing them just makes it impossible to do anything meaningful with RTL text. But you can make things easier for them, as long as you leave them in control.

Just my 2c.
> The only person that knows what the characters represent is the app dev, so they should be the one that gets to decide what order they are output. Second guessing them just makes it impossible to do anything meaningful with RTL text. But you can make things easier for them, as long as you leave them in control.

i agree with this wholeheartedly, and am doing this. without the LTR forcings, *we don't know which way it'll be drawn*. thus it is impossible for me to reliably fulfill the app dev's goals. by forcing LTR, i allow them to lay things out exactly as they want.
now what i could add is something like `ncplane_putstr_rtl()`. but then we're Cartesianing our output functions by another dimension and that is gonna explode very soon.
what gets ignored here a lot is BtT, as mongolian is written. here again, we allow it to be accomplished via exact positioning and a universal direction rule.
> by forcing LTR, i allow them to lay things out exactly as they want.

Sorry, I should have said, I think what you're doing now is perfect. There's nothing stopping someone writing a bidi-aware application on top of a framework that stays out of the way. My suggestion was only if you wanted additional RTL-specific functionality on top of what you already have, and I didn't want you taking the auto-RTL approach that some terminals seem to think was a good idea.
@eyalroz , what are your thoughts, having read this? perhaps the real change needed is one to the FAQ?
My thoughts are that I need to get to know the library more to better understand planes and multiplanarity - because I didn't get the intuition you intended to evoke with your example. 

My intuition comes from real life :-P If I wrote ABCD on a small sign and put it on top of a bigger sign where I wrote 9876543210, at offset 2 from the left, then what I'll see from above is 98ABCD3210. 

> So it seems just as valid for ABCD to have knocked out 2345 rather than 7654. 

why is that?

(You don't have to answer, I'll go read the other issue pages.)
> My intuition comes from real life :-P If I wrote ABCD on a small sign and put it on top of a bigger sign where I wrote 9876543210, at offset 2 from the left, then what I'll see from above is 98ABCD3210.
> > So it seems just as valid for ABCD to have knocked out 2345 rather than 7654.
> why is that?

because it depends on whether the plane was intended to cover *cells 2, 3, 4 and 5 of the visual area* or *glyphs 2, 3, 4 and 5 of the text*. for instance, take a look at the text in the first minutes of the [Notcurses III hype video](https://www.youtube.com/watch?v=dcjkezf1ARY). if you look closely at the expository text, you'll see that it's being dimmed towards the back. new text is brightest, oldest text is dimmest.

i implemented this by placing a `NCALPHA_BLEND` plane atop the text plane, where this blending plane was dimmed according to the desired scheme. there, the upper left is darker than the upper right. if i had RtL text, i would presumably want that inverted, no? since the oldest text would be on the upper right. so here we want to make the decision based on the text layout. we want the darkest on the right, not on the left. this implies that **i need to know the direction the text is flowing**. see earlier comments: by relying on the terminal's arbitrary BiDi implementation, **i cannot know this**. thus **the app dev must explicitly drive this layout**, which is absolutely possible to do; you just write in reverse, or start from the right.

which is why i think maybe the wording in the FAQ is what's wrong. it's not that you can't write right-to-left as intended; it's that you must consciously do so, since otherwise we can't know whether your intention is being honored or not. if you write Gimel, Bet, Alef, it's not going to reorder it as Alef Bet Gimel, but if you write Alef Bet Gimel, it's also not going to be reordered as Gimel Bet Alef.
here's another issue: let's say i have two 1x1 planes, one at {1, 0} and one at {1, 0}. the first has an Alef. the second has a Bet. I then move the second to {1, 1}. ought this show up as Bet Alef, or Alef Bet? if we honored RtL straight (and the terminal also did), a 2x1 plane into which one wrote "Alef Bet" would show Bet Alef. what of the logical 2x1 equivalent made up of 2x 1x1s? and what will the terminal do?

the only sane answer seems to be forcing LtR presentation, and allowing the app dev to place things precisely.
Well, once you have BiDi text, you need to be able to specify things like "start side" and "end side" rather than just "right" or "left". So, in the ABCD example - it depends on how you defined things. If you wanted "offset from left", you would say that and get that. If you wanted "offset from start side", you would say that. This is not the same as the app explicitly driving the layout: The app does not have to remember what the bottom plane text direction choice is, it can just say "align my second plane to the start side".

About the Alef-Bet planes - if they are two planes, then they don't constitute a single BiDi run. Aligning planes is not the same as uniting planes. It's just the same with, say, HTML table rendering. Even if there are no borders, cells are rendered individually. That being said - you could very easily create tricky examples; see the shocking [Text rendering hates you](https://gankra.github.io/blah/text-hates-you/) if you want to be brought to tears...

> About the Alef-Bet planes - if they are two planes, then they don't constitute a single BiDi run. Aligning planes is not the same as uniting planes. It's just the same with, say, HTML table rendering. Even if there are no borders, cells are rendered individually. That being said - you could very easily create tricky examples; see the shocking [Text rendering hates you](https://gankra.github.io/blah/text-hates-you/) if you want to be brought to tears...

i agree with this, but **the terminal might not**. test if you don't believe me: open a terminal, generate an Alef, hit enter, `cup` to the cell following the Alef, and enter a Bet. numerous terminals will swap the two glyphs (and others will not).

html would be easier to work with, but alas, that's not what i'm working with.

all such schemes break down because different terminals do different things. again, i bring up Konsole: it has a config option, which can be changed at runtime, as to whether it "honors" BiDi. i have absolutely no way of knowing what it's going to do at a given time if i have two RtL glyphs next to one another. the only way i can control things is by inserting LtR forcings to break up any possible RtL chain. i see no way around this, nor any way that user hints could help--the user must take desired order explicitly into account, and make up for it.

**now**, maybe there's a middle ground: when given a chunk of RtL text, i could lay it out myself RtL, just as i'm suggesting the user ought. that way, they needn't think about it. i've disliked this idea because it means

`ncplane_putstr(n, "rtlfun");`

and

`for(const char* s = "rtlfun" ; *s ; ++s){ ncplane_putchar(n, *s); }`

generate different output (the former would be written "nufltr", the latter "rtlfun"). so i put the question to you, oh speaker of the sinistral: would this be helpful? would it eliminate most of your annoyance? or would this end up being a false balm, one that suggests rich RtL support but doesn't actually imply it? ought i handle Mongolian text bottom-to-top?
> **now**, maybe there's a middle ground: when given a chunk of RtL text, i could lay it out myself RtL, just as i'm suggesting the user ought. that way, they needn't think about it. i've disliked this idea because it means

another thing that makes this unattractive, or at least worrying, is that i have no rich knowledge of right-to-left languages, and would be kinda shooting blind despite now being responsible for deciding what ought be RtL and what oughtn't (like, numbers are always LtR no matter what language they're surrounded by, right?). but those would of course just be bugs that can be reported and fixed.

if you think this scheme is a good direction to take, let me know, and i can code it up, probably for 3.0.
> deciding what ought be RtL and what oughtn't

As far as I remember, the Unicode standard documents all of this stuff in great detail. But there are also libraries you can probably use that will do a lot of the work for you (I've not tried it myself, but I've heard good things about https://github.com/fribidi/fribidi).

Edit: This is the Unicode doc I was thinking of: http://www.unicode.org/reports/tr9/


mintty, it ought be noted, has a mode where it alerts us whenever a glyph wider than a single column is emitted. unfortunately, i don't see how it can be effectively used, as we'd essentially need to emit a character and then wait (how long?) to see whether it was wide or not.
let's finally try this out and see what happens.
we can get a quick preview by simply commenting out the `return 0;` in `goto_location()`, let's see...
in kitty, *always* dropping an hpa:
* resolves all mess in `uniblock`
* resolves the overeat in `mojibake`
* completely fixes `whiteout`
* has negligible effect on total `notcurses-demo` runtime
succeeding rather beyond my wildest dreams. like, this at first glance fixes all outstanding width problems, with no significant cost. and that's the naive version, that issues hpa every move--we could cut it down to plane transitions, as speced here.

Alacritty has some small boundary problems in `uniblock` resolved.

XTerm gets much crappier whenever we're clearing sixel. here's another place where we'd really like DECERA to apply to sixel =[. but with the plane-transition change mentioned above, this goes away.
so i think this solves a very real problem, and solves it elegantly and completely. i'm honestly surprised by just how much this improves the situation -- literally every alignment problem i was aware of has been resolved in one fell swoop.

i think it's sensible to add a stat for this, so we can track the number of "unnecessary" `hpa`s being inserted. it might furthermore be reasonable to only insert the `hpa` when we've seen a possibly-wide character, though then we're back to guessing what terminals are going to do.

XTerm's `[xray]` does get substantially crappier, despite the change to only perform on a plane transition -- perhaps we fucked this up? hence the value of the proposed stat:

before:
```c
             runtime│ frames│output(B)│    FPS│%r│%a│%w│TheoFPS║                
══╤════════╤════════╪═══════╪═════════╪═══════╪══╪══╪══╪═══════╣                
 1│   intro│   5.95s│    298│   3.72Mi│   50.1│ 0│ 0│46│ 105.79║                
 2│    xray│  28.97s│    486│  23.93Mi│   16.8│ 0│ 0│67│  24.60║                
 3│   eagle│  13.77s│    368│   7.03Mi│   26.7│ 0│ 0│81│  32.62║                
══╧════════╧════════╪═══════╪═════════╪═══════╧══╧══╧══╧═══════╝                
              48.69s│   1152│  34.68Mi│   
1155 renders, 228.28ms (69.72µs min, 197.65µs avg, 5.40ms max)
1155 rasters, 47.51ms (20.53µs min, 41.14µs avg, 104.79µs max)
1155 writes, 33.58s (20.55µs min, 29.07ms avg, 160.14ms max)
34.69MiB (96B min, 30.75KiB avg, 503.88KiB max) 0 inputs
0 failed renders, 0 failed rasters, 0 refreshes, 0 input errors
RGB emits:elides: def 1180:810404 fg 208921:318784 bg 218348:1120940
Cell emits:elides: 1339288:6592488 (83.11%) 99.85% 60.41% 83.70%
Bitmap emits:elides: 14:761 (98.19%) 25.45MiB (73.37%) SuM: 0 (0.00%)
```

after:

```c
             runtime│ frames│output(B)│    FPS│%r│%a│%w│TheoFPS║                
══╤════════╤════════╪═══════╪═════════╪═══════╪══╪══╪══╪═══════╣                
 1│   intro│   6.72s│    280│   3.56Mi│   41.7│ 0│ 0│53│  76.74║                
 2│    xray│  84.63s│    486│  28.30Mi│    5.7│ 0│ 0│99│   5.76║                
 3│   eagle│  14.70s│    368│   7.93Mi│   25.0│ 0│ 0│83│  29.93║                
══╧════════╧════════╪═══════╪═════════╪═══════╧══╧══╧══╧═══════╝                
             106.05s│   1134│  39.79Mi│                                         
1137 renders, 239.77ms (91.40µs min, 210.88µs avg, 3.08ms max)
1137 rasters, 48.44ms (21.35µs min, 42.60µs avg, 132.83µs max)
1137 writes, 100.10s (20.68µs min, 88.03ms avg, 293.69ms max)
39.83MiB (96B min, 35.87KiB avg, 506.29KiB max) 0 inputs
0 failed renders, 0 failed rasters, 0 refreshes, 0 input errors
RGB emits:elides: def 1180:810404 fg 197140:303059 bg 211390:1100392
Cell emits:elides: 1311782:6496162 (83.20%) 99.85% 60.59% 83.89%
Bitmap emits:elides: 14:748 (98.16%) 25.45MiB (63.90%) SuM: 0 (0.00%)
```

so let's add the stat, and see what we can do about `[xray]` with sixel, but this is happening, and it's happening for 2.4.9.
maaaaaaaaaaaaaaaaaaaaaaaaaaaybe we even make this a per-terminal thing, since some terminals have a single width for all characters?
looks like we're definitely releasing too many, which would explain continued sixel slowdowns:

`49G🚱^[[51G✊🏿69^[[55G🔬^[[57G🧬^[[59G🏴`
> looks like we're definitely releasing too many, which would explain continued sixel slowdowns:
> 
> `49G🚱^[[51G✊🏿69^[[55G🔬^[[57G🧬^[[59G🏴`

this of course means we must verify continued effectiveness once we eliminate the unnecessary `hpa`s.
i'm so fuckin' excited about this! a major, major improvement.
mintty, it ought be noted, has a mode where it alerts us whenever a glyph wider than a single column is emitted. unfortunately, i don't see how it can be effectively used, as we'd essentially need to emit a character and then wait (how long?) to see whether it was wide or not.
let's finally try this out and see what happens.
we can get a quick preview by simply commenting out the `return 0;` in `goto_location()`, let's see...
in kitty, *always* dropping an hpa:
* resolves all mess in `uniblock`
* resolves the overeat in `mojibake`
* completely fixes `whiteout`
* has negligible effect on total `notcurses-demo` runtime
succeeding rather beyond my wildest dreams. like, this at first glance fixes all outstanding width problems, with no significant cost. and that's the naive version, that issues hpa every move--we could cut it down to plane transitions, as speced here.

Alacritty has some small boundary problems in `uniblock` resolved.

XTerm gets much crappier whenever we're clearing sixel. here's another place where we'd really like DECERA to apply to sixel =[. but with the plane-transition change mentioned above, this goes away.
so i think this solves a very real problem, and solves it elegantly and completely. i'm honestly surprised by just how much this improves the situation -- literally every alignment problem i was aware of has been resolved in one fell swoop.

i think it's sensible to add a stat for this, so we can track the number of "unnecessary" `hpa`s being inserted. it might furthermore be reasonable to only insert the `hpa` when we've seen a possibly-wide character, though then we're back to guessing what terminals are going to do.

XTerm's `[xray]` does get substantially crappier, despite the change to only perform on a plane transition -- perhaps we fucked this up? hence the value of the proposed stat:

before:
```c
             runtime│ frames│output(B)│    FPS│%r│%a│%w│TheoFPS║                
══╤════════╤════════╪═══════╪═════════╪═══════╪══╪══╪══╪═══════╣                
 1│   intro│   5.95s│    298│   3.72Mi│   50.1│ 0│ 0│46│ 105.79║                
 2│    xray│  28.97s│    486│  23.93Mi│   16.8│ 0│ 0│67│  24.60║                
 3│   eagle│  13.77s│    368│   7.03Mi│   26.7│ 0│ 0│81│  32.62║                
══╧════════╧════════╪═══════╪═════════╪═══════╧══╧══╧══╧═══════╝                
              48.69s│   1152│  34.68Mi│   
1155 renders, 228.28ms (69.72µs min, 197.65µs avg, 5.40ms max)
1155 rasters, 47.51ms (20.53µs min, 41.14µs avg, 104.79µs max)
1155 writes, 33.58s (20.55µs min, 29.07ms avg, 160.14ms max)
34.69MiB (96B min, 30.75KiB avg, 503.88KiB max) 0 inputs
0 failed renders, 0 failed rasters, 0 refreshes, 0 input errors
RGB emits:elides: def 1180:810404 fg 208921:318784 bg 218348:1120940
Cell emits:elides: 1339288:6592488 (83.11%) 99.85% 60.41% 83.70%
Bitmap emits:elides: 14:761 (98.19%) 25.45MiB (73.37%) SuM: 0 (0.00%)
```

after:

```c
             runtime│ frames│output(B)│    FPS│%r│%a│%w│TheoFPS║                
══╤════════╤════════╪═══════╪═════════╪═══════╪══╪══╪══╪═══════╣                
 1│   intro│   6.72s│    280│   3.56Mi│   41.7│ 0│ 0│53│  76.74║                
 2│    xray│  84.63s│    486│  28.30Mi│    5.7│ 0│ 0│99│   5.76║                
 3│   eagle│  14.70s│    368│   7.93Mi│   25.0│ 0│ 0│83│  29.93║                
══╧════════╧════════╪═══════╪═════════╪═══════╧══╧══╧══╧═══════╝                
             106.05s│   1134│  39.79Mi│                                         
1137 renders, 239.77ms (91.40µs min, 210.88µs avg, 3.08ms max)
1137 rasters, 48.44ms (21.35µs min, 42.60µs avg, 132.83µs max)
1137 writes, 100.10s (20.68µs min, 88.03ms avg, 293.69ms max)
39.83MiB (96B min, 35.87KiB avg, 506.29KiB max) 0 inputs
0 failed renders, 0 failed rasters, 0 refreshes, 0 input errors
RGB emits:elides: def 1180:810404 fg 197140:303059 bg 211390:1100392
Cell emits:elides: 1311782:6496162 (83.20%) 99.85% 60.59% 83.89%
Bitmap emits:elides: 14:748 (98.16%) 25.45MiB (63.90%) SuM: 0 (0.00%)
```

so let's add the stat, and see what we can do about `[xray]` with sixel, but this is happening, and it's happening for 2.4.9.
maaaaaaaaaaaaaaaaaaaaaaaaaaaybe we even make this a per-terminal thing, since some terminals have a single width for all characters?
looks like we're definitely releasing too many, which would explain continued sixel slowdowns:

`49G🚱^[[51G✊🏿69^[[55G🔬^[[57G🧬^[[59G🏴`
> looks like we're definitely releasing too many, which would explain continued sixel slowdowns:
> 
> `49G🚱^[[51G✊🏿69^[[55G🔬^[[57G🧬^[[59G🏴`

this of course means we must verify continued effectiveness once we eliminate the unnecessary `hpa`s.
i'm so fuckin' excited about this! a major, major improvement.

Fix patch:
diff --git a/NEWS.md b/NEWS.md
index 3d80e3c744..d9f9333692 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -11,6 +11,16 @@ rearrangements of Notcurses.
     `notcurses_options`, for which I make no apology. If you've been avoiding
     deprecated functionality, ABI3 ought require small changes, if any.
 
+* 2.4.9 (not yet released)
+  * On transition between `ncplane`s (on terminals implementing complex wide
+    glyphs), Notcurses now always issues an `hpa` sequence to force horizontal
+    positioning. This fixes a number of longstanding bugs in e.g. the
+    `[uniblock]` and `[whiteout]` demos at the cost of some extra control
+    sequences. For more information, see
+    [issue 2199](https://github.com/dankamongmen/notcurses/issues/2199). The
+    number of `hpa`s issued in this manner is tracked in a new stat,
+    `hpa_gratuitous`.
+
 * 2.4.8 (2021-10-23)
   * Added new functions `notcurses_canpixel()` and `notcurses_osversion()`.
   * `notcurses_get()` now evaluates its timeout against `CLOCK_MONOTONIC`
@@ -186,7 +196,7 @@ rearrangements of Notcurses.
   * Added `ncplane_moverel()`.
   * Documented `ncplane_move_yx()` in `notcurses_plane.3`, and removed the
     false comment that "passing -1 as a coordinate will hold that axis
-    constant" from `USGAE.md` and `notcurses.h`. This has never been true.
+    constant" from `USAGE.md` and `notcurses.h`. This has never been true.
   * Added `ncdirect_putegc()` to perform Unicode segmentation. It returns
     the number of columns consumed, and makes available the number of bytes
     used by the EGC.
@@ -578,10 +588,7 @@ rearrangements of Notcurses.
     * Add new function `ncpile_render()`, which renders the pile containing the
       specified plane to the specified buffer. Add new function
       `ncpile_rasterize()` to rasterize the specified buffer to output.
-  * Added `NCSTYLE_STRUCK` for strikethrough. Note that this is not supported
-    by terminfo, and we instead just hardcode the control sequence. Use at your
-    own risk! If your terminal doesn't support this control sequence, behavior
-    is undefined.
+  * Added `NCSTYLE_STRUCK` for strikethrough.
 
 * 2.0.7 (2020-11-21)
   * The `horiz` union of `ncplane_options` has been discarded; the `int x`
diff --git a/README.md b/README.md
index 290bafa6b9..af169b4361 100644
--- a/README.md
+++ b/README.md
@@ -117,7 +117,6 @@ may well be possible to use still older versions. Let me know of any successes!
 * (OPTIONAL) (testing) [Doctest](https://github.com/onqtam/doctest) 2.3.5+
 * (OPTIONAL) (documentation) [pandoc](https://pandoc.org/index.html) 1.19.2+
 * (OPTIONAL) (python bindings): Python 3.7+, [CFFI](https://pypi.org/project/cffi/) 1.13.2+, [pypandoc](https://pypi.org/project/pypandoc/) 1.5+
-* (OPTIONAL) (rust bindings): rust 1.47.0+, [bindgen](https://crates.io/crates/bindgen) 0.55.1+, pkg-config 0.3.18+, cty 0.2.1+
 * (runtime) Linux 5.3+, FreeBSD 11+, DragonFly BSD 5.9+, Windows Vista+, or macOS 11.4+
 
 [Here's more information](INSTALL.md) on building and installation.
@@ -450,7 +449,9 @@ If things break or seem otherwise lackluster, **please** consult the
 <details>
   <summary>Will there ever be Java wrappers?</summary>
   I should hope not. If you want a Java solution, try Autumn Lamonte's
-  <a href="https://jexer.sourceforge.io/">Jexer</a>.
+  <a href="https://jexer.sourceforge.io/">Jexer</a>. Autumn's a good
+  woman, and thorough. We seem to have neatly partitioned the language
+  space.
 </details>
 
 <details>
@@ -462,13 +463,6 @@ If things break or seem otherwise lackluster, **please** consult the
   so it isn't happening.
 </details>
 
-<details>
-  <summary>Why does my right-to-left text appear left-to-right?</summary>
-  Notcurses doesn't honor the BiDi state machine, and in fact forces
-  left-to-right with BiDi codes. Likewise, ultra-wide glyphs will have
-  interesting effects. ﷽!
-</details>
-
 <details>
   <summary>I get linker errors when statically linking.</summary>
   Are you linking all necessary libraries? Use
diff --git a/USAGE.md b/USAGE.md
index 00ab4002b3..8fa554b621 100644
--- a/USAGE.md
+++ b/USAGE.md
@@ -3587,6 +3587,7 @@ typedef struct ncstats {
   uint64_t sprixelbytes;     // sprixel bytes emitted
   uint64_t appsync_updates;  // application-synchronized updates
   uint64_t input_errors;     // errors processing control sequences/utf8
+  uint64_t hpa_gratuitous;   // unnecessary hpas issued
 
   // current state -- these can decrease
   uint64_t fbbytes;          // total bytes devoted to all active framebuffers
diff --git a/doc/man/man3/notcurses_stats.3.md b/doc/man/man3/notcurses_stats.3.md
index e4e3edd3ea..e2b78bb024 100644
--- a/doc/man/man3/notcurses_stats.3.md
+++ b/doc/man/man3/notcurses_stats.3.md
@@ -44,6 +44,7 @@ typedef struct ncstats {
   uint64_t appsync_updates;  // application-synchronized updates
   uint64_t input_events;     // inputs received or synthesized
   uint64_t input_errors;     // errors processing input
+  uint64_t hpa_gratuitous;   // gratuitous HPAs issued
 
   // current state -- these can decrease
   uint64_t fbbytes;          // bytes devoted to framebuffers
@@ -117,6 +118,15 @@ include move/delete operations, nor glyphs used to erase sprixels.
 **input_errors** is the number of errors while processing input, e.g.
 malformed control sequences or invalid UTF-8 (see **utf8(7)**).
 
+**hpa_gratuitous** is the number of **hpa** (horizontal position absolute,
+see **terminfo(5)**) control sequences issued where not strictly necessary.
+This is done to cope with fundamental ambiguities regarding glyph
+width. It is not generally possible to know how wide a glyph will be rendered
+on a given combination of font, font rendering engine, and terminal. Indeed, it
+is not even generally possible to know how many glyphs will result from a
+sequence of EGCs. As a result, Notcurses sometimes issues "gratuitous" **hpa**
+controls.
+
 # NOTES
 
 Unsuccessful render operations do not contribute to the render timing stats.
@@ -138,4 +148,5 @@ object on success, or **NULL** on failure.
 **mmap(2)**,
 **notcurses(3)**,
 **notcurses_render(3)**,
+**terminfo(5)**,
 **utf8(7)**
diff --git a/include/notcurses/notcurses.h b/include/notcurses/notcurses.h
index b13d08a317..4e5eedd1b1 100644
--- a/include/notcurses/notcurses.h
+++ b/include/notcurses/notcurses.h
@@ -1541,6 +1541,7 @@ typedef struct ncstats {
   uint64_t sprixelbytes;     // sprixel bytes emitted
   uint64_t input_errors;     // errors processing control sequences/utf8
   uint64_t input_events;     // characters returned to userspace
+  uint64_t hpa_gratuitous;   // unnecessary hpas issued
 } ncstats;
 
 // Allocate an ncstats object. Use this rather than allocating your own, since
diff --git a/src/lib/in.c b/src/lib/in.c
index b6e24cfd31..9b41b6abd9 100644
--- a/src/lib/in.c
+++ b/src/lib/in.c
@@ -1035,6 +1035,8 @@ da2_cb(inputctx* ictx){
   if(pv == 0){
     return 2;
   }
+  // modern XTerm replies to XTVERSION, but older versions require extracting
+  // the version from secondary DA
   if(ictx->initdata->qterm == TERMINAL_XTERM){
     if(ictx->initdata->version == NULL){
       char ver[8];
@@ -1217,11 +1219,14 @@ tcap_cb(inputctx* ictx){
   // 'TN' (Terminal Name)
   if(strncasecmp(str, "544e=", 5) == 0){
     const char* tn = str + 5;
+    // FIXME clean this crap up
     if(strcasecmp(tn, "6D6C7465726D") == 0){
       ictx->initdata->qterm = TERMINAL_MLTERM;
+    }else if(strcasecmp(tn, "787465726d") == 0){
+      ictx->initdata->qterm = TERMINAL_XTERM; // "xterm"
     }else if(strcasecmp(tn, "787465726d2d6b69747479") == 0){
-      ictx->initdata->qterm = TERMINAL_KITTY;
-    }else if(strcasecmp(tn, "787465726D2D323536636F6C6F72") == 0){
+      ictx->initdata->qterm = TERMINAL_KITTY; // "xterm-kitty"
+    }else if(strcasecmp(tn, "787465726d2d323536636f6c6f72") == 0){
       ictx->initdata->qterm = TERMINAL_XTERM; // "xterm-256color"
     }else{
       logdebug("unknown terminal name %s\n", tn);
diff --git a/src/lib/internal.h b/src/lib/internal.h
index 5a7cc9c5ea..ce089bb9fc 100644
--- a/src/lib/internal.h
+++ b/src/lib/internal.h
@@ -128,6 +128,8 @@ typedef struct rasterstate {
   // modified by: output, cursor moves, clearing the screen (during refresh).
   int y, x;
 
+  const ncplane* lastsrcp; // last source plane (we emit hpa on plane changes)
+
   unsigned lastr;   // foreground rgb, overloaded for palindexed fg
   unsigned lastg;
   unsigned lastb;
@@ -1148,13 +1150,14 @@ mouse_disable(tinfo* ti, fbuf* f){
 // sync the drawing position to the specified location with as little overhead
 // as possible (with nothing, if already at the right location). we prefer
 // absolute horizontal moves (hpa) to relative ones, in the rare event that
-// our understanding of our horizontal location is faulty.
+// our understanding of our horizontal location is faulty. if we're moving from
+// one plane to another, we emit an hpa no matter what.
 // FIXME fall back to synthesized moves in the absence of capabilities (i.e.
 // textronix lacks cup; fake it with horiz+vert moves)
 // if hardcursorpos is non-zero, we always perform a cup. this is done when we
 // don't know where the cursor currently is =].
 static inline int
-goto_location(notcurses* nc, fbuf* f, int y, int x){
+goto_location(notcurses* nc, fbuf* f, int y, int x, const ncplane* srcp){
 //fprintf(stderr, "going to %d/%d from %d/%d hard: %u\n", y, x, nc->rstate.y, nc->rstate.x, nc->rstate.hardcursorpos);
   int ret = 0;
   // if we don't have hpa, force a cup even if we're only 1 char away. the only
@@ -1162,8 +1165,11 @@ goto_location(notcurses* nc, fbuf* f, int y, int x){
   // you can't use cuf for backwards moves anyway; again, vt100 can suck it.
   const char* hpa = get_escape(&nc->tcache, ESCAPE_HPA);
   if(nc->rstate.y == y && hpa && !nc->rstate.hardcursorpos){ // only need move x
-    if(nc->rstate.x == x){ // needn't move shit
-      return 0;
+    if(nc->rstate.x == x){
+      if(nc->rstate.lastsrcp == srcp || !nc->tcache.gratuitous_hpa){
+        return 0; // needn't move shit
+      }
+      ++nc->stats.s.hpa_gratuitous;
     }
     if(fbuf_emit(f, tiparm(hpa, x))){
       return -1;
@@ -1182,6 +1188,7 @@ goto_location(notcurses* nc, fbuf* f, int y, int x){
   nc->rstate.x = x;
   nc->rstate.y = y;
   nc->rstate.hardcursorpos = 0;
+  nc->rstate.lastsrcp = srcp;
   return ret;
 }
 
@@ -1317,33 +1324,6 @@ cell_blend_bchannel(nccell* cl, unsigned channel, unsigned* blends){
   return cell_set_bchannel(cl, channels_blend(cell_bchannel(cl), channel, blends));
 }
 
-// examine the UTF-8 EGC in the first |*bytes| bytes of |egc|. if the EGC is
-// right-to-left, we make a copy, appending an U+200E to force left-to-right.
-// only the first unicode char of the EGC is currently checked FIXME. if the
-// EGC is not RTL, we return NULL.
-ALLOC static inline char*
-egc_rtl(const char* egc, int* bytes){
-  wchar_t w;
-  mbstate_t mbstate = { };
-  size_t r = mbrtowc(&w, egc, *bytes, &mbstate);
-  if(r == (size_t)-1 || r == (size_t)-2){
-    return NULL;
-  }
-  const int bidic = uc_bidi_category(w);
-//fprintf(stderr, "BIDI CAT (%lc): %d\n", w, bidic);
-  if(bidic != UC_BIDI_R && bidic != UC_BIDI_AL && bidic != UC_BIDI_RLE && bidic != UC_BIDI_RLO){
-    return NULL;
-  }
-  // insert U+200E, "LEFT-TO-RIGHT MARK". This ought reset the text direction
-  // after emitting a potentially RTL EGC.
-  const char LTRMARK[] = "\xe2\x80\x8e";
-  char* s = (char*)malloc(*bytes + sizeof(LTRMARK)); // cast for C++ callers
-  memcpy(s, egc, *bytes);
-  memcpy(s + *bytes, LTRMARK, sizeof(LTRMARK));
-  *bytes += strlen(LTRMARK);
-  return s;
-}
-
 // a sprixel occupies the entirety of its associated plane, usually an entirely
 // new, purpose-specific plane. |leny| and |lenx| are output geometry in pixels.
 static inline int
@@ -1411,21 +1391,12 @@ pool_blit_direct(egcpool* pool, nccell* c, const char* gcluster, int bytes, int
   return bytes;
 }
 
-// Do an RTL-check, reset the quadrant occupancy bits, and pass the cell down to
+// Reset the quadrant occupancy bits, and pass the cell down to
 // pool_blit_direct(). Returns the number of bytes loaded.
 static inline int
 pool_load_direct(egcpool* pool, nccell* c, const char* gcluster, int bytes, int cols){
-  char* rtl = NULL;
   c->channels &= ~NC_NOBACKGROUND_MASK;
-  if(bytes >= 0){
-    rtl = egc_rtl(gcluster, &bytes); // checks for RTL and adds U+200E if so
-  }
-  if(rtl){
-    gcluster = rtl;
-  }
-  int r = pool_blit_direct(pool, c, gcluster, bytes, cols);
-  free(rtl);
-  return r;
+  return pool_blit_direct(pool, c, gcluster, bytes, cols);
 }
 
 static inline int
diff --git a/src/lib/kitty.c b/src/lib/kitty.c
index 7f66f93af5..c13071e2a9 100644
--- a/src/lib/kitty.c
+++ b/src/lib/kitty.c
@@ -1183,7 +1183,7 @@ int kitty_move(sprixel* s, fbuf* f, unsigned noscroll, int yoff, int xoff){
   const int targx = s->n->absx;
   logdebug("moving %u to %d %d\n", s->id, targy, targx);
   int ret = 0;
-  if(goto_location(ncplane_notcurses(s->n), f, targy + yoff, targx + xoff)){
+  if(goto_location(ncplane_notcurses(s->n), f, targy + yoff, targx + xoff, s->n)){
     ret = -1;
   }else if(fbuf_printf(f, "\e_Ga=p,i=%d,p=1,q=2%s\e\\", s->id,
                        noscroll ? ",C=1" : "") < 0){
diff --git a/src/lib/notcurses.c b/src/lib/notcurses.c
index fe22c12309..b90eaf9830 100644
--- a/src/lib/notcurses.c
+++ b/src/lib/notcurses.c
@@ -1093,7 +1093,7 @@ notcurses* notcurses_core_init(const notcurses_options* opts, FILE* outfp){
     // the u7 led the queries so that we would get a cursor position
     // unaffected by any query spill (unconsumed control sequences). move
     // us back to that location, in case there was any such spillage.
-    if(goto_location(ret, &ret->rstate.f, *cursory, *cursorx)){
+    if(goto_location(ret, &ret->rstate.f, *cursory, *cursorx, NULL)){
       goto err;
     }
   }
@@ -1227,7 +1227,7 @@ int notcurses_stop(notcurses* nc){
         fbuf_putc(&nc->rstate.f, '\n');
         --targy;
       }
-      goto_location(nc, &nc->rstate.f, targy, 0);
+      goto_location(nc, &nc->rstate.f, targy, 0, NULL);
       fbuf_finalize(&nc->rstate.f, stdout);
     }
     if(nc->stdplane){
diff --git a/src/lib/render.c b/src/lib/render.c
index fa10696896..506cb8e63a 100644
--- a/src/lib/render.c
+++ b/src/lib/render.c
@@ -976,7 +976,7 @@ rasterize_scrolls(const ncpile* p, fbuf* f){
   if(p->nc->tcache.pixel_scroll){
     p->nc->tcache.pixel_scroll(p, &p->nc->tcache, scrolls);
   }
-  if(goto_location(p->nc, f, p->dimy, 0)){
+  if(goto_location(p->nc, f, p->dimy, 0, NULL)){
     return -1;
   }
   // terminals advertising 'bce' will scroll in the current background color;
@@ -1018,7 +1018,7 @@ rasterize_sprixels(notcurses* nc, ncpile* p, fbuf* f){
       if(nc->tcache.pixel_commit){
         int y, x;
         ncplane_abs_yx(s->n, &y, &x);
-        if(goto_location(nc, f, y + nc->margin_t, x + nc->margin_l)){
+        if(goto_location(nc, f, y + nc->margin_t, x + nc->margin_l, NULL)){
           return -1;
         }
         if(sprite_commit(&nc->tcache, f, s, false)){
@@ -1107,7 +1107,7 @@ rasterize_core(notcurses* nc, const ncpile* p, fbuf* f, unsigned phase){
         // was not above a sprixel (and the cell is damaged). in the second
         // phase, we draw everything that remains damaged.
         ++nc->stats.s.cellemissions;
-        if(goto_location(nc, f, y, x)){
+        if(goto_location(nc, f, y, x, rvec[damageidx].p)){
           return -1;
         }
         // set the style. this can change the color back to the default; if it
@@ -1322,7 +1322,7 @@ notcurses_rasterize(notcurses* nc, ncpile* p, fbuf* f){
   if(cursory >= 0){
     notcurses_cursor_enable(nc, cursory, cursorx);
   }else if(nc->rstate.logendy >= 0){
-    goto_location(nc, f, nc->rstate.logendy, nc->rstate.logendx);
+    goto_location(nc, f, nc->rstate.logendy, nc->rstate.logendx, nc->rstate.lastsrcp);
     if(fbuf_flush(f, nc->ttyfp)){
       ret = -1;
     }
@@ -1341,18 +1341,10 @@ int clear_and_home(notcurses* nc, tinfo* ti, fbuf* f){
       goto success;
     }
   }
-  const ncplane* stdn = notcurses_stdplane_const(nc);
-  // clearscr didn't fly. try scrolling everything off. first, go to the
-  // bottom of the screen, then write N newlines.
-  if(goto_location(nc, f, ncplane_dim_y(stdn) - 1, 0)){
+  if(emit_scrolls(ti, ncplane_dim_y(notcurses_stdplane_const(nc)), f)){
     return -1;
   }
-  for(int y = 0 ; y < ncplane_dim_y(stdn) ; ++y){
-    if(fbuf_putc(f, '\n') < 0){
-      return -1;
-    }
-  }
-  if(goto_location(nc, f, 0, 0)){
+  if(goto_location(nc, f, 0, 0, NULL)){
     return -1;
   }
 
@@ -1682,7 +1674,7 @@ int notcurses_cursor_enable(notcurses* nc, int y, int x){
     return -1;
   }
   // updates nc->rstate.cursor{y,x}
-  if(goto_location(nc, &f, y + nc->margin_t, x + nc->margin_l)){
+  if(goto_location(nc, &f, y + nc->margin_t, x + nc->margin_l, nc->rstate.lastsrcp)){
     fbuf_free(&f);
     return -1;
   }
diff --git a/src/lib/sixel.c b/src/lib/sixel.c
index bdfdfd24f7..90b815e9fe 100644
--- a/src/lib/sixel.c
+++ b/src/lib/sixel.c
@@ -1010,7 +1010,7 @@ int sixel_draw(const tinfo* ti, const ncpile* p, sprixel* s, fbuf* f,
   if(p){
     const int targy = s->n->absy + yoff;
     const int targx = s->n->absx + xoff;
-    if(goto_location(p->nc, f, targy, targx)){
+    if(goto_location(p->nc, f, targy, targx, NULL)){
       return -1;
     }
     if(s->invalidated == SPRIXEL_MOVED){
diff --git a/src/lib/stats.c b/src/lib/stats.c
index c10cec48b4..559d41fa0b 100644
--- a/src/lib/stats.c
+++ b/src/lib/stats.c
@@ -152,6 +152,7 @@ void notcurses_stats_reset(notcurses* nc, ncstats* stats){
     stash->appsync_updates += nc->stats.s.appsync_updates;
     stash->input_errors += nc->stats.s.input_errors;
     stash->input_events += nc->stats.s.input_events;
+    stash->hpa_gratuitous += nc->stats.s.hpa_gratuitous;
 
     stash->fbbytes = nc->stats.s.fbbytes;
     stash->planes = nc->stats.s.planes;
@@ -203,17 +204,20 @@ void summarize_stats(notcurses* nc){
             1, minbuf, 1),
     bprefix(stats->renders ? stats->render_bytes / stats->renders : 0, 1, avgbuf, 1);
     bprefix(stats->render_max_bytes, 1, maxbuf, 1),
-    fprintf(stderr, "%s%sB (%sB min, %sB avg, %sB max) %"PRIu64" input%s" NL,
+    fprintf(stderr, "%s%sB (%sB min, %sB avg, %sB max) %"PRIu64" input%s Ghpa: %"PRIu64 NL,
             clreol, totalbuf, minbuf, avgbuf, maxbuf,
             stats->input_events,
-            stats->input_events == 1 ? "" : "s");
+            stats->input_events == 1 ? "" : "s",
+            stats->hpa_gratuitous);
   }
-  fprintf(stderr, "%s%"PRIu64" failed render%s, %"PRIu64" failed raster%s, %"PRIu64" refresh%s, %"PRIu64" input error%s" NL,
+  fprintf(stderr, "%s%"PRIu64" failed render%s, %"PRIu64" failed raster%s, %"
+                  PRIu64" refresh%s, %"PRIu64" input error%s" NL,
           clreol, stats->failed_renders, stats->failed_renders == 1 ? "" : "s",
           stats->failed_writeouts, stats->failed_writeouts == 1 ? "" : "s",
           stats->refreshes, stats->refreshes == 1 ? "" : "es",
           stats->input_errors, stats->input_errors == 1 ? "" : "s");
-  fprintf(stderr, "%sRGB emits:elides: def %"PRIu64":%"PRIu64" fg %"PRIu64":%"PRIu64" bg %"PRIu64":%"PRIu64"" NL,
+  fprintf(stderr, "%sRGB emits:elides: def %"PRIu64":%"PRIu64" fg %"PRIu64":%"
+                  PRIu64" bg %"PRIu64":%"PRIu64 NL,
           clreol, stats->defaultemissions,
           stats->defaultelisions,
           stats->fgemissions,
diff --git a/src/lib/termdesc.c b/src/lib/termdesc.c
index 5f4ee138d4..d8a3952fb6 100644
--- a/src/lib/termdesc.c
+++ b/src/lib/termdesc.c
@@ -423,7 +423,7 @@ send_initial_queries(int fd, bool minimal, bool noaltscreen){
     }
   }
   size_t len = strlen(queries);
-  loginfo("sending %lluB queries\n", (unsigned long long)len);
+  loginfo("sending %lluB\n", (unsigned long long)len);
   if(blocking_write(fd, queries, len)){
     return -1;
   }
@@ -585,6 +585,7 @@ apply_term_heuristics(tinfo* ti, const char* termname, queried_terminals_e qterm
     // kitty SUM doesn't want long sequences, which is exactly where we use
     // it. remove support (we pick it up from queries).
     kill_appsync_escapes(ti);
+    ti->gratuitous_hpa = true;
   }else if(qterm == TERMINAL_ALACRITTY){
     termname = "Alacritty";
     ti->caps.quadrants = true;
diff --git a/src/lib/termdesc.h b/src/lib/termdesc.h
index 116f7ea8ad..3ab1545536 100644
--- a/src/lib/termdesc.h
+++ b/src/lib/termdesc.h
@@ -167,6 +167,8 @@ typedef struct tinfo {
   struct termios *tpreserved;// terminal state upon entry
   struct inputctx* ictx;     // new input layer
   unsigned stdio_blocking_save; // was stdio blocking at entry? restore on stop.
+  // ought we issue gratuitous HPAs to work around ambiguous widths?
+  unsigned gratuitous_hpa;
 
   // if we get a reply to our initial \e[18t cell geometry query, it will
   // replace these values. note that LINES/COLUMNS cannot be used to limit


Test patch:
diff --git a/src/tests/bitmap.cpp b/src/tests/bitmap.cpp
index 95ba136445..ce794c98ad 100644
--- a/src/tests/bitmap.cpp
+++ b/src/tests/bitmap.cpp
@@ -185,6 +185,7 @@ TEST_CASE("Bitmaps") {
     vopts.n = n_;
     vopts.blitter = NCBLIT_PIXEL;
     vopts.flags = NCVISUAL_OPTION_NODEGRADE | NCVISUAL_OPTION_CHILDPLANE;
+    vopts.scaling = NCSCALE_STRETCH;
     auto newn = ncvisual_blit(nc_, ncv, &vopts);
     CHECK(newn);
     CHECK(0 == notcurses_render(nc_));
diff --git a/src/tests/sixel.cpp b/src/tests/sixel.cpp
index 84c9a16f9f..129e84b2a1 100644
--- a/src/tests/sixel.cpp
+++ b/src/tests/sixel.cpp
@@ -165,8 +165,9 @@ TEST_CASE("Sixels") {
     vopts.n = n_;
     vopts.blitter = NCBLIT_PIXEL;
     vopts.flags = NCVISUAL_OPTION_NODEGRADE | NCVISUAL_OPTION_CHILDPLANE;
+    vopts.scaling = NCSCALE_STRETCH;
     auto newn = ncvisual_blit(nc_, ncv, &vopts);
-    CHECK(newn);
+    REQUIRE(nullptr != newn);
     CHECK(0 == notcurses_render(nc_));
     auto rgb = sixel_to_rgb(newn->sprite->glyph.buf, newn->sprite->glyph.used,
                             newn->sprite->pixy, newn->sprite->pixx);
@@ -189,7 +190,7 @@ TEST_CASE("Sixels") {
     vopts.blitter = NCBLIT_PIXEL;
     vopts.flags = NCVISUAL_OPTION_NODEGRADE | NCVISUAL_OPTION_CHILDPLANE;
     auto newn = ncvisual_blit(nc_, ncv, &vopts);
-    CHECK(newn);
+    REQUIRE(nullptr != newn);
     auto rgbold = sixel_to_rgb(newn->sprite->glyph.buf, newn->sprite->glyph.used,
                                newn->sprite->pixy, newn->sprite->pixx);
 //print_bmap(rgbold, newn->sprite->pixy, newn->sprite->pixx);