Repository: kovidgoyal/kitty

Issue:
Feature request: Disabling bidi support
TL;DR: Some applications (e.g. vim) have their own bidi support, which conflicts
with kitty's support. Please offer a way of turning off kitty's bidi support.

Kitty's bidi seems to cut the line into words, shape each word, then place them
in one line left-to-right, whereas vim's bidi just places letters one after the
other in the specified direction.
(Side note: I can't find kitty's behaviour here documented anywhere)

Hence, kitty's bidi works reasonably well for interactive shell use and the
like, where there is a prevailing directionality and the reversed text is
relatively rare. Conversely, vim's bidi works well in longform text where
changes in direction occur in multi-word streaks.

(Hypothetically, if a good bidi algorithm were implemented (e.g.
[UAX9](https://www.unicode.org/reports/tr9/)), it would deprecate vim's bidi in
_some_ scenarios, but not all – in cases where extra markup exists, such
algorithms would probably get confused by the markup and give wrong results)

However, kitty+vim is broken in this regard – kitty lies to vim about what's
under the cursor, editing longform text written in this zigzag layout is tiring,
and I don't even have the usual vim workaround of opening the file and its
reflection, since then the words are also reflected.


Discussion:
dup of #2109
Not a dup - am asking for doc + config for a feature already existing in kitty today. 

On 5 January 2020 5:28:43 GMT+02:00, Kovid Goyal <notifications@github.com> wrote:
>Closed #2252.
>
>-- 
>You are receiving this because you authored the thread.
>Reply to this email directly or view it on GitHub:
>https://github.com/kovidgoyal/kitty/issues/2252#event-2922190109

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
There is no bidi supportin kitty at all. What you see if just harfbuzz
reversing rtl text based on unicode code points.

Ah. So implementing this would be a matter of toggling between HarfBuzz and a naive shaper, the development effort of which would be better served improving the bidi support. I will submit, though, that HarfBuzz-style shaping is inappropriate when either the underlying application does its own reordering,  or when one wants to see RR he text in logical order (eg when debugging compilation errors).

Only thing that's unclear is why bidi text is displayed in reverse word order, ie the equivalent of WORLD HELLO. Is kitty splitting words before passing them? 

On 5 January 2020 12:42:49 GMT+02:00, Kovid Goyal <notifications@github.com> wrote:
>There is no bidi supportin kitty at all. What you see if just harfbuzz
>reversing rtl text based on unicode code points.
>
>
>-- 
>You are receiving this because you authored the thread.
>Reply to this email directly or view it on GitHub:
>https://github.com/kovidgoyal/kitty/issues/2252#issuecomment-570898843

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Sun, Jan 05, 2020 at 04:57:08AM -0800, hseg wrote:
> Ah. So implementing this would be a matter of toggling between HarfBuzz and a naive shaper, the development effort of which would be better served improving the bidi support. I will submit, though, that HarfBuzz-style shaping is inappropriate when either the underlying application does its own reordering,  or when one wants to see RR he text in logical order (eg when debugging compilation errors).

Yeah and proper bidi support means having some way for the application
to inform the terminal what level of bidi support it needs/wants.
Without that its just whack-a-mole.

> 
> Only thing that's unclear is why bidi text is displayed in reverse word order, ie the equivalent of WORLD HELLO. Is kitty splitting words before passing them? 

Yup, kitty rendering fundamentally works at a character cell level, not
at the level of words/lines/whatever other logical unit you can name.
And character cell rendering works on runs, runs that are broken up at
spaces, font changes, various other things.

This allows for many optimisations that give it its stellar performance,
but it means support for complex scripts/bidi/etc. is never going to be 
that good.




On 5 January 2020 15:04:31 GMT+02:00, Kovid Goyal <notifications@github.com> wrote:
>On Sun, Jan 05, 2020 at 04:57:08AM -0800, hseg wrote:
>> Ah. So implementing this would be a matter of toggling between
>HarfBuzz and a naive shaper, the development effort of which would be
>better served improving the bidi support. I will submit, though, that
>HarfBuzz-style shaping is inappropriate when either the underlying
>application does its own reordering,  or when one wants to see RR he
>text in logical order (eg when debugging compilation errors).
>
>Yeah and proper bidi support means having some way for the application
>to inform the terminal what level of bidi support it needs/wants.
>Without that its just whack-a-mole.

Which is why I suggested toggling shaping - that way I can at least wrap the relevant programs. 

>> Only thing that's unclear is why bidi text is displayed in reverse
>word order, ie the equivalent of WORLD HELLO. Is kitty splitting words
>before passing them? 
>
>Yup, kitty rendering fundamentally works at a character cell level, not
>at the level of words/lines/whatever other logical unit you can name.
>And character cell rendering works on runs, runs that are broken up at
>spaces, font changes, various other things.
>
>This allows for many optimisations that give it its stellar
>performance,
>but it means support for complex scripts/bidi/etc. is never going to be
>
>that good.

Oh... and I suppose joining runs separated by weak characters would be problematic for that? Just that would already be nice (with the caveat that lines should be processed LTR by default due to mostly-English command line) 
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

On Sun, Jan 05, 2020 at 05:46:24AM -0800, hseg wrote:
> 
> 
> On 5 January 2020 15:04:31 GMT+02:00, Kovid Goyal <notifications@github.com> wrote:
> >On Sun, Jan 05, 2020 at 04:57:08AM -0800, hseg wrote:
> >> Ah. So implementing this would be a matter of toggling between
> >HarfBuzz and a naive shaper, the development effort of which would be
> >better served improving the bidi support. I will submit, though, that
> >HarfBuzz-style shaping is inappropriate when either the underlying
> >application does its own reordering,  or when one wants to see RR he
> >text in logical order (eg when debugging compilation errors).
> >
> >Yeah and proper bidi support means having some way for the application
> >to inform the terminal what level of bidi support it needs/wants.
> >Without that its just whack-a-mole.
> 
> Which is why I suggested toggling shaping - that way I can at least wrap the relevant programs. 

Yes, but that requires a fair amount of work and is just a bandaid. If
I am going to put in the work, I am not going to do it for a bandaid.
I encourage you to contribute code to kitty for bidi support i it is
important to you, personally, its not a high priority.

> 
> >> Only thing that's unclear is why bidi text is displayed in reverse
> >word order, ie the equivalent of WORLD HELLO. Is kitty splitting words
> >before passing them? 
> >
> >Yup, kitty rendering fundamentally works at a character cell level, not
> >at the level of words/lines/whatever other logical unit you can name.
> >And character cell rendering works on runs, runs that are broken up at
> >spaces, font changes, various other things.
> >
> >This allows for many optimisations that give it its stellar
> >performance,
> >but it means support for complex scripts/bidi/etc. is never going to be
> >
> >that good.
> 
> Oh... and I suppose joining runs separated by weak characters would be problematic for that? Just that would already be nice (with the caveat that lines should be processed LTR by default due to mostly-English command line) 

It might not be, depends on the details, and I dont understand bidi well
enough to comment authoritatively.


OK, thanks. Have a workaround ATM (use gvim), but will put this on the back burner for if this gets annoying enough before you fix bidi.

On 5 January 2020 16:32:28 GMT+02:00, Kovid Goyal <notifications@github.com> wrote:
>On Sun, Jan 05, 2020 at 05:46:24AM -0800, hseg wrote:
>> 
>> 
>> On 5 January 2020 15:04:31 GMT+02:00, Kovid Goyal
><notifications@github.com> wrote:
>> >On Sun, Jan 05, 2020 at 04:57:08AM -0800, hseg wrote:
>> >> Ah. So implementing this would be a matter of toggling between
>> >HarfBuzz and a naive shaper, the development effort of which would
>be
>> >better served improving the bidi support. I will submit, though,
>that
>> >HarfBuzz-style shaping is inappropriate when either the underlying
>> >application does its own reordering,  or when one wants to see RR he
>> >text in logical order (eg when debugging compilation errors).
>> >
>> >Yeah and proper bidi support means having some way for the
>application
>> >to inform the terminal what level of bidi support it needs/wants.
>> >Without that its just whack-a-mole.
>> 
>> Which is why I suggested toggling shaping - that way I can at least
>wrap the relevant programs. 
>
>Yes, but that requires a fair amount of work and is just a bandaid. If
>I am going to put in the work, I am not going to do it for a bandaid.
>I encourage you to contribute code to kitty for bidi support i it is
>important to you, personally, its not a high priority.
>
>> 
>> >> Only thing that's unclear is why bidi text is displayed in reverse
>> >word order, ie the equivalent of WORLD HELLO. Is kitty splitting
>words
>> >before passing them? 
>> >
>> >Yup, kitty rendering fundamentally works at a character cell level,
>not
>> >at the level of words/lines/whatever other logical unit you can
>name.
>> >And character cell rendering works on runs, runs that are broken up
>at
>> >spaces, font changes, various other things.
>> >
>> >This allows for many optimisations that give it its stellar
>> >performance,
>> >but it means support for complex scripts/bidi/etc. is never going to
>be
>> >
>> >that good.
>> 
>> Oh... and I suppose joining runs separated by weak characters would
>be problematic for that? Just that would already be nice (with the
>caveat that lines should be processed LTR by default due to
>mostly-English command line) 
>
>It might not be, depends on the details, and I dont understand bidi
>well
>enough to comment authoritatively.
>
>
>
>-- 
>You are receiving this because you authored the thread.
>Reply to this email directly or view it on GitHub:
>https://github.com/kovidgoyal/kitty/issues/2252#issuecomment-570917707

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Thanks!

On 5 January 2020 16:32:28 GMT+02:00, Kovid Goyal <notifications@github.com> wrote:
>On Sun, Jan 05, 2020 at 05:46:24AM -0800, hseg wrote:
>> 
>> 
>> On 5 January 2020 15:04:31 GMT+02:00, Kovid Goyal
><notifications@github.com> wrote:
>> >On Sun, Jan 05, 2020 at 04:57:08AM -0800, hseg wrote:
>> >> Ah. So implementing this would be a matter of toggling between
>> >HarfBuzz and a naive shaper, the development effort of which would
>be
>> >better served improving the bidi support. I will submit, though,
>that
>> >HarfBuzz-style shaping is inappropriate when either the underlying
>> >application does its own reordering,  or when one wants to see RR he
>> >text in logical order (eg when debugging compilation errors).
>> >
>> >Yeah and proper bidi support means having some way for the
>application
>> >to inform the terminal what level of bidi support it needs/wants.
>> >Without that its just whack-a-mole.
>> 
>> Which is why I suggested toggling shaping - that way I can at least
>wrap the relevant programs. 
>
>Yes, but that requires a fair amount of work and is just a bandaid. If
>I am going to put in the work, I am not going to do it for a bandaid.
>I encourage you to contribute code to kitty for bidi support i it is
>important to you, personally, its not a high priority.
>
>> 
>> >> Only thing that's unclear is why bidi text is displayed in reverse
>> >word order, ie the equivalent of WORLD HELLO. Is kitty splitting
>words
>> >before passing them? 
>> >
>> >Yup, kitty rendering fundamentally works at a character cell level,
>not
>> >at the level of words/lines/whatever other logical unit you can
>name.
>> >And character cell rendering works on runs, runs that are broken up
>at
>> >spaces, font changes, various other things.
>> >
>> >This allows for many optimisations that give it its stellar
>> >performance,
>> >but it means support for complex scripts/bidi/etc. is never going to
>be
>> >
>> >that good.
>> 
>> Oh... and I suppose joining runs separated by weak characters would
>be problematic for that? Just that would already be nice (with the
>caveat that lines should be processed LTR by default due to
>mostly-English command line) 
>
>It might not be, depends on the details, and I dont understand bidi
>well
>enough to comment authoritatively.
>
>
>
>-- 
>You are receiving this because you authored the thread.
>Reply to this email directly or view it on GitHub:
>https://github.com/kovidgoyal/kitty/issues/2252#issuecomment-570917707

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Resolving patch:
diff --git a/kitty/config_data.py b/kitty/config_data.py
--- a/kitty/config_data.py
+++ b/kitty/config_data.py
@@ -223,6 +223,23 @@ def to_font_size(x):
 
 o('font_size', 11.0, long_text=_('Font size (in pts)'), option_type=to_font_size)
 
+o('force_ltr', False, long_text=_("""
+Kitty does not yet support BIDI (bidirectional text), however, for RTL scripts,
+HarfBuzz will "helpfully" shape text as if we were laying it out RTL. That is
+to say, in an RTL script, the words "HELLO WORLD" display in Kitty as "WORLD
+HELLO", and if you try to select a substring of an RTL-shaped string, you will
+get the character that _would_ be there were the string LTR. For example,
+assuming the Hebrew word ירושלים, selecting the character that on the screen
+appears to be ם actually writes into the selection buffer the character י.
+
+Kitty's default behavior is useful in conjunction with a filter to reverse the
+word order, however, if you wish to manipulate RTL glyphs, it can be very
+challenging to work with, so this option is provided to turn it off.
+Furthermore, this option can be used with the command line program
+:link:`GNU FriBidi <https://github.com/fribidi/fribidi#executable>` to get BIDI
+support, because it will force HarfBuzz to always treat the text as LTR, which
+FriBidi expects for terminals."""))
+
 
 def adjust_line_height(x):
     if x.endswith('%'):
@@ -315,6 +332,12 @@ def disable_ligatures(x):
 font) breaks up monotony::
 
     font_features TT2020StyleB-Regular -liga +calt
+
+In conjunction with :opt:`force_ltr`, you may want to disable Arabic shaping
+entirely, and only look at their isolated forms if they show up in a document.
+You can do this with e.g.::
+
+    font_features UnifontMedium +isol -medi -fina -init
 '''))
 
 


Test patch: