Repository: crate-ci/typos
Pull request #465 (2022-04-25 02:44:29)

Fix misalignment in report highlight and indent when there are double width characters
Previous method misaligns highlights when there are double width asian characters
```
39 |  한글 eglish
   |    ^^^^^^
```
This commit fixes the highlight to have correct alignment.
```
39 |  한글 eglish
   |      ^^^^^^
```
`unicode-rs` crate is used by the Rust compiler [1].
[1]: https://github.com/rust-lang/rust/blob/34a6c9f26e2ce32cad0d71f5e342365b09f4d12c/compiler/rustc_errors/src/emitter.rs#L861

I've intentionally left `column` variable as is since many IDE counts by grapheme clusters.

foriequal0: ```rust
fn main() {
    let 한=abc1;
    // The variable is NFC, 1 grpahemes, 1 codepoints.
    // Many Koreans are familiar with this.
    // Rust compiler, VSCode, JetBrains IDE, Notepad++ agree on that 'a' is at column number 11.
    // Most terminal emulators render it as fixed width (2 column width) characters.
    // Old editors used to render it as fixed width too.
    // Many old Korean programmers are accustomed to this.
    // So, Rust compiler puts highlight at column 12 expecting it to be rendered on terminals.
    // 
    // However, many modern editors, and browsers render the variable as variable width characters.
    // So they are slightly misaligned on some environment such as https://play.rust-lang.org/

    let 한=abc2;
    // The variable is NFD. 1 graphemes, 3 codepoints.
    // I expect it should look same and be treated as above.
    // It is an unfamiliar form even for Korean, but It is how macOS handles a filename.
    // It was due to the limitation of HFS+, but filesystem API keep denormalizes filename on APFS which doesn't have the limitation.
    // macOS renders filenames correctly, but Windows renders as decomposed form, which annoys many Koreans.
    //
    // Anyway, all editors fail to handle correctly in different way.
    // - Rust compiler:
    //   It puts highlight at column 12, but says 'a' is at column 13.
    //   Rust playground https://play.rust-lang.org/ that uses https://ace.c9.io/, and it
    //    - displays the variable as 'ㅎㅏㄴㄱㅡㄹ', which is incorrect,
    //    - and puts a cursor in an incorrect position. The cursor is put between decomposed letters,
    //      and the visible cursor position is different with the editing position.
    //    - It requires 3 strokes to delete a grapheme cluster as if it is compositing,
    //      while many Korean IME deletes a NFC character at once after the composition is completed. 
    // - VSCode:
    //   It says 'a' is at column 13.
    //   It correctly displays the variable and puts a cursor in a correct position.
    //   It requires 3 keystrokes to delete.
    // - Notepad++
    //   It says 'a' is at column 13.
    //   It is displayed as 'ㅎㅏㄴㄱㅡㄹ', but in a slightly different form.
    //   It puts a cursor in an incorrect position. The cursor is put between decomposed letters.
    //   It requires 3 keystrokes to delete.
    // - JetBrains IDE
    //   It says 'a' is at column 13.
    //   It correctly displays the variable.
    //   It fails to put a cursor in a correct position. The cursor is put in the middle of the character (at 1/3 of the character, etc.)
    //   It requires 3 strokes to delete.

    let 👁️‍🗨️=abc3;
    // The variable is "U+1F441 U+FE0F U+200D U+1F5E8 U+FE0F". 1 graphemes, 6 codepoints.
    // Also, all editors fail to handle correctly in different way.
    // - Rust compiler:
    //   It puts highlight at column 12, but says 'a' is at column 15.
    //   Rust playground treats the variable as 3 characters '👁️ 🗨️' in the editor, but it is displayed as 2 characters '👁️🗨️' in the compilation result panel.
    //   It requires 3 keystrokes to navigate across the variable with the cursor.
    //   It required 5 keystrokes to delete.
    // - VSCode:
    //   It says 'a' is at column 15.
    //   It correctly displays the variable and puts a cursor in a correct position.
    //   However, deleting the character requires 2 keystrokes.
    // - Notepad++
    //   It says 'a' is at column 15.
    //   It correctly displays the variable.
    //   It requires 5 keystrokes to navigate across the variable with the cursor, and the cursor sits in the middle of the character (1/5 of the character, etc.).
    //   It required 5 keystrokes to delete.
    // - JetBrains IDE
    //   It says 'a' is at column 15.
    //   It displays as 2 characters '👁️🗨️'.
    //   It requires 2/1(ZWJ)/2, keystrokes to navigate across the variable with the cursor, and the cursor sits in the middle of the character (1/2 of the character, etc.).
    //   It required 5 keystrokes to delete.

    let 😵‍💫=abc4;
    // The variable is "U+1F635 U+200D U+1F4AB". 1 graphemes, 3 codepoints.
    // Also, all editors fail to handle correctly in different way.
    // - Rust compiler:
    //   It puts highlight at column 14, but says 'a' is at column 13.
    //   Rust playground treats the variable as 3 characters '😵 💫' in the editor, but it is displayed as 2 characters '😵💫' in the compilation result panel.
    //   It requires 3 keystrokes to navigate across the variable with the cursor.
    //   It required 3 keystrokes to delete.
    // - VSCode:
    //   It says 'a' is at column 13.
    //   It incorrectly displays the variable as '😵💫'.
    //   It requires 1 keystroke to navigate across the variable with the cursor.
    //   It required 2 keystrokes to delete.
    // - Notepad++
    //   It says 'a' is at column 13.
    //   It correctly displays the variable.
    //   It requires 1/1(ZWJ)/1 keystrokes to navigate across the variable with the cursor.
    //   It required 3 keystrokes to delete.
    // - JetBrains IDE
    //   It says 'a' is at column 13.
    //   It displays as 2 characters '😵💫'.
    //   It requires 1/1(ZWJ)/1 keystrokes to navigate across the variable with the cursor.
    //   It required 3 keystrokes to delete.

	abc5
    // The above line starts with a horizontal tab ('\t', 0x09), and I'm using 4 spaces tab.
    // - Rust compiler:
    //   It puts highlight at column 5, but says 'a' is at column 2.
    // - VSCode:
    //   It says 'a' is at column 5.
    // - Notepad++
    //   It says 'a' is at column 5.
    // - JetBrains IDE
    //   It says 'a' is at column 5.
}
```

Rust playground:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=6494413102a03a812fd193476c16d2aa
![image](https://user-images.githubusercontent.com/5465658/165226606-1d4bac97-e913-4f6c-a58b-256dfa03919e.png)

Windows Terminal
![image](https://user-images.githubusercontent.com/5465658/165226845-4c8c1efe-1e1d-47aa-9fe6-7721ea0b5139.png)

VSCode
![image](https://user-images.githubusercontent.com/5465658/165226975-c0727c61-1146-4d75-be36-f454d8032e22.png)

Notepad++
![image](https://user-images.githubusercontent.com/5465658/165226899-68d1142f-2670-4140-b62f-7be3d63186fe.png)

JetBrains Rider
![image](https://user-images.githubusercontent.com/5465658/165226938-f8d75a3a-2de8-41db-a2bc-b218a6fcf52a.png)

Rust compiler output in Rust playground (Firefox, 100)
![image](https://user-images.githubusercontent.com/5465658/165226645-52569946-f420-4f72-b678-7bd6b9078cae.png)

Rust compiler output in Windows Terminal, WSL
![image](https://user-images.githubusercontent.com/5465658/165227014-112f3882-822c-4e8d-8a06-811ee99b9444.png)


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8cc87f9..f4a866e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,13 +7,6 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 <!-- next-header -->
 ## [Unreleased] - ReleaseDate
 
-## [1.7.1] - 2022-04-25
-
-### Fixes
-- Ignore uppercase UUID because Microsoft
-- Correct `unencyrpted`
-- Correct `signign`
-
 ## [1.7.0] - 2022-04-18
 
 ### Fixes
@@ -340,8 +333,7 @@ Note: MSRV is now 1.54
 *   Only do hex check if digits are in identifiers ([68cd36d0](https://github.com/crate-ci/typos/commit/68cd36d0de90226dbc9d31c2ce6d8bf6b69adb5c))
 
 <!-- next-url -->
-[Unreleased]: https://github.com/crate-ci/typos/compare/v1.7.1...HEAD
-[1.7.1]: https://github.com/crate-ci/typos/compare/v1.7.0...v1.7.1
+[Unreleased]: https://github.com/crate-ci/typos/compare/v1.7.0...HEAD
 [1.7.0]: https://github.com/crate-ci/typos/compare/v1.6.0...v1.7.0
 [1.6.0]: https://github.com/crate-ci/typos/compare/v1.5.0...v1.6.0
 [1.5.0]: https://github.com/crate-ci/typos/compare/v1.4.1...v1.5.0
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 46dc6de..4456277 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -31,7 +31,7 @@ to re-work some of it and the discouragement that goes along with that.
 
 `typos` dictionary is a mapping of typos to a list of possible corrections (see [Design](docs/design.md)).  To add to the dictionary:
 
-1. Add your typo to our data file `crates/typos-dict/assets/words.csv`
+1. Add your type to our data file `crates/typos-dict/assets/words.csv`
 
 Format: `typo,correction[,correction...]`
 
diff --git a/Cargo.lock b/Cargo.lock
index b08c507..3fdc389 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1523,7 +1523,7 @@ checksum = "0685c84d5d54d1c26f7d3eb96cd41550adb97baed141a761cf335d3d33bcd0ae"
 
 [[package]]
 name = "typos"
-version = "0.9.1"
+version = "0.9.0"
 dependencies = [
  "anyhow",
  "bstr",
@@ -1540,7 +1540,7 @@ dependencies = [
 
 [[package]]
 name = "typos-cli"
-version = "1.7.1"
+version = "1.7.0"
 dependencies = [
  "ahash",
  "anyhow",
@@ -1576,13 +1576,14 @@ dependencies = [
  "typos-vars",
  "unicase",
  "unicode-segmentation",
+ "unicode-width",
  "varcon-core",
  "yansi",
 ]
 
 [[package]]
 name = "typos-dict"
-version = "0.8.1"
+version = "0.8.0"
 dependencies = [
  "dictgen",
  "log",
diff --git a/Cargo.toml b/Cargo.toml
index 84ca8fb..7b6c86c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -12,7 +12,7 @@ members = [
 
 [package]
 name = "typos-cli"
-version = "1.7.1"
+version = "1.7.0"
 description = "Source Code Spelling Correction"
 authors = ["Ed Page <eopage@gmail.com>"]
 license = "MIT OR Apache-2.0"
@@ -91,6 +91,7 @@ encoding = "0.2"
 kstring = { version = "2.0.0", features = ["serde"] }
 typed-arena = "2.0.1"
 maplit = "1.0"
+unicode-width = "0.1.9"
 
 [dev-dependencies]
 assert_fs = "1.0"
diff --git a/crates/typos-dict/Cargo.toml b/crates/typos-dict/Cargo.toml
index 0866ddb..0b0fc3d 100644
--- a/crates/typos-dict/Cargo.toml
+++ b/crates/typos-dict/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "typos-dict"
-version = "0.8.1"
+version = "0.8.0"
 description = "Source Code Spelling Correction"
 authors = ["Ed Page <eopage@gmail.com>"]
 license = "MIT OR Apache-2.0"
diff --git a/crates/typos-dict/assets/words.csv b/crates/typos-dict/assets/words.csv
index 8cb9aa6..ad68a85 100644
--- a/crates/typos-dict/assets/words.csv
+++ b/crates/typos-dict/assets/words.csv
@@ -43653,7 +43653,6 @@ signifigant,significant
 signifigantly,significantly
 signifikant,significant
 signifiy,signify
-signign,signing
 signigns,signings
 signins,signings
 signitories,signatories
@@ -49983,7 +49982,6 @@ unempolyed,unemployed
 unempolyment,unemployment
 unencrpt,unencrypt
 unencrpted,unencrypted
-unencyrpted,unencrypted
 unenforcable,unenforceable
 unenployment,unemployment
 unepected,unexpected
diff --git a/crates/typos-dict/src/dict_codegen.rs b/crates/typos-dict/src/dict_codegen.rs
index 6e938d4..f3fc28b 100644
--- a/crates/typos-dict/src/dict_codegen.rs
+++ b/crates/typos-dict/src/dict_codegen.rs
@@ -8969,14 +8969,12 @@ pub static WORD_UNEN_CHILDREN: dictgen::DictTable<&'static [&'static str]> = dic
     keys: &[
         dictgen::InsensitiveStr::Ascii("crpt"),
         dictgen::InsensitiveStr::Ascii("crpted"),
-        dictgen::InsensitiveStr::Ascii("cyrpted"),
         dictgen::InsensitiveStr::Ascii("forcable"),
         dictgen::InsensitiveStr::Ascii("ployment"),
     ],
     values: &[
         &["unencrypt"],
         &["unencrypted"],
-        &["unencrypted"],
         &["unenforceable"],
         &["unemployment"],
     ],
@@ -30109,7 +30107,6 @@ pub static WORD_SIG_CHILDREN: dictgen::DictTable<&'static [&'static str]> = dict
         dictgen::InsensitiveStr::Ascii("nifigantly"),
         dictgen::InsensitiveStr::Ascii("nifikant"),
         dictgen::InsensitiveStr::Ascii("nifiy"),
-        dictgen::InsensitiveStr::Ascii("nign"),
         dictgen::InsensitiveStr::Ascii("nigns"),
         dictgen::InsensitiveStr::Ascii("nins"),
         dictgen::InsensitiveStr::Ascii("nitories"),
@@ -30172,7 +30169,6 @@ pub static WORD_SIG_CHILDREN: dictgen::DictTable<&'static [&'static str]> = dict
         &["significantly"],
         &["significant"],
         &["signify"],
-        &["signing"],
         &["signings"],
         &["signings"],
         &["signatories"],
diff --git a/crates/typos/Cargo.toml b/crates/typos/Cargo.toml
index 2751ef8..6aa8e52 100644
--- a/crates/typos/Cargo.toml
+++ b/crates/typos/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "typos"
-version = "0.9.1"
+version = "0.9.0"
 description = "Source Code Spelling Correction"
 authors = ["Ed Page <eopage@gmail.com>"]
 license = "MIT OR Apache-2.0"
diff --git a/crates/typos/src/tokens.rs b/crates/typos/src/tokens.rs
index 734f3a5..83241ae 100644
--- a/crates/typos/src/tokens.rs
+++ b/crates/typos/src/tokens.rs
@@ -319,29 +319,16 @@ mod parser {
         <T as nom::InputTakeAtPosition>::Item: AsChar + Copy,
         <T as nom::InputIter>::Item: AsChar + Copy,
     {
-        recognize(alt((
-            tuple((
-                take_while_m_n(8, 8, is_lower_hex_digit),
-                char('-'),
-                take_while_m_n(4, 4, is_lower_hex_digit),
-                char('-'),
-                take_while_m_n(4, 4, is_lower_hex_digit),
-                char('-'),
-                take_while_m_n(4, 4, is_lower_hex_digit),
-                char('-'),
-                take_while_m_n(12, 12, is_lower_hex_digit),
-            )),
-            tuple((
-                take_while_m_n(8, 8, is_upper_hex_digit),
-                char('-'),
-                take_while_m_n(4, 4, is_upper_hex_digit),
-                char('-'),
-                take_while_m_n(4, 4, is_upper_hex_digit),
-                char('-'),
-                take_while_m_n(4, 4, is_upper_hex_digit),
-                char('-'),
-                take_while_m_n(12, 12, is_upper_hex_digit),
-            )),
+        recognize(tuple((
+            take_while_m_n(8, 8, is_lower_hex_digit),
+            char('-'),
+            take_while_m_n(4, 4, is_lower_hex_digit),
+            char('-'),
+            take_while_m_n(4, 4, is_lower_hex_digit),
+            char('-'),
+            take_while_m_n(4, 4, is_lower_hex_digit),
+            char('-'),
+            take_while_m_n(12, 12, is_lower_hex_digit),
         )))(input)
     }
 
@@ -1100,21 +1087,6 @@ mod test {
         assert_eq!(expected, actual);
     }
 
-    #[test]
-    fn tokenize_ignore_uuid_uppercase() {
-        let parser = TokenizerBuilder::new().build();
-
-        let input = "Hello 123E4567-E89B-12D3-A456-426652340000 World";
-        let expected: Vec<Identifier> = vec![
-            Identifier::new_unchecked("Hello", Case::None, 0),
-            Identifier::new_unchecked("World", Case::None, 43),
-        ];
-        let actual: Vec<_> = parser.parse_bytes(input.as_bytes()).collect();
-        assert_eq!(expected, actual);
-        let actual: Vec<_> = parser.parse_str(input).collect();
-        assert_eq!(expected, actual);
-    }
-
     #[test]
     fn tokenize_ignore_hash() {
         let parser = TokenizerBuilder::new().build();
diff --git a/docker/Dockerfile b/docker/Dockerfile
index b78b01e..1b833c1 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -1,5 +1,5 @@
 FROM ubuntu:20.04
-ARG VERSION=1.7.1
+ARG VERSION=1.7.0
 ENV VERSION=${VERSION}
 RUN apt-get update && apt-get install -y wget
 RUN wget https://github.com/crate-ci/typos/releases/download/v${VERSION}/typos-v${VERSION}-x86_64-unknown-linux-musl.tar.gz && \
diff --git a/docs/pre-commit.md b/docs/pre-commit.md
index 6b10c02..d58f849 100644
--- a/docs/pre-commit.md
+++ b/docs/pre-commit.md
@@ -6,7 +6,7 @@ config at this repository:
 ```yaml
 repos:
   - repo: https://github.com/crate-ci/typos
-    rev: v1.7.1
+    rev: v1.7.0
     hooks:
       - id: typos
 ```
diff --git a/setup.py b/setup.py
index 7bf64a3..a4fc0fa 100644
--- a/setup.py
+++ b/setup.py
@@ -15,7 +15,7 @@ from typing import Tuple
 from setuptools import setup
 from setuptools.command.install import install as orig_install
 
-TYPOS_VERSION = '1.7.1'
+TYPOS_VERSION = '1.7.0'
 POSTFIX_SHA256 = {
     'linux': (
         'x86_64-unknown-linux-musl.tar.gz',
diff --git a/src/bin/typos-cli/report.rs b/src/bin/typos-cli/report.rs
index 1eb6318..87e7e75 100644
--- a/src/bin/typos-cli/report.rs
+++ b/src/bin/typos-cli/report.rs
@@ -3,6 +3,8 @@
 use std::io::{self, Write};
 use std::sync::atomic;
 
+use unicode_width::UnicodeWidthStr;
+
 use typos_cli::report::{Context, Message, Report, Typo};
 
 #[derive(Copy, Clone, Debug)]
@@ -218,8 +220,11 @@ fn print_long_correction(msg: &Typo, palette: Palette) -> Result<(), std::io::Er
         let line_num = context.line_num.to_string();
         let line_indent: String = itertools::repeat_n(" ", line_num.len()).collect();
 
-        let hl_indent: String = itertools::repeat_n(" ", column).collect();
-        let hl: String = itertools::repeat_n("^", msg.typo.len()).collect();
+        let visible_column = UnicodeWidthStr::width(start.as_ref());
+        let visible_len = UnicodeWidthStr::width(msg.typo);
+
+        let hl_indent: String = itertools::repeat_n(" ", visible_column).collect();
+        let hl: String = itertools::repeat_n("^", visible_len).collect();
 
         writeln!(handle, "{} |", line_indent)?;
         writeln!(


Test patch:
diff --git a/tests/cmd/stdin-failure-multiwidth.stdin b/tests/cmd/stdin-failure-multiwidth.stdin
new file mode 100644
index 0000000..1af6419
--- /dev/null
+++ b/tests/cmd/stdin-failure-multiwidth.stdin
@@ -0,0 +1,34 @@
+---
+Korean character (NFC)
+Grapheme clusters: 1, codepoints: 1, UnicodeWidthStr::width() == 2
+
+한 Apropriate world
+   ^^^^^^^^^^ highlight here
+
+---
+Korean character (NFD).
+Grapheme clusters: 1, codepoints: 3, UnicodeWidthStr::width() == 2
+
+한 Apropriate world
+   ^^^^^^^^^^ highlight here
+
+---
+Eye in Speech Bubble Emoji (U+1F441 U+FE0F U+200D U+1F5E8 U+FE0F, Recommended Emoji ZWJ Sequences, v2.0)
+Grapheme clusters: 1, codepoints: 5, UnicodeWidthStr::width() == 2 (Read NOTE: https://github.com/unicode-rs/unicode-width)
+
+👁️‍🗨️ Apropriate world
+   ^^^^^^^^^^ highlight here
+
+---
+Face with spiral eyes (U+1F635 U+200D U+1F4AB, Recommended Emoji ZWJ Sequence, v13.1)
+Grapheme clusters: 1, codepoints: 3, UnicodeWidthStr::width() == 4 (Read NOTE: https://github.com/unicode-rs/unicode-width)
+
+😵‍💫 Apropriate world
+     ^^^^^^^^^^ highlight here
+
+---
+Horizontal tab (\t, U+09)
+Grapheme clusters: 1, codepoints: 1, UnicodeWidthStr::width() == 0
+
+	Apropriate world
+^^^^^^^^^^ highlight here
\ No newline at end of file
diff --git a/tests/cmd/stdin-failure-multiwidth.stdout b/tests/cmd/stdin-failure-multiwidth.stdout
new file mode 100644
index 0000000..e6cf44a
--- /dev/null
+++ b/tests/cmd/stdin-failure-multiwidth.stdout
@@ -0,0 +1,30 @@
+error: `Apropriate` should be `Appropriate`
+  --> -:5:2
+  |
+5 | 한 Apropriate world
+  |    ^^^^^^^^^^
+  |
+error: `Apropriate` should be `Appropriate`
+  --> -:12:2
+   |
+12 | 한 Apropriate world
+   |    ^^^^^^^^^^
+   |
+error: `Apropriate` should be `Appropriate`
+  --> -:19:2
+   |
+19 | 👁️‍🗨️ Apropriate world
+   |    ^^^^^^^^^^
+   |
+error: `Apropriate` should be `Appropriate`
+  --> -:26:2
+   |
+26 | 😵‍💫 Apropriate world
+   |      ^^^^^^^^^^
+   |
+error: `Apropriate` should be `Appropriate`
+  --> -:33:1
+   |
+33 |  Apropriate world
+   | ^^^^^^^^^^
+   |
diff --git a/tests/cmd/stdin-failure-multiwidth.toml b/tests/cmd/stdin-failure-multiwidth.toml
new file mode 100644
index 0000000..18f517f
--- /dev/null
+++ b/tests/cmd/stdin-failure-multiwidth.toml
@@ -0,0 +1,3 @@
+bin.name = "typos"
+args = "-"
+status.code = 2
\ No newline at end of file