Repository: ocaml-community/sedlex
Pull request #127: Add utf8 support for string literal
Author: login: hhugo
Created: 2023-02-25T21:12:52Z | Merged: 2025-05-28T15:09:15Z

The PR adds support for utf-8 encoded string. It fixes #90.

This only work if we assume the source is encoded in utf8.

Note that OCaml 5.3 has the following entry in its changelog.

> - #11736, #12664: Support utf-8 encoded source files and latin-9 compatible
  identifiers.



Linked issue #90: Matching a unicode character without codepoint
Currently,
`
match%sedlex lexbuf with
 | "ρ" -> ..
`
does not match, although
`
match%sedlex lexbuf with
 | math -> if Sedlexing.Utf8.lexeme lexbuf = "ρ" then ..
`
does.

Is there any way of making the first variant work, without having to replace "ρ" with its underlying codepoint, so that I can use it as part of a more complex regexp?
Linked issue comments: user: login: ssfrr
created_at: 2022-01-12T21:23:01Z
body: just ran into this also. I tried two different ways:

```ocaml
open Printf

let next_tok buf =
  let open Sedlexing.Utf8 in
  let fn = [%sedlex.regexp? Chars "-+×÷"] in
  match%sedlex buf with
  | Chars "+-×÷" -> sprintf "with Chars: %s" (lexeme buf)
  | "+"|"-"|"×"|"÷" -> sprintf "with Bars: %s" (lexeme buf)
  | _ -> failwith (sprintf "Unexpected character: %s" (lexeme buf))

let test_tok =
  Sedlexing.Utf8.from_string "+" |> next_tok |> print_string; print_newline ();
  Sedlexing.Utf8.from_string "÷" |> next_tok |> print_string; print_newline ();
```

This prints `with Chars: +` for the first line and then errors out with `exception Failure("Unexpected character: ")` for the second. 

It's not clear to me whether it's a problem with the match cases or the character iteration (or both) It's not printing the unexpected character which seems to indicate that the lexeme being processed doesn't include the whole unicode codepoint.

Is there a better way to do this?
author_association: NONE
user: login: ssfrr
created_at: 2022-01-12T21:31:10Z
body: looks like I can put the raw codepoints like
```
  | 0x00D7 | 0x00F7 -> sprintf "with Codepoints: %s" (lexeme buf)
```

This is a bit of a hassle to generate the codepoints for all the characters I need, but I think is a reasonable workaround.

~It's also still puzzling to me why the "unexpected character" case isn't printing the correct character.~
edit: it looks like `lexeme buf` is the empty string here, which makes sense for cases where nothing matched.
author_association: NONE
user: login: pmetzger
created_at: 2022-01-12T21:52:24Z
body: OCaml does not currently provide a guarantee that Unicode can be embedded without trouble in source files. It would be nice if everyone agreed that source files were encoded as UTF-8, but that is not yet the case.
author_association: MEMBER
user: login: ssfrr
created_at: 2022-01-12T22:03:59Z
body: ah, I see. Yes that's unfortunate.

Could sedlex just make that assumption and document that any strings that show up in the `match%sedlex` clause are assumed to be in UTF8? Alternatively could it use the system's locale to decide?

How does `ocamlc` interpret source files? If you have a file encoded in UTF16 would it work?
author_association: NONE
user: login: pmetzger
created_at: 2022-01-13T01:15:09Z
body: > Could sedlex just make that assumption

No, it can't, unfortunately. Among other things, as things stand, using js_of_ocaml puts you into a situation where strings are interpreted as UTF-16. The situation is messy.

> Alternatively could it use the system's locale to decide?

Not really.

> How does ocamlc interpret source files?

It used to assume that things were in ISO Latin-1. Then that got very partially obsoleted but without any definitive move to setting an actual reasonable permanent standard. Most programming languages have now adopted UTF-8 as an encoding for source files, but at this point OCaml hasn't.

A lot of people will claim that if you just use UTF-8 in strings this will work most of the time and should be fine. In fact, it can break in subtle ways. It's important that OCaml adopt an actual policy on what the encoding is, but it hasn't.

For now, just use the codepoint for Sedlex and you'll be much happier.
author_association: MEMBER
user: login: ssfrr
created_at: 2022-01-13T03:43:28Z
body: I see. Thanks for taking the time to explain. I'll use the codepoints.
author_association: NONE
user: login: hhugo
created_at: 2022-01-13T07:15:31Z
body: > No, it can't, unfortunately. Among other things, as things stand, using js_of_ocaml puts you into a situation where strings are interpreted as UTF-16. The situation is messy.

I don't think this is correct. Javascript encodes its strings at runtime as utf-16, javascript source files are usually utf-8.
Js_of_ocaml treats OCaml strings as sequence of bytes and even assuming they are utf-8 encoded when converting them to javascript utf-16 ones.  
author_association: COLLABORATOR
user: login: hhugo
created_at: 2022-01-13T07:32:51Z
body: What about providing a new constructor Utf8 and treat ocaml strings  inside it as utf8 encoded
```ocaml
let next_tok buf =
  let open Sedlexing.Utf8 in
  let fn = [%sedlex.regexp? Chars "-+×÷"] in
  match%sedlex buf with
  | Utf8 (Chars "+-×÷") -> sprintf "with Chars: %s" (lexeme buf)
  | Utf8 ("+"|"-"|"×"|"÷") -> sprintf "with Bars: %s" (lexeme buf)
  | _ -> failwith (sprintf "Unexpected character: %s" (lexeme buf))
```
author_association: COLLABORATOR
user: login: hhugo
created_at: 2022-01-13T08:52:14Z
body: I've implemented a PoC in https://github.com/hhugo/sedlex/tree/utf8
author_association: COLLABORATOR
user: login: pmetzger
created_at: 2022-01-13T13:11:09Z
body: You cannot alter the OCaml lexer through the use of a constructor written at the level of the language. The fact that something "usually" works isn't a guarantee that it will work consistently. 
author_association: MEMBER
user: login: alainfrisch
created_at: 2022-01-13T13:17:09Z
body: I don't think there is a need to change the OCaml lexer, all special characters not interpreted verbatim in strings literal are ASCII, and any occurrence of those bytes in the string literal encoded in utf-8 really correspond to those ASCII characters.
author_association: COLLABORATOR
user: login: pmetzger
created_at: 2022-01-13T15:55:06Z
body: > I don't think there is a need to change the OCaml lexer, all special characters not interpreted verbatim in strings literal are ASCII,

The lexer itself doesn't need to be changed much to adopt a policy of utf-8 encoding throughout, but it does currently happily take Latin-1, including in identifiers and strings, and if you want to hand Unicode code points to code dealing with strings you want tools that can safely presume valid utf-8 is going to be presented to them, meaning one needs to validate that (for example) input strings are valid utf-8.
author_association: MEMBER
user: login: pmetzger
created_at: 2022-01-13T15:56:14Z
body: (Note that I proposed patches on this a couple of years ago and they got a bunch of pushback. If there's a desire to do this, I'm happy to support it and to get my patches to apply to the current compiler.)
author_association: MEMBER
user: login: hhugo
created_at: 2024-10-29T14:16:40Z
body: @pmetzger we can probably make some progress now that https://github.com/ocaml/ocaml/pull/12664 is merged 
author_association: COLLABORATOR

Diff:
diff --git a/CHANGES.md b/CHANGES.md
index a1ef56f..1f5ac42 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,5 +1,6 @@
 # dev
 - Implement Corrigendum #1: UTF-8 Shortest Form
+- Add utf8 support for string literal (#127)
 
 # 3.4 (2025-03-28)
 - Make the library compatibility with ppxlib.0.36 (#166)
diff --git a/README.md b/README.md
index 4a2b249..595a95a 100644
--- a/README.md
+++ b/README.md
@@ -84,7 +84,7 @@ The actions can call functions from the Sedlexing module to extract
 
 Regular expressions are syntactically OCaml patterns:
 
-- `"...."` (string constant): recognize the specified string
+- `"...."` (string constant): recognize the specified string.
 - `'....'` (character constant) : recognize the specified character
 - `i` (integer constant) : recognize the specified codepoint
 - `'...' .. '...'`: character range
@@ -103,6 +103,9 @@ Regular expressions are syntactically OCaml patterns:
   and recognize the set of items in `R1` but not in `R2` ("subtract")
 - `Intersect (R1,R2)` : assume that `R` is a single-character length regexp (see
   below) and recognize the set of items which are in both `R1` and `R2`
+- `Utf8 R` : string literals inside R are assumed to be utf-8 encoded.
+- `Latin1 R` : string literals inside R are assumed to be latin1 encoded.
+- `Ascii R` : string literals inside R are assumed to be ascii encoded.
 - `lid` (lowercase identifier) : reference a named regexp (see below)
 
 A single-character length regexp is a regexp which does not contain (after
@@ -112,8 +115,9 @@ with a length different from one.
 
 
 Note:
- - The OCaml source is assumed to be encoded in Latin1 (for string
-   and character literals).
+ - The OCaml source is assumed to be encoded in UTF-8.
+ - Strings and chars litterals will be interpreted in ASCII unless otherwise
+   specified by the `Latin1`,`Ascii` and `Utf8` constructors in patterns.
 
 
 It is possible to define named regular expressions with the following
diff --git a/src/lib/sedlexing.mli b/src/lib/sedlexing.mli
index 2332f7c..ad08c2f 100644
--- a/src/lib/sedlexing.mli
+++ b/src/lib/sedlexing.mli
@@ -274,6 +274,13 @@ module Utf8 : sig
 
   (** As [Sedlexing.sub_lexeme] with a result encoded in UTF-8. *)
   val sub_lexeme : lexbuf -> int -> int -> string
+
+  module Helper : sig
+    val width : char -> int
+    val check_two : int -> int -> int
+    val check_three : int -> int -> int -> int
+    val check_four : int -> int -> int -> int -> int
+  end
 end
 
 module Utf16 : sig
diff --git a/src/syntax/ppx_sedlex.ml b/src/syntax/ppx_sedlex.ml
index f37a3bd..1503643 100644
--- a/src/syntax/ppx_sedlex.ml
+++ b/src/syntax/ppx_sedlex.ml
@@ -13,7 +13,6 @@ module Cset = Sedlex_cset
 (* Decision tree for partitions *)
 
 let default_loc = Location.none
-let lident_loc ~loc s = { loc; txt = lident s }
 
 type decision_tree =
   | Lte of int * decision_tree * decision_tree
@@ -286,17 +285,46 @@ let codepoint i =
     failwith (Printf.sprintf "Invalid Unicode code point: %i" i);
   i
 
-let regexp_for_char c = Sedlex.chars (Cset.singleton (Char.code c))
-
-let regexp_for_string s =
-  let rec aux n =
-    if n = String.length s then Sedlex.eps
-    else Sedlex.seq (regexp_for_char s.[n]) (aux (succ n))
-  in
-  aux 0
-
-let err loc s =
-  raise (Location.Error (Location.Error.createf ~loc "Sedlex: %s" s))
+let char c = Cset.singleton (Char.code c)
+let uchar c = Cset.singleton (Uchar.to_int c)
+
+let err loc fmt =
+  Printf.ksprintf
+    (fun s ->
+      raise (Location.Error (Location.Error.createf ~loc "Sedlex: %s" s)))
+    fmt
+
+type encoding = Utf8 | Latin1 | Ascii
+
+let string_of_encoding = function
+  | Utf8 -> "UTF-8"
+  | Latin1 -> "Latin-1"
+  | Ascii -> "ASCII"
+
+let rev_csets_of_string ~loc ~encoding s =
+  match encoding with
+    | Utf8 ->
+        Utf8.fold
+          ~f:(fun acc _ x ->
+            match x with
+              | `Malformed _ ->
+                  err loc "Malformed %s string" (string_of_encoding encoding)
+              | `Uchar c -> uchar c :: acc)
+          [] s
+    | Latin1 ->
+        let l = ref [] in
+        for i = 0 to String.length s - 1 do
+          l := char s.[i] :: !l
+        done;
+        !l
+    | Ascii ->
+        let l = ref [] in
+        for i = 0 to String.length s - 1 do
+          match s.[i] with
+            | '\x00' .. '\x7F' as c -> l := char c :: !l
+            | _ -> err loc "Malformed %s string" (string_of_encoding encoding)
+        done;
+        !l
 
 let rec repeat r = function
   | 0, 0 -> Sedlex.eps
@@ -304,30 +332,39 @@ let rec repeat r = function
   | n, m -> Sedlex.seq r (repeat r (n - 1, m - 1))
 
 let regexp_of_pattern env =
-  let rec char_pair_op func name p tuple =
+  let rec char_pair_op func name ~encoding p tuple =
     (* Construct something like Sub(a,b) *)
     match tuple with
       | Some { ppat_desc = Ppat_tuple [p0; p1] } -> begin
-          match func (aux p0) (aux p1) with
+          match func (aux ~encoding p0) (aux ~encoding p1) with
             | Some r -> r
             | None ->
-                err p.ppat_loc @@ "the " ^ name
-                ^ " operator can only applied to single-character length \
-                   regexps"
+                err p.ppat_loc
+                  "the %s operator can only applied to single-character \
+                   length regexps"
+                  name
         end
       | _ ->
-          err p.ppat_loc @@ "the " ^ name
-          ^ " operator requires two arguments, like " ^ name ^ "(a,b)"
-  and aux p =
+          err p.ppat_loc "the %s operator requires two arguments, like %s(a,b)"
+            name name
+  and aux ~encoding p =
     (* interpret one pattern node *)
     match p.ppat_desc with
-      | Ppat_or (p1, p2) -> Sedlex.alt (aux p1) (aux p2)
+      | Ppat_or (p1, p2) -> Sedlex.alt (aux ~encoding p1) (aux ~encoding p2)
       | Ppat_tuple (p :: pl) ->
-          List.fold_left (fun r p -> Sedlex.seq r (aux p)) (aux p) pl
+          List.fold_left
+            (fun r p -> Sedlex.seq r (aux ~encoding p))
+            (aux ~encoding p) pl
       | Ppat_construct ({ txt = Lident "Star" }, Some (_, p)) ->
-          Sedlex.rep (aux p)
+          Sedlex.rep (aux ~encoding p)
       | Ppat_construct ({ txt = Lident "Plus" }, Some (_, p)) ->
-          Sedlex.plus (aux p)
+          Sedlex.plus (aux ~encoding p)
+      | Ppat_construct ({ txt = Lident "Utf8" }, Some (_, p)) ->
+          aux ~encoding:Utf8 p
+      | Ppat_construct ({ txt = Lident "Latin1" }, Some (_, p)) ->
+          aux ~encoding:Latin1 p
+      | Ppat_construct ({ txt = Lident "Ascii" }, Some (_, p)) ->
+          aux ~encoding:Ascii p
       | Ppat_construct
           ( { txt = Lident "Rep" },
             Some
@@ -347,7 +384,7 @@ let regexp_of_pattern env =
             | Pconst_integer (i1, _), Pconst_integer (i2, _) ->
                 let i1 = int_of_string i1 in
                 let i2 = int_of_string i2 in
-                if 0 <= i1 && i1 <= i2 then repeat (aux p0) (i1, i2)
+                if 0 <= i1 && i1 <= i2 then repeat (aux ~encoding p0) (i1, i2)
                 else err p.ppat_loc "Invalid range for Rep operator"
             | _ ->
                 err p.ppat_loc "Rep must take an integer constant or interval"
@@ -355,11 +392,11 @@ let regexp_of_pattern env =
       | Ppat_construct ({ txt = Lident "Rep" }, _) ->
           err p.ppat_loc "the Rep operator takes 2 arguments"
       | Ppat_construct ({ txt = Lident "Opt" }, Some (_, p)) ->
-          Sedlex.alt Sedlex.eps (aux p)
+          Sedlex.alt Sedlex.eps (aux ~encoding p)
       | Ppat_construct ({ txt = Lident "Compl" }, arg) -> begin
           match arg with
             | Some (_, p0) -> begin
-                match Sedlex.compl (aux p0) with
+                match Sedlex.compl (aux ~encoding p0) with
                   | Some r -> r
                   | None ->
                       err p.ppat_loc
@@ -369,10 +406,10 @@ let regexp_of_pattern env =
             | _ -> err p.ppat_loc "the Compl operator requires an argument"
         end
       | Ppat_construct ({ txt = Lident "Sub" }, arg) ->
-          char_pair_op Sedlex.subtract "Sub" p
+          char_pair_op ~encoding Sedlex.subtract "Sub" p
             (Option.map (fun (_, arg) -> arg) arg)
       | Ppat_construct ({ txt = Lident "Intersect" }, arg) ->
-          char_pair_op Sedlex.intersection "Intersect" p
+          char_pair_op ~encoding Sedlex.intersection "Intersect" p
             (Option.map (fun (_, arg) -> arg) arg)
       | Ppat_construct ({ txt = Lident "Chars" }, arg) -> (
           let const =
@@ -382,16 +419,26 @@ let regexp_of_pattern env =
           in
           match const with
             | Some (Pconst_string (s, _, _)) ->
-                let c = ref Cset.empty in
-                for i = 0 to String.length s - 1 do
-                  c := Cset.union !c (Cset.singleton (Char.code s.[i]))
-                done;
-                Sedlex.chars !c
+                let l = rev_csets_of_string ~loc:p.ppat_loc ~encoding s in
+                let chars = List.fold_left Cset.union Cset.empty l in
+                Sedlex.chars chars
             | _ ->
                 err p.ppat_loc "the Chars operator requires a string argument")
       | Ppat_interval (i_start, i_end) -> begin
           match (i_start, i_end) with
             | Pconst_char c1, Pconst_char c2 ->
+                let valid =
+                  match encoding with
+                    (* utf8 char interval can only match ascii because
+                       of the OCaml lexer. *)
+                    | Ascii | Utf8 -> (
+                        function '\x00' .. '\x7f' -> true | _ -> false)
+                    | Latin1 -> ( function _ -> true)
+                in
+                if not (valid c1 && valid c2) then
+                  err p.ppat_loc
+                    "this pattern is not a valid %s interval regexp"
+                    (string_of_encoding encoding);
                 Sedlex.chars (Cset.interval (Char.code c1) (Char.code c2))
             | Pconst_integer (i1, _), Pconst_integer (i2, _) ->
                 Sedlex.chars
@@ -402,20 +449,23 @@ let regexp_of_pattern env =
         end
       | Ppat_constant const -> begin
           match const with
-            | Pconst_string (s, _, _) -> regexp_for_string s
-            | Pconst_char c -> regexp_for_char c
+            | Pconst_string (s, _, _) ->
+                let rev_l = rev_csets_of_string s ~loc:p.ppat_loc ~encoding in
+                List.fold_left
+                  (fun acc cset -> Sedlex.seq (Sedlex.chars cset) acc)
+                  Sedlex.eps rev_l
+            | Pconst_char c -> Sedlex.chars (char c)
             | Pconst_integer (i, _) ->
                 Sedlex.chars (Cset.singleton (codepoint (int_of_string i)))
             | _ -> err p.ppat_loc "this pattern is not a valid regexp"
         end
       | Ppat_var { txt = x } -> begin
           try StringMap.find x env
-          with Not_found ->
-            err p.ppat_loc (Printf.sprintf "unbound regexp %s" x)
+          with Not_found -> err p.ppat_loc "unbound regexp %s" x
         end
       | _ -> err p.ppat_loc "this pattern is not a valid regexp"
   in
-  aux
+  aux ~encoding:Ascii
 
 let previous = ref []
 let regexps = ref []
@@ -468,7 +518,7 @@ let mapper =
             (this#define_regexp name p)#expression body
         | [%expr [%sedlex [%e? _]]] ->
             err e.pexp_loc
-              "the %sedlex extension is only recognized on match expressions"
+              "the %%sedlex extension is only recognized on match expressions"
         | _ -> super#expression e
 
     val toplevel = true
diff --git a/src/syntax/utf8.ml b/src/syntax/utf8.ml
new file mode 100644
index 0000000..1a0be54
--- /dev/null
+++ b/src/syntax/utf8.ml
@@ -0,0 +1,49 @@
+open Sedlexing
+
+let unsafe_byte s j = Char.code (String.unsafe_get s j)
+let malformed s j l = `Malformed (String.sub s j l)
+
+let r_utf_8 s j l =
+  (* assert (0 <= j && 0 <= l && j + l <= String.length s); *)
+  let uchar c = `Uchar (Uchar.unsafe_of_int c) in
+  match l with
+    | 1 -> uchar (unsafe_byte s j)
+    | 2 -> (
+        let b0 = unsafe_byte s j in
+        let b1 = unsafe_byte s (j + 1) in
+        match Utf8.Helper.check_two b0 b1 with
+          | i -> uchar i
+          | exception MalFormed -> malformed s j l)
+    | 3 -> (
+        let b0 = unsafe_byte s j in
+        let b1 = unsafe_byte s (j + 1) in
+        let b2 = unsafe_byte s (j + 2) in
+        match Utf8.Helper.check_three b0 b1 b2 with
+          | i -> uchar i
+          | exception MalFormed -> malformed s j l)
+    | 4 -> (
+        let b0 = unsafe_byte s j in
+        let b1 = unsafe_byte s (j + 1) in
+        let b2 = unsafe_byte s (j + 2) in
+        let b3 = unsafe_byte s (j + 3) in
+        match Utf8.Helper.check_four b0 b1 b2 b3 with
+          | i -> uchar i
+          | exception MalFormed -> malformed s j l)
+    | _ -> assert false
+
+let fold ~f acc s =
+  let rec loop acc f s i last =
+    if i > last then acc
+    else (
+      match Utf8.Helper.width (String.unsafe_get s i) with
+        | exception MalFormed ->
+            loop (f acc i (malformed s i 1)) f s (i + 1) last
+        | need ->
+            let rem = last - i + 1 in
+            if rem < need then f acc i (malformed s i rem)
+            else loop (f acc i (r_utf_8 s i need)) f s (i + need) last)
+  in
+  let pos = 0 in
+  let len = String.length s in
+  let last = pos + len - 1 in
+  loop acc f s pos last
diff --git a/src/syntax/utf8.mli b/src/syntax/utf8.mli
new file mode 100644
index 0000000..c5cf2dd
--- /dev/null
+++ b/src/syntax/utf8.mli
@@ -0,0 +1,5 @@
+val fold :
+  f:('a -> int -> [> `Malformed of string | `Uchar of Uchar.t ] -> 'a) ->
+  'a ->
+  string ->
+  'a
diff --git a/test/utf8.ml b/test/utf8.ml
new file mode 100644
index 0000000..58b315e
--- /dev/null
+++ b/test/utf8.ml
@@ -0,0 +1,33 @@
+open Printf
+
+let next_tok buf =
+  let open Sedlexing.Utf8 in
+  match%sedlex buf with
+    | "a", Utf8 (Chars "+-×÷") -> sprintf "with Chars: %s" (lexeme buf)
+    | "b", Utf8 ("+" | "-" | "×" | "÷") ->
+        sprintf "with or_pattern: %s" (lexeme buf)
+    | Latin1 "\xc0", Utf8 "À", Ascii (Utf8 (Latin1 (Utf8 (Chars "À")))) ->
+        sprintf "mixed encoding: %s" (lexeme buf)
+    | Ascii (Star '\x00' .. '\x7f') -> sprintf "only ascii: %s" (lexeme buf)
+    | Utf8 (Star '\x00' .. '\x7f') ->
+        assert false
+        (* utf8 char interval can only match ascii because of the OCaml lexer. The regexp above should match instead  *)
+    | Latin1 (Star '\x00' .. '\xff') -> sprintf "only latin1: %s" (lexeme buf)
+    | _ -> failwith (sprintf "Unexpected character: %s" (lexeme buf))
+
+let%expect_test _ =
+  Sedlexing.Utf8.from_string "a+" |> next_tok |> print_string;
+  [%expect {| with Chars: a+ |}];
+  Sedlexing.Utf8.from_string "a÷" |> next_tok |> print_string;
+  [%expect {| with Chars: a÷ |}];
+  Sedlexing.Utf8.from_string "b+" |> next_tok |> print_string;
+  [%expect {| with or_pattern: b+ |}];
+  Sedlexing.Utf8.from_string "b÷" |> next_tok |> print_string;
+  [%expect {| with or_pattern: b÷ |}];
+  Sedlexing.Utf8.from_string "ÀÀÀ" |> next_tok |> print_string;
+  [%expect {| mixed encoding: ÀÀÀ |}];
+  Sedlexing.Utf8.from_string "az\x7f"
+  |> next_tok |> String.escaped |> print_string;
+  [%expect {| only ascii: az\127 |}];
+  Sedlexing.Utf8.from_string "az\u{c0}" |> next_tok |> print_string;
+  [%expect {| only latin1: azÀ |}]


Reviews:
user: login: toots
body: This looks good! The new checks from https://github.com/ocaml-community/sedlex/pull/168 should be added here once it's merged (which I'll do today).

Also, you should document the new `Utf8` in `README.md`. 

Other than that, this seems good to go for me.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-20T14:02:42Z
user: login: hhugo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-20T14:48:24Z
user: login: hhugo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-20T14:48:32Z
user: login: toots
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T18:34:14Z
user: login: toots
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T18:35:30Z
user: login: toots
body: This is looking great, just minor comments and one about valid latin1 chars.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-27T14:14:33Z
user: login: hhugo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-27T14:44:53Z
user: login: toots
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-27T14:50:24Z
user: login: hhugo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-27T21:14:23Z
user: login: toots
body: Looking good!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-27T22:16:36Z
user: login: toots
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-30T13:47:40Z
user: login: hhugo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-31T21:07:37Z

Review comments:
diff_hunk: @@ -382,11 +394,9 @@ let regexp_of_pattern env =
           in
           match const with
             | Some (Pconst_string (s, _, _)) ->
-                let c = ref Cset.empty in
-                for i = 0 to String.length s - 1 do
-                  c := Cset.union !c (Cset.singleton (Char.code s.[i]))
-                done;
-                Sedlex.chars !c
+                let l = rev_csets_of_string ~loc:p.ppat_loc ~utf8 s in
+                let chars = List.fold_left Cset.union Cset.empty l in
+                Sedlex.chars chars
path: src/syntax/ppx_sedlex.ml
user: login: toots
body: Nice
created_at: 2025-05-20T14:00:36Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,73 @@
+let unsafe_byte s j = Char.code (String.unsafe_get s j)
+let malformed s j l = `Malformed (String.sub s j l)
+
+let width = function
+  | '\000' .. '\127' -> 1
+  | '\192' .. '\223' -> 2
+  | '\224' .. '\239' -> 3
+  | '\240' .. '\247' -> 4
+  | _ -> 0
path: src/syntax/utf8.ml
user: login: toots
body: Do you need to raise here?
created_at: 2025-05-20T14:01:06Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,73 @@
+let unsafe_byte s j = Char.code (String.unsafe_get s j)
+let malformed s j l = `Malformed (String.sub s j l)
+
+let width = function
+  | '\000' .. '\127' -> 1
+  | '\192' .. '\223' -> 2
+  | '\224' .. '\239' -> 3
+  | '\240' .. '\247' -> 4
+  | _ -> 0
+
+let r_utf_8 s j l =
+  (* assert (0 <= j && 0 <= l && j + l <= String.length s); *)
+  let uchar c = `Uchar (Uchar.unsafe_of_int c) in
+  match l with
+    | 1 -> uchar (unsafe_byte s j)
+    | 2 ->
+        let b0 = unsafe_byte s j in
+        let b1 = unsafe_byte s (j + 1) in
+        if b1 lsr 6 != 0b10 then malformed s j l
path: src/syntax/utf8.ml
user: login: toots
body: This should be re-using the new checks in https://github.com/ocaml-community/sedlex/pull/168 when they get merged
created_at: 2025-05-20T14:01:26Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,73 @@
+let unsafe_byte s j = Char.code (String.unsafe_get s j)
+let malformed s j l = `Malformed (String.sub s j l)
+
+let width = function
+  | '\000' .. '\127' -> 1
+  | '\192' .. '\223' -> 2
+  | '\224' .. '\239' -> 3
+  | '\240' .. '\247' -> 4
+  | _ -> 0
path: src/syntax/utf8.ml
user: login: hhugo
body: now using `Sedlexing.Utf8.Helper.width`
created_at: 2025-05-20T14:48:23Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,73 @@
+let unsafe_byte s j = Char.code (String.unsafe_get s j)
+let malformed s j l = `Malformed (String.sub s j l)
+
+let width = function
+  | '\000' .. '\127' -> 1
+  | '\192' .. '\223' -> 2
+  | '\224' .. '\239' -> 3
+  | '\240' .. '\247' -> 4
+  | _ -> 0
+
+let r_utf_8 s j l =
+  (* assert (0 <= j && 0 <= l && j + l <= String.length s); *)
+  let uchar c = `Uchar (Uchar.unsafe_of_int c) in
+  match l with
+    | 1 -> uchar (unsafe_byte s j)
+    | 2 ->
+        let b0 = unsafe_byte s j in
+        let b1 = unsafe_byte s (j + 1) in
+        if b1 lsr 6 != 0b10 then malformed s j l
path: src/syntax/utf8.ml
user: login: hhugo
body: done
created_at: 2025-05-20T14:48:32Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,19 @@
+open Printf
+
+let next_tok buf =
+  let open Sedlexing.Utf8 in
+  match%sedlex buf with
+    | "a", Utf8 (Chars "+-×÷") -> sprintf "with Chars: %s" (lexeme buf)
+    | "b", Utf8 ("+" | "-" | "×" | "÷") ->
path: test/utf8.ml
user: login: toots
body: Could be nice to have an explicitely non-ascii character here to test all the way.
created_at: 2025-05-20T18:34:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,19 @@
+open Printf
+
+let next_tok buf =
+  let open Sedlexing.Utf8 in
+  match%sedlex buf with
+    | "a", Utf8 (Chars "+-×÷") -> sprintf "with Chars: %s" (lexeme buf)
+    | "b", Utf8 ("+" | "-" | "×" | "÷") ->
path: test/utf8.ml
user: login: toots
body: My bad `÷` is!
created_at: 2025-05-20T18:35:30Z
author_association: MEMBER
diff_hunk: @@ -286,29 +285,49 @@ let codepoint i =
     failwith (Printf.sprintf "Invalid Unicode code point: %i" i);
   i
 
-let regexp_for_char c = Sedlex.chars (Cset.singleton (Char.code c))
-
-let regexp_for_string s =
-  let rec aux n =
-    if n = String.length s then Sedlex.eps
-    else Sedlex.seq (regexp_for_char s.[n]) (aux (succ n))
-  in
-  aux 0
+let char c = Cset.singleton (Char.code c)
+let uchar c = Cset.singleton (Uchar.to_int c)
 
 let err loc s =
   raise (Location.Error (Location.Error.createf ~loc "Sedlex: %s" s))
 
+type encoding = Utf8 | Latin1 | Ascii
+
+let rev_csets_of_string ~loc ~encoding s =
+  match encoding with
+    | Utf8 ->
+        Utf8.fold
+          ~f:(fun acc _ x ->
+            match x with
+              | `Malformed _ -> err loc "Malformed utf-8 string"
+              | `Uchar c -> uchar c :: acc)
+          [] s
+    | Latin1 ->
+        let l = ref [] in
+        for i = 0 to String.length s - 1 do
+          l := char s.[i] :: !l
path: src/syntax/ppx_sedlex.ml
user: login: toots
body: According to https://en.wikipedia.org/wiki/ISO/IEC_8859-1, not all values are proper `latin1` characters.
created_at: 2025-05-27T14:10:10Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,19 @@
+open Printf
+
+let next_tok buf =
+  let open Sedlexing.Utf8 in
+  match%sedlex buf with
+    | "a", Utf8 (Chars "+-×÷") -> sprintf "with Chars: %s" (lexeme buf)
+    | "b", Utf8 ("+" | "-" | "×" | "÷") ->
+        sprintf "with or_pattern: %s" (lexeme buf)
+    | _ -> failwith (sprintf "Unexpected character: %s" (lexeme buf))
+
+let%expect_test _ =
+  Sedlexing.Utf8.from_string "a+" |> next_tok |> print_string;
+  [%expect {| with Chars: a+ |}];
+  Sedlexing.Utf8.from_string "a÷" |> next_tok |> print_string;
+  [%expect {| with Chars: a÷ |}];
+  Sedlexing.Utf8.from_string "b+" |> next_tok |> print_string;
+  [%expect {| with or_pattern: b+ |}];
+  Sedlexing.Utf8.from_string "b÷" |> next_tok |> print_string;
+  [%expect {| with or_pattern: b÷ |}]
path: test/utf8.ml
user: login: toots
body: Should we have test for cases that do not work? And `latin1`/`ascii` tests?
created_at: 2025-05-27T14:11:11Z
author_association: MEMBER
diff_hunk: @@ -286,29 +285,49 @@ let codepoint i =
     failwith (Printf.sprintf "Invalid Unicode code point: %i" i);
   i
 
-let regexp_for_char c = Sedlex.chars (Cset.singleton (Char.code c))
-
-let regexp_for_string s =
-  let rec aux n =
-    if n = String.length s then Sedlex.eps
-    else Sedlex.seq (regexp_for_char s.[n]) (aux (succ n))
-  in
-  aux 0
+let char c = Cset.singleton (Char.code c)
+let uchar c = Cset.singleton (Uchar.to_int c)
 
 let err loc s =
   raise (Location.Error (Location.Error.createf ~loc "Sedlex: %s" s))
 
+type encoding = Utf8 | Latin1 | Ascii
+
+let rev_csets_of_string ~loc ~encoding s =
+  match encoding with
+    | Utf8 ->
+        Utf8.fold
+          ~f:(fun acc _ x ->
+            match x with
+              | `Malformed _ -> err loc "Malformed utf-8 string"
+              | `Uchar c -> uchar c :: acc)
+          [] s
+    | Latin1 ->
+        let l = ref [] in
+        for i = 0 to String.length s - 1 do
+          l := char s.[i] :: !l
path: src/syntax/ppx_sedlex.ml
user: login: hhugo
body: >In 1990, the first version of [Unicode](https://en.wikipedia.org/wiki/Unicode) used the code points of ISO-8859-1 as the first 256 Unicode code points.
>
> In 1992, the [IANA](https://en.wikipedia.org/wiki/Internet_Assigned_Numbers_Authority) registered the character map ISO_8859-1:1987, more commonly known by its preferred [MIME](https://en.wikipedia.org/wiki/MIME) name of ISO-8859-1 (note the extra hyphen over ISO 8859-1), a superset of ISO 8859-1, for use on the [Internet](https://en.wikipedia.org/wiki/Internet). This map assigns the [C0 and C1 control codes](https://en.wikipedia.org/wiki/C0_and_C1_control_codes) to the unassigned code values thus provides for 256 characters via every possible 8-bit value.
created_at: 2025-05-27T14:44:53Z
author_association: COLLABORATOR
diff_hunk: @@ -286,29 +285,49 @@ let codepoint i =
     failwith (Printf.sprintf "Invalid Unicode code point: %i" i);
   i
 
-let regexp_for_char c = Sedlex.chars (Cset.singleton (Char.code c))
-
-let regexp_for_string s =
-  let rec aux n =
-    if n = String.length s then Sedlex.eps
-    else Sedlex.seq (regexp_for_char s.[n]) (aux (succ n))
-  in
-  aux 0
+let char c = Cset.singleton (Char.code c)
+let uchar c = Cset.singleton (Uchar.to_int c)
 
 let err loc s =
   raise (Location.Error (Location.Error.createf ~loc "Sedlex: %s" s))
 
+type encoding = Utf8 | Latin1 | Ascii
+
+let rev_csets_of_string ~loc ~encoding s =
+  match encoding with
+    | Utf8 ->
+        Utf8.fold
+          ~f:(fun acc _ x ->
+            match x with
+              | `Malformed _ -> err loc "Malformed utf-8 string"
+              | `Uchar c -> uchar c :: acc)
+          [] s
+    | Latin1 ->
+        let l = ref [] in
+        for i = 0 to String.length s - 1 do
+          l := char s.[i] :: !l
path: src/syntax/ppx_sedlex.ml
user: login: toots
body: ok!
created_at: 2025-05-27T14:50:24Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,19 @@
+open Printf
+
+let next_tok buf =
+  let open Sedlexing.Utf8 in
+  match%sedlex buf with
+    | "a", Utf8 (Chars "+-×÷") -> sprintf "with Chars: %s" (lexeme buf)
+    | "b", Utf8 ("+" | "-" | "×" | "÷") ->
+        sprintf "with or_pattern: %s" (lexeme buf)
+    | _ -> failwith (sprintf "Unexpected character: %s" (lexeme buf))
+
+let%expect_test _ =
+  Sedlexing.Utf8.from_string "a+" |> next_tok |> print_string;
+  [%expect {| with Chars: a+ |}];
+  Sedlexing.Utf8.from_string "a÷" |> next_tok |> print_string;
+  [%expect {| with Chars: a÷ |}];
+  Sedlexing.Utf8.from_string "b+" |> next_tok |> print_string;
+  [%expect {| with or_pattern: b+ |}];
+  Sedlexing.Utf8.from_string "b÷" |> next_tok |> print_string;
+  [%expect {| with or_pattern: b÷ |}]
path: test/utf8.ml
user: login: hhugo
body: I've added some tests for Latin1 and Ascii.

It's not super easy to tests for errors currently.
created_at: 2025-05-27T21:14:23Z
author_association: COLLABORATOR
diff_hunk: @@ -112,8 +115,9 @@ with a length different from one.
 
 
 Note:
- - The OCaml source is assumed to be encoded in Latin1 (for string
-   and character literals).
+ - The OCaml source is assumed to be encoded in UTF-8.
+ - Strings and chars litterals will be interpreted in ASCII unless otherwise
+   specified by the `Latin1`,`Ascii` and `Utf8` constructors in patterns.
path: README.md
user: login: toots
body: @hhugo What was the old behavior with this? Looks like defaulting to ascii is breaking a lot of projects depending on this library: https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/3050523ae8c96624eafb2c77bfc91136fa223c2d
created_at: 2025-05-30T13:47:39Z
author_association: MEMBER
diff_hunk: @@ -112,8 +115,9 @@ with a length different from one.
 
 
 Note:
- - The OCaml source is assumed to be encoded in Latin1 (for string
-   and character literals).
+ - The OCaml source is assumed to be encoded in UTF-8.
+ - Strings and chars litterals will be interpreted in ASCII unless otherwise
+   specified by the `Latin1`,`Ascii` and `Utf8` constructors in patterns.
path: README.md
user: login: hhugo
body: The old behavior was to expect latin1 but it seems some people were also feeding utf8 https://github.com/FStarLang/FStar/blob/3ff998c60bb0efe9925fc94e8fb8b785b9485af0/src/ml/FStarC_Parser_LexFStar.ml#L530 or using the 0x80..0xff range weirdly https://github.com/janestreet/ppx_css/blob/3837ab33ee0c4635826db2c746e9da163a80a360/vendor/css_parser/src/lexer.ml#L93. It seems that theses are all incorrectly using sedlex but I can give this a deeper look next week 
created_at: 2025-05-31T21:07:37Z
author_association: COLLABORATOR

PR comments:
user: login: pmetzger
created_at: 2024-11-03T19:13:40Z
body: So the documentation doesn't (yet!) officially say if we can assume source files are UTF-8, but I presume that's imminent.
author_association: MEMBER
user: login: toots
created_at: 2024-11-04T16:13:23Z
body: This is a much missed feature, thanks for working on it and fingers crossed we can make it work soon.

My guess is: if you're not using this feature, we would not be requiring any assumption regarding the file's encoding? So this would be opt-in only correct?
author_association: MEMBER
user: login: hhugo
created_at: 2024-12-19T14:56:49Z
body: https://github.com/ocaml/ocaml/pull/13668
author_association: COLLABORATOR
user: login: toots
created_at: 2025-05-19T21:14:59Z
body: Any interest on reviving and finishing this?
author_association: MEMBER
user: login: hhugo
created_at: 2025-05-20T07:53:00Z
body: I've debased the PR. code review welcome.
author_association: COLLABORATOR
user: login: toots
created_at: 2025-05-20T15:31:28Z
body: Looking good!
author_association: MEMBER
user: login: hhugo
created_at: 2025-05-20T16:38:06Z
body: I think this PR mostly need documentation now
author_association: COLLABORATOR
user: login: toots
created_at: 2025-05-26T15:55:40Z
body: Are you considering adding documentation @hhugo ?
author_association: MEMBER
user: login: hhugo
created_at: 2025-05-27T09:40:56Z
body: While adding some doc I figured the previous state of this PR was weird. We were saying that the file is expected to be utf-8 encoded but we would process strings as if they were latint1 encoded by default.

I've decided to change the default behavior and only accept ASCII by default and let users be explicit about the encoding of their literals if they need more than ASCII. The PR extend the syntax with 3 constructors `Ascii, `Latin1` and `Utf8`.
author_association: COLLABORATOR
user: login: toots
created_at: 2025-05-28T14:20:46Z
body: @hhugo good to merge on your end?
author_association: MEMBER
user: login: hhugo
created_at: 2025-05-28T15:07:48Z
body: Yrs
author_association: COLLABORATOR