nebius/SWE-rebench-V2-PRs huggingface dataset:
Repository: pest-parser/pest
Pull request #874 (2023-06-21 08:00:21)

fix: expose the unicode emoji builtin rules
ref: https://github.com/pest-parser/pest/discussions/872

tomtau: > `EMOJI, EMOJI_COMPONENT, EMOJI_MODIFIER, EMOJI_MODIFIER_BASE, EMOJI_PRESENTATION, EXTENDED_PICTOGRAPHIC` where are you get this code?
> 
> In Pest, there have a script (`update_unicode.sh`) to generate the Unicode names.

from here: https://pest.rs/book/grammars/built-ins.html#binary-properties
huacnlee: Oh, never mind on my last comment.
You do the right way. The Emoji unicode script has been generated before, we just not publish in pest `mod.rs`

Patch:
diff --git a/pest/src/unicode/mod.rs b/pest/src/unicode/mod.rs
index 8a56bd6..6d7cb1d 100644
--- a/pest/src/unicode/mod.rs
+++ b/pest/src/unicode/mod.rs
@@ -56,6 +56,7 @@ char_property_functions! {
         ALPHABETIC, BIDI_CONTROL, CASE_IGNORABLE, CASED, CHANGES_WHEN_CASEFOLDED,
         CHANGES_WHEN_CASEMAPPED, CHANGES_WHEN_LOWERCASED, CHANGES_WHEN_TITLECASED,
         CHANGES_WHEN_UPPERCASED, DASH, DEFAULT_IGNORABLE_CODE_POINT, DEPRECATED, DIACRITIC,
+        EMOJI, EMOJI_COMPONENT, EMOJI_MODIFIER, EMOJI_MODIFIER_BASE, EMOJI_PRESENTATION, EXTENDED_PICTOGRAPHIC,
         EXTENDER, GRAPHEME_BASE, GRAPHEME_EXTEND, GRAPHEME_LINK, HEX_DIGIT, HYPHEN,
         IDS_BINARY_OPERATOR, IDS_TRINARY_OPERATOR, ID_CONTINUE, ID_START, IDEOGRAPHIC, JOIN_CONTROL,
         LOGICAL_ORDER_EXCEPTION, LOWERCASE, MATH, NONCHARACTER_CODE_POINT, OTHER_ALPHABETIC,


Test patch:
diff --git a/derive/tests/grammar.pest b/derive/tests/grammar.pest
index ab670be..5d64d5d 100644
--- a/derive/tests/grammar.pest
+++ b/derive/tests/grammar.pest
@@ -68,6 +68,7 @@ han = { HAN+ }
 hangul = { HANGUL+ }
 hiragana = { HIRAGANA+ }
 arabic = { ARABIC+ }
+emoji = { EMOJI+ }
 
 WHITESPACE = _{ " " }
 COMMENT = _{ "$"+ }
diff --git a/vm/tests/grammar.rs b/vm/tests/grammar.rs
index 0128db6..3ab29e5 100644
--- a/vm/tests/grammar.rs
+++ b/vm/tests/grammar.rs
@@ -1005,3 +1005,15 @@ fn arabic() {
         ]
     }
 }
+
+#[test]
+fn emoji() {
+    parses_to! {
+        parser: vm(),
+        input: "👶",
+        rule: "emoji",
+        tokens: [
+            emoji(0, 4)
+        ]
+    }
+}