Repository: ruby/prism
Pull request #2486 (2024-02-22 22:05:46)

Set Regexp encoding flags
This PR sets the encoding flags on `Regexp` instances much like we do for `String` and `Symbol`. Unlike `String` and `Symbol`, there's a third way to set a `Regexp` encoding: encoding modifiers. There are four encoding modifiers:

| Modifier | Encoding |
|--------|--------|
| n | ASCII-8BIT |
| u | UTF-8 |
| e | EUC-JP |
| s | Windows-31J |

## Background

The meaning of these flags appears to have shifted over time. In my copy of "The Ruby Programming Language", which covers Ruby 1.8 and the soon-to-be-released 1.9, the `n` modifier meant "ASCII" and the `s` modifier meant "Shift_JIS". I think the `n` flag still has some of its heritage as it will adjust the regex encoding to be US-ASCII, rather than ASCII-8BIT, in certain circumstances.

`String`, `Symbol`, and `Regexp` all have a flag to force the object's encoding to US-ASCII, ASCII-8BIT, or UTF-8. Rather than introduce new flag values to force a `Regexp` encoding to either EUC-JP or Windows-31J, I opted to use the encoding modifiers to indicate the `Regexp` encoding. To avoid any confusion, if an encoding modifier option is used, then the "force encoding" flags are only set if they would result in a different encoding. I.e., the "/u" option would never result in the "force UTF-8" flag being set. But, the "/n" option may result in the "force US-ASCII" flag being set.

## Client Usage

To determine the `Regexp` encoding would require flag checks like the following (and used in _encoding_test.rb_):

```ruby
if regexp.forced_utf8_encoding?
  Encoding::UTF_8
elsif regexp.forced_binary_encoding?
  Encoding::ASCII_8BIT
elsif regexp.forced_us_ascii_encoding?
  Encoding::US_ASCII
elsif regexp.ascii_8bit?
  Encoding::ASCII_8BIT
elsif regexp.utf_8?
  Encoding::UTF_8
elsif regexp.euc_jp?
  Encoding::EUC_JP
elsif regexp.windows_31j?
  Encoding::Windows_31J
else
  encoding
end
```

The order here matters. The "forced encoding" flags must be checked before the encoding modifiers. The encoding modifiers may differ, but must be kept around for Ruby semantics. The order of the "force encoding" flags and the encoding modifiers is irrelevant, however, since each flag value is mutually exclusive within that class.

## Design Notes

`Regexp` encoding validation is rather complex. I'm not sure that there's a unifying rule coming from CRuby. I'm sure there's a philosophy on what should be an error but some of the details appear to be happenstance. While I did consult the existing implementations of CRuby, JRuby, and TruffleRuby for what needs to be done, I mostly arrived at the rules embodied in this code by extensive testing. This PR adds rudimentary `Regexp` validation. To be 100% accurate would require scanning the bytes of the source string and performing validation against the associated encoding. Given Prism doesn't currently do that at large, I decided not to add it for this PR. However, there are other validation issues trivially detectable by mismatched combination of encoding modifiers and source file or `Regexp` source string encoding. In cases where I could I added that validation.

In several places I added placeholder code. These sections use intent-hinting variable names but set the value conservatively to match Prism's current semantics. This was done largely to help me keep track of the complex set of rules involved in validation. However, I retained the code because I think it'll make subsequent improvements much easier to slot into the overarching design. If this is controversial I can safely remove that code. It would necessitate the removal of some error messages that are otherwise unused in Prism but used by CRuby during `Regexp` validation.

I had initially planned on deferring validation to a subsequent pass. However, I found it very confusing to have flags set to nonsense values that should be syntax errors that we weren't yet handling. Adding this rudimentary validation avoids those situations. I've tried to avoid adjusting any of the flags if there is a syntax error, but given validation can happen in stages I think Prism as a whole should not make any guarantees about flag values in the presence of syntax errors.

I also combined the computation of a `Regexp`'s encoding flags and the `Regexp` validation into a joint function. While this may not be architecturally pure, I think it's much easier to follow the convoluted set of rules that determine the resulting encoding. This is in contrast to how we handle `String` and `Symbol`. In both of those other cases the validation is much simpler: does this byte array map into the associate encoding? While we could split the two processes for `Regexp`, we would end up in a situation where we could have to process some values multiple times or set flags and have to clear them later on.

Finally, while this PR does not compute the validity of a `Regexp` source string in a given encoding, it does scan the string to determine if it is ASCII-only. In some error cases this may happen multiple times. I expect all of this will clean up when we add the code range calculation to strings in Prism. If it's required I can compute the value once and pass it around, but we should retain the laziness and that complicates things when passing the value across function boundaries.

enebo: @nirvdrum The flags are needed for clients to know how an encoding was arrived at but I feel like we should consider storing the arrived at encoding.  Checking n state flags to figure out encoding feels like something is missing like a value representing the encoding it is.  Or can we be more clever here and use flag value with a lookup table (for Ruby lib if these conditionals add clarity I am not against them staying there but I am asking about how JRuby can do this without that many conditionals)?

(not for this PR but for when CR is calculated):
If we can't table lookup then we should consider how to make encoding a single lookup value to remove the need for the if/else chain.   We cannot CR until we know encoding so it is probably the right time to add encoding value (since parser already arrived at actual encoding).
nirvdrum: > The flags are needed for clients to know how an encoding was arrived at but I feel like we should consider storing the arrived at encoding.

@enebo I agree, but that's a larger design change. If we go that route, I think it makes sense to do it for `String` and `Symbol` as well. This PR adds missing functionality adhering to the current API philosophy and adds no new flags. I had previously spoken with @kddnewton about the API for determining the `Regexp` encoding and he was in favor of cutting that out for this PR to help keep it scoped. With that said, I think this PR helps to frame that discussion better since we can clearly see what is required of clients if we don't have an `encoding` field on `RegularExpressionNode`.
nirvdrum: I've fixed the `make test-all` issues. It turns out one of the variables I was using for a `define_method` loop was being used as something of a global variable elsewhere. Depending on execution order the tests could end up working with incorrect data. I've renamed the variable to avoid that.

The memory leaks are taken care of now, as well. I hadn't realized that `parser->current_string` isn't cleaned up on the next pass. I'm reasonably certain that `parser->current_regular_expression_source` as well as the associated token buffer are only used for regular expressions now. The `valgrind` tests would raise an issue if `parser->current_regular_expression_source` were being populated and unused by `String` and `Symbol` parsing.

Patch:
diff --git a/config.yml b/config.yml
index 34086cd8..08d4f78c 100644
--- a/config.yml
+++ b/config.yml
@@ -1,5 +1,4 @@
 errors:
-  - CANNOT_PARSE_EXPRESSION
   - ALIAS_ARGUMENT
   - AMPAMPEQ_MULTI_ASSIGN
   - ARGUMENT_AFTER_BLOCK
@@ -34,6 +33,7 @@ errors:
   - BLOCK_PARAM_PIPE_TERM
   - BLOCK_TERM_BRACE
   - BLOCK_TERM_END
+  - CANNOT_PARSE_EXPRESSION
   - CANNOT_PARSE_STRING_PART
   - CASE_EXPRESSION_AFTER_CASE
   - CASE_EXPRESSION_AFTER_WHEN
@@ -82,13 +82,13 @@ errors:
   - EXPECT_ARGUMENT
   - EXPECT_EOL_AFTER_STATEMENT
   - EXPECT_EXPRESSION_AFTER_AMPAMPEQ
-  - EXPECT_EXPRESSION_AFTER_PIPEPIPEEQ
   - EXPECT_EXPRESSION_AFTER_COMMA
   - EXPECT_EXPRESSION_AFTER_EQUAL
   - EXPECT_EXPRESSION_AFTER_LESS_LESS
   - EXPECT_EXPRESSION_AFTER_LPAREN
-  - EXPECT_EXPRESSION_AFTER_QUESTION
   - EXPECT_EXPRESSION_AFTER_OPERATOR
+  - EXPECT_EXPRESSION_AFTER_PIPEPIPEEQ
+  - EXPECT_EXPRESSION_AFTER_QUESTION
   - EXPECT_EXPRESSION_AFTER_SPLAT
   - EXPECT_EXPRESSION_AFTER_SPLAT_HASH
   - EXPECT_EXPRESSION_AFTER_STAR
@@ -113,23 +113,25 @@ errors:
   - HASH_VALUE
   - HEREDOC_TERM
   - INCOMPLETE_QUESTION_MARK
-  - INCOMPLETE_VARIABLE_CLASS_3_3_0
   - INCOMPLETE_VARIABLE_CLASS
-  - INCOMPLETE_VARIABLE_INSTANCE_3_3_0
+  - INCOMPLETE_VARIABLE_CLASS_3_3_0
   - INCOMPLETE_VARIABLE_INSTANCE
+  - INCOMPLETE_VARIABLE_INSTANCE_3_3_0
+  - INVALID_CHARACTER
   - INVALID_ENCODING_MAGIC_COMMENT
   - INVALID_FLOAT_EXPONENT
+  - INVALID_MULTIBYTE_CHAR
+  - INVALID_MULTIBYTE_CHARACTER
+  - INVALID_MULTIBYTE_ESCAPE
   - INVALID_NUMBER_BINARY
   - INVALID_NUMBER_DECIMAL
   - INVALID_NUMBER_HEXADECIMAL
   - INVALID_NUMBER_OCTAL
   - INVALID_NUMBER_UNDERSCORE
-  - INVALID_CHARACTER
-  - INVALID_MULTIBYTE_CHARACTER
-  - INVALID_PRINTABLE_CHARACTER
   - INVALID_PERCENT
-  - INVALID_VARIABLE_GLOBAL_3_3_0
+  - INVALID_PRINTABLE_CHARACTER
   - INVALID_VARIABLE_GLOBAL
+  - INVALID_VARIABLE_GLOBAL_3_3_0
   - IT_NOT_ALLOWED_NUMBERED
   - IT_NOT_ALLOWED_ORDINARY
   - LAMBDA_OPEN
@@ -150,8 +152,8 @@ errors:
   - MODULE_TERM
   - MULTI_ASSIGN_MULTI_SPLATS
   - MULTI_ASSIGN_UNEXPECTED_REST
-  - NOT_EXPRESSION
   - NO_LOCAL_VARIABLE
+  - NOT_EXPRESSION
   - NUMBER_LITERAL_UNDERSCORE
   - NUMBERED_PARAMETER_IT
   - NUMBERED_PARAMETER_ORDINARY
@@ -173,8 +175,8 @@ errors:
   - PARAMETER_UNEXPECTED_FWD
   - PARAMETER_WILD_LOOSE_COMMA
   - PATTERN_EXPRESSION_AFTER_BRACKET
-  - PATTERN_EXPRESSION_AFTER_HROCKET
   - PATTERN_EXPRESSION_AFTER_COMMA
+  - PATTERN_EXPRESSION_AFTER_HROCKET
   - PATTERN_EXPRESSION_AFTER_IN
   - PATTERN_EXPRESSION_AFTER_KEY
   - PATTERN_EXPRESSION_AFTER_PAREN
@@ -191,7 +193,12 @@ errors:
   - PATTERN_TERM_BRACKET
   - PATTERN_TERM_PAREN
   - PIPEPIPEEQ_MULTI_ASSIGN
+  - REGEXP_ENCODING_OPTION_MISMATCH
+  - REGEXP_INCOMPAT_CHAR_ENCODING
+  - REGEXP_INVALID_UNICODE_RANGE
+  - REGEXP_NON_ESCAPED_MBC
   - REGEXP_TERM
+  - REGEXP_UTF8_CHAR_NON_UTF8_REGEXP
   - RESCUE_EXPRESSION
   - RESCUE_MODIFIER_VALUE
   - RESCUE_TERM
@@ -213,9 +220,9 @@ errors:
   - TERNARY_EXPRESSION_FALSE
   - TERNARY_EXPRESSION_TRUE
   - UNARY_RECEIVER
+  - UNDEF_ARGUMENT
   - UNEXPECTED_TOKEN_CLOSE_CONTEXT
   - UNEXPECTED_TOKEN_IGNORE
-  - UNDEF_ARGUMENT
   - UNTIL_TERM
   - VOID_EXPRESSION
   - WHILE_TERM
diff --git a/include/prism/encoding.h b/include/prism/encoding.h
index d0f947ea..0850e291 100644
--- a/include/prism/encoding.h
+++ b/include/prism/encoding.h
@@ -248,10 +248,22 @@ extern const pm_encoding_t pm_encodings[PM_ENCODING_MAXIMUM];
 /**
  * This is the ASCII-8BIT encoding. We need a reference to it so that pm_strpbrk
  * can compare against it because invalid multibyte characters are not a thing
- * in this encoding.
+ * in this encoding. It is also needed for handling Regexp encoding flags.
  */
 #define PM_ENCODING_ASCII_8BIT_ENTRY (&pm_encodings[PM_ENCODING_ASCII_8BIT])
 
+/**
+ * This is the EUC-JP encoding. We need a reference to it to quickly process
+ * regular expression modifiers.
+ */
+#define PM_ENCODING_EUC_JP_ENTRY (&pm_encodings[PM_ENCODING_EUC_JP])
+
+/**
+ * This is the Windows-31J encoding. We need a reference to it to quickly
+ * process regular expression modifiers.
+ */
+#define PM_ENCODING_WINDOWS_31J_ENTRY (&pm_encodings[PM_ENCODING_WINDOWS_31J])
+
 /**
  * Parse the given name of an encoding and return a pointer to the corresponding
  * encoding struct if one can be found, otherwise return NULL.
diff --git a/include/prism/parser.h b/include/prism/parser.h
index 02f60192..cf5f702a 100644
--- a/include/prism/parser.h
+++ b/include/prism/parser.h
@@ -663,6 +663,17 @@ struct pm_parser {
      */
     pm_string_t current_string;
 
+    /**
+     * This string is used to pass information from the lexer to the parser. When
+     * processing regular expressions we must track the string source for the expression
+     * as well as its unescaped representation. In that case, `current_string` will hold
+     * the unescaped value while this field will hold the translated source value. There
+     * are some escape sequences in regular expressions that will cause the associated
+     * source string to have a different value than the content of the expression so we
+     * must track this state separately.
+     */
+    pm_string_t current_regular_expression_source;
+
     /**
      * The line number at the start of the parse. This will be used to offset
      * the line numbers of all of the locations.
diff --git a/lib/prism/translation/parser/lexer.rb b/lib/prism/translation/parser/lexer.rb
index 7585aadd..62ce17c7 100644
--- a/lib/prism/translation/parser/lexer.rb
+++ b/lib/prism/translation/parser/lexer.rb
@@ -296,7 +296,7 @@ module Prism
                 start_offset = offset_cache[token.location.start_offset]
                 lines.map do |line|
                   end_offset = start_offset + line.length
-                  tokens << [:tSTRING_CONTENT, [line, Range.new(source_buffer, offset_cache[start_offset], offset_cache[end_offset])]]
+                  tokens << [:tSTRING_CONTENT, [line, Range.new(source_buffer, start_offset, offset_cache[end_offset])]]
                   start_offset = end_offset
                 end
                 next
diff --git a/src/prism.c b/src/prism.c
index 6921feac..903c7511 100644
--- a/src/prism.c
+++ b/src/prism.c
@@ -5949,6 +5949,130 @@ parse_symbol_encoding(const pm_parser_t *parser, const pm_string_t *contents) {
     return 0;
 }
 
+static pm_node_flags_t
+parse_and_validate_regular_expression_encoding_modifier(pm_parser_t *parser, const pm_string_t *source, const pm_string_t *contents, pm_node_flags_t flags, char modifier, const pm_encoding_t *modifier_encoding) {
+    assert ((modifier == 'n' && modifier_encoding == PM_ENCODING_ASCII_8BIT_ENTRY) ||
+            (modifier == 'u' && modifier_encoding == PM_ENCODING_UTF_8_ENTRY) ||
+            (modifier == 'e' && modifier_encoding == PM_ENCODING_EUC_JP_ENTRY) ||
+            (modifier == 's' && modifier_encoding == PM_ENCODING_WINDOWS_31J_ENTRY));
+
+    // There's special validation logic used if a string does not contain any character escape sequences.
+    if (parser->explicit_encoding == NULL) {
+        // If an ASCII-only string without character escapes is used with an encoding modifier, then resulting Regexp
+        // has the modifier encoding, unless the ASCII-8BIT modifier is used, in which case the Regexp "downgrades" to
+        // the US-ASCII encoding.
+        bool ascii_only = pm_ascii_only_p(contents);
+        if (ascii_only) {
+            return modifier == 'n' ? PM_REGULAR_EXPRESSION_FLAGS_FORCED_US_ASCII_ENCODING : flags;
+        }
+
+        if (parser->encoding == PM_ENCODING_US_ASCII_ENTRY) {
+            if (!ascii_only) {
+                PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->current, PM_ERR_INVALID_MULTIBYTE_CHAR, parser->encoding->name);
+            }
+        } else if (parser->encoding != modifier_encoding) {
+            PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->current, PM_ERR_REGEXP_ENCODING_OPTION_MISMATCH, modifier, parser->encoding->name);
+
+            if (modifier == 'n' && !ascii_only) {
+                PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->current, PM_ERR_REGEXP_NON_ESCAPED_MBC, (int) pm_string_length(source), (const char *) pm_string_source(source));
+            }
+        }
+
+        return flags;
+    }
+
+    // TODO (nirvdrum 21-Feb-2024): To validate regexp sources with character escape sequences we need to know whether hex or Unicode escape sequences were used and Prism doesn't currently provide that data. We handle a subset of unambiguous cases in the meanwhile.
+    bool mixed_encoding = false;
+
+    if (mixed_encoding) {
+        PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->current, PM_ERR_INVALID_MULTIBYTE_ESCAPE, (int) pm_string_length(source), (const char *) pm_string_source(source));
+    } else if (modifier != 'n' && parser->explicit_encoding == PM_ENCODING_ASCII_8BIT_ENTRY) {
+        // TODO (nirvdrum 21-Feb-2024): Validate the content is valid in the modifier encoding. Do this on-demand so we don't pay the cost of computation unnecessarily.
+        bool valid_string_in_modifier_encoding = true;
+
+        if (!valid_string_in_modifier_encoding) {
+            PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->current, PM_ERR_INVALID_MULTIBYTE_ESCAPE, (int) pm_string_length(source), (const char *) pm_string_source(source));
+        }
+    } else if (modifier != 'u' && parser->explicit_encoding == PM_ENCODING_UTF_8_ENTRY) {
+        // TODO (nirvdrum 21-Feb-2024): There's currently no way to tell if the source used hex or Unicode character escapes from `explicit_encoding` alone. If the source encoding was already UTF-8, both character escape types would set `explicit_encoding` to UTF-8, but need to be processed differently. Skip for now.
+        if (parser->encoding != PM_ENCODING_UTF_8_ENTRY) {
+            PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->current, PM_ERR_REGEXP_INCOMPAT_CHAR_ENCODING, (int) pm_string_length(source), (const char *) pm_string_source(source));
+        }
+    }
+
+    // We've determined the encoding would naturally be EUC-JP and there is no need to force the encoding to anything else.
+    return flags;
+}
+
+/**
+ * Ruby "downgrades" the encoding of Regexps to US-ASCII if the associated encoding is ASCII-compatible and
+ * the unescaped representation of a Regexp source consists only of US-ASCII code points. This is true even
+ * when the Regexp is explicitly given an ASCII-8BIT encoding via the (/n) modifier. Otherwise, the encoding
+ * may be explicitly set with an escape sequence.
+ */
+static pm_node_flags_t
+parse_and_validate_regular_expression_encoding(pm_parser_t *parser, const pm_string_t *source, const pm_string_t *contents, pm_node_flags_t flags) {
+    // TODO (nirvdrum 22-Feb-2024): CRuby reports a special Regexp-specific error for invalid Unicode ranges. We either need to scan again or modify the "invalid Unicode escape sequence" message we already report.
+    bool valid_unicode_range = true;
+    if (parser->explicit_encoding == PM_ENCODING_UTF_8_ENTRY && !valid_unicode_range) {
+        PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->current, PM_ERR_REGEXP_INVALID_UNICODE_RANGE, (int) pm_string_length(source), (const char *) pm_string_source(source));
+        return flags;
+    }
+
+    // US-ASCII strings do not admit multi-byte character literals. However, character escape sequences corresponding
+    // to multi-byte characters are allowed.
+    if (parser->encoding == PM_ENCODING_US_ASCII_ENTRY && parser->explicit_encoding == NULL && !pm_ascii_only_p(contents)) {
+        // CRuby will continue processing even though a SyntaxError has already been detected. It may result in the
+        // following error message appearing twice. We do the same for compatibility.
+        PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->current, PM_ERR_INVALID_MULTIBYTE_CHAR, parser->encoding->name);
+    }
+
+    /**
+     * Start checking modifier flags. We need to process these before considering any explicit encodings that may have
+     * been set by character literals. The order in which the encoding modifiers is checked does not matter. In the
+     * event that both an encoding modifier and an explicit encoding would result in the same encoding we do not set
+     * the corresponding "forced_<encoding>" flag. Instead, the caller should check the encoding modifier flag and
+     * determine the encoding that way.
+     */
+
+    if (flags & PM_REGULAR_EXPRESSION_FLAGS_ASCII_8BIT) {
+        return parse_and_validate_regular_expression_encoding_modifier(parser, source, contents, flags, 'n', PM_ENCODING_ASCII_8BIT_ENTRY);
+    }
+
+    if (flags & PM_REGULAR_EXPRESSION_FLAGS_UTF_8) {
+        return parse_and_validate_regular_expression_encoding_modifier(parser, source, contents, flags, 'u', PM_ENCODING_UTF_8_ENTRY);
+    }
+
+    if (flags & PM_REGULAR_EXPRESSION_FLAGS_EUC_JP) {
+        return parse_and_validate_regular_expression_encoding_modifier(parser, source, contents, flags, 'e', PM_ENCODING_EUC_JP_ENTRY);
+    }
+
+    if (flags & PM_REGULAR_EXPRESSION_FLAGS_WINDOWS_31J) {
+        return parse_and_validate_regular_expression_encoding_modifier(parser, source, contents, flags, 's', PM_ENCODING_WINDOWS_31J_ENTRY);
+    }
+
+    // At this point no encoding modifiers will be present on the regular expression as they would have already
+    // been processed. Ruby stipulates that all source files must use an ASCII-compatible encoding. Thus, all
+    // regular expressions without an encoding modifier appearing in source are eligible for "downgrading" to US-ASCII.
+    if (pm_ascii_only_p(contents)) {
+        return PM_REGULAR_EXPRESSION_FLAGS_FORCED_US_ASCII_ENCODING;
+    }
+
+    // A Regexp may optionally have its encoding explicitly set via a character escape sequence in the source string
+    // or by specifying a modifier.
+    //
+    // NB: an explicitly set encoding is ignored by Ruby if the Regexp consists of only US ASCII code points.
+    if (parser->explicit_encoding != NULL) {
+        if (parser->explicit_encoding == PM_ENCODING_UTF_8_ENTRY) {
+            return PM_REGULAR_EXPRESSION_FLAGS_FORCED_UTF8_ENCODING;
+        } else if (parser->encoding == PM_ENCODING_US_ASCII_ENTRY) {
+            return PM_REGULAR_EXPRESSION_FLAGS_FORCED_BINARY_ENCODING;
+        }
+    }
+
+    return 0;
+}
+
 /**
  * Allocate and initialize a new SymbolNode node with the given unescaped
  * string.
@@ -8130,34 +8254,34 @@ escape_write_escape_encoded(pm_parser_t *parser, pm_buffer_t *buffer) {
  * source so that the regular expression engine will perform its own unescaping.
  */
 static inline void
-escape_write_byte(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags, uint8_t byte) {
+escape_write_byte(pm_parser_t *parser, pm_buffer_t *buffer, pm_buffer_t *regular_expression_buffer, uint8_t flags, uint8_t byte) {
     if (flags & PM_ESCAPE_FLAG_REGEXP) {
-        pm_buffer_append_bytes(buffer, (const uint8_t *) "\\x", 2);
+        pm_buffer_append_bytes(regular_expression_buffer, (const uint8_t *) "\\x", 2);
 
         uint8_t byte1 = (uint8_t) ((byte >> 4) & 0xF);
         uint8_t byte2 = (uint8_t) (byte & 0xF);
 
         if (byte1 >= 0xA) {
-            pm_buffer_append_byte(buffer, (uint8_t) ((byte1 - 0xA) + 'A'));
+            pm_buffer_append_byte(regular_expression_buffer, (uint8_t) ((byte1 - 0xA) + 'A'));
         } else {
-            pm_buffer_append_byte(buffer, (uint8_t) (byte1 + '0'));
+            pm_buffer_append_byte(regular_expression_buffer, (uint8_t) (byte1 + '0'));
         }
 
         if (byte2 >= 0xA) {
-            pm_buffer_append_byte(buffer, (uint8_t) (byte2 - 0xA + 'A'));
+            pm_buffer_append_byte(regular_expression_buffer, (uint8_t) (byte2 - 0xA + 'A'));
         } else {
-            pm_buffer_append_byte(buffer, (uint8_t) (byte2 + '0'));
+            pm_buffer_append_byte(regular_expression_buffer, (uint8_t) (byte2 + '0'));
         }
-    } else {
-        escape_write_byte_encoded(parser, buffer, byte);
     }
+
+    escape_write_byte_encoded(parser, buffer, byte);
 }
 
 /**
  * Read the value of an escape into the buffer.
  */
 static void
-escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
+escape_read(pm_parser_t *parser, pm_buffer_t *buffer, pm_buffer_t *regular_expression_buffer, uint8_t flags) {
     switch (peek(parser)) {
         case '\\': {
             parser->current.end++;
@@ -8248,10 +8372,10 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
                 }
 
                 if (flags & PM_ESCAPE_FLAG_REGEXP) {
-                    pm_buffer_append_bytes(buffer, start, (size_t) (parser->current.end - start));
-                } else {
-                    escape_write_byte_encoded(parser, buffer, value);
+                    pm_buffer_append_bytes(regular_expression_buffer, start, (size_t) (parser->current.end - start));
                 }
+
+                escape_write_byte_encoded(parser, buffer, value);
             } else {
                 pm_parser_err_current(parser, PM_ERR_ESCAPE_INVALID_HEXADECIMAL);
             }
@@ -8272,10 +8396,9 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
                 uint32_t value = escape_unicode(parser->current.end, 4);
 
                 if (flags & PM_ESCAPE_FLAG_REGEXP) {
-                    pm_buffer_append_bytes(buffer, start, (size_t) (parser->current.end + 4 - start));
-                } else {
-                    escape_write_unicode(parser, buffer, flags, start, parser->current.end + 4, value);
+                    pm_buffer_append_bytes(regular_expression_buffer, start, (size_t) (parser->current.end + 4 - start));
                 }
+                escape_write_unicode(parser, buffer, flags, start, parser->current.end + 4, value);
 
                 parser->current.end += 4;
             } else if (peek(parser) == '{') {
@@ -8306,10 +8429,8 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
                         extra_codepoints_start = unicode_start;
                     }
 
-                    if (!(flags & PM_ESCAPE_FLAG_REGEXP)) {
-                        uint32_t value = escape_unicode(unicode_start, hexadecimal_length);
-                        escape_write_unicode(parser, buffer, flags, unicode_start, parser->current.end, value);
-                    }
+                    uint32_t value = escape_unicode(unicode_start, hexadecimal_length);
+                    escape_write_unicode(parser, buffer, flags, unicode_start, parser->current.end, value);
 
                     parser->current.end += pm_strspn_whitespace(parser->current.end, parser->end - parser->current.end);
                 }
@@ -8327,7 +8448,7 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
                 }
 
                 if (flags & PM_ESCAPE_FLAG_REGEXP) {
-                    pm_buffer_append_bytes(buffer, unicode_codepoints_start, (size_t) (parser->current.end - unicode_codepoints_start));
+                    pm_buffer_append_bytes(regular_expression_buffer, unicode_codepoints_start, (size_t) (parser->current.end - unicode_codepoints_start));
                 }
             } else {
                 pm_parser_err_current(parser, PM_ERR_ESCAPE_INVALID_UNICODE);
@@ -8346,7 +8467,7 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
             switch (peeked) {
                 case '?': {
                     parser->current.end++;
-                    escape_write_byte(parser, buffer, flags, escape_byte(0x7f, flags));
+                    escape_write_byte(parser, buffer, regular_expression_buffer, flags, escape_byte(0x7f, flags));
                     return;
                 }
                 case '\\':
@@ -8355,7 +8476,7 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
                         return;
                     }
                     parser->current.end++;
-                    escape_read(parser, buffer, flags | PM_ESCAPE_FLAG_CONTROL);
+                    escape_read(parser, buffer, regular_expression_buffer, flags | PM_ESCAPE_FLAG_CONTROL);
                     return;
                 default: {
                     if (!char_is_ascii_printable(peeked)) {
@@ -8364,7 +8485,7 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
                     }
 
                     parser->current.end++;
-                    escape_write_byte(parser, buffer, flags, escape_byte(peeked, flags | PM_ESCAPE_FLAG_CONTROL));
+                    escape_write_byte(parser, buffer, regular_expression_buffer, flags, escape_byte(peeked, flags | PM_ESCAPE_FLAG_CONTROL));
                     return;
                 }
             }
@@ -8386,7 +8507,7 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
             switch (peeked) {
                 case '?': {
                     parser->current.end++;
-                    escape_write_byte(parser, buffer, flags, escape_byte(0x7f, flags));
+                    escape_write_byte(parser, buffer, regular_expression_buffer, flags, escape_byte(0x7f, flags));
                     return;
                 }
                 case '\\':
@@ -8395,7 +8516,7 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
                         return;
                     }
                     parser->current.end++;
-                    escape_read(parser, buffer, flags | PM_ESCAPE_FLAG_CONTROL);
+                    escape_read(parser, buffer, regular_expression_buffer, flags | PM_ESCAPE_FLAG_CONTROL);
                     return;
                 default: {
                     if (!char_is_ascii_printable(peeked)) {
@@ -8404,7 +8525,7 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
                     }
 
                     parser->current.end++;
-                    escape_write_byte(parser, buffer, flags, escape_byte(peeked, flags | PM_ESCAPE_FLAG_CONTROL));
+                    escape_write_byte(parser, buffer, regular_expression_buffer, flags, escape_byte(peeked, flags | PM_ESCAPE_FLAG_CONTROL));
                     return;
                 }
             }
@@ -8429,7 +8550,7 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
                     return;
                 }
                 parser->current.end++;
-                escape_read(parser, buffer, flags | PM_ESCAPE_FLAG_META);
+                escape_read(parser, buffer, regular_expression_buffer, flags | PM_ESCAPE_FLAG_META);
                 return;
             }
 
@@ -8439,7 +8560,7 @@ escape_read(pm_parser_t *parser, pm_buffer_t *buffer, uint8_t flags) {
             }
 
             parser->current.end++;
-            escape_write_byte(parser, buffer, flags, escape_byte(peeked, flags | PM_ESCAPE_FLAG_META));
+            escape_write_byte(parser, buffer, regular_expression_buffer, flags, escape_byte(peeked, flags | PM_ESCAPE_FLAG_META));
             return;
         }
         case '\r': {
@@ -8510,7 +8631,7 @@ lex_question_mark(pm_parser_t *parser) {
         pm_buffer_t buffer;
         pm_buffer_init_capacity(&buffer, 3);
 
-        escape_read(parser, &buffer, PM_ESCAPE_FLAG_SINGLE);
+        escape_read(parser, &buffer, NULL, PM_ESCAPE_FLAG_SINGLE);
         pm_string_owned_init(&parser->current_string, (uint8_t *) buffer.value, buffer.length);
 
         return PM_TOKEN_CHARACTER_LITERAL;
@@ -8724,7 +8845,7 @@ parser_end_of_line_p(const pm_parser_t *parser) {
  *     "foo\n"
  *
  * then the bytes in the string are "f", "o", "o", "\", "n", but we want to
- * provide out consumers with the string content "f", "o", "o", "\n". In these
+ * provide our consumers with the string content "f", "o", "o", "\n". In these
  * cases, when we find the first escape sequence, we initialize a pm_buffer_t
  * to keep track of the string content. Then in the parser, it will
  * automatically attach the string content to the node that it belongs to.
@@ -8743,6 +8864,26 @@ typedef struct {
     const uint8_t *cursor;
 } pm_token_buffer_t;
 
+/**
+ * In order to properly set a regular expression's encoding and to validate
+ * the byte sequence for the underlying encoding we must process any escape
+ * sequences. The unescaped byte sequence will be stored in `buffer` just like
+ * for other string-like types. However, we also need to store the regular
+ * expression's source string. That string may be different from what we see
+ * during lexing because some escape sequences rewrite the source.
+ *
+ * This value will only be initialized for regular expressions and only if we
+ * receive an escape sequence. It will contain the regular expression's source
+ * string's byte sequence.
+ */
+typedef struct {
+    /** The embedded base buffer. */
+    pm_token_buffer_t base;
+
+    /** The buffer holding the regexp source. */
+    pm_buffer_t regexp_buffer;
+} pm_regexp_token_buffer_t;
+
 /**
  * Push the given byte into the token buffer.
  */
@@ -8751,20 +8892,16 @@ pm_token_buffer_push_byte(pm_token_buffer_t *token_buffer, uint8_t byte) {
     pm_buffer_append_byte(&token_buffer->buffer, byte);
 }
 
-/**
- * Append the given bytes into the token buffer.
- */
 static inline void
-pm_token_buffer_push_bytes(pm_token_buffer_t *token_buffer, const uint8_t *bytes, size_t length) {
-    pm_buffer_append_bytes(&token_buffer->buffer, bytes, length);
+pm_regexp_token_buffer_push_byte(pm_regexp_token_buffer_t *token_buffer, uint8_t byte) {
+    pm_buffer_append_byte(&token_buffer->regexp_buffer, byte);
 }
 
 /**
- * Push an escaped character into the token buffer.
+ * Return the width of the character at the end of the current token.
  */
-static inline void
-pm_token_buffer_push_escaped(pm_token_buffer_t *token_buffer, pm_parser_t *parser) {
-    // First, determine the width of the character to be escaped.
+static inline size_t
+parser_char_width(const pm_parser_t *parser) {
     size_t width;
     if (parser->encoding_changed) {
         width = parser->encoding->char_width(parser->current.end, parser->end - parser->current.end);
@@ -8774,10 +8911,24 @@ pm_token_buffer_push_escaped(pm_token_buffer_t *token_buffer, pm_parser_t *parse
 
     // TODO: If the character is invalid in the given encoding, then we'll just
     // push one byte into the buffer. This should actually be an error.
-    width = (width == 0 ? 1 : width);
+    return (width == 0 ? 1 : width);
+}
+
+/**
+ * Push an escaped character into the token buffer.
+ */
+static void
+pm_token_buffer_push_escaped(pm_token_buffer_t *token_buffer, pm_parser_t *parser) {
+    size_t width = parser_char_width(parser);
+    pm_buffer_append_bytes(&token_buffer->buffer, parser->current.end, width);
+    parser->current.end += width;
+}
 
-    // Now, push the bytes into the buffer.
-    pm_token_buffer_push_bytes(token_buffer, parser->current.end, width);
+static void
+pm_regexp_token_buffer_push_escaped(pm_regexp_token_buffer_t *token_buffer, pm_parser_t *parser) {
+    size_t width = parser_char_width(parser);
+    pm_buffer_append_bytes(&token_buffer->base.buffer, parser->current.end, width);
+    pm_buffer_append_bytes(&token_buffer->regexp_buffer, parser->current.end, width);
     parser->current.end += width;
 }
 
@@ -8789,7 +8940,13 @@ pm_token_buffer_push_escaped(pm_token_buffer_t *token_buffer, pm_parser_t *parse
  */
 static inline void
 pm_token_buffer_copy(pm_parser_t *parser, pm_token_buffer_t *token_buffer) {
-    pm_string_owned_init(&parser->current_string, (uint8_t *) token_buffer->buffer.value, token_buffer->buffer.length);
+    pm_string_owned_init(&parser->current_string, (uint8_t *) pm_buffer_value(&token_buffer->buffer), pm_buffer_length(&token_buffer->buffer));
+}
+
+static inline void
+pm_regexp_token_buffer_copy(pm_parser_t *parser, pm_regexp_token_buffer_t *token_buffer) {
+    pm_string_owned_init(&parser->current_string, (uint8_t *) pm_buffer_value(&token_buffer->base.buffer), pm_buffer_length(&token_buffer->base.buffer));
+    pm_string_owned_init(&parser->current_regular_expression_source, (uint8_t *) pm_buffer_value(&token_buffer->regexp_buffer), pm_buffer_length(&token_buffer->regexp_buffer));
 }
 
 /**
@@ -8811,6 +8968,20 @@ pm_token_buffer_flush(pm_parser_t *parser, pm_token_buffer_t *token_buffer) {
     }
 }
 
+static void
+pm_regexp_token_buffer_flush(pm_parser_t *parser, pm_regexp_token_buffer_t *token_buffer) {
+    if (token_buffer->base.cursor == NULL) {
+        pm_string_shared_init(&parser->current_string, parser->current.start, parser->current.end);
+        pm_string_shared_init(&parser->current_regular_expression_source, parser->current.start, parser->current.end);
+    } else {
+        pm_buffer_append_bytes(&token_buffer->base.buffer, token_buffer->base.cursor, (size_t) (parser->current.end - token_buffer->base.cursor));
+        pm_buffer_append_bytes(&token_buffer->regexp_buffer, token_buffer->base.cursor, (size_t) (parser->current.end - token_buffer->base.cursor));
+        pm_regexp_token_buffer_copy(parser, token_buffer);
+    }
+}
+
+#define PM_TOKEN_BUFFER_DEFAULT_SIZE 16
+
 /**
  * When we've found an escape sequence, we need to copy everything up to this
  * point into the buffer because we're about to provide a string that has
@@ -8823,7 +8994,7 @@ static void
 pm_token_buffer_escape(pm_parser_t *parser, pm_token_buffer_t *token_buffer) {
     const uint8_t *start;
     if (token_buffer->cursor == NULL) {
-        pm_buffer_init_capacity(&token_buffer->buffer, 16);
+        pm_buffer_init_capacity(&token_buffer->buffer, PM_TOKEN_BUFFER_DEFAULT_SIZE);
         start = parser->current.start;
     } else {
         start = token_buffer->cursor;
@@ -8835,6 +9006,26 @@ pm_token_buffer_escape(pm_parser_t *parser, pm_token_buffer_t *token_buffer) {
     token_buffer->cursor = end;
 }
 
+static void
+pm_regexp_token_buffer_escape(pm_parser_t *parser, pm_regexp_token_buffer_t *token_buffer) {
+    const uint8_t *start;
+    if (token_buffer->base.cursor == NULL) {
+        pm_buffer_init_capacity(&token_buffer->base.buffer, PM_TOKEN_BUFFER_DEFAULT_SIZE);
+        pm_buffer_init_capacity(&token_buffer->regexp_buffer, PM_TOKEN_BUFFER_DEFAULT_SIZE);
+        start = parser->current.start;
+    } else {
+        start = token_buffer->base.cursor;
+    }
+
+    const uint8_t *end = parser->current.end - 1;
+    pm_buffer_append_bytes(&token_buffer->base.buffer, start, (size_t) (end - start));
+    pm_buffer_append_bytes(&token_buffer->regexp_buffer, start, (size_t) (end - start));
+
+    token_buffer->base.cursor = end;
+}
+
+#undef PM_TOKEN_BUFFER_DEFAULT_SIZE
+
 /**
  * Effectively the same thing as pm_strspn_inline_whitespace, but in the case of
  * a tilde heredoc expands out tab characters to the nearest tab boundaries.
@@ -10143,7 +10334,7 @@ parser_lex(pm_parser_t *parser) {
 
             // If we haven't found an escape yet, then this buffer will be
             // unallocated since we can refer directly to the source string.
-            pm_token_buffer_t token_buffer = { { 0 }, 0 };
+            pm_token_buffer_t token_buffer = { 0 };
 
             while (breakpoint != NULL) {
                 // If we hit a null byte, skip directly past it.
@@ -10242,7 +10433,7 @@ parser_lex(pm_parser_t *parser) {
                                 pm_token_buffer_push_byte(&token_buffer, peeked);
                                 parser->current.end++;
                             } else if (lex_mode->as.list.interpolation) {
-                                escape_read(parser, &token_buffer.buffer, PM_ESCAPE_FLAG_NONE);
+                                escape_read(parser, &token_buffer.buffer, NULL, PM_ESCAPE_FLAG_NONE);
                             } else {
                                 pm_token_buffer_push_byte(&token_buffer, '\\');
                                 pm_token_buffer_push_escaped(&token_buffer, parser);
@@ -10306,8 +10497,8 @@ parser_lex(pm_parser_t *parser) {
                 parser->next_start = NULL;
             }
 
-            // We'll check if we're at the end of the file. If we are, then we need to
-            // return the EOF token.
+            // We'll check if we're at the end of the file. If we are, then we
+            // need to return the EOF token.
             if (parser->current.end >= parser->end) {
                 LEX(PM_TOKEN_EOF);
             }
@@ -10320,7 +10511,7 @@ parser_lex(pm_parser_t *parser) {
             // characters.
             const uint8_t *breakpoints = lex_mode->as.regexp.breakpoints;
             const uint8_t *breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, false);
-            pm_token_buffer_t token_buffer = { { 0 }, 0 };
+            pm_regexp_token_buffer_t token_buffer = { 0 };
 
             while (breakpoint != NULL) {
                 // If we hit a null byte, skip directly past it.
@@ -10368,7 +10559,7 @@ parser_lex(pm_parser_t *parser) {
                     // first.
                     if (breakpoint > parser->current.start) {
                         parser->current.end = breakpoint;
-                        pm_token_buffer_flush(parser, &token_buffer);
+                        pm_regexp_token_buffer_flush(parser, &token_buffer);
                         LEX(PM_TOKEN_STRING_CONTENT);
                     }
 
@@ -10395,7 +10586,7 @@ parser_lex(pm_parser_t *parser) {
                         continue;
                     }
 
-                    pm_token_buffer_escape(parser, &token_buffer);
+                    pm_regexp_token_buffer_escape(parser, &token_buffer);
                     uint8_t peeked = peek(parser);
 
                     switch (peeked) {
@@ -10403,9 +10594,10 @@ parser_lex(pm_parser_t *parser) {
                             parser->current.end++;
                             if (peek(parser) != '\n') {
                                 if (lex_mode->as.regexp.terminator != '\r') {
-                                    pm_token_buffer_push_byte(&token_buffer, '\\');
+                                    pm_regexp_token_buffer_push_byte(&token_buffer, '\\');
                                 }
-                                pm_token_buffer_push_byte(&token_buffer, '\r');
+                                pm_token_buffer_push_byte(&token_buffer.base, '\r');
+                                pm_regexp_token_buffer_push_byte(&token_buffer, '\r');
                                 break;
                             }
                         /* fallthrough */
@@ -10415,7 +10607,7 @@ parser_lex(pm_parser_t *parser) {
                                 // flush the heredoc and continue parsing after
                                 // heredoc_end.
                                 parser_flush_heredoc_end(parser);
-                                pm_token_buffer_copy(parser, &token_buffer);
+                                pm_regexp_token_buffer_copy(parser, &token_buffer);
                                 LEX(PM_TOKEN_STRING_CONTENT);
                             } else {
                                 // ... else track the newline.
@@ -10429,7 +10621,7 @@ parser_lex(pm_parser_t *parser) {
                         case 'M':
                         case 'u':
                         case 'x':
-                            escape_read(parser, &token_buffer.buffer, PM_ESCAPE_FLAG_REGEXP);
+                            escape_read(parser, &token_buffer.base.buffer, &token_buffer.regexp_buffer, PM_ESCAPE_FLAG_REGEXP);
                             break;
                         default:
                             if (lex_mode->as.regexp.terminator == peeked) {
@@ -10440,23 +10632,24 @@ parser_lex(pm_parser_t *parser) {
                                     case '$': case ')': case '*': case '+':
                                     case '.': case '>': case '?': case ']':
                                     case '^': case '|': case '}':
-                                        pm_token_buffer_push_byte(&token_buffer, '\\');
+                                        pm_regexp_token_buffer_push_byte(&token_buffer, '\\');
                                         break;
                                     default:
                                         break;
                                 }
 
-                                pm_token_buffer_push_byte(&token_buffer, peeked);
+                                pm_token_buffer_push_byte(&token_buffer.base, peeked);
+                                pm_regexp_token_buffer_push_byte(&token_buffer, peeked);
                                 parser->current.end++;
                                 break;
                             }
 
-                            if (peeked < 0x80) pm_token_buffer_push_byte(&token_buffer, '\\');
-                            pm_token_buffer_push_escaped(&token_buffer, parser);
+                            if (peeked < 0x80) pm_regexp_token_buffer_push_byte(&token_buffer, '\\');
+                            pm_regexp_token_buffer_push_escaped(&token_buffer, parser);
                             break;
                     }
 
-                    token_buffer.cursor = parser->current.end;
+                    token_buffer.base.cursor = parser->current.end;
                     breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, false);
                     continue;
                 }
@@ -10475,7 +10668,7 @@ parser_lex(pm_parser_t *parser) {
                     }
 
                     if (type == PM_TOKEN_STRING_CONTENT) {
-                        pm_token_buffer_flush(parser, &token_buffer);
+                        pm_regexp_token_buffer_flush(parser, &token_buffer);
                     }
 
                     LEX(type);
@@ -10491,14 +10684,14 @@ parser_lex(pm_parser_t *parser) {
             }
 
             if (parser->current.end > parser->current.start) {
-                pm_token_buffer_flush(parser, &token_buffer);
+                pm_regexp_token_buffer_flush(parser, &token_buffer);
                 LEX(PM_TOKEN_STRING_CONTENT);
             }
 
             // If we were unable to find a breakpoint, then this token hits the
             // end of the file.
             parser->current.end = parser->end;
-            pm_token_buffer_flush(parser, &token_buffer);
+            pm_regexp_token_buffer_flush(parser, &token_buffer);
             LEX(PM_TOKEN_STRING_CONTENT);
         }
         case PM_LEX_STRING: {
@@ -10525,7 +10718,7 @@ parser_lex(pm_parser_t *parser) {
 
             // If we haven't found an escape yet, then this buffer will be
             // unallocated since we can refer directly to the source string.
-            pm_token_buffer_t token_buffer = { { 0 }, 0 };
+            pm_token_buffer_t token_buffer = { 0 };
 
             while (breakpoint != NULL) {
                 // If we hit the incrementor, then we'll increment then nesting and
@@ -10660,7 +10853,7 @@ parser_lex(pm_parser_t *parser) {
                                     pm_token_buffer_push_byte(&token_buffer, peeked);
                                     parser->current.end++;
                                 } else if (lex_mode->as.string.interpolation) {
-                                    escape_read(parser, &token_buffer.buffer, PM_ESCAPE_FLAG_NONE);
+                                    escape_read(parser, &token_buffer.buffer, NULL, PM_ESCAPE_FLAG_NONE);
                                 } else {
                                     pm_token_buffer_push_byte(&token_buffer, '\\');
                                     pm_token_buffer_push_escaped(&token_buffer, parser);
@@ -10813,7 +11006,7 @@ parser_lex(pm_parser_t *parser) {
             }
 
             const uint8_t *breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
-            pm_token_buffer_t token_buffer = { { 0 }, 0 };
+            pm_token_buffer_t token_buffer = { 0 };
             bool was_line_continuation = false;
 
             while (breakpoint != NULL) {
@@ -10972,7 +11165,7 @@ parser_lex(pm_parser_t *parser) {
                                     breakpoint = parser->current.end;
                                     continue;
                                 default:
-                                    escape_read(parser, &token_buffer.buffer, PM_ESCAPE_FLAG_NONE);
+                                    escape_read(parser, &token_buffer.buffer, NULL, PM_ESCAPE_FLAG_NONE);
                                     break;
                             }
                         }
@@ -16948,10 +17141,14 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
                 };
 
                 parser_lex(parser);
-                return (pm_node_t *) pm_regular_expression_node_create(parser, &opening, &content, &parser->previous);
+
+                pm_node_t *node = (pm_node_t *) pm_regular_expression_node_create(parser, &opening, &content, &parser->previous);
+                pm_node_flag_set(node, PM_REGULAR_EXPRESSION_FLAGS_FORCED_US_ASCII_ENCODING);
+
+                return node;
             }
 
-            pm_interpolated_regular_expression_node_t *node;
+            pm_interpolated_regular_expression_node_t *interpolated;
 
             if (match1(parser, PM_TOKEN_STRING_CONTENT)) {
                 // In this case we've hit string content so we know the regular
@@ -16960,36 +17157,56 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
                 // regular expression) or if it's not then it has interpolation.
                 pm_string_t unescaped = parser->current_string;
                 pm_token_t content = parser->current;
+
+                pm_string_t source = parser->current_regular_expression_source;
+                pm_string_constant_init(&parser->current_regular_expression_source, "", 0);
+
                 parser_lex(parser);
 
                 // If we hit an end, then we can create a regular expression node
                 // without interpolation, which can be represented more succinctly and
                 // more easily compiled.
                 if (accept1(parser, PM_TOKEN_REGEXP_END)) {
-                    return (pm_node_t *) pm_regular_expression_node_create_unescaped(parser, &opening, &content, &parser->previous, &unescaped);
+                    pm_node_t *node = (pm_node_t *) pm_regular_expression_node_create_unescaped(parser, &opening, &content, &parser->previous, &source);
+                    pm_node_flag_set(node, parse_and_validate_regular_expression_encoding(parser, &source, &unescaped, node->flags));
+                    pm_string_free(&unescaped);
+                    return node;
                 }
 
                 // If we get here, then we have interpolation so we'll need to create
                 // a regular expression node with interpolation.
-                node = pm_interpolated_regular_expression_node_create(parser, &opening);
+                interpolated = pm_interpolated_regular_expression_node_create(parser, &opening);
 
                 pm_token_t opening = not_provided(parser);
                 pm_token_t closing = not_provided(parser);
-                pm_node_t *part = (pm_node_t *) pm_string_node_create_unescaped(parser, &opening, &parser->previous, &closing, &unescaped);
-                pm_interpolated_regular_expression_node_append(node, part);
+                pm_node_t *part = (pm_node_t *) pm_string_node_create_unescaped(parser, &opening, &parser->previous, &closing, &source);
+                pm_interpolated_regular_expression_node_append(interpolated, part);
+                pm_string_free(&unescaped);
             } else {
                 // If the first part of the body of the regular expression is not a
                 // string content, then we have interpolation and we need to create an
                 // interpolated regular expression node.
-                node = pm_interpolated_regular_expression_node_create(parser, &opening);
+                interpolated = pm_interpolated_regular_expression_node_create(parser, &opening);
             }
 
             // Now that we're here and we have interpolation, we'll parse all of the
             // parts into the list.
             pm_node_t *part;
             while (!match2(parser, PM_TOKEN_REGEXP_END, PM_TOKEN_EOF)) {
-                if ((part = parse_string_part(parser)) != NULL) {
-                    pm_interpolated_regular_expression_node_append(node, part);
+                if (match1(parser, PM_TOKEN_STRING_CONTENT)) {
+                    pm_token_t opening = not_provided(parser);
+                    pm_token_t closing = not_provided(parser);
+
+                    pm_node_t *node = (pm_node_t *) pm_string_node_create_unescaped(parser, &opening, &parser->current, &closing, &parser->current_regular_expression_source);
+                    pm_node_flag_set(node, parse_unescaped_encoding(parser));
+
+                    pm_string_free(&parser->current_string);
+                    pm_string_constant_init(&parser->current_regular_expression_source, "", 0);
+
+                    parser_lex(parser);
+                    pm_interpolated_regular_expression_node_append(interpolated, node);
+                } else if ((part = parse_string_part(parser)) != NULL) {
+                    pm_interpolated_regular_expression_node_append(interpolated, part);
                 }
             }
 
@@ -17000,9 +17217,9 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
             } else {
                 expect1(parser, PM_TOKEN_REGEXP_END, PM_ERR_REGEXP_TERM);
             }
-            pm_interpolated_regular_expression_node_closing_set(node, &closing);
 
-            return (pm_node_t *) node;
+            pm_interpolated_regular_expression_node_closing_set(interpolated, &closing);
+            return (pm_node_t *) interpolated;
         }
         case PM_TOKEN_BACKTICK:
         case PM_TOKEN_PERCENT_LOWER_X: {
@@ -18527,6 +18744,7 @@ pm_parser_init(pm_parser_t *parser, const uint8_t *source, size_t size, const pm
         .newline_list = { 0 },
         .integer_base = 0,
         .current_string = PM_STRING_EMPTY,
+        .current_regular_expression_source = PM_STRING_EMPTY,
         .start_line = 1,
         .explicit_encoding = NULL,
         .command_line = 0,
diff --git a/templates/src/diagnostic.c.erb b/templates/src/diagnostic.c.erb
index c8c35bd4..64e1a1d7 100644
--- a/templates/src/diagnostic.c.erb
+++ b/templates/src/diagnostic.c.erb
@@ -204,7 +204,9 @@ static const pm_diagnostic_data_t diagnostic_messages[PM_DIAGNOSTIC_ID_MAX] = {
     [PM_ERR_INVALID_NUMBER_OCTAL]               = { "invalid octal number", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_INVALID_NUMBER_UNDERSCORE]          = { "invalid underscore placement in number", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_INVALID_CHARACTER]                  = { "invalid character 0x%X", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_INVALID_MULTIBYTE_CHAR]             = { "invalid multibyte char (%s)", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_INVALID_MULTIBYTE_CHARACTER]        = { "invalid multibyte character 0x%X", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_INVALID_MULTIBYTE_ESCAPE]           = { "invalid multibyte escape: /%.*s/", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_INVALID_PRINTABLE_CHARACTER]        = { "invalid character `%c`", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_INVALID_PERCENT]                    = { "invalid `%` token", PM_ERROR_LEVEL_FATAL }, // TODO WHAT?
     [PM_ERR_INVALID_VARIABLE_GLOBAL_3_3_0]      = { "`%.*s' is not allowed as a global variable name", PM_ERROR_LEVEL_FATAL },
@@ -270,7 +272,12 @@ static const pm_diagnostic_data_t diagnostic_messages[PM_DIAGNOSTIC_ID_MAX] = {
     [PM_ERR_PATTERN_TERM_BRACKET]               = { "expected a `]` to close the pattern expression", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_PATTERN_TERM_PAREN]                 = { "expected a `)` to close the pattern expression", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_PIPEPIPEEQ_MULTI_ASSIGN]            = { "unexpected `||=` in a multiple assignment", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_REGEXP_ENCODING_OPTION_MISMATCH]    = { "regexp encoding option '%c' differs from source encoding '%s'", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_REGEXP_INCOMPAT_CHAR_ENCODING]      = { "incompatible character encoding: /%.*s/", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_REGEXP_NON_ESCAPED_MBC]             = { "/.../n has a non escaped non ASCII character in non ASCII-8BIT script: /%.*s/", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_REGEXP_INVALID_UNICODE_RANGE]       = { "invalid Unicode range: /%.*s/", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_REGEXP_TERM]                        = { "expected a closing delimiter for the regular expression", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_REGEXP_UTF8_CHAR_NON_UTF8_REGEXP]   = { "UTF-8 character in non UTF-8 regexp: /%s/", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_RESCUE_EXPRESSION]                  = { "expected a rescued expression", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_RESCUE_MODIFIER_VALUE]              = { "expected a value after the `rescue` modifier", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_RESCUE_TERM]                        = { "expected a closing delimiter for the `rescue` clause", PM_ERROR_LEVEL_FATAL },


Test patch:
diff --git a/test/prism/encoding_test.rb b/test/prism/encoding_test.rb
index 44491bf0..c23bb8c2 100644
--- a/test/prism/encoding_test.rb
+++ b/test/prism/encoding_test.rb
@@ -149,23 +149,42 @@ module Prism
     escapes = ["\\x00", "\\x7F", "\\x80", "\\xFF", "\\u{00}", "\\u{7F}", "\\u{80}", "\\M-\\C-?"]
     escapes = escapes.concat(escapes.product(escapes).map(&:join))
     symbols = [:a, :ą, :+]
+    regexps = [/a/, /ą/, //]
 
     encodings.each_key do |encoding|
       define_method(:"test_encoding_flags_#{encoding.name}") do
         assert_encoding_flags(encoding, escapes)
       end
-    end
 
-    encodings.each_key do |encoding|
       define_method(:"test_symbol_encoding_flags_#{encoding.name}") do
         assert_symbol_encoding_flags(encoding, symbols)
       end
-    end
 
-    encodings.each_key do |encoding|
       define_method(:"test_symbol_character_escape_encoding_flags_#{encoding.name}") do
         assert_symbol_character_escape_encoding_flags(encoding, escapes)
       end
+
+      define_method(:"test_regular_expression_encoding_flags_#{encoding.name}") do
+        assert_regular_expression_encoding_flags(encoding, regexps.map(&:inspect))
+      end
+
+      define_method(:"test_regular_expression_escape_encoding_flags_#{encoding.name}") do
+        assert_regular_expression_encoding_flags(encoding, escapes.map { |e| "/#{e}/" })
+      end
+    end
+
+    encoding_modifiers = { ascii_8bit: "n", utf_8: "u", euc_jp: "e", windows_31j: "s" }
+    regexp_sources = ["abc", "garçon", "\\x80", "gar\\xC3\\xA7on", "gar\\u{E7}on", "abc\\u{FFFFFF}", "\\x80\\u{80}" ]
+
+    encoding_modifiers.each_value do |modifier|
+      encodings.each_key do |encoding|
+        define_method(:"test_regular_expression_encoding_modifiers_/#{modifier}_#{encoding.name}") do
+          assert_regular_expression_encoding_flags(
+            encoding,
+            regexp_sources.product(encoding_modifiers.values).map { |r, modifier| "/#{r}/#{modifier}" }
+          )
+        end
+      end
     end
 
     def test_coding
@@ -454,5 +473,100 @@ module Prism
         assert_equal expected, actual
       end
     end
+
+    def assert_regular_expression_encoding_flags(encoding, regexps)
+      regexps.each do |regexp|
+        regexp_modifier_used = regexp.end_with?("/u") || regexp.end_with?("/e") || regexp.end_with?("/s") || regexp.end_with?("/n")
+        source = "# encoding: #{encoding.name}\n#{regexp}"
+
+        encoding_errors = ["invalid multibyte char", "escaped non ASCII character in UTF-8 regexp", "differs from source encoding"]
+        skipped_errors = ["invalid multibyte escape", "incompatible character encoding", "UTF-8 character in non UTF-8 regexp", "invalid Unicode range", "invalid Unicode list"]
+
+        # TODO (nirvdrum 21-Feb-2024): Prism currently does not handle Regexp validation unless modifiers are used. So, skip processing those errors for now: https://github.com/ruby/prism/issues/2104
+        unless regexp_modifier_used
+          skipped_errors += encoding_errors
+          encoding_errors.clear
+        end
+
+        expected =
+          begin
+            eval(source).encoding
+          rescue SyntaxError => error
+            if encoding_errors.find { |e| error.message.include?(e) }
+              error.message.split("\n").map { |m| m[/: (.+?)$/, 1] }
+            elsif skipped_errors.find { |e| error.message.include?(e) }
+              next
+            else
+              raise
+            end
+          end
+
+        actual =
+          Prism.parse(source).then do |result|
+            if result.success?
+              regexp = result.value.statements.body.first
+
+              actual_encoding = if regexp.forced_utf8_encoding?
+                Encoding::UTF_8
+              elsif regexp.forced_binary_encoding?
+                Encoding::ASCII_8BIT
+              elsif regexp.forced_us_ascii_encoding?
+                Encoding::US_ASCII
+              elsif regexp.ascii_8bit?
+                Encoding::ASCII_8BIT
+              elsif regexp.utf_8?
+                Encoding::UTF_8
+              elsif regexp.euc_jp?
+                Encoding::EUC_JP
+              elsif regexp.windows_31j?
+                Encoding::Windows_31J
+              else
+                encoding
+              end
+
+              if regexp.utf_8? && actual_encoding != Encoding::UTF_8
+                raise "expected regexp encoding to be UTF-8 due to '/u' modifier, but got #{actual_encoding.name}"
+              elsif regexp.ascii_8bit? && (actual_encoding != Encoding::ASCII_8BIT && actual_encoding != Encoding::US_ASCII)
+                raise "expected regexp encoding to be ASCII-8BIT or US-ASCII due to '/n' modifier, but got #{actual_encoding.name}"
+              elsif regexp.euc_jp? && actual_encoding != Encoding::EUC_JP
+                raise "expected regexp encoding to be EUC-JP due to '/e' modifier, but got #{actual_encoding.name}"
+              elsif regexp.windows_31j? && actual_encoding != Encoding::Windows_31J
+                raise "expected regexp encoding to be Windows-31J due to '/s' modifier, but got #{actual_encoding.name}"
+              end
+
+              if regexp.utf_8? && regexp.forced_utf8_encoding?
+                raise "the forced_utf8 flag should not be set when the UTF-8 modifier (/u) is used"
+              elsif regexp.ascii_8bit? && regexp.forced_binary_encoding?
+                raise "the forced_ascii_8bit flag should not be set when the UTF-8 modifier (/u) is used"
+              end
+
+              actual_encoding
+            else
+              errors = result.errors.map(&:message)
+
+              if errors.last&.include?("UTF-8 mixed within")
+                nil
+              else
+                errors
+              end
+            end
+          end
+
+        # TODO (nirvdrum 22-Feb-2024): Remove this workaround once Prism better maps CRuby's error messages.
+        # This class of error message is tricky. The part not being compared is a representation of the regexp.
+        # Depending on the source encoding and any encoding modifiers being used, CRuby alters how the regexp is represented.
+        # Sometimes it's an MBC string. Other times it uses hexadecimal character escapes. And in other cases it uses
+        # the long-form Unicode escape sequences. This short-circuit checks that the error message is mostly correct.
+        if expected.is_a?(Array) && actual.is_a?(Array)
+          if expected.last.start_with?("/.../n has a non escaped non ASCII character in non ASCII-8BIT script:") &&
+              actual.last.start_with?("/.../n has a non escaped non ASCII character in non ASCII-8BIT script:")
+              expected.last.clear
+              actual.last.clear
+          end
+        end
+
+        assert_equal expected, actual
+      end
+    end
   end
 end
diff --git a/test/prism/snapshots/heredoc_with_escaped_newline_at_start.txt b/test/prism/snapshots/heredoc_with_escaped_newline_at_start.txt
index 5d17559e..acc6b082 100644
--- a/test/prism/snapshots/heredoc_with_escaped_newline_at_start.txt
+++ b/test/prism/snapshots/heredoc_with_escaped_newline_at_start.txt
@@ -21,7 +21,7 @@
         │   │   ├── flags: ∅
         │   │   └── arguments: (length: 2)
         │   │       ├── @ RegularExpressionNode (location: (1,15)-(1,21))
-        │   │       │   ├── flags: ∅
+        │   │       │   ├── flags: forced_us_ascii_encoding
         │   │       │   ├── opening_loc: (1,15)-(1,16) = "/"
         │   │       │   ├── content_loc: (1,16)-(1,20) = "^\\s{"
         │   │       │   ├── closing_loc: (1,20)-(1,21) = "/"
@@ -52,7 +52,7 @@
             │   ├── flags: ∅
             │   └── arguments: (length: 2)
             │       ├── @ RegularExpressionNode (location: (5,15)-(5,21))
-            │       │   ├── flags: ∅
+            │       │   ├── flags: forced_us_ascii_encoding
             │       │   ├── opening_loc: (5,15)-(5,16) = "/"
             │       │   ├── content_loc: (5,16)-(5,20) = "^\\s{"
             │       │   ├── closing_loc: (5,20)-(5,21) = "/"
diff --git a/test/prism/snapshots/newline_terminated.txt b/test/prism/snapshots/newline_terminated.txt
index 496e86fb..6a3b28db 100644
--- a/test/prism/snapshots/newline_terminated.txt
+++ b/test/prism/snapshots/newline_terminated.txt
@@ -100,7 +100,7 @@
         │   ├── closing_loc: (37,3)-(38,0) = "\n"
         │   └── unescaped: "foo"
         └── @ RegularExpressionNode (location: (39,0)-(41,0))
-            ├── flags: ∅
+            ├── flags: forced_us_ascii_encoding
             ├── opening_loc: (39,0)-(40,0) = "%r\n"
             ├── content_loc: (40,0)-(40,3) = "foo"
             ├── closing_loc: (40,3)-(41,0) = "\n"
diff --git a/test/prism/snapshots/patterns.txt b/test/prism/snapshots/patterns.txt
index 96205349..4efd3159 100644
--- a/test/prism/snapshots/patterns.txt
+++ b/test/prism/snapshots/patterns.txt
@@ -165,7 +165,7 @@
         │   │   └── block: ∅
         │   ├── pattern:
         │   │   @ RegularExpressionNode (location: (9,7)-(9,12))
-        │   │   ├── flags: ∅
+        │   │   ├── flags: forced_us_ascii_encoding
         │   │   ├── opening_loc: (9,7)-(9,8) = "/"
         │   │   ├── content_loc: (9,8)-(9,11) = "foo"
         │   │   ├── closing_loc: (9,11)-(9,12) = "/"
@@ -719,14 +719,14 @@
         │   │   ├── flags: ∅
         │   │   ├── left:
         │   │   │   @ RegularExpressionNode (location: (35,7)-(35,12))
-        │   │   │   ├── flags: ∅
+        │   │   │   ├── flags: forced_us_ascii_encoding
         │   │   │   ├── opening_loc: (35,7)-(35,8) = "/"
         │   │   │   ├── content_loc: (35,8)-(35,11) = "foo"
         │   │   │   ├── closing_loc: (35,11)-(35,12) = "/"
         │   │   │   └── unescaped: "foo"
         │   │   ├── right:
         │   │   │   @ RegularExpressionNode (location: (35,16)-(35,21))
-        │   │   │   ├── flags: ∅
+        │   │   │   ├── flags: forced_us_ascii_encoding
         │   │   │   ├── opening_loc: (35,16)-(35,17) = "/"
         │   │   │   ├── content_loc: (35,17)-(35,20) = "foo"
         │   │   │   ├── closing_loc: (35,20)-(35,21) = "/"
@@ -2543,7 +2543,7 @@
         │   │   └── block: ∅
         │   ├── pattern:
         │   │   @ RegularExpressionNode (location: (112,7)-(112,12))
-        │   │   ├── flags: ∅
+        │   │   ├── flags: forced_us_ascii_encoding
         │   │   ├── opening_loc: (112,7)-(112,8) = "/"
         │   │   ├── content_loc: (112,8)-(112,11) = "foo"
         │   │   ├── closing_loc: (112,11)-(112,12) = "/"
@@ -3126,7 +3126,7 @@
         │   │   └── @ InNode (location: (143,10)-(143,23))
         │   │       ├── pattern:
         │   │       │   @ RegularExpressionNode (location: (143,13)-(143,18))
-        │   │       │   ├── flags: ∅
+        │   │       │   ├── flags: forced_us_ascii_encoding
         │   │       │   ├── opening_loc: (143,13)-(143,14) = "/"
         │   │       │   ├── content_loc: (143,14)-(143,17) = "foo"
         │   │       │   ├── closing_loc: (143,17)-(143,18) = "/"
@@ -3914,7 +3914,7 @@
         │   │       │   │   @ StatementsNode (location: (170,13)-(170,18))
         │   │       │   │   └── body: (length: 1)
         │   │       │   │       └── @ RegularExpressionNode (location: (170,13)-(170,18))
-        │   │       │   │           ├── flags: ∅
+        │   │       │   │           ├── flags: forced_us_ascii_encoding
         │   │       │   │           ├── opening_loc: (170,13)-(170,14) = "/"
         │   │       │   │           ├── content_loc: (170,14)-(170,17) = "foo"
         │   │       │   │           ├── closing_loc: (170,17)-(170,18) = "/"
diff --git a/test/prism/snapshots/regex.txt b/test/prism/snapshots/regex.txt
index 8f3e2595..9e19bbb1 100644
--- a/test/prism/snapshots/regex.txt
+++ b/test/prism/snapshots/regex.txt
@@ -15,7 +15,7 @@
         │   │   ├── flags: ∅
         │   │   └── arguments: (length: 1)
         │   │       └── @ RegularExpressionNode (location: (1,4)-(1,9))
-        │   │           ├── flags: ∅
+        │   │           ├── flags: forced_us_ascii_encoding
         │   │           ├── opening_loc: (1,4)-(1,5) = "/"
         │   │           ├── content_loc: (1,5)-(1,8) = "bar"
         │   │           ├── closing_loc: (1,8)-(1,9) = "/"
@@ -23,13 +23,13 @@
         │   ├── closing_loc: ∅
         │   └── block: ∅
         ├── @ RegularExpressionNode (location: (3,0)-(3,8))
-        │   ├── flags: ignore_case
+        │   ├── flags: ignore_case, forced_us_ascii_encoding
         │   ├── opening_loc: (3,0)-(3,3) = "%r{"
         │   ├── content_loc: (3,3)-(3,6) = "abc"
         │   ├── closing_loc: (3,6)-(3,8) = "}i"
         │   └── unescaped: "abc"
         ├── @ RegularExpressionNode (location: (5,0)-(5,5))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (5,0)-(5,1) = "/"
         │   ├── content_loc: (5,1)-(5,4) = "a\\b"
         │   ├── closing_loc: (5,4)-(5,5) = "/"
@@ -92,7 +92,7 @@
         │   │   │   │   ├── flags: ∅
         │   │   │   │   ├── receiver:
         │   │   │   │   │   @ RegularExpressionNode (location: (11,1)-(11,14))
-        │   │   │   │   │   ├── flags: ∅
+        │   │   │   │   │   ├── flags: forced_us_ascii_encoding
         │   │   │   │   │   ├── opening_loc: (11,1)-(11,2) = "/"
         │   │   │   │   │   ├── content_loc: (11,2)-(11,13) = "(?<foo>bar)"
         │   │   │   │   │   ├── closing_loc: (11,13)-(11,14) = "/"
@@ -127,31 +127,31 @@
         │   ├── opening_loc: (11,0)-(11,1) = "["
         │   └── closing_loc: (11,26)-(11,27) = "]"
         ├── @ RegularExpressionNode (location: (13,0)-(13,6))
-        │   ├── flags: ignore_case
+        │   ├── flags: ignore_case, forced_us_ascii_encoding
         │   ├── opening_loc: (13,0)-(13,1) = "/"
         │   ├── content_loc: (13,1)-(13,4) = "abc"
         │   ├── closing_loc: (13,4)-(13,6) = "/i"
         │   └── unescaped: "abc"
         ├── @ RegularExpressionNode (location: (15,0)-(15,26))
-        │   ├── flags: ignore_case
+        │   ├── flags: ignore_case, forced_us_ascii_encoding
         │   ├── opening_loc: (15,0)-(15,3) = "%r/"
         │   ├── content_loc: (15,3)-(15,24) = "[a-z$._?][\\w$.?\#@~]*:"
         │   ├── closing_loc: (15,24)-(15,26) = "/i"
         │   └── unescaped: "[a-z$._?][\\w$.?\#@~]*:"
         ├── @ RegularExpressionNode (location: (17,0)-(17,37))
-        │   ├── flags: ignore_case
+        │   ├── flags: ignore_case, forced_us_ascii_encoding
         │   ├── opening_loc: (17,0)-(17,3) = "%r/"
         │   ├── content_loc: (17,3)-(17,35) = "([a-z$._?][\\w$.?\#@~]*)(\\s+)(equ)"
         │   ├── closing_loc: (17,35)-(17,37) = "/i"
         │   └── unescaped: "([a-z$._?][\\w$.?\#@~]*)(\\s+)(equ)"
         ├── @ RegularExpressionNode (location: (19,0)-(19,25))
-        │   ├── flags: ignore_case
+        │   ├── flags: ignore_case, forced_us_ascii_encoding
         │   ├── opening_loc: (19,0)-(19,3) = "%r/"
         │   ├── content_loc: (19,3)-(19,23) = "[a-z$._?][\\w$.?\#@~]*"
         │   ├── closing_loc: (19,23)-(19,25) = "/i"
         │   └── unescaped: "[a-z$._?][\\w$.?\#@~]*"
         ├── @ RegularExpressionNode (location: (21,0)-(24,1))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (21,0)-(21,3) = "%r("
         │   ├── content_loc: (21,3)-(24,0) = "\n(?:[\\w\#$%_']|\\(\\)|\\(,\\)|\\[\\]|[0-9])*\n  (?:[\\w\#$%_']+)\n"
         │   ├── closing_loc: (24,0)-(24,1) = ")"
@@ -160,7 +160,7 @@
         │   ├── flags: ∅
         │   ├── receiver:
         │   │   @ RegularExpressionNode (location: (26,0)-(26,8))
-        │   │   ├── flags: ∅
+        │   │   ├── flags: forced_us_ascii_encoding
         │   │   ├── opening_loc: (26,0)-(26,1) = "/"
         │   │   ├── content_loc: (26,1)-(26,7) = "(?#\\))"
         │   │   ├── closing_loc: (26,7)-(26,8) = "/"
@@ -182,7 +182,7 @@
         │   ├── closing_loc: ∅
         │   └── block: ∅
         ├── @ RegularExpressionNode (location: (28,0)-(28,9))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (28,0)-(28,3) = "%r#"
         │   ├── content_loc: (28,3)-(28,8) = "pound"
         │   ├── closing_loc: (28,8)-(28,9) = "#"
@@ -220,7 +220,7 @@
         │   │   ├── flags: ∅
         │   │   ├── receiver:
         │   │   │   @ RegularExpressionNode (location: (32,0)-(33,4))
-        │   │   │   ├── flags: ∅
+        │   │   │   ├── flags: forced_us_ascii_encoding
         │   │   │   ├── opening_loc: (32,0)-(32,1) = "/"
         │   │   │   ├── content_loc: (32,1)-(33,3) = "(?<a\\\nb>)"
         │   │   │   ├── closing_loc: (33,3)-(33,4) = "/"
@@ -254,7 +254,7 @@
         │   │   ├── flags: ∅
         │   │   ├── receiver:
         │   │   │   @ RegularExpressionNode (location: (35,0)-(35,18))
-        │   │   │   ├── flags: ∅
+        │   │   │   ├── flags: forced_us_ascii_encoding
         │   │   │   ├── opening_loc: (35,0)-(35,1) = "/"
         │   │   │   ├── content_loc: (35,1)-(35,17) = "(?<abc>)(?<abc>)"
         │   │   │   ├── closing_loc: (35,17)-(35,18) = "/"
@@ -286,7 +286,7 @@
         │   ├── flags: ∅
         │   ├── receiver:
         │   │   @ RegularExpressionNode (location: (37,0)-(37,10))
-        │   │   ├── flags: ∅
+        │   │   ├── flags: forced_us_ascii_encoding
         │   │   ├── opening_loc: (37,0)-(37,1) = "/"
         │   │   ├── content_loc: (37,1)-(37,9) = "(?<a b>)"
         │   │   ├── closing_loc: (37,9)-(37,10) = "/"
@@ -338,7 +338,7 @@
                 │           │   ├── flags: ∅
                 │           │   ├── receiver:
                 │           │   │   @ RegularExpressionNode (location: (40,6)-(40,14))
-                │           │   │   ├── flags: ∅
+                │           │   │   ├── flags: forced_us_ascii_encoding
                 │           │   │   ├── opening_loc: (40,6)-(40,7) = "/"
                 │           │   │   ├── content_loc: (40,7)-(40,13) = "(?<a>)"
                 │           │   │   ├── closing_loc: (40,13)-(40,14) = "/"
diff --git a/test/prism/snapshots/seattlerb/TestRubyParserShared.txt b/test/prism/snapshots/seattlerb/TestRubyParserShared.txt
index 4a2a48b7..fabc92e4 100644
--- a/test/prism/snapshots/seattlerb/TestRubyParserShared.txt
+++ b/test/prism/snapshots/seattlerb/TestRubyParserShared.txt
@@ -70,7 +70,7 @@
         │   ├── opening_loc: (26,0)-(26,3) = "%i["
         │   └── closing_loc: (29,0)-(29,1) = "]"
         ├── @ RegularExpressionNode (location: (31,0)-(34,1))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (31,0)-(31,3) = "%r["
         │   ├── content_loc: (31,3)-(34,0) = "\n\n\n"
         │   ├── closing_loc: (34,0)-(34,1) = "]"
diff --git a/test/prism/snapshots/seattlerb/bug190.txt b/test/prism/snapshots/seattlerb/bug190.txt
index f7eaefa5..b261a166 100644
--- a/test/prism/snapshots/seattlerb/bug190.txt
+++ b/test/prism/snapshots/seattlerb/bug190.txt
@@ -4,7 +4,7 @@
     @ StatementsNode (location: (1,0)-(1,6))
     └── body: (length: 1)
         └── @ RegularExpressionNode (location: (1,0)-(1,6))
-            ├── flags: ∅
+            ├── flags: forced_us_ascii_encoding
             ├── opening_loc: (1,0)-(1,3) = "%r'"
             ├── content_loc: (1,3)-(1,5) = "\\'"
             ├── closing_loc: (1,5)-(1,6) = "'"
diff --git a/test/prism/snapshots/seattlerb/bug_case_when_regexp.txt b/test/prism/snapshots/seattlerb/bug_case_when_regexp.txt
index f6a6f41c..0cc1ca05 100644
--- a/test/prism/snapshots/seattlerb/bug_case_when_regexp.txt
+++ b/test/prism/snapshots/seattlerb/bug_case_when_regexp.txt
@@ -16,7 +16,7 @@
             │       ├── keyword_loc: (1,9)-(1,13) = "when"
             │       ├── conditions: (length: 1)
             │       │   └── @ RegularExpressionNode (location: (1,14)-(1,17))
-            │       │       ├── flags: ∅
+            │       │       ├── flags: forced_us_ascii_encoding
             │       │       ├── opening_loc: (1,14)-(1,15) = "/"
             │       │       ├── content_loc: (1,15)-(1,16) = "x"
             │       │       ├── closing_loc: (1,16)-(1,17) = "/"
diff --git a/test/prism/snapshots/seattlerb/bug_cond_pct.txt b/test/prism/snapshots/seattlerb/bug_cond_pct.txt
index 73cb18f5..cbf3bc3e 100644
--- a/test/prism/snapshots/seattlerb/bug_cond_pct.txt
+++ b/test/prism/snapshots/seattlerb/bug_cond_pct.txt
@@ -10,7 +10,7 @@
             │       ├── keyword_loc: (1,6)-(1,10) = "when"
             │       ├── conditions: (length: 1)
             │       │   └── @ RegularExpressionNode (location: (1,11)-(1,23))
-            │       │       ├── flags: ∅
+            │       │       ├── flags: forced_us_ascii_encoding
             │       │       ├── opening_loc: (1,11)-(1,14) = "%r%"
             │       │       ├── content_loc: (1,14)-(1,22) = "blahblah"
             │       │       ├── closing_loc: (1,22)-(1,23) = "%"
diff --git a/test/prism/snapshots/seattlerb/case_in.txt b/test/prism/snapshots/seattlerb/case_in.txt
index 6a1cc56d..e7e291c6 100644
--- a/test/prism/snapshots/seattlerb/case_in.txt
+++ b/test/prism/snapshots/seattlerb/case_in.txt
@@ -338,7 +338,7 @@
         │   │   └── @ InNode (location: (46,0)-(46,11))
         │   │       ├── pattern:
         │   │       │   @ RegularExpressionNode (location: (46,3)-(46,11))
-        │   │       │   ├── flags: ∅
+        │   │       │   ├── flags: forced_us_ascii_encoding
         │   │       │   ├── opening_loc: (46,3)-(46,4) = "/"
         │   │       │   ├── content_loc: (46,4)-(46,10) = "regexp"
         │   │       │   ├── closing_loc: (46,10)-(46,11) = "/"
diff --git a/test/prism/snapshots/seattlerb/regexp.txt b/test/prism/snapshots/seattlerb/regexp.txt
index abe17918..06cf9926 100644
--- a/test/prism/snapshots/seattlerb/regexp.txt
+++ b/test/prism/snapshots/seattlerb/regexp.txt
@@ -4,31 +4,31 @@
     @ StatementsNode (location: (1,0)-(9,13))
     └── body: (length: 5)
         ├── @ RegularExpressionNode (location: (1,0)-(1,5))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (1,0)-(1,1) = "/"
         │   ├── content_loc: (1,1)-(1,4) = "wtf"
         │   ├── closing_loc: (1,4)-(1,5) = "/"
         │   └── unescaped: "wtf"
         ├── @ RegularExpressionNode (location: (3,0)-(3,6))
-        │   ├── flags: multi_line
+        │   ├── flags: multi_line, forced_us_ascii_encoding
         │   ├── opening_loc: (3,0)-(3,1) = "/"
         │   ├── content_loc: (3,1)-(3,4) = "wtf"
         │   ├── closing_loc: (3,4)-(3,6) = "/m"
         │   └── unescaped: "wtf"
         ├── @ RegularExpressionNode (location: (5,0)-(5,6))
-        │   ├── flags: ascii_8bit
+        │   ├── flags: ascii_8bit, forced_us_ascii_encoding
         │   ├── opening_loc: (5,0)-(5,1) = "/"
         │   ├── content_loc: (5,1)-(5,4) = "wtf"
         │   ├── closing_loc: (5,4)-(5,6) = "/n"
         │   └── unescaped: "wtf"
         ├── @ RegularExpressionNode (location: (7,0)-(7,7))
-        │   ├── flags: multi_line, ascii_8bit
+        │   ├── flags: multi_line, ascii_8bit, forced_us_ascii_encoding
         │   ├── opening_loc: (7,0)-(7,1) = "/"
         │   ├── content_loc: (7,1)-(7,4) = "wtf"
         │   ├── closing_loc: (7,4)-(7,7) = "/nm"
         │   └── unescaped: "wtf"
         └── @ RegularExpressionNode (location: (9,0)-(9,13))
-            ├── flags: multi_line, ascii_8bit
+            ├── flags: multi_line, ascii_8bit, forced_us_ascii_encoding
             ├── opening_loc: (9,0)-(9,1) = "/"
             ├── content_loc: (9,1)-(9,4) = "wtf"
             ├── closing_loc: (9,4)-(9,13) = "/nmnmnmnm"
diff --git a/test/prism/snapshots/seattlerb/regexp_esc_C_slash.txt b/test/prism/snapshots/seattlerb/regexp_esc_C_slash.txt
index b6bf2426..4dbedc44 100644
--- a/test/prism/snapshots/seattlerb/regexp_esc_C_slash.txt
+++ b/test/prism/snapshots/seattlerb/regexp_esc_C_slash.txt
@@ -4,7 +4,7 @@
     @ StatementsNode (location: (1,0)-(1,7))
     └── body: (length: 1)
         └── @ RegularExpressionNode (location: (1,0)-(1,7))
-            ├── flags: ∅
+            ├── flags: forced_us_ascii_encoding
             ├── opening_loc: (1,0)-(1,1) = "/"
             ├── content_loc: (1,1)-(1,6) = "\\cC\\d"
             ├── closing_loc: (1,6)-(1,7) = "/"
diff --git a/test/prism/snapshots/seattlerb/regexp_esc_u.txt b/test/prism/snapshots/seattlerb/regexp_esc_u.txt
index 7b1ebdc6..bca451eb 100644
--- a/test/prism/snapshots/seattlerb/regexp_esc_u.txt
+++ b/test/prism/snapshots/seattlerb/regexp_esc_u.txt
@@ -4,7 +4,7 @@
     @ StatementsNode (location: (1,0)-(1,17))
     └── body: (length: 1)
         └── @ RegularExpressionNode (location: (1,0)-(1,17))
-            ├── flags: ∅
+            ├── flags: forced_us_ascii_encoding
             ├── opening_loc: (1,0)-(1,1) = "/"
             ├── content_loc: (1,1)-(1,16) = "[\\u0021-\\u0027]"
             ├── closing_loc: (1,16)-(1,17) = "/"
diff --git a/test/prism/snapshots/seattlerb/regexp_unicode_curlies.txt b/test/prism/snapshots/seattlerb/regexp_unicode_curlies.txt
index 8dd265af..487161b4 100644
--- a/test/prism/snapshots/seattlerb/regexp_unicode_curlies.txt
+++ b/test/prism/snapshots/seattlerb/regexp_unicode_curlies.txt
@@ -4,13 +4,13 @@
     @ StatementsNode (location: (1,0)-(3,8))
     └── body: (length: 2)
         ├── @ RegularExpressionNode (location: (1,0)-(1,15))
-        │   ├── flags: ∅
+        │   ├── flags: forced_utf8_encoding
         │   ├── opening_loc: (1,0)-(1,1) = "/"
         │   ├── content_loc: (1,1)-(1,14) = "\\u{c0de babe}"
         │   ├── closing_loc: (1,14)-(1,15) = "/"
         │   └── unescaped: "\\u{c0de babe}"
         └── @ RegularExpressionNode (location: (3,0)-(3,8))
-            ├── flags: ∅
+            ├── flags: forced_utf8_encoding
             ├── opening_loc: (3,0)-(3,1) = "/"
             ├── content_loc: (3,1)-(3,7) = "\\u{df}"
             ├── closing_loc: (3,7)-(3,8) = "/"
diff --git a/test/prism/snapshots/spanning_heredoc_newlines.txt b/test/prism/snapshots/spanning_heredoc_newlines.txt
index 171b0ff9..e3609ddb 100644
--- a/test/prism/snapshots/spanning_heredoc_newlines.txt
+++ b/test/prism/snapshots/spanning_heredoc_newlines.txt
@@ -46,7 +46,7 @@
         │   │   ├── flags: ∅
         │   │   └── arguments: (length: 1)
         │   │       └── @ RegularExpressionNode (location: (5,4)-(8,0))
-        │   │           ├── flags: ∅
+        │   │           ├── flags: forced_us_ascii_encoding
         │   │           ├── opening_loc: (5,4)-(6,0) = "%r\n"
         │   │           ├── content_loc: (6,0)-(6,0) = ""
         │   │           ├── closing_loc: (7,0)-(8,0) = "\n"
diff --git a/test/prism/snapshots/unescaping.txt b/test/prism/snapshots/unescaping.txt
index 00c5f59c..456ef226 100644
--- a/test/prism/snapshots/unescaping.txt
+++ b/test/prism/snapshots/unescaping.txt
@@ -15,7 +15,7 @@
         │   ├── opening_loc: (1,0)-(1,1) = "["
         │   └── closing_loc: (1,9)-(1,10) = "]"
         ├── @ RegularExpressionNode (location: (3,0)-(3,8))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (3,0)-(3,1) = "/"
         │   ├── content_loc: (3,1)-(3,7) = "\\c\#{1}"
         │   ├── closing_loc: (3,7)-(3,8) = "/"
diff --git a/test/prism/snapshots/unparser/corpus/literal/if.txt b/test/prism/snapshots/unparser/corpus/literal/if.txt
index 6a78779d..00eeba17 100644
--- a/test/prism/snapshots/unparser/corpus/literal/if.txt
+++ b/test/prism/snapshots/unparser/corpus/literal/if.txt
@@ -7,7 +7,7 @@
         │   ├── if_keyword_loc: (1,0)-(1,2) = "if"
         │   ├── predicate:
         │   │   @ MatchLastLineNode (location: (1,3)-(1,8))
-        │   │   ├── flags: ∅
+        │   │   ├── flags: forced_us_ascii_encoding
         │   │   ├── opening_loc: (1,3)-(1,4) = "/"
         │   │   ├── content_loc: (1,4)-(1,7) = "foo"
         │   │   ├── closing_loc: (1,7)-(1,8) = "/"
diff --git a/test/prism/snapshots/unparser/corpus/literal/literal.txt b/test/prism/snapshots/unparser/corpus/literal/literal.txt
index 8ed1bf5f..ba7dd70b 100644
--- a/test/prism/snapshots/unparser/corpus/literal/literal.txt
+++ b/test/prism/snapshots/unparser/corpus/literal/literal.txt
@@ -566,13 +566,13 @@
         │   ├── closing_loc: (48,2)-(48,3) = "\""
         │   └── unescaped: ""
         ├── @ RegularExpressionNode (location: (49,0)-(49,5))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (49,0)-(49,1) = "/"
         │   ├── content_loc: (49,1)-(49,4) = "foo"
         │   ├── closing_loc: (49,4)-(49,5) = "/"
         │   └── unescaped: "foo"
         ├── @ RegularExpressionNode (location: (50,0)-(50,28))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (50,0)-(50,1) = "/"
         │   ├── content_loc: (50,1)-(50,27) = "[^-+',.\\/:@[:alnum:]\\[\\]]+"
         │   ├── closing_loc: (50,27)-(50,28) = "/"
@@ -633,25 +633,25 @@
         │   │       └── closing_loc: (53,11)-(53,12) = "}"
         │   └── closing_loc: (53,12)-(53,13) = "/"
         ├── @ RegularExpressionNode (location: (54,0)-(54,4))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (54,0)-(54,1) = "/"
         │   ├── content_loc: (54,1)-(54,3) = "\\n"
         │   ├── closing_loc: (54,3)-(54,4) = "/"
         │   └── unescaped: "\\n"
         ├── @ RegularExpressionNode (location: (55,0)-(55,4))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (55,0)-(55,1) = "/"
         │   ├── content_loc: (55,1)-(55,3) = "\\n"
         │   ├── closing_loc: (55,3)-(55,4) = "/"
         │   └── unescaped: "\\n"
         ├── @ RegularExpressionNode (location: (56,0)-(56,5))
-        │   ├── flags: extended
+        │   ├── flags: extended, forced_us_ascii_encoding
         │   ├── opening_loc: (56,0)-(56,1) = "/"
         │   ├── content_loc: (56,1)-(56,3) = "\\n"
         │   ├── closing_loc: (56,3)-(56,5) = "/x"
         │   └── unescaped: "\\n"
         ├── @ RegularExpressionNode (location: (57,0)-(57,7))
-        │   ├── flags: extended
+        │   ├── flags: extended, forced_us_ascii_encoding
         │   ├── opening_loc: (57,0)-(57,1) = "/"
         │   ├── content_loc: (57,1)-(57,5) = "\\/\\/"
         │   ├── closing_loc: (57,5)-(57,7) = "/x"
diff --git a/test/prism/snapshots/unparser/corpus/literal/send.txt b/test/prism/snapshots/unparser/corpus/literal/send.txt
index 2fa4fd62..b7eb0647 100644
--- a/test/prism/snapshots/unparser/corpus/literal/send.txt
+++ b/test/prism/snapshots/unparser/corpus/literal/send.txt
@@ -425,7 +425,7 @@
         │   │   │           ├── flags: ∅
         │   │   │           ├── receiver:
         │   │   │           │   @ RegularExpressionNode (location: (37,1)-(37,6))
-        │   │   │           │   ├── flags: ∅
+        │   │   │           │   ├── flags: forced_us_ascii_encoding
         │   │   │           │   ├── opening_loc: (37,1)-(37,2) = "/"
         │   │   │           │   ├── content_loc: (37,2)-(37,5) = "bar"
         │   │   │           │   ├── closing_loc: (37,5)-(37,6) = "/"
@@ -511,7 +511,7 @@
         │   │   │           │   ├── flags: ∅
         │   │   │           │   └── arguments: (length: 1)
         │   │   │           │       └── @ RegularExpressionNode (location: (39,8)-(39,13))
-        │   │   │           │           ├── flags: ∅
+        │   │   │           │           ├── flags: forced_us_ascii_encoding
         │   │   │           │           ├── opening_loc: (39,8)-(39,9) = "/"
         │   │   │           │           ├── content_loc: (39,9)-(39,12) = "bar"
         │   │   │           │           ├── closing_loc: (39,12)-(39,13) = "/"
@@ -531,7 +531,7 @@
         │   ├── flags: ∅
         │   ├── receiver:
         │   │   @ RegularExpressionNode (location: (40,0)-(40,5))
-        │   │   ├── flags: ∅
+        │   │   ├── flags: forced_us_ascii_encoding
         │   │   ├── opening_loc: (40,0)-(40,1) = "/"
         │   │   ├── content_loc: (40,1)-(40,4) = "bar"
         │   │   ├── closing_loc: (40,4)-(40,5) = "/"
@@ -556,7 +556,7 @@
         │   ├── flags: ∅
         │   ├── receiver:
         │   │   @ RegularExpressionNode (location: (41,0)-(41,5))
-        │   │   ├── flags: ∅
+        │   │   ├── flags: forced_us_ascii_encoding
         │   │   ├── opening_loc: (41,0)-(41,1) = "/"
         │   │   ├── content_loc: (41,1)-(41,4) = "bar"
         │   │   ├── closing_loc: (41,4)-(41,5) = "/"
@@ -758,7 +758,7 @@
         │   │   ├── flags: ∅
         │   │   └── arguments: (length: 1)
         │   │       └── @ RegularExpressionNode (location: (49,7)-(49,12))
-        │   │           ├── flags: ∅
+        │   │           ├── flags: forced_us_ascii_encoding
         │   │           ├── opening_loc: (49,7)-(49,8) = "/"
         │   │           ├── content_loc: (49,8)-(49,11) = "bar"
         │   │           ├── closing_loc: (49,11)-(49,12) = "/"
@@ -1007,7 +1007,7 @@
         │   │           │   ├── flags: ∅
         │   │           │   └── arguments: (length: 1)
         │   │           │       └── @ RegularExpressionNode (location: (57,11)-(57,16))
-        │   │           │           ├── flags: ∅
+        │   │           │           ├── flags: forced_us_ascii_encoding
         │   │           │           ├── opening_loc: (57,11)-(57,12) = "/"
         │   │           │           ├── content_loc: (57,12)-(57,15) = "bar"
         │   │           │           ├── closing_loc: (57,15)-(57,16) = "/"
diff --git a/test/prism/snapshots/unparser/corpus/semantic/literal.txt b/test/prism/snapshots/unparser/corpus/semantic/literal.txt
index 7f76e2f5..59e02be6 100644
--- a/test/prism/snapshots/unparser/corpus/semantic/literal.txt
+++ b/test/prism/snapshots/unparser/corpus/semantic/literal.txt
@@ -31,13 +31,13 @@
         │   ├── closing_loc: ∅
         │   └── unescaped: "c"
         ├── @ RegularExpressionNode (location: (9,0)-(9,5))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (9,0)-(9,3) = "%r("
         │   ├── content_loc: (9,3)-(9,4) = "/"
         │   ├── closing_loc: (9,4)-(9,5) = ")"
         │   └── unescaped: "/"
         ├── @ RegularExpressionNode (location: (10,0)-(10,6))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (10,0)-(10,3) = "%r("
         │   ├── content_loc: (10,3)-(10,5) = "\\)"
         │   ├── closing_loc: (10,5)-(10,6) = ")"
diff --git a/test/prism/snapshots/whitequark/bug_regex_verification.txt b/test/prism/snapshots/whitequark/bug_regex_verification.txt
index 5ca85e34..4464b66e 100644
--- a/test/prism/snapshots/whitequark/bug_regex_verification.txt
+++ b/test/prism/snapshots/whitequark/bug_regex_verification.txt
@@ -4,7 +4,7 @@
     @ StatementsNode (location: (1,0)-(1,5))
     └── body: (length: 1)
         └── @ RegularExpressionNode (location: (1,0)-(1,5))
-            ├── flags: extended
+            ├── flags: extended, forced_us_ascii_encoding
             ├── opening_loc: (1,0)-(1,1) = "/"
             ├── content_loc: (1,1)-(1,3) = "#)"
             ├── closing_loc: (1,3)-(1,5) = "/x"
diff --git a/test/prism/snapshots/whitequark/cond_match_current_line.txt b/test/prism/snapshots/whitequark/cond_match_current_line.txt
index fb5ff33e..700d0966 100644
--- a/test/prism/snapshots/whitequark/cond_match_current_line.txt
+++ b/test/prism/snapshots/whitequark/cond_match_current_line.txt
@@ -7,7 +7,7 @@
         │   ├── flags: ∅
         │   ├── receiver:
         │   │   @ MatchLastLineNode (location: (1,1)-(1,6))
-        │   │   ├── flags: ∅
+        │   │   ├── flags: forced_us_ascii_encoding
         │   │   ├── opening_loc: (1,1)-(1,2) = "/"
         │   │   ├── content_loc: (1,2)-(1,5) = "wat"
         │   │   ├── closing_loc: (1,5)-(1,6) = "/"
@@ -23,7 +23,7 @@
             ├── if_keyword_loc: (3,0)-(3,2) = "if"
             ├── predicate:
             │   @ MatchLastLineNode (location: (3,3)-(3,8))
-            │   ├── flags: ∅
+            │   ├── flags: forced_us_ascii_encoding
             │   ├── opening_loc: (3,3)-(3,4) = "/"
             │   ├── content_loc: (3,4)-(3,7) = "wat"
             │   ├── closing_loc: (3,7)-(3,8) = "/"
diff --git a/test/prism/snapshots/whitequark/interp_digit_var.txt b/test/prism/snapshots/whitequark/interp_digit_var.txt
index 6c34760b..09d90981 100644
--- a/test/prism/snapshots/whitequark/interp_digit_var.txt
+++ b/test/prism/snapshots/whitequark/interp_digit_var.txt
@@ -106,13 +106,13 @@
         │   ├── closing_loc: (23,8)-(23,9) = "}"
         │   └── unescaped: "\#@@1"
         ├── @ RegularExpressionNode (location: (25,1)-(25,8))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (25,1)-(25,4) = "%r{"
         │   ├── content_loc: (25,4)-(25,7) = "\#@1"
         │   ├── closing_loc: (25,7)-(25,8) = "}"
         │   └── unescaped: "\#@1"
         ├── @ RegularExpressionNode (location: (27,1)-(27,9))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (27,1)-(27,4) = "%r{"
         │   ├── content_loc: (27,4)-(27,8) = "\#@@1"
         │   ├── closing_loc: (27,8)-(27,9) = "}"
@@ -188,13 +188,13 @@
         │   ├── closing_loc: (47,6)-(47,7) = "'"
         │   └── unescaped: "\#@@1"
         ├── @ RegularExpressionNode (location: (49,1)-(49,6))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (49,1)-(49,2) = "/"
         │   ├── content_loc: (49,2)-(49,5) = "\#@1"
         │   ├── closing_loc: (49,5)-(49,6) = "/"
         │   └── unescaped: "\#@1"
         ├── @ RegularExpressionNode (location: (51,1)-(51,7))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (51,1)-(51,2) = "/"
         │   ├── content_loc: (51,2)-(51,6) = "\#@@1"
         │   ├── closing_loc: (51,6)-(51,7) = "/"
diff --git a/test/prism/snapshots/whitequark/lvar_injecting_match.txt b/test/prism/snapshots/whitequark/lvar_injecting_match.txt
index a1d70e9c..0d1df23d 100644
--- a/test/prism/snapshots/whitequark/lvar_injecting_match.txt
+++ b/test/prism/snapshots/whitequark/lvar_injecting_match.txt
@@ -9,7 +9,7 @@
         │   │   ├── flags: ∅
         │   │   ├── receiver:
         │   │   │   @ RegularExpressionNode (location: (1,0)-(1,15))
-        │   │   │   ├── flags: ∅
+        │   │   │   ├── flags: forced_us_ascii_encoding
         │   │   │   ├── opening_loc: (1,0)-(1,1) = "/"
         │   │   │   ├── content_loc: (1,1)-(1,14) = "(?<match>bar)"
         │   │   │   ├── closing_loc: (1,14)-(1,15) = "/"
diff --git a/test/prism/snapshots/whitequark/parser_bug_830.txt b/test/prism/snapshots/whitequark/parser_bug_830.txt
index e9201087..e52b291d 100644
--- a/test/prism/snapshots/whitequark/parser_bug_830.txt
+++ b/test/prism/snapshots/whitequark/parser_bug_830.txt
@@ -4,7 +4,7 @@
     @ StatementsNode (location: (1,0)-(1,4))
     └── body: (length: 1)
         └── @ RegularExpressionNode (location: (1,0)-(1,4))
-            ├── flags: ∅
+            ├── flags: forced_us_ascii_encoding
             ├── opening_loc: (1,0)-(1,1) = "/"
             ├── content_loc: (1,1)-(1,3) = "\\("
             ├── closing_loc: (1,3)-(1,4) = "/"
diff --git a/test/prism/snapshots/whitequark/parser_slash_slash_n_escaping_in_literals.txt b/test/prism/snapshots/whitequark/parser_slash_slash_n_escaping_in_literals.txt
index 96cc5671..080d4d0e 100644
--- a/test/prism/snapshots/whitequark/parser_slash_slash_n_escaping_in_literals.txt
+++ b/test/prism/snapshots/whitequark/parser_slash_slash_n_escaping_in_literals.txt
@@ -55,7 +55,7 @@
         │   ├── closing_loc: (17,1)-(17,2) = "}"
         │   └── unescaped: "a\\\nb"
         ├── @ RegularExpressionNode (location: (19,0)-(20,2))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (19,0)-(19,3) = "%r{"
         │   ├── content_loc: (19,3)-(20,1) = "a\\\nb"
         │   ├── closing_loc: (20,1)-(20,2) = "}"
@@ -96,7 +96,7 @@
         │   ├── closing_loc: (35,1)-(35,2) = "'"
         │   └── unescaped: "a\\\nb"
         ├── @ RegularExpressionNode (location: (37,0)-(38,2))
-        │   ├── flags: ∅
+        │   ├── flags: forced_us_ascii_encoding
         │   ├── opening_loc: (37,0)-(37,1) = "/"
         │   ├── content_loc: (37,1)-(38,1) = "a\\\nb"
         │   ├── closing_loc: (38,1)-(38,2) = "/"
diff --git a/test/prism/snapshots/whitequark/regex_plain.txt b/test/prism/snapshots/whitequark/regex_plain.txt
index 34fe61c6..df771f7a 100644
--- a/test/prism/snapshots/whitequark/regex_plain.txt
+++ b/test/prism/snapshots/whitequark/regex_plain.txt
@@ -4,7 +4,7 @@
     @ StatementsNode (location: (1,0)-(1,10))
     └── body: (length: 1)
         └── @ RegularExpressionNode (location: (1,0)-(1,10))
-            ├── flags: ignore_case, multi_line
+            ├── flags: ignore_case, multi_line, forced_us_ascii_encoding
             ├── opening_loc: (1,0)-(1,1) = "/"
             ├── content_loc: (1,1)-(1,7) = "source"
             ├── closing_loc: (1,7)-(1,10) = "/im"
diff --git a/test/prism/snapshots/whitequark/ruby_bug_11873.txt b/test/prism/snapshots/whitequark/ruby_bug_11873.txt
index af04f59b..2999662c 100644
--- a/test/prism/snapshots/whitequark/ruby_bug_11873.txt
+++ b/test/prism/snapshots/whitequark/ruby_bug_11873.txt
@@ -112,7 +112,7 @@
         │   │       │   ├── closing_loc: (3,7)-(3,8) = ")"
         │   │       │   └── block: ∅
         │   │       └── @ RegularExpressionNode (location: (3,10)-(3,13))
-        │   │           ├── flags: ∅
+        │   │           ├── flags: forced_us_ascii_encoding
         │   │           ├── opening_loc: (3,10)-(3,11) = "/"
         │   │           ├── content_loc: (3,11)-(3,12) = "x"
         │   │           ├── closing_loc: (3,12)-(3,13) = "/"
@@ -173,7 +173,7 @@
         │   │       │   ├── closing_loc: (5,7)-(5,8) = ")"
         │   │       │   └── block: ∅
         │   │       └── @ RegularExpressionNode (location: (5,10)-(5,14))
-        │   │           ├── flags: multi_line
+        │   │           ├── flags: multi_line, forced_us_ascii_encoding
         │   │           ├── opening_loc: (5,10)-(5,11) = "/"
         │   │           ├── content_loc: (5,11)-(5,12) = "x"
         │   │           ├── closing_loc: (5,12)-(5,14) = "/m"
@@ -295,7 +295,7 @@
         │   │       │   ├── closing_loc: (9,8)-(9,9) = ")"
         │   │       │   └── block: ∅
         │   │       └── @ RegularExpressionNode (location: (9,11)-(9,14))
-        │   │           ├── flags: ∅
+        │   │           ├── flags: forced_us_ascii_encoding
         │   │           ├── opening_loc: (9,11)-(9,12) = "/"
         │   │           ├── content_loc: (9,12)-(9,13) = "x"
         │   │           ├── closing_loc: (9,13)-(9,14) = "/"
@@ -356,7 +356,7 @@
         │   │       │   ├── closing_loc: (11,8)-(11,9) = ")"
         │   │       │   └── block: ∅
         │   │       └── @ RegularExpressionNode (location: (11,11)-(11,15))
-        │   │           ├── flags: multi_line
+        │   │           ├── flags: multi_line, forced_us_ascii_encoding
         │   │           ├── opening_loc: (11,11)-(11,12) = "/"
         │   │           ├── content_loc: (11,12)-(11,13) = "x"
         │   │           ├── closing_loc: (11,13)-(11,15) = "/m"
@@ -488,7 +488,7 @@
         │   │       │       ├── opening_loc: (15,3)-(15,4) = "{"
         │   │       │       └── closing_loc: (15,7)-(15,8) = "}"
         │   │       └── @ RegularExpressionNode (location: (15,10)-(15,13))
-        │   │           ├── flags: ∅
+        │   │           ├── flags: forced_us_ascii_encoding
         │   │           ├── opening_loc: (15,10)-(15,11) = "/"
         │   │           ├── content_loc: (15,11)-(15,12) = "x"
         │   │           ├── closing_loc: (15,12)-(15,13) = "/"
@@ -554,7 +554,7 @@
         │   │       │       ├── opening_loc: (17,3)-(17,4) = "{"
         │   │       │       └── closing_loc: (17,7)-(17,8) = "}"
         │   │       └── @ RegularExpressionNode (location: (17,10)-(17,14))
-        │   │           ├── flags: multi_line
+        │   │           ├── flags: multi_line, forced_us_ascii_encoding
         │   │           ├── opening_loc: (17,10)-(17,11) = "/"
         │   │           ├── content_loc: (17,11)-(17,12) = "x"
         │   │           ├── closing_loc: (17,12)-(17,14) = "/m"
@@ -686,7 +686,7 @@
         │   │       │       ├── opening_loc: (21,3)-(21,4) = "{"
         │   │       │       └── closing_loc: (21,8)-(21,9) = "}"
         │   │       └── @ RegularExpressionNode (location: (21,11)-(21,14))
-        │   │           ├── flags: ∅
+        │   │           ├── flags: forced_us_ascii_encoding
         │   │           ├── opening_loc: (21,11)-(21,12) = "/"
         │   │           ├── content_loc: (21,12)-(21,13) = "x"
         │   │           ├── closing_loc: (21,13)-(21,14) = "/"
@@ -752,7 +752,7 @@
             │       │       ├── opening_loc: (23,3)-(23,4) = "{"
             │       │       └── closing_loc: (23,8)-(23,9) = "}"
             │       └── @ RegularExpressionNode (location: (23,11)-(23,15))
-            │           ├── flags: multi_line
+            │           ├── flags: multi_line, forced_us_ascii_encoding
             │           ├── opening_loc: (23,11)-(23,12) = "/"
             │           ├── content_loc: (23,12)-(23,13) = "x"
             │           ├── closing_loc: (23,13)-(23,15) = "/m"
diff --git a/test/prism/unescape_test.rb b/test/prism/unescape_test.rb
index 72ad780d..2a352c52 100644
--- a/test/prism/unescape_test.rb
+++ b/test/prism/unescape_test.rb
@@ -230,8 +230,6 @@ module Prism
       else
         assert_equal expected.bytes, actual.bytes, message
       end
-    rescue Exception
-      binding.irb
     end
   end
 end