Repository: ruby/prism
Pull request #2381 (2024-02-08 16:51:52)

Serialize the newline_list to avoid recomputing it again later
* Fixes https://github.com/ruby/prism/issues/2380

Related: https://github.com/ruby/prism/issues/2402

eregon: Should be green and ready for review now.
enebo: @eregon Also simple error in build-java failing in CI but you obviously will fix that.
eregon: @enebo I think it's a lot cleaner and less error-prone this way, and it's also exactly how it works in the Ruby API (how `Prism.parse` takes a String of code and returns a ParseResult which includes the Prism::Source).
The main concern there is that if someone creates a Source object they would end up with invalid lineOffsets, unless the same object is also passed to Loader.
If the Source object is created by Loader it's not possible to have this issue.
Also creating the Source object in the caller felt awkward, I'd rather Loader takes just a `(byte[] serialize, byte[] sourceBytes)`.

> Do we actually need startLine or could this be boiled into the linecalcs as well?

It cannot be stored in the lineOffsets (which are offsets in the byte[]/char* source, so 0-based), it needs to be a separate field.

Regarding once we don't need the source bytes anymore for Loader, I think that's fine, then one would be able to just pass `null` for the `sourceBytes` (or have an overload doing that). And then the Source object would only contain startLine + lineOffsets, which seems all good (well except the name maybe, but its main purpose is to provide *source* line & column info).
In fact the only usage for `bytes` in the Source Java class currently is `bytes.length`, which we could do another way or avoid needing entirely in `findLine()`.

It is also something this PR helps significantly, since the lineOffsets now come from the serialized form and there is no longer a need to use the `sourceBytes` to find them.

If it's problematic later for some unforeseen we can of course always rework this, it's pretty easy to change, b836f031cad936826b5a5e6fcb1cb071cd665ded is a pretty small commit.

Are you OK with that?
If not I can remove that commit, but IMO it is significantly cleaner this way and also more convenient to call Loader. I was confused for a while how startLine was set before, if the Loader creates the Source there is no hidden magic and it's obvious.
enebo: @eregon yeah I forgot we are just getting an index as well so I did not think through the startLine comment.

Will you be keeping the source?  I don't see anyone doing this with location fields removed but for FFI consumers who have all the location data they probably do want the source (although they do already have it). 

I am ok though with landing this.  We can argue out how we deal with source bytes once we actually don't need it for anything.  I am isolating this code outside the JRuby codebase so these APIs can change as much as they want and it won't require a new JRuby release.  Just a new gem release containing a newly made provider.

enebo: @eregon I am only approving Java changes as I did not look very closely at the rest.
eregon: > Will you be keeping the source?

Until TruffleRuby can cache the serialized form on disk (TBD where/how, not for the 24.0 release) it needs to read the source as a byte[] anyway to pass it to parseAndSerialize().
(and of course currently the source bytes are needed for Loader so it's not yet possible to skip reading the source file)

>  I am isolating this code outside the JRuby codebase so these APIs can change as much as they want and it won't require a new JRuby release. Just a new gem release containing a newly made provider.

Don't you vendor a specific version/commit of Prism in JRuby when using it as the parser for Ruby code?
That's what TruffleRuby and CRuby do. Then it doesn't matter which version(s) of Prism are installed as gems, it's independent.

Or do you also use the Java Loader when Prism is used as a gem? I wouldn't expect that since what's tested in CI on JRuby is the Ruby API using FFI (so not using the Java Loader).
enebo: > Don't you vendor a specific version/commit of Prism in JRuby when using it as the parser for Ruby code? That's what TruffleRuby and CRuby do. Then it doesn't matter which version(s) of Prism are installed as gems, it's independent.
> 
> Or do you also use the Java Loader when Prism is used as a gem? I wouldn't expect that since what's tested in CI on JRuby is the Ruby API using FFI (so not using the Java Loader).

I am isolating the Java part as an artifact and it is included into a Ruby gem which is effectively a "barely" fork of prism gem that has been renamed.  That gem includes the artifact and will compile the source with the ENVs we want when the gem is installed (later we will probably pre-compile to remove compile requirement on main platforms).

The rationale for doing this is if there is a CVE or just a good fix there is a high likelihood I only have to do a gem release and not a full JRuby release.
eregon: I see, makes sense. Could be worth documenting at https://github.com/ruby/prism/blob/main/docs/build_system.md#building-prism-as-part-of-jruby
We cannot do this for TruffleRuby because it needs to work for `--native` mode and that needs to know all Java code ahead-of-time.
eregon: @kddnewton Could you merge this?

Patch:
diff --git a/docs/serialization.md b/docs/serialization.md
index 151d209d..cc87b63e 100644
--- a/docs/serialization.md
+++ b/docs/serialization.md
@@ -85,6 +85,8 @@ The header is structured like the following table:
 | `1` | 1 indicates only semantics fields were serialized, 0 indicates all fields were serialized (including location fields) |
 | string | the encoding name |
 | varsint | the start line |
+| varuint | number of newline offsets |
+| varuint* | newline offsets |
 | varuint | number of comments |
 | comment* | comments |
 | varuint | number of magic comments |
diff --git a/ext/prism/extension.c b/ext/prism/extension.c
index c20ce5b5..cf3c7edc 100644
--- a/ext/prism/extension.c
+++ b/ext/prism/extension.c
@@ -542,8 +542,9 @@ parse_lex_input(pm_string_t *input, const pm_options_t *options, bool return_nod
     pm_parser_init(&parser, pm_string_source(input), pm_string_length(input), options);
     pm_parser_register_encoding_changed_callback(&parser, parse_lex_encoding_changed_callback);
 
+    VALUE source_string = rb_str_new((const char *) pm_string_source(input), pm_string_length(input));
     VALUE offsets = rb_ary_new();
-    VALUE source_argv[] = { rb_str_new((const char *) pm_string_source(input), pm_string_length(input)), ULONG2NUM(parser.start_line), offsets };
+    VALUE source_argv[] = { source_string, LONG2NUM(parser.start_line), offsets };
     VALUE source = rb_class_new_instance(3, source_argv, rb_cPrismSource);
 
     parse_lex_data_t parse_lex_data = {
@@ -561,17 +562,21 @@ parse_lex_input(pm_string_t *input, const pm_options_t *options, bool return_nod
     parser.lex_callback = &lex_callback;
     pm_node_t *node = pm_parse(&parser);
 
-    // Here we need to update the source range to have the correct newline
-    // offsets. We do it here because we've already created the object and given
-    // it over to all of the tokens.
+    // Here we need to update the Source object to have the correct
+    // encoding for the source string and the correct newline offsets.
+    // We do it here because we've already created the Source object and given
+    // it over to all of the tokens, and both of these are only set after pm_parse().
+    rb_encoding *encoding = rb_enc_find(parser.encoding->name);
+    rb_enc_associate(source_string, encoding);
+
     for (size_t index = 0; index < parser.newline_list.size; index++) {
-        rb_ary_push(offsets, INT2FIX(parser.newline_list.offsets[index]));
+        rb_ary_push(offsets, ULONG2NUM(parser.newline_list.offsets[index]));
     }
 
     VALUE value;
     if (return_nodes) {
         value = rb_ary_new_capa(2);
-        rb_ary_push(value, pm_ast_new(&parser, node, parse_lex_data.encoding));
+        rb_ary_push(value, pm_ast_new(&parser, node, parse_lex_data.encoding, source));
         rb_ary_push(value, parse_lex_data.tokens);
     } else {
         value = parse_lex_data.tokens;
@@ -650,7 +655,7 @@ parse_input(pm_string_t *input, const pm_options_t *options) {
 
     VALUE source = pm_source_new(&parser, encoding);
     VALUE result_argv[] = {
-        pm_ast_new(&parser, node, encoding),
+        pm_ast_new(&parser, node, encoding, source),
         parser_comments(&parser, source),
         parser_magic_comments(&parser, source),
         parser_data_loc(&parser, source),
diff --git a/ext/prism/extension.h b/ext/prism/extension.h
index ba97583e..0003b469 100644
--- a/ext/prism/extension.h
+++ b/ext/prism/extension.h
@@ -9,7 +9,7 @@
 
 VALUE pm_source_new(pm_parser_t *parser, rb_encoding *encoding);
 VALUE pm_token_new(pm_parser_t *parser, pm_token_t *token, rb_encoding *encoding, VALUE source);
-VALUE pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding);
+VALUE pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding, VALUE source);
 
 void Init_prism_api_node(void);
 void Init_prism_pack(void);
diff --git a/include/prism/diagnostic.h b/include/prism/diagnostic.h
index bf47e8db..ebd4a8bb 100644
--- a/include/prism/diagnostic.h
+++ b/include/prism/diagnostic.h
@@ -281,7 +281,10 @@ typedef enum {
     PM_ERR_TERNARY_COLON,
     PM_ERR_TERNARY_EXPRESSION_FALSE,
     PM_ERR_TERNARY_EXPRESSION_TRUE,
-    PM_ERR_UNARY_RECEIVER,
+    PM_ERR_UNARY_RECEIVER_BANG,
+    PM_ERR_UNARY_RECEIVER_MINUS,
+    PM_ERR_UNARY_RECEIVER_PLUS,
+    PM_ERR_UNARY_RECEIVER_TILDE,
     PM_ERR_UNEXPECTED_TOKEN_CLOSE_CONTEXT,
     PM_ERR_UNEXPECTED_TOKEN_IGNORE,
     PM_ERR_UNDEF_ARGUMENT,
diff --git a/include/prism/encoding.h b/include/prism/encoding.h
index d0f947ea..7ba1695d 100644
--- a/include/prism/encoding.h
+++ b/include/prism/encoding.h
@@ -245,13 +245,6 @@ extern const pm_encoding_t pm_encodings[PM_ENCODING_MAXIMUM];
  */
 #define PM_ENCODING_US_ASCII_ENTRY (&pm_encodings[PM_ENCODING_US_ASCII])
 
-/**
- * This is the ASCII-8BIT encoding. We need a reference to it so that pm_strpbrk
- * can compare against it because invalid multibyte characters are not a thing
- * in this encoding.
- */
-#define PM_ENCODING_ASCII_8BIT_ENTRY (&pm_encodings[PM_ENCODING_ASCII_8BIT])
-
 /**
  * Parse the given name of an encoding and return a pointer to the corresponding
  * encoding struct if one can be found, otherwise return NULL.
diff --git a/include/prism/util/pm_strpbrk.h b/include/prism/util/pm_strpbrk.h
index f387bd57..c1cf0d54 100644
--- a/include/prism/util/pm_strpbrk.h
+++ b/include/prism/util/pm_strpbrk.h
@@ -7,7 +7,6 @@
 #define PRISM_STRPBRK_H
 
 #include "prism/defines.h"
-#include "prism/diagnostic.h"
 #include "prism/parser.h"
 
 #include <stddef.h>
@@ -36,11 +35,9 @@
  * @param source The source to search.
  * @param charset The charset to search for.
  * @param length The maximum number of bytes to search.
- * @param validate Whether to validate that the source string is valid in the
- *     current encoding of the parser.
  * @return A pointer to the first character in the source string that is in the
  *     charset, or NULL if no such character exists.
  */
-const uint8_t * pm_strpbrk(pm_parser_t *parser, const uint8_t *source, const uint8_t *charset, ptrdiff_t length, bool validate);
+const uint8_t * pm_strpbrk(const pm_parser_t *parser, const uint8_t *source, const uint8_t *charset, ptrdiff_t length);
 
 #endif
diff --git a/java/org/prism/ParseResult.java b/java/org/prism/ParseResult.java
index dc9008c9..d2a6f2ef 100644
--- a/java/org/prism/ParseResult.java
+++ b/java/org/prism/ParseResult.java
@@ -62,13 +62,15 @@ public final class ParseResult {
     public final Nodes.Location dataLocation;
     public final Error[] errors;
     public final Warning[] warnings;
+    public final Nodes.Source source;
 
-    public ParseResult(Nodes.Node value, MagicComment[] magicComments, Nodes.Location dataLocation, Error[] errors, Warning[] warnings) {
+    public ParseResult(Nodes.Node value, MagicComment[] magicComments, Nodes.Location dataLocation, Error[] errors, Warning[] warnings, Nodes.Source source) {
         this.value = value;
         this.magicComments = magicComments;
         this.dataLocation = dataLocation;
         this.errors = errors;
         this.warnings = warnings;
+        this.source = source;
     }
 }
 // @formatter:on
diff --git a/javascript/src/parsePrism.js b/javascript/src/parsePrism.js
index aeaa1c15..7c59aca0 100644
--- a/javascript/src/parsePrism.js
+++ b/javascript/src/parsePrism.js
@@ -68,7 +68,7 @@ function dumpOptions(options) {
   values.push(options.frozen_string_literal === undefined ? 0 : 1);
 
   template.push("C");
-  if (!options.version || options.version === "latest" || options.version === "3.4.0") {
+  if (!options.version || options.version === "latest") {
     values.push(0);
   } else if (options.version === "3.3.0") {
     values.push(1);
diff --git a/lib/prism/ffi.rb b/lib/prism/ffi.rb
index 3418d787..617c469d 100644
--- a/lib/prism/ffi.rb
+++ b/lib/prism/ffi.rb
@@ -317,7 +317,7 @@ module Prism
       values << (options.fetch(:frozen_string_literal, false) ? 1 : 0)
 
       template << "C"
-      values << { nil => 0, "3.3.0" => 1, "3.4.0" => 0, "latest" => 0 }.fetch(options[:version])
+      values << { nil => 0, "3.3.0" => 1, "latest" => 0 }.fetch(options[:version])
 
       template << "L"
       if (scopes = options[:scopes])
diff --git a/lib/prism/parse_result.rb b/lib/prism/parse_result.rb
index 4b0c57ed..9cc60d49 100644
--- a/lib/prism/parse_result.rb
+++ b/lib/prism/parse_result.rb
@@ -9,18 +9,16 @@ module Prism
     attr_reader :source
 
     # The line number where this source starts.
-    attr_accessor :start_line
+    attr_reader :start_line
 
     # The list of newline byte offsets in the source code.
     attr_reader :offsets
 
-    # Create a new source object with the given source code and newline byte
-    # offsets. If no newline byte offsets are given, they will be computed from
-    # the source code.
-    def initialize(source, start_line = 1, offsets = compute_offsets(source))
+    # Create a new source object with the given source code.
+    def initialize(source, start_line = 1, offsets = [])
       @source = source
-      @start_line = start_line
-      @offsets = offsets
+      @start_line = start_line # set after parsing is done
+      @offsets = offsets # set after parsing is done
     end
 
     # Perform a byteslice on the source code using the given byte offset and
@@ -56,23 +54,6 @@ module Prism
       character_offset(byte_offset) - character_offset(line_start(byte_offset))
     end
 
-    # Returns the offset from the start of the file for the given byte offset
-    # counting in code units for the given encoding.
-    #
-    # This method is tested with UTF-8, UTF-16, and UTF-32. If there is the
-    # concept of code units that differs from the number of characters in other
-    # encodings, it is not captured here.
-    def code_units_offset(byte_offset, encoding)
-      byteslice = source.byteslice(0, byte_offset).encode(encoding)
-      (encoding == Encoding::UTF_16LE || encoding == Encoding::UTF_16BE) ? (byteslice.bytesize / 2) : byteslice.length
-    end
-
-    # Returns the column number in code units for the given encoding for the
-    # given byte offset.
-    def code_units_column(byte_offset, encoding)
-      code_units_offset(byte_offset, encoding) - code_units_offset(line_start(byte_offset), encoding)
-    end
-
     private
 
     # Binary search through the offsets to find the line number for the given
@@ -94,14 +75,6 @@ module Prism
 
       left - 1
     end
-
-    # Find all of the newlines in the source code and return their byte offsets
-    # from the start of the string an array.
-    def compute_offsets(code)
-      offsets = [0]
-      code.b.scan("\n") { offsets << $~.end(0) }
-      offsets
-    end
   end
 
   # This represents a location in the source.
@@ -155,11 +128,6 @@ module Prism
       source.character_offset(start_offset)
     end
 
-    # The offset from the start of the file in code units of the given encoding.
-    def start_code_units_offset(encoding = Encoding::UTF_16LE)
-      source.code_units_offset(start_offset, encoding)
-    end
-
     # The byte offset from the beginning of the source where this location ends.
     def end_offset
       start_offset + length
@@ -171,11 +139,6 @@ module Prism
       source.character_offset(end_offset)
     end
 
-    # The offset from the start of the file in code units of the given encoding.
-    def end_code_units_offset(encoding = Encoding::UTF_16LE)
-      source.code_units_offset(end_offset, encoding)
-    end
-
     # The line number where this location starts.
     def start_line
       source.line(start_offset)
@@ -204,12 +167,6 @@ module Prism
       source.character_column(start_offset)
     end
 
-    # The column number in code units of the given encoding where this location
-    # starts from the start of the line.
-    def start_code_units_column(encoding = Encoding::UTF_16LE)
-      source.code_units_column(start_offset, encoding)
-    end
-
     # The column number in bytes where this location ends from the start of the
     # line.
     def end_column
@@ -222,12 +179,6 @@ module Prism
       source.character_column(end_offset)
     end
 
-    # The column number in code units of the given encoding where this location
-    # ends from the start of the line.
-    def end_code_units_column(encoding = Encoding::UTF_16LE)
-      source.code_units_column(end_offset, encoding)
-    end
-
     # Implement the hash pattern matching interface for Location.
     def deconstruct_keys(keys)
       { start_offset: start_offset, end_offset: end_offset }
diff --git a/rakelib/serialization.rake b/rakelib/serialization.rake
index 5a60544d..a57a7ba7 100644
--- a/rakelib/serialization.rake
+++ b/rakelib/serialization.rake
@@ -19,7 +19,6 @@ task "test:java_loader:internal" => :compile do
   require "fileutils"
   require 'java'
   require_relative '../tmp/prism.jar'
-  java_import 'org.prism.Nodes$Source'
 
   Dir["**/*.txt", base: fixtures].each do |relative|
     path = "#{fixtures}/#{relative}"
@@ -27,8 +26,7 @@ task "test:java_loader:internal" => :compile do
     puts path
     serialized = Prism.dump_file(path)
     source_bytes = File.binread(path).unpack('c*')
-    source = Source.new(source_bytes.to_java(:byte))
-    parse_result = org.prism.Loader.load(serialized.unpack('c*'), source)
+    parse_result = org.prism.Loader.load(serialized.unpack('c*'), source_bytes)
     puts parse_result.value
   end
 end
diff --git a/src/diagnostic.c b/src/diagnostic.c
index 80805c9b..957cc47d 100644
--- a/src/diagnostic.c
+++ b/src/diagnostic.c
@@ -184,7 +184,7 @@ static const pm_diagnostic_data_t diagnostic_messages[PM_DIAGNOSTIC_ID_LEN] = {
     [PM_ERR_FOR_IN]                             = { "expected an `in` after the index in a `for` statement", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_FOR_TERM]                           = { "expected an `end` to close the `for` loop", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_HASH_EXPRESSION_AFTER_LABEL]        = { "expected an expression after the label in a hash", PM_ERROR_LEVEL_FATAL },
-    [PM_ERR_HASH_KEY]                           = { "unexpected %s, expecting '}' or a key in the hash literal", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_HASH_KEY]                           = { "expected a key in the hash literal", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_HASH_ROCKET]                        = { "expected a `=>` between the hash key and value", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_HASH_TERM]                          = { "expected a `}` to close the hash literal", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_HASH_VALUE]                         = { "expected a value in the hash literal", PM_ERROR_LEVEL_FATAL },
@@ -284,9 +284,12 @@ static const pm_diagnostic_data_t diagnostic_messages[PM_DIAGNOSTIC_ID_LEN] = {
     [PM_ERR_TERNARY_EXPRESSION_FALSE]           = { "expected an expression after `:` in the ternary operator", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_TERNARY_EXPRESSION_TRUE]            = { "expected an expression after `?` in the ternary operator", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_UNDEF_ARGUMENT]                     = { "invalid argument being passed to `undef`; expected a bare word, constant, or symbol argument", PM_ERROR_LEVEL_FATAL },
-    [PM_ERR_UNARY_RECEIVER]                     = { "unexpected %s, expected a receiver for unary `%c`", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_UNARY_RECEIVER_BANG]                = { "expected a receiver for unary `!`", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_UNARY_RECEIVER_MINUS]               = { "expected a receiver for unary `-`", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_UNARY_RECEIVER_PLUS]                = { "expected a receiver for unary `+`", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_UNEXPECTED_TOKEN_CLOSE_CONTEXT]     = { "unexpected %s, assuming it is closing the parent %s", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_UNEXPECTED_TOKEN_IGNORE]            = { "unexpected %s, ignoring it", PM_ERROR_LEVEL_FATAL },
+    [PM_ERR_UNARY_RECEIVER_TILDE]               = { "expected a receiver for unary `~`", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_UNTIL_TERM]                         = { "expected an `end` to close the `until` statement", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_VOID_EXPRESSION]                    = { "unexpected void value expression", PM_ERROR_LEVEL_FATAL },
     [PM_ERR_WHILE_TERM]                         = { "expected an `end` to close the `while` statement", PM_ERROR_LEVEL_FATAL },
diff --git a/src/encoding.c b/src/encoding.c
index 1d455c24..c8e0941d 100644
--- a/src/encoding.c
+++ b/src/encoding.c
@@ -2253,12 +2253,12 @@ static const uint8_t pm_utf_8_dfa[] = {
 static pm_unicode_codepoint_t
 pm_utf_8_codepoint(const uint8_t *b, ptrdiff_t n, size_t *width) {
     assert(n >= 0);
+    size_t maximum = (size_t) n;
 
-    size_t maximum = (n > 4) ? 4 : ((size_t) n);
     uint32_t codepoint;
     uint32_t state = 0;
 
-    for (size_t index = 0; index < maximum; index++) {
+    for (size_t index = 0; index < 4 && index < maximum; index++) {
         uint32_t byte = b[index];
         uint32_t type = pm_utf_8_dfa[byte];
 
@@ -2267,7 +2267,7 @@ pm_utf_8_codepoint(const uint8_t *b, ptrdiff_t n, size_t *width) {
             (0xffu >> type) & (byte);
 
         state = pm_utf_8_dfa[256 + (state * 16) + type];
-        if (state == 0) {
+        if (!state) {
             *width = index + 1;
             return (pm_unicode_codepoint_t) codepoint;
         }
@@ -2282,17 +2282,9 @@ pm_utf_8_codepoint(const uint8_t *b, ptrdiff_t n, size_t *width) {
  */
 size_t
 pm_encoding_utf_8_char_width(const uint8_t *b, ptrdiff_t n) {
-    assert(n >= 0);
-
-    size_t maximum = (n > 4) ? 4 : ((size_t) n);
-    uint32_t state = 0;
-
-    for (size_t index = 0; index < maximum; index++) {
-        state = pm_utf_8_dfa[256 + (state * 16) + pm_utf_8_dfa[b[index]]];
-        if (state == 0) return index + 1;
-    }
-
-    return 0;
+    size_t width;
+    pm_utf_8_codepoint(b, n, &width);
+    return width;
 }
 
 /**
@@ -4194,6 +4186,15 @@ pm_encoding_gbk_char_width(const uint8_t *b, ptrdiff_t n) {
     return 0;
 }
 
+/**
+ * Returns the size of the next character in the KOI-8 encoding. This means
+ * checking if it's a valid codepoint in KOI-8 and if it is returning 1.
+ */
+static size_t
+pm_encoding_koi8_char_width(const uint8_t *b, PRISM_ATTRIBUTE_UNUSED ptrdiff_t n) {
+    return ((*b >= 0x20 && *b <= 0x7E) || (*b >= 0x80)) ? 1 : 0;
+}
+
 /**
  * Returns the size of the next character in the Shift_JIS encoding, or 0 if a
  * character cannot be decoded from the given bytes.
@@ -4651,7 +4652,7 @@ const pm_encoding_t pm_encodings[] = {
     },
     [PM_ENCODING_KOI8_R] = {
         .name = "KOI8-R",
-        .char_width = pm_encoding_single_char_width,
+        .char_width = pm_encoding_koi8_char_width,
         .alnum_char = pm_encoding_koi8_r_alnum_char,
         .alpha_char = pm_encoding_koi8_r_alpha_char,
         .isupper_char = pm_encoding_koi8_r_isupper_char,
@@ -4659,7 +4660,7 @@ const pm_encoding_t pm_encodings[] = {
     },
     [PM_ENCODING_KOI8_U] = {
         .name = "KOI8-U",
-        .char_width = pm_encoding_single_char_width,
+        .char_width = pm_encoding_koi8_char_width,
         .alnum_char = pm_encoding_koi8_u_alnum_char,
         .alpha_char = pm_encoding_koi8_u_alpha_char,
         .isupper_char = pm_encoding_koi8_u_isupper_char,
diff --git a/src/options.c b/src/options.c
index 35d3869a..4187abf9 100644
--- a/src/options.c
+++ b/src/options.c
@@ -45,22 +45,17 @@ pm_options_version_set(pm_options_t *options, const char *version, size_t length
     }
 
     if (length == 5) {
-        if (strncmp(version, "3.3.0", length) == 0) {
+        if (strncmp(version, "3.3.0", 5) == 0) {
             options->version = PM_OPTIONS_VERSION_CRUBY_3_3_0;
             return true;
         }
 
-        if (strncmp(version, "3.4.0", length) == 0) {
+        if (strncmp(version, "latest", 6) == 0) {
             options->version = PM_OPTIONS_VERSION_LATEST;
             return true;
         }
     }
 
-    if (length == 6 && strncmp(version, "latest", length) == 0) {
-        options->version = PM_OPTIONS_VERSION_LATEST;
-        return true;
-    }
-
     return false;
 }
 
diff --git a/src/prism.c b/src/prism.c
index 972f813b..7e1fae5e 100644
--- a/src/prism.c
+++ b/src/prism.c
@@ -9737,7 +9737,7 @@ parser_lex(pm_parser_t *parser) {
             // and then find the first one.
             pm_lex_mode_t *lex_mode = parser->lex_modes.current;
             const uint8_t *breakpoints = lex_mode->as.list.breakpoints;
-            const uint8_t *breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+            const uint8_t *breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
 
             // If we haven't found an escape yet, then this buffer will be
             // unallocated since we can refer directly to the source string.
@@ -9746,7 +9746,7 @@ parser_lex(pm_parser_t *parser) {
             while (breakpoint != NULL) {
                 // If we hit a null byte, skip directly past it.
                 if (*breakpoint == '\0') {
-                    breakpoint = pm_strpbrk(parser, breakpoint + 1, breakpoints, parser->end - (breakpoint + 1), true);
+                    breakpoint = pm_strpbrk(parser, breakpoint + 1, breakpoints, parser->end - (breakpoint + 1));
                     continue;
                 }
 
@@ -9765,7 +9765,7 @@ parser_lex(pm_parser_t *parser) {
                     // we need to continue on past it.
                     if (lex_mode->as.list.nesting > 0) {
                         parser->current.end = breakpoint + 1;
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         lex_mode->as.list.nesting--;
                         continue;
                     }
@@ -9850,7 +9850,7 @@ parser_lex(pm_parser_t *parser) {
                     }
 
                     token_buffer.cursor = parser->current.end;
-                    breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                    breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                     continue;
                 }
 
@@ -9863,7 +9863,7 @@ parser_lex(pm_parser_t *parser) {
                         // that looked like an interpolated class or instance variable
                         // like "#@" but wasn't actually. In this case we'll just skip
                         // to the next breakpoint.
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         continue;
                     }
 
@@ -9878,7 +9878,7 @@ parser_lex(pm_parser_t *parser) {
                 // and find the next breakpoint.
                 assert(*breakpoint == lex_mode->as.list.incrementor);
                 parser->current.end = breakpoint + 1;
-                breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                 lex_mode->as.list.nesting++;
                 continue;
             }
@@ -9917,14 +9917,14 @@ parser_lex(pm_parser_t *parser) {
             // regular expression. We'll use strpbrk to find the first of these
             // characters.
             const uint8_t *breakpoints = lex_mode->as.regexp.breakpoints;
-            const uint8_t *breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, false);
+            const uint8_t *breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
             pm_token_buffer_t token_buffer = { { 0 }, 0 };
 
             while (breakpoint != NULL) {
                 // If we hit a null byte, skip directly past it.
                 if (*breakpoint == '\0') {
                     parser->current.end = breakpoint + 1;
-                    breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, false);
+                    breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                     continue;
                 }
 
@@ -9946,7 +9946,7 @@ parser_lex(pm_parser_t *parser) {
                         // If the terminator is not a newline, then we can set
                         // the next breakpoint and continue.
                         parser->current.end = breakpoint + 1;
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, false);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         continue;
                     }
                 }
@@ -9956,7 +9956,7 @@ parser_lex(pm_parser_t *parser) {
                 if (*breakpoint == lex_mode->as.regexp.terminator) {
                     if (lex_mode->as.regexp.nesting > 0) {
                         parser->current.end = breakpoint + 1;
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, false);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         lex_mode->as.regexp.nesting--;
                         continue;
                     }
@@ -10055,7 +10055,7 @@ parser_lex(pm_parser_t *parser) {
                     }
 
                     token_buffer.cursor = parser->current.end;
-                    breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, false);
+                    breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                     continue;
                 }
 
@@ -10068,7 +10068,7 @@ parser_lex(pm_parser_t *parser) {
                         // something that looked like an interpolated class or
                         // instance variable like "#@" but wasn't actually. In
                         // this case we'll just skip to the next breakpoint.
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, false);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         continue;
                     }
 
@@ -10083,7 +10083,7 @@ parser_lex(pm_parser_t *parser) {
                 // and find the next breakpoint.
                 assert(*breakpoint == lex_mode->as.regexp.incrementor);
                 parser->current.end = breakpoint + 1;
-                breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, false);
+                breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                 lex_mode->as.regexp.nesting++;
                 continue;
             }
@@ -10119,7 +10119,7 @@ parser_lex(pm_parser_t *parser) {
             // string. We'll use strpbrk to find the first of these characters.
             pm_lex_mode_t *lex_mode = parser->lex_modes.current;
             const uint8_t *breakpoints = lex_mode->as.string.breakpoints;
-            const uint8_t *breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+            const uint8_t *breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
 
             // If we haven't found an escape yet, then this buffer will be
             // unallocated since we can refer directly to the source string.
@@ -10131,7 +10131,7 @@ parser_lex(pm_parser_t *parser) {
                 if (lex_mode->as.string.incrementor != '\0' && *breakpoint == lex_mode->as.string.incrementor) {
                     lex_mode->as.string.nesting++;
                     parser->current.end = breakpoint + 1;
-                    breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                    breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                     continue;
                 }
 
@@ -10143,7 +10143,7 @@ parser_lex(pm_parser_t *parser) {
                     // to continue on past it.
                     if (lex_mode->as.string.nesting > 0) {
                         parser->current.end = breakpoint + 1;
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         lex_mode->as.string.nesting--;
                         continue;
                     }
@@ -10185,7 +10185,7 @@ parser_lex(pm_parser_t *parser) {
                     if (parser->heredoc_end == NULL) {
                         pm_newline_list_append(&parser->newline_list, breakpoint);
                         parser->current.end = breakpoint + 1;
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         continue;
                     } else {
                         parser->current.end = breakpoint + 1;
@@ -10199,7 +10199,7 @@ parser_lex(pm_parser_t *parser) {
                     case '\0':
                         // Skip directly past the null character.
                         parser->current.end = breakpoint + 1;
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         break;
                     case '\\': {
                         // Here we hit escapes.
@@ -10268,7 +10268,7 @@ parser_lex(pm_parser_t *parser) {
                         }
 
                         token_buffer.cursor = parser->current.end;
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         break;
                     }
                     case '#': {
@@ -10279,7 +10279,7 @@ parser_lex(pm_parser_t *parser) {
                             // looked like an interpolated class or instance variable like "#@"
                             // but wasn't actually. In this case we'll just skip to the next
                             // breakpoint.
-                            breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                            breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                             break;
                         }
 
@@ -10407,7 +10407,7 @@ parser_lex(pm_parser_t *parser) {
                 breakpoints[2] = '\0';
             }
 
-            const uint8_t *breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+            const uint8_t *breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
             pm_token_buffer_t token_buffer = { { 0 }, 0 };
             bool was_escaped_newline = false;
 
@@ -10416,7 +10416,7 @@ parser_lex(pm_parser_t *parser) {
                     case '\0':
                         // Skip directly past the null character.
                         parser->current.end = breakpoint + 1;
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         break;
                     case '\n': {
                         if (parser->heredoc_end != NULL && (parser->heredoc_end > breakpoint)) {
@@ -10491,7 +10491,7 @@ parser_lex(pm_parser_t *parser) {
                         // Otherwise we hit a newline and it wasn't followed by
                         // a terminator, so we can continue parsing.
                         parser->current.end = breakpoint + 1;
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         break;
                     }
                     case '\\': {
@@ -10555,7 +10555,7 @@ parser_lex(pm_parser_t *parser) {
                         }
 
                         token_buffer.cursor = parser->current.end;
-                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                        breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                         break;
                     }
                     case '#': {
@@ -10567,7 +10567,7 @@ parser_lex(pm_parser_t *parser) {
                             // or instance variable like "#@" but wasn't
                             // actually. In this case we'll just skip to the
                             // next breakpoint.
-                            breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end, true);
+                            breakpoint = pm_strpbrk(parser, parser->current.end, breakpoints, parser->end - parser->current.end);
                             break;
                         }
 
@@ -14367,29 +14367,6 @@ parse_strings(pm_parser_t *parser, pm_node_t *current) {
     return current;
 }
 
-/**
- * Append an error to the error list on the parser using the given diagnostic
- * ID. This function is a specialization that handles formatting the specific
- * kind of error that is being appended.
- */
-static void
-pm_parser_err_prefix(pm_parser_t *parser, pm_diagnostic_id_t diag_id) {
-    switch (diag_id) {
-        case PM_ERR_HASH_KEY: {
-            PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->previous, diag_id, pm_token_type_human(parser->previous.type));
-            break;
-        }
-        case PM_ERR_UNARY_RECEIVER: {
-            const char *human = (parser->current.type == PM_TOKEN_EOF ? "end-of-input" : pm_token_type_human(parser->current.type));
-            PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->previous, diag_id, human, parser->previous.start[0]);
-            break;
-        }
-        default:
-            pm_parser_err_previous(parser, diag_id);
-            break;
-    }
-}
-
 /**
  * Parse an expression that begins with the previous node that we just lexed.
  */
@@ -16517,7 +16494,7 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
             // context of a multiple assignment. We enforce that here. We'll
             // still lex past it though and create a missing node place.
             if (binding_power != PM_BINDING_POWER_STATEMENT) {
-                pm_parser_err_prefix(parser, diag_id);
+                pm_parser_err_previous(parser, diag_id);
                 return (pm_node_t *) pm_missing_node_create(parser, parser->previous.start, parser->previous.end);
             }
 
@@ -16540,7 +16517,7 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
             parser_lex(parser);
 
             pm_token_t operator = parser->previous;
-            pm_node_t *receiver = parse_expression(parser, pm_binding_powers[parser->previous.type].right, binding_power < PM_BINDING_POWER_MATCH, PM_ERR_UNARY_RECEIVER);
+            pm_node_t *receiver = parse_expression(parser, pm_binding_powers[parser->previous.type].right, binding_power < PM_BINDING_POWER_MATCH, PM_ERR_UNARY_RECEIVER_BANG);
             pm_call_node_t *node = pm_call_node_unary_create(parser, &operator, receiver, "!");
 
             pm_conditional_predicate(receiver);
@@ -16550,7 +16527,7 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
             parser_lex(parser);
 
             pm_token_t operator = parser->previous;
-            pm_node_t *receiver = parse_expression(parser, pm_binding_powers[parser->previous.type].right, false, PM_ERR_UNARY_RECEIVER);
+            pm_node_t *receiver = parse_expression(parser, pm_binding_powers[parser->previous.type].right, false, PM_ERR_UNARY_RECEIVER_TILDE);
             pm_call_node_t *node = pm_call_node_unary_create(parser, &operator, receiver, "~");
 
             return (pm_node_t *) node;
@@ -16559,7 +16536,7 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
             parser_lex(parser);
 
             pm_token_t operator = parser->previous;
-            pm_node_t *receiver = parse_expression(parser, pm_binding_powers[parser->previous.type].right, false, PM_ERR_UNARY_RECEIVER);
+            pm_node_t *receiver = parse_expression(parser, pm_binding_powers[parser->previous.type].right, false, PM_ERR_UNARY_RECEIVER_MINUS);
             pm_call_node_t *node = pm_call_node_unary_create(parser, &operator, receiver, "-@");
 
             return (pm_node_t *) node;
@@ -16568,7 +16545,7 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
             parser_lex(parser);
 
             pm_token_t operator = parser->previous;
-            pm_node_t *node = parse_expression(parser, pm_binding_powers[parser->previous.type].right, false, PM_ERR_UNARY_RECEIVER);
+            pm_node_t *node = parse_expression(parser, pm_binding_powers[parser->previous.type].right, false, PM_ERR_UNARY_RECEIVER_MINUS);
 
             if (accept1(parser, PM_TOKEN_STAR_STAR)) {
                 pm_token_t exponent_operator = parser->previous;
@@ -16684,7 +16661,7 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
             parser_lex(parser);
 
             pm_token_t operator = parser->previous;
-            pm_node_t *receiver = parse_expression(parser, pm_binding_powers[parser->previous.type].right, false, PM_ERR_UNARY_RECEIVER);
+            pm_node_t *receiver = parse_expression(parser, pm_binding_powers[parser->previous.type].right, false, PM_ERR_UNARY_RECEIVER_PLUS);
             pm_call_node_t *node = pm_call_node_unary_create(parser, &operator, receiver, "+@");
 
             return (pm_node_t *) node;
@@ -16707,7 +16684,7 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
                 // here because it will provide more context in addition to the
                 // recoverable error that we will also add.
                 if (diag_id != PM_ERR_CANNOT_PARSE_EXPRESSION) {
-                    pm_parser_err_prefix(parser, diag_id);
+                    pm_parser_err_previous(parser, diag_id);
                 }
 
                 // If we get here, then we are assuming this token is closing a
@@ -16720,7 +16697,7 @@ parse_expression_prefix(pm_parser_t *parser, pm_binding_power_t binding_power, b
                 // have an unexpected token.
                 PM_PARSER_ERR_TOKEN_FORMAT(parser, parser->current, PM_ERR_UNEXPECTED_TOKEN_IGNORE, pm_token_type_human(parser->current.type));
             } else {
-                pm_parser_err_prefix(parser, diag_id);
+                pm_parser_err_previous(parser, diag_id);
             }
 
             return (pm_node_t *) pm_missing_node_create(parser, parser->previous.start, parser->previous.end);
diff --git a/src/util/pm_strpbrk.c b/src/util/pm_strpbrk.c
index 6c8dea18..115eba1f 100644
--- a/src/util/pm_strpbrk.c
+++ b/src/util/pm_strpbrk.c
@@ -1,76 +1,10 @@
 #include "prism/util/pm_strpbrk.h"
 
-/**
- * Add an invalid multibyte character error to the parser.
- */
-static inline void
-pm_strpbrk_invalid_multibyte_character(pm_parser_t *parser, const uint8_t *start, const uint8_t *end) {
-    pm_diagnostic_list_append_format(&parser->error_list, start, end, PM_ERR_INVALID_MULTIBYTE_CHARACTER, *start);
-}
-
-/**
- * This is the default path.
- */
-static inline const uint8_t *
-pm_strpbrk_utf8(pm_parser_t *parser, const uint8_t *source, const uint8_t *charset, size_t maximum, bool validate) {
-    size_t index = 0;
-
-    while (index < maximum) {
-        if (strchr((const char *) charset, source[index]) != NULL) {
-            return source + index;
-        }
-
-        if (source[index] < 0x80) {
-            index++;
-        } else {
-            size_t width = pm_encoding_utf_8_char_width(source + index, (ptrdiff_t) (maximum - index));
-
-            if (width > 0) {
-                index += width;
-            } else if (!validate) {
-                index++;
-            } else {
-                // At this point we know we have an invalid multibyte character.
-                // We'll walk forward as far as we can until we find the next
-                // valid character so that we don't spam the user with a ton of
-                // the same kind of error.
-                const size_t start = index;
-
-                do {
-                    index++;
-                } while (index < maximum && pm_encoding_utf_8_char_width(source + index, (ptrdiff_t) (maximum - index)) == 0);
-
-                pm_strpbrk_invalid_multibyte_character(parser, source + start, source + index);
-            }
-        }
-    }
-
-    return NULL;
-}
-
-/**
- * This is the path when the encoding is ASCII-8BIT.
- */
-static inline const uint8_t *
-pm_strpbrk_ascii_8bit(const uint8_t *source, const uint8_t *charset, size_t maximum) {
-    size_t index = 0;
-
-    while (index < maximum) {
-        if (strchr((const char *) charset, source[index]) != NULL) {
-            return source + index;
-        }
-
-        index++;
-    }
-
-    return NULL;
-}
-
 /**
  * This is the slow path that does care about the encoding.
  */
 static inline const uint8_t *
-pm_strpbrk_multi_byte(pm_parser_t *parser, const uint8_t *source, const uint8_t *charset, size_t maximum, bool validate) {
+pm_strpbrk_multi_byte(const pm_parser_t *parser, const uint8_t *source, const uint8_t *charset, size_t maximum) {
     size_t index = 0;
 
     while (index < maximum) {
@@ -78,40 +12,22 @@ pm_strpbrk_multi_byte(pm_parser_t *parser, const uint8_t *source, const uint8_t
             return source + index;
         }
 
-        if (source[index] < 0x80) {
-            index++;
-        } else {
-            size_t width = parser->encoding->char_width(source + index, (ptrdiff_t) (maximum - index));
-
-            if (width > 0) {
-                index += width;
-            } else if (!validate) {
-                index++;
-            } else {
-                // At this point we know we have an invalid multibyte character.
-                // We'll walk forward as far as we can until we find the next
-                // valid character so that we don't spam the user with a ton of
-                // the same kind of error.
-                const size_t start = index;
-
-                do {
-                    index++;
-                } while (index < maximum && parser->encoding->char_width(source + index, (ptrdiff_t) (maximum - index)) == 0);
-
-                pm_strpbrk_invalid_multibyte_character(parser, source + start, source + index);
-            }
+        size_t width = parser->encoding->char_width(source + index, (ptrdiff_t) (maximum - index));
+        if (width == 0) {
+            return NULL;
         }
+
+        index += width;
     }
 
     return NULL;
 }
 
 /**
- * This is the fast path that does not care about the encoding because we know
- * the encoding only supports single-byte characters.
+ * This is the fast path that does not care about the encoding.
  */
 static inline const uint8_t *
-pm_strpbrk_single_byte(pm_parser_t *parser, const uint8_t *source, const uint8_t *charset, size_t maximum, bool validate) {
+pm_strpbrk_single_byte(const uint8_t *source, const uint8_t *charset, size_t maximum) {
     size_t index = 0;
 
     while (index < maximum) {
@@ -119,27 +35,7 @@ pm_strpbrk_single_byte(pm_parser_t *parser, const uint8_t *source, const uint8_t
             return source + index;
         }
 
-        if (source[index] < 0x80 || !validate) {
-            index++;
-        } else {
-            size_t width = parser->encoding->char_width(source + index, (ptrdiff_t) (maximum - index));
-
-            if (width > 0) {
-                index += width;
-            } else {
-                // At this point we know we have an invalid multibyte character.
-                // We'll walk forward as far as we can until we find the next
-                // valid character so that we don't spam the user with a ton of
-                // the same kind of error.
-                const size_t start = index;
-
-                do {
-                    index++;
-                } while (index < maximum && parser->encoding->char_width(source + index, (ptrdiff_t) (maximum - index)) == 0);
-
-                pm_strpbrk_invalid_multibyte_character(parser, source + start, source + index);
-            }
-        }
+        index++;
     }
 
     return NULL;
@@ -161,20 +57,16 @@ pm_strpbrk_single_byte(pm_parser_t *parser, const uint8_t *source, const uint8_t
  *
  * Finally, we want to support encodings wherein the charset could contain
  * characters that are trailing bytes of multi-byte characters. For example, in
- * Shift_JIS, the backslash character can be a trailing byte. In that case we
+ * Shift-JIS, the backslash character can be a trailing byte. In that case we
  * need to take a slower path and iterate one multi-byte character at a time.
  */
 const uint8_t *
-pm_strpbrk(pm_parser_t *parser, const uint8_t *source, const uint8_t *charset, ptrdiff_t length, bool validate) {
+pm_strpbrk(const pm_parser_t *parser, const uint8_t *source, const uint8_t *charset, ptrdiff_t length) {
     if (length <= 0) {
         return NULL;
-    } else if (!parser->encoding_changed) {
-        return pm_strpbrk_utf8(parser, source, charset, (size_t) length, validate);
-    } else if (parser->encoding == PM_ENCODING_ASCII_8BIT_ENTRY) {
-        return pm_strpbrk_ascii_8bit(source, charset, (size_t) length);
-    } else if (parser->encoding->multibyte) {
-        return pm_strpbrk_multi_byte(parser, source, charset, (size_t) length, validate);
+    } else if (parser->encoding_changed && parser->encoding->multibyte) {
+        return pm_strpbrk_multi_byte(parser, source, charset, (size_t) length);
     } else {
-        return pm_strpbrk_single_byte(parser, source, charset, (size_t) length, validate);
+        return pm_strpbrk_single_byte(source, charset, (size_t) length);
     }
 }
diff --git a/templates/ext/prism/api_node.c.erb b/templates/ext/prism/api_node.c.erb
index 20b38107..f52d537d 100644
--- a/templates/ext/prism/api_node.c.erb
+++ b/templates/ext/prism/api_node.c.erb
@@ -36,17 +36,17 @@ pm_string_new(pm_string_t *string, rb_encoding *encoding) {
     return rb_enc_str_new((const char *) pm_string_source(string), pm_string_length(string), encoding);
 }
 
-// Create a Prism::Source object from the given parser.
+// Create a Prism::Source object from the given parser, after pm_parse() was called.
 VALUE
 pm_source_new(pm_parser_t *parser, rb_encoding *encoding) {
-    VALUE source = rb_enc_str_new((const char *) parser->start, parser->end - parser->start, encoding);
-    VALUE offsets = rb_ary_new_capa(parser->newline_list.size);
+    VALUE source_string = rb_enc_str_new((const char *) parser->start, parser->end - parser->start, encoding);
 
+    VALUE offsets = rb_ary_new_capa(parser->newline_list.size);
     for (size_t index = 0; index < parser->newline_list.size; index++) {
-        rb_ary_push(offsets, INT2FIX(parser->newline_list.offsets[index]));
+        rb_ary_push(offsets, ULONG2NUM(parser->newline_list.offsets[index]));
     }
 
-    VALUE source_argv[] = { source, LONG2NUM(parser->start_line), offsets };
+    VALUE source_argv[] = { source_string, LONG2NUM(parser->start_line), offsets };
     return rb_class_new_instance(3, source_argv, rb_cPrismSource);
 }
 
@@ -77,8 +77,7 @@ pm_node_stack_pop(pm_node_stack_node_t **stack) {
 }
 
 VALUE
-pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding) {
-    VALUE source = pm_source_new(parser, encoding);
+pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding, VALUE source) {
     ID *constants = calloc(parser->constant_pool.size, sizeof(ID));
 
     for (uint32_t index = 0; index < parser->constant_pool.size; index++) {
diff --git a/templates/java/org/prism/Loader.java.erb b/templates/java/org/prism/Loader.java.erb
index fe0df910..b6a7650d 100644
--- a/templates/java/org/prism/Loader.java.erb
+++ b/templates/java/org/prism/Loader.java.erb
@@ -1,6 +1,8 @@
 <%- string_type = Prism::JAVA_STRING_TYPE -%>
 package org.prism;
 
+import org.prism.Nodes;
+
 import java.lang.Short;
 import java.nio.ByteBuffer;
 import java.nio.ByteOrder;
@@ -12,8 +14,8 @@ import java.util.Locale;
 // @formatter:off
 public class Loader {
 
-    public static ParseResult load(byte[] serialized, Nodes.Source source) {
-        return new Loader(serialized, source).load();
+    public static ParseResult load(byte[] serialized, byte[] sourceBytes) {
+        return new Loader(serialized, sourceBytes).load();
     }
 
     // Overridable methods
@@ -85,9 +87,9 @@ public class Loader {
     <%- end -%>
     private ConstantPool constantPool;
 
-    protected Loader(byte[] serialized, Nodes.Source source) {
+    protected Loader(byte[] serialized, byte[] sourceBytes) {
         this.buffer = ByteBuffer.wrap(serialized).order(ByteOrder.nativeOrder());
-        this.source = source;
+        this.source = new Nodes.Source(sourceBytes);
     }
 
     protected ParseResult load() {
@@ -113,6 +115,7 @@ public class Loader {
         <%- end -%>
 
         source.setStartLine(loadVarSInt());
+        source.setLineOffsets(loadLineOffsets());
 
         ParseResult.MagicComment[] magicComments = loadMagicComments();
         Nodes.Location dataLocation = loadOptionalLocation();
@@ -134,7 +137,7 @@ public class Loader {
         MarkNewlinesVisitor visitor = new MarkNewlinesVisitor(source, newlineMarked);
         node.accept(visitor);
 
-        return new ParseResult(node, magicComments, dataLocation, errors, warnings);
+        return new ParseResult(node, magicComments, dataLocation, errors, warnings, source);
     }
 
     private byte[] loadEmbeddedString() {
@@ -159,6 +162,15 @@ public class Loader {
         }
     }
 
+    private int[] loadLineOffsets() {
+        int count = loadVarUInt();
+        int[] lineOffsets = new int[count];
+        for (int i = 0; i < count; i++) {
+            lineOffsets[i] = loadVarUInt();
+        }
+        return lineOffsets;
+    }
+
     private ParseResult.MagicComment[] loadMagicComments() {
         int count = loadVarUInt();
         ParseResult.MagicComment[] magicComments = new ParseResult.MagicComment[count];
diff --git a/templates/java/org/prism/Nodes.java.erb b/templates/java/org/prism/Nodes.java.erb
index 25a155aa..2f265c64 100644
--- a/templates/java/org/prism/Nodes.java.erb
+++ b/templates/java/org/prism/Nodes.java.erb
@@ -42,38 +42,19 @@ public abstract class Nodes {
 
     public static final class Source {
         public final byte[] bytes;
-        private int startLine;
-        private final int[] lineOffsets;
+        private int startLine = 1;
+        private int[] lineOffsets = null;
 
-        public Source(byte[] bytes) {
-            this(bytes, 1, computeLineOffsets(bytes));
+        Source(byte[] bytes) {
+          this.bytes = bytes;
         }
 
-        public Source(byte[] bytes, int startLine, int[] lineOffsets) {
-            assert lineOffsets[0] == 0;
-            this.bytes = bytes;
-            this.startLine = startLine;
-            this.lineOffsets = lineOffsets;
-        }
-
-        public void setStartLine(int startLine) {
+        void setStartLine(int startLine) {
             this.startLine = startLine;
         }
 
-        public static int[] computeLineOffsets(byte[] bytes) {
-            int[] lineOffsets = new int[8];
-            int lineOffsetsSize = 0;
-            lineOffsets[lineOffsetsSize++] = 0;
-
-            for (int i = 0; i < bytes.length; i++) {
-                if (bytes[i] == '\n') {
-                    if (lineOffsetsSize == lineOffsets.length) {
-                        lineOffsets = Arrays.copyOf(lineOffsets, lineOffsets.length * 2);
-                    }
-                    lineOffsets[lineOffsetsSize++] = i + 1;
-                }
-            }
-            return Arrays.copyOf(lineOffsets, lineOffsetsSize);
+        void setLineOffsets(int[] lineOffsets) {
+            this.lineOffsets = lineOffsets;
         }
 
         // 1-based
diff --git a/templates/javascript/src/deserialize.js.erb b/templates/javascript/src/deserialize.js.erb
index 2c979eff..4c85075d 100644
--- a/templates/javascript/src/deserialize.js.erb
+++ b/templates/javascript/src/deserialize.js.erb
@@ -218,6 +218,13 @@ export function deserialize(source, array) {
   // JavaScript.
   buffer.readVarInt();
 
+  // Skip past the line offsets, as there is no Source object yet in JavaScript.
+  // const lineOffsets = Array.from({ length: buffer.readVarInt() }, () => buffer.readVarInt());
+  const lineOffsetsCount = buffer.readVarInt();
+  for (let i = 0; i < lineOffsetsCount; i ++) {
+    buffer.readVarInt();
+  }
+
   const comments = Array.from({ length: buffer.readVarInt() }, () => ({
     type: buffer.readVarInt(),
     location: buffer.readLocation()
diff --git a/templates/lib/prism/serialize.rb.erb b/templates/lib/prism/serialize.rb.erb
index 58ec6e85..8568e537 100644
--- a/templates/lib/prism/serialize.rb.erb
+++ b/templates/lib/prism/serialize.rb.erb
@@ -79,7 +79,11 @@ module Prism
       end
 
       def load_start_line
-        source.start_line = load_varsint
+        source.instance_variable_set :@start_line, load_varsint
+      end
+
+      def load_line_offsets
+        source.instance_variable_set :@offsets, load_varuint.times.map { load_varuint }
       end
 
       def load_comments
@@ -118,6 +122,7 @@ module Prism
         tokens = load_tokens
         encoding = load_encoding
         load_start_line
+        load_line_offsets
         comments, magic_comments, data_loc, errors, warnings = load_metadata
         tokens.each { |token,| token.value.force_encoding(encoding) }
 
@@ -129,6 +134,7 @@ module Prism
         load_header
         load_encoding
         load_start_line
+        load_line_offsets
 
         comments, magic_comments, data_loc, errors, warnings = load_metadata
 
diff --git a/templates/src/serialize.c.erb b/templates/src/serialize.c.erb
index deda01f2..c9511bbf 100644
--- a/templates/src/serialize.c.erb
+++ b/templates/src/serialize.c.erb
@@ -128,6 +128,17 @@ pm_serialize_node(pm_parser_t *parser, pm_node_t *node, pm_buffer_t *buffer) {
     }
 }
 
+static void
+pm_serialize_newline_list(pm_newline_list_t *list, pm_buffer_t *buffer) {
+    uint32_t size = pm_sizet_to_u32(list->size);
+    pm_buffer_append_varuint(buffer, size);
+
+    for (uint32_t i = 0; i < size; i++) {
+        uint32_t offset = pm_sizet_to_u32(list->offsets[i]);
+        pm_buffer_append_varuint(buffer, offset);
+    }
+}
+
 static void
 pm_serialize_comment(pm_parser_t *parser, pm_comment_t *comment, pm_buffer_t *buffer) {
     // serialize type
@@ -214,14 +225,11 @@ pm_serialize_encoding(const pm_encoding_t *encoding, pm_buffer_t *buffer) {
     pm_buffer_append_string(buffer, encoding->name, encoding_length);
 }
 
-#line <%= __LINE__ + 1 %> "<%= File.basename(__FILE__) %>"
-/**
- * Serialize the encoding, metadata, nodes, and constant pool.
- */
-void
-pm_serialize_content(pm_parser_t *parser, pm_node_t *node, pm_buffer_t *buffer) {
+static void
+pm_serialize_metadata(pm_parser_t *parser, pm_buffer_t *buffer) {
     pm_serialize_encoding(parser->encoding, buffer);
     pm_buffer_append_varsint(buffer, parser->start_line);
+    pm_serialize_newline_list(&parser->newline_list, buffer);
 <%- unless Prism::SERIALIZE_ONLY_SEMANTICS_FIELDS -%>
     pm_serialize_comment_list(parser, &parser->comment_list, buffer);
 <%- end -%>
@@ -229,6 +237,15 @@ pm_serialize_content(pm_parser_t *parser, pm_node_t *node, pm_buffer_t *buffer)
     pm_serialize_data_loc(parser, buffer);
     pm_serialize_diagnostic_list(parser, &parser->error_list, buffer);
     pm_serialize_diagnostic_list(parser, &parser->warning_list, buffer);
+}
+
+#line <%= __LINE__ + 1 %> "<%= File.basename(__FILE__) %>"
+/**
+ * Serialize the metadata, nodes, and constant pool.
+ */
+void
+pm_serialize_content(pm_parser_t *parser, pm_node_t *node, pm_buffer_t *buffer) {
+    pm_serialize_metadata(parser, buffer);
 
     // Here we're going to leave space for the offset of the constant pool in
     // the buffer.
@@ -319,13 +336,7 @@ pm_serialize_lex(pm_buffer_t *buffer, const uint8_t *source, size_t size, const
     // Append 0 to mark end of tokens.
     pm_buffer_append_byte(buffer, 0);
 
-    pm_serialize_encoding(parser.encoding, buffer);
-    pm_buffer_append_varsint(buffer, parser.start_line);
-    pm_serialize_comment_list(&parser, &parser.comment_list, buffer);
-    pm_serialize_magic_comment_list(&parser, &parser.magic_comment_list, buffer);
-    pm_serialize_data_loc(&parser, buffer);
-    pm_serialize_diagnostic_list(&parser, &parser.error_list, buffer);
-    pm_serialize_diagnostic_list(&parser, &parser.warning_list, buffer);
+    pm_serialize_metadata(&parser, buffer);
 
     pm_node_destroy(&parser, node);
     pm_parser_free(&parser);
diff --git a/templates/src/token_type.c.erb b/templates/src/token_type.c.erb
index 493d46f1..ebcbb5e0 100644
--- a/templates/src/token_type.c.erb
+++ b/templates/src/token_type.c.erb
@@ -350,7 +350,7 @@ pm_token_type_human(pm_token_type_t token_type) {
         case PM_TOKEN_UPLUS:
             return "'+'";
         case PM_TOKEN_USTAR:
-            return "*";
+            return "'*'";
         case PM_TOKEN_USTAR_STAR:
             return "'**'";
         case PM_TOKEN_WORDS_SEP:


Test patch:
diff --git a/java-wasm/src/test/java/org/prism/DummyTest.java b/java-wasm/src/test/java/org/prism/DummyTest.java
index 69c99e8e..72230e84 100644
--- a/java-wasm/src/test/java/org/prism/DummyTest.java
+++ b/java-wasm/src/test/java/org/prism/DummyTest.java
@@ -6,6 +6,7 @@ import com.dylibso.chicory.wasm.types.Value;
 import org.junit.jupiter.api.Test;
 
 import java.io.File;
+import java.nio.charset.StandardCharsets;
 import java.util.Arrays;
 
 import static org.junit.jupiter.api.Assertions.assertEquals;
@@ -26,6 +27,7 @@ public class DummyTest {
 
         // The Ruby source code to be processed
         var source = "1 + 1";
+        var sourceBytes = source.getBytes(StandardCharsets.US_ASCII);
 
         var sourcePointer = calloc.apply(Value.i32(1), Value.i32(source.length()));
         memory.writeString(sourcePointer[0].asInt(), source);
@@ -46,9 +48,7 @@ public class DummyTest {
 
         System.out.println("RESULT: " + new String(result));
 
-        var byteResults = result;
-        Nodes.Source nodeSource = new Nodes.Source(byteResults);
-        ParseResult pr = Loader.load(byteResults, nodeSource);
+        ParseResult pr = Loader.load(result, sourceBytes);
 
         assertEquals(1, pr.value.childNodes().length);
         System.out.println("Nodes:");
@@ -69,6 +69,7 @@ public class DummyTest {
 
         // The Ruby source code to be processed
         var source = "puts \"h\ne\nl\nl\no\n\"";
+        var sourceBytes = source.getBytes(StandardCharsets.US_ASCII);
 
         var sourcePointer = calloc.apply(Value.i32(1), Value.i32(source.length()));
         memory.writeString(sourcePointer[0].asInt(), source);
@@ -89,9 +90,7 @@ public class DummyTest {
 
         System.out.println("RESULT: " + new String(result));
 
-        var byteResults = result;
-        Nodes.Source nodeSource = new Nodes.Source(byteResults);
-        ParseResult pr = Loader.load(byteResults, nodeSource);
+        ParseResult pr = Loader.load(result, sourceBytes);
 
         assertEquals(1, pr.value.childNodes().length);
 
diff --git a/test/prism/location_test.rb b/test/prism/location_test.rb
index c6071705..e6ef82b2 100644
--- a/test/prism/location_test.rb
+++ b/test/prism/location_test.rb
@@ -178,10 +178,6 @@ module Prism
       assert_location(CallNode, "-> { it }", 5...7, version: "3.3.0") do |node|
         node.body.body.first
       end
-
-      assert_location(LocalVariableReadNode, "-> { it }", 5...7, version: "3.4.0") do |node|
-        node.body.body.first
-      end
     end
 
     def test_CallAndWriteNode
diff --git a/test/prism/parse_test.rb b/test/prism/parse_test.rb
index 3a2bc371..574a1c17 100644
--- a/test/prism/parse_test.rb
+++ b/test/prism/parse_test.rb
@@ -52,6 +52,9 @@ module Prism
 
       assert_equal line, result.value.location.start_line
       assert_equal line + 1, find_source_file_node(result.value).location.start_line
+
+      result = Prism.parse_lex("def foo\n __FILE__\nend", line: line)
+      assert_equal line, result.value.first.location.start_line
     end
 
     def test_parse_takes_negative_lines
@@ -60,6 +63,9 @@ module Prism
 
       assert_equal line, result.value.location.start_line
       assert_equal line + 1, find_source_file_node(result.value).location.start_line
+
+      result = Prism.parse_lex("def foo\n __FILE__\nend", line: line)
+      assert_equal line, result.value.first.location.start_line
     end
 
     def test_parse_lex
diff --git a/test/prism/ruby_api_test.rb b/test/prism/ruby_api_test.rb
index ff69ef54..cf7ea437 100644
--- a/test/prism/ruby_api_test.rb
+++ b/test/prism/ruby_api_test.rb
@@ -116,86 +116,6 @@ module Prism
       assert_equal 7, location.end_character_column
     end
 
-    def test_location_code_units
-      program = Prism.parse("😀 + 😀\n😍 ||= 😍").value
-
-      # first 😀
-      location = program.statements.body.first.receiver.location
-
-      assert_equal 0, location.start_code_units_offset(Encoding::UTF_8)
-      assert_equal 0, location.start_code_units_offset(Encoding::UTF_16LE)
-      assert_equal 0, location.start_code_units_offset(Encoding::UTF_32LE)
-
-      assert_equal 1, location.end_code_units_offset(Encoding::UTF_8)
-      assert_equal 2, location.end_code_units_offset(Encoding::UTF_16LE)
-      assert_equal 1, location.end_code_units_offset(Encoding::UTF_32LE)
-
-      assert_equal 0, location.start_code_units_column(Encoding::UTF_8)
-      assert_equal 0, location.start_code_units_column(Encoding::UTF_16LE)
-      assert_equal 0, location.start_code_units_column(Encoding::UTF_32LE)
-
-      assert_equal 1, location.end_code_units_column(Encoding::UTF_8)
-      assert_equal 2, location.end_code_units_column(Encoding::UTF_16LE)
-      assert_equal 1, location.end_code_units_column(Encoding::UTF_32LE)
-
-      # second 😀
-      location = program.statements.body.first.arguments.arguments.first.location
-
-      assert_equal 4, location.start_code_units_offset(Encoding::UTF_8)
-      assert_equal 5, location.start_code_units_offset(Encoding::UTF_16LE)
-      assert_equal 4, location.start_code_units_offset(Encoding::UTF_32LE)
-
-      assert_equal 5, location.end_code_units_offset(Encoding::UTF_8)
-      assert_equal 7, location.end_code_units_offset(Encoding::UTF_16LE)
-      assert_equal 5, location.end_code_units_offset(Encoding::UTF_32LE)
-
-      assert_equal 4, location.start_code_units_column(Encoding::UTF_8)
-      assert_equal 5, location.start_code_units_column(Encoding::UTF_16LE)
-      assert_equal 4, location.start_code_units_column(Encoding::UTF_32LE)
-
-      assert_equal 5, location.end_code_units_column(Encoding::UTF_8)
-      assert_equal 7, location.end_code_units_column(Encoding::UTF_16LE)
-      assert_equal 5, location.end_code_units_column(Encoding::UTF_32LE)
-
-      # first 😍
-      location = program.statements.body.last.name_loc
-
-      assert_equal 6, location.start_code_units_offset(Encoding::UTF_8)
-      assert_equal 8, location.start_code_units_offset(Encoding::UTF_16LE)
-      assert_equal 6, location.start_code_units_offset(Encoding::UTF_32LE)
-
-      assert_equal 7, location.end_code_units_offset(Encoding::UTF_8)
-      assert_equal 10, location.end_code_units_offset(Encoding::UTF_16LE)
-      assert_equal 7, location.end_code_units_offset(Encoding::UTF_32LE)
-
-      assert_equal 0, location.start_code_units_column(Encoding::UTF_8)
-      assert_equal 0, location.start_code_units_column(Encoding::UTF_16LE)
-      assert_equal 0, location.start_code_units_column(Encoding::UTF_32LE)
-
-      assert_equal 1, location.end_code_units_column(Encoding::UTF_8)
-      assert_equal 2, location.end_code_units_column(Encoding::UTF_16LE)
-      assert_equal 1, location.end_code_units_column(Encoding::UTF_32LE)
-
-      # second 😍
-      location = program.statements.body.last.value.location
-
-      assert_equal 12, location.start_code_units_offset(Encoding::UTF_8)
-      assert_equal 15, location.start_code_units_offset(Encoding::UTF_16LE)
-      assert_equal 12, location.start_code_units_offset(Encoding::UTF_32LE)
-
-      assert_equal 13, location.end_code_units_offset(Encoding::UTF_8)
-      assert_equal 17, location.end_code_units_offset(Encoding::UTF_16LE)
-      assert_equal 13, location.end_code_units_offset(Encoding::UTF_32LE)
-
-      assert_equal 6, location.start_code_units_column(Encoding::UTF_8)
-      assert_equal 7, location.start_code_units_column(Encoding::UTF_16LE)
-      assert_equal 6, location.start_code_units_column(Encoding::UTF_32LE)
-
-      assert_equal 7, location.end_code_units_column(Encoding::UTF_8)
-      assert_equal 9, location.end_code_units_column(Encoding::UTF_16LE)
-      assert_equal 7, location.end_code_units_column(Encoding::UTF_32LE)
-    end
-
     def test_heredoc?
       refute parse_expression("\"foo\"").heredoc?
       refute parse_expression("\"foo \#{1}\"").heredoc?