Repository: segmentio/parquet-go
Pull request #242 (2022-06-16 05:38:15)

Update documentation + add convenience APIs
This PR updates the README and code documentation to reflect the changes we introduced in #186, and add a few new APIs to simplify reading and writing parquet files.


Patch:
diff --git a/README.md b/README.md
index 2992351..d61c00b 100644
--- a/README.md
+++ b/README.md
@@ -35,7 +35,9 @@ dependency on and install with the following command:
 go get github.com/segmentio/parquet-go
 ```
 
-Go 1.17 or later is required to use the package.
+Go 1.18 or later is required to use the package. As a backward-compatibility
+mechanism, the package can also be built with Go 1.17, in which case the APIs
+based on Generics are disabled.
 
 ### Compatibility Guarantees
 
@@ -46,34 +48,46 @@ use cases that the library solves for. These occurrences are expected to be rare
 in frequency and documentation will be produce to guide users on how to adapt
 their programs to breaking changes.
 
-One planned breaking change will be to support generics, which were released in
-Go 1.18 and will simplify the code and provide better type safety.
-The targetted timeline for this change is end of 2022; programs depending on
-parquet-go will be expected to compile with Go 1.18 by then, at which point
-we will drop compatibility with Go 1.17.
-
 ## Usage
 
 The following sections describe how to use APIs exposed by the library,
 highlighting the use cases with code examples to demonstrate how they are used
 in practice.
 
-### Writing Parquet Files: [parquet.Writer](https://pkg.go.dev/github.com/segmentio/parquet-go#Writer)
+### Writing Parquet Files: [parquet.GenericWriter[T]](https://pkg.go.dev/github.com/segmentio/parquet-go#GenericWriter)
 
 A parquet file is a collection of rows sharing the same schema, arranged in
 columns to support faster scan operations on subsets of the data set.
 
-The `parquet.Writer` type denormalizes rows into columns, then encodes the
-columns into a parquet file, generating row groups, column chunks, and pages
+For simple use cases, the `parquet.WriteFile[T]` function allows the creation
+of parquet files on the file system from a slice of Go values representing the
+rows to write to the file.
+
+```go
+type RowType struct { FirstName, LastName string }
+
+if err := parquet.WriteFile("file.parquet", []RowType{
+    {FirstName: "Bob"},
+    {FirstName: "Alice"},
+}); err != nil {
+    ...
+}
+```
+
+The `parquet.GenericWriter[T]` type denormalizes rows into columns, then encodes
+the columns into a parquet file, generating row groups, column chunks, and pages
 based on configurable heuristics.
 
 ```go
-writer := parquet.NewWriter(file)
+type RowType struct { FirstName, LastName string }
 
-for _, row := range rows {
-    if err := writer.Write(row); err != nil {
-        ...
-    }
+writer := parquet.NewGenericWriter[RowType](output)
+
+_, err := writer.Write([]RowType{
+    ...
+})
+if err != nil {
+    ...
 }
 
 // Closing the writer is necessary to flush buffers and write the file footer.
@@ -82,49 +96,35 @@ if err := writer.Close(); err != nil {
 }
 ```
 
-By default, the writer will lazily determine the schema of rows by introspecting
-the struct types of row values.
-
 Explicit declaration of the parquet schema on a writer is useful when the
 application needs to ensure that data written to a file adheres to a predefined
-schema and the rows come from dynamic or user input. The `parquet.Schema` type
-is a in-memory representation of the schema of parquet rows, translated from the
-type of Go values, and can be used for this purpose.
+schema, which may differ from the schema derived from the writer's type
+parameter. The `parquet.Schema` type is a in-memory representation of the schema
+of parquet rows, translated from the type of Go values, and can be used for this
+purpose.
 
 ```go
-schema := parquet.SchemaOf(rows[0])
-writer := parquet.NewWriter(file, schema)
+schema := parquet.SchemaOf(new(RowType))
+writer := parquet.NewGenericWriter[any](output, schema)
 ...
 ```
 
-### Reading Parquet Files: [parquet.Reader](https://pkg.go.dev/github.com/segmentio/parquet-go#Reader)
-
-The `parquet.Reader` type supports reading rows from parquet files into Go
-values. When reading rows, the schema is already determined by metadata within
-the file; the reader knows how to leverage this information so the application
-does not need to explicitly declare the schema of values that will be read.
-However, the reader will validate that the schemas of the file and Go value
-are compatible.
+### Reading Parquet Files: [parquet.GenericReader[T]](https://pkg.go.dev/github.com/segmentio/parquet-go#GenericReader)
 
-This example shows how a `parquet.Reader` is typically used:
+For simple use cases where the data set fits in memory and the program will
+read most rows of the file, the `parquet.ReadFile[T]` function returns a slice
+of Go values representing the rows read from the file.
 
 ```go
-reader := parquet.NewReader(file)
+type RowType struct { FirstName, LastName string }
 
-for {
-    row := new(RowType)
-    err := reader.Read(row)
-    if err != nil {
-        if err == io.EOF {
-            break
-        }
-        ...
-    }
+rows, err := parquet.ReadFile[RowType]("file.parquet")
+if err != nil {
     ...
 }
 
-if err := reader.Close(); err != nil {
-    ...
+for _, c := range rows {
+    fmt.Printf("%+v\n", c)
 }
 ```
 
@@ -189,8 +189,11 @@ rules from a source to a target schema. The function is used to build converted
 views of `parquet.RowReader` or `parquet.RowGroup`, for example:
 
 ```go
-source := parquet.NewSchema(&RowTypeV1{})
-target := parquet.NewSchema(&RowTypeV2{})
+type RowTypeV1 struct { ID int64; FirstName string }
+type RowTypeV2 struct { ID int64; FirstName, LastName string }
+
+source := parquet.NewSchema(RowTypeV1{})
+target := parquet.NewSchema(RowTypeV2{})
 
 conversion, err := parquet.Convert(target, source)
 if err != nil {
@@ -212,48 +215,52 @@ At this time, conversion rules only supports adding or removing columns from
 the schemas, there are no type conversions performed, nor ways to rename
 columns, etc... More advanced conversion rules may be added in the future.
 
-### Sorting Row Groups: [parquet.Buffer](https://pkg.go.dev/github.com/segmentio/parquet-go#Buffer)
+### Sorting Row Groups: [parquet.GenericBuffer[T]](https://pkg.go.dev/github.com/segmentio/parquet-go#Buffer)
 
-The `parquet.Writer` type is optimized for minimal memory usage, keeping the
-order rows unchanged and flushing pages as soon as they are filled.
+The `parquet.GenericWriter[T]` type is optimized for minimal memory usage,
+keeping the order of rows unchanged and flushing pages as soon as they are filled.
 
 Parquet supports expressing columns by which rows are sorted through the
 declaration of _sorting columns_ on row groups. Sorting row groups requires
 buffering all rows before ordering and writing them to a parquet file.
 
 To help with those use cases, the `segmentio/parquet-go` package exposes the
-`parquet.Buffer` type which acts as a buffer of rows and implements
+`parquet.GenericBuffer[T]` type which acts as a buffer of rows and implements
 `sort.Interface` to allow applications to sort rows prior to writing them
 to a file.
 
 The columns that rows are ordered by are configured when creating
-`parquet.Buffer` instances using the `parquet.SortingColumns` function to
-construct row group options configuring the buffer. The type of parquet columns
-defines how values are compared, see [Parquet Logical Types](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md)
+`parquet.GenericBuffer[T]` instances using the `parquet.SortingColumns` function
+to construct row group options configuring the buffer. The type of parquet
+columns defines how values are compared, see [Parquet Logical Types](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md)
 for details.
 
 When written to a file, the buffer is materialized into a single row group with
 the declared sorting columns. After being written, buffers can be reused by
 calling their `Reset` method.
 
-The following example shows how to use a `parquet.Buffer` to order rows written
-to a parquet file:
+The following example shows how to use a `parquet.GenericBuffer[T]` to order rows
+written to a parquet file:
 
 ```go
-buffer := parquet.NewBuffer(
+type RowType struct { FirstName, LastName string }
+
+buffer := parquet.NewGenericBuffer[RowType](
     parquet.SortingColumns(
         parquet.Ascending("LastName"),
         parquet.Ascending("FistName"),
     ),
 )
 
-buffer.Write(&Character{FirstName: "Luke", LastName: "Skywalker"})
-buffer.Write(&Character{FirstName: "Han", LastName: "Solo"})
-buffer.Write(&Character{FirstName: "Anakin", LastName: "Skywalker"})
+buffer.Write([]RowType{
+    {FirstName: "Luke", LastName: "Skywalker"},
+    {FirstName: "Han", LastName: "Solo"},
+    {FirstName: "Anakin", LastName: "Skywalker"},
+})
 
 sort.Sort(buffer)
 
-writer := parquet.NewWriter(output)
+writer := parquet.NewGenericWriter[RowType](output)
 _, err := parquet.CopyRows(writer, buffer.Rows())
 if err != nil {
     ...
@@ -295,7 +302,7 @@ if err != nil {
     ...
 }
 
-writer := parquet.NewWriter(output)
+writer := parquet.NewGenericWriter[RowType](output)
 _, err := parquet.CopyRows(writer, merge)
 if err != nil {
     ...
@@ -316,7 +323,12 @@ configure the list of columns to create filters for using the `parquet.BloomFilt
 option when instantiating writers; for example:
 
 ```go
-writer := parquet.NewWriter(output,
+type RowType struct {
+    FirstName string `parquet:"first_name"`
+    LastName  string `parquet:"last_name"`
+}
+
+writer := parquet.NewGenericWriter[RowType](output,
     parquet.BloomFilters(
         // Configures the write to generate split-block bloom filters for the
         // "first_name" and "last_name" columns of the parquet schema of rows
@@ -331,11 +343,11 @@ writer := parquet.NewWriter(output,
 Generating bloom filters requires to know how many values exist in a column
 chunk in order to properly size the filter, which requires buffering all the
 values written to the column in memory. Because of it, the memory footprint
-of `parquet.Writer` increases linearly with the number of columns that the
-writer needs to generate filters for. This extra cost is optimized away when
-rows are copied from a `parquet.Buffer` to a writer, since in this case the
-number of values per column in known since the buffer already holds all the
-values in memory.
+of `parquet.GenericWriter[T]` increases linearly with the number of columns
+that the writer needs to generate filters for. This extra cost is optimized
+away when rows are copied from a `parquet.GenericBuffer[T]` to a writer, since
+in this case the number of values per column in known since the buffer already
+holds all the values in memory.
 
 When reading parquet files, column chunks expose the generated bloom filters
 with the `parquet.ColumnChunk.BloomFilter` method, returning a
@@ -432,11 +444,11 @@ values, for example slices of primitive Go types like `[]float32`; this would be
 the case if the application is built on top of a framework like
 [Apache Arrow](https://pkg.go.dev/github.com/apache/arrow/go/arrow).
 
-`parquet.Buffer` is an implementation of the `parquet.RowGroup` interface which
-maintains in-memory buffers of column values. Rows can be written by either
-boxing primitive values into arrays of `parquet.Value`, or type asserting the
-columns to a access specialized versions of the write methods accepting arrays
-of Go primitive types.
+`parquet.GenericBuffer[T]` is an implementation of the `parquet.RowGroup`
+interface which maintains in-memory buffers of column values. Rows can be
+written by either boxing primitive values into arrays of `parquet.Value`,
+or type asserting the columns to a access specialized versions of the write
+methods accepting arrays of Go primitive types.
 
 When using either of these models, the application is responsible for ensuring
 that the same number of rows are written to each column or the resulting parquet
@@ -446,23 +458,22 @@ The following examples demonstrate how to use these two models to write columns
 of Go values:
 
 ```go
-func writeColumns(buffer *parquet.Buffer, columns [3][]interface{}) error {
-    values := make([]parquet.Value, len(columns[0]))
-    for i := range columns {
-        c := buffer.ColumnBuffers()[i]
-        for j, v := range columns[i] {
-            values[j] = parquet.ValueOf(v)
-        }
-        if _, err := c.WriteValues(values); err != nil {
-            return err
-        }
+type RowType struct { FirstName, LastName string }
+
+func writeColumns(buffer *parquet.GenericBuffer[RowType], firstNames []string) error {
+    values := make([]parquet.Value, len(firstNames))
+    for i := range firstNames {
+        values[i] = parquet.ValueOf(firstNames[i])
     }
-    return nil
+    _, err := buffer.ColumnBuffers()[0].WriteValues(values)
+    return err
 }
 ```
 
 ```go
-func writeColumns(buffer *parquet.Buffer, ids []int64, values []float32) error {
+type RowType struct { ID int64; Value float32 }
+
+func writeColumns(buffer *parquet.GenericBuffer[RowType], ids []int64, values []float32) error {
     if len(ids) != len(values) {
         return fmt.Errorf("number of ids and values mismatch: ids=%d values=%d", len(ids), len(values))
     }
@@ -491,7 +502,7 @@ to expose column values of the row group.
 
 This model can be preferable when the underlying storage or in-memory
 representation of the data needs to be optimized further than what can be
-achieved by using an intermediary buffering layer with `parquet.Buffer`.
+achieved by using an intermediary buffering layer with `parquet.GenericBuffer[T]`.
 
 See [parquet.RowGroup](https://pkg.go.dev/github.com/segmentio/parquet-go#RowGroup)
 for the full interface documentation.
@@ -503,18 +514,21 @@ can create the row group. This may require significant amounts of memory as the
 entire file content must be buffered prior to generating it. In some cases, the
 files might even be larger than the amount of memory available to the program.
 
-The `parquet.Writer` can be configured to use disk storage instead as a scratch
-buffer when generating files, by configuring a different page buffer pool using
-the `parquet.ColumnPageBuffers` option and `parquet.PageBufferPool` interface.
+The `parquet.GenericWriter[T]` can be configured to use disk storage instead as
+a scratch buffer when generating files, by configuring a different page buffer
+pool using the `parquet.ColumnPageBuffers` option and `parquet.PageBufferPool`
+interface.
 
-The parquet-go package provides an implementation of the interface which uses
-temporary files to store pages while a file is generated, allowing programs to
-use local storage as swap space to hold pages and keep memory utilization to a
-minimum. The following example demonstrates how to configure a parquet writer
-to use on-disk page buffers:
+The `segmentio/parquet-go` package provides an implementation of the interface
+which uses temporary files to store pages while a file is generated, allowing
+programs to use local storage as swap space to hold pages and keep memory
+utilization to a minimum. The following example demonstrates how to configure
+a parquet writer to use on-disk page buffers:
 
 ```go
-writer := parquet.NewWriter(output,
+type RowType struct { ... }
+
+writer := parquet.NewGenericWriter[RowType](output,
     parquet.ColumnPageBuffers(
         parquet.NewFileBufferPool("", "buffers.*"),
     ),
diff --git a/bloom.go b/bloom.go
index b2ea7ec..8918e54 100644
--- a/bloom.go
+++ b/bloom.go
@@ -177,7 +177,7 @@ func (splitBlockEncoding) EncodeByteArray(dst, src []byte) ([]byte, error) {
 	filter := bloom.MakeSplitBlockFilter(dst)
 	buffer := make([]uint64, 0, filterEncodeBufferSize)
 
-	err := plain.RangeByteArrays(src, func(value []byte) error {
+	err := plain.RangeByteArray(src, func(value []byte) error {
 		if len(buffer) == cap(buffer) {
 			filter.InsertBulk(buffer)
 			buffer = buffer[:0]
diff --git a/column_buffer.go b/column_buffer.go
index dfb07c3..f3587ac 100644
--- a/column_buffer.go
+++ b/column_buffer.go
@@ -1507,7 +1507,7 @@ func (col *byteArrayColumnBuffer) WriteByteArrays(values []byte) (int, error) {
 func (col *byteArrayColumnBuffer) writeByteArrays(values []byte) (count, bytes int, err error) {
 	baseCount, baseBytes := len(col.offsets), len(col.values)
 
-	err = plain.RangeByteArrays(values, func(value []byte) error {
+	err = plain.RangeByteArray(values, func(value []byte) error {
 		col.append(unsafecast.BytesToString(value))
 		return nil
 	})
diff --git a/column_index.go b/column_index.go
index efbc9eb..8df2702 100644
--- a/column_index.go
+++ b/column_index.go
@@ -693,13 +693,13 @@ func isMaxByteArrayValue(value []byte) bool {
 
 func splitByteArrays(data []byte) [][]byte {
 	length := 0
-	plain.RangeByteArrays(data, func([]byte) error {
+	plain.RangeByteArray(data, func([]byte) error {
 		length++
 		return nil
 	})
 	buffer := make([]byte, 0, len(data)-(4*length))
 	values := make([][]byte, 0, length)
-	plain.RangeByteArrays(data, func(value []byte) error {
+	plain.RangeByteArray(data, func(value []byte) error {
 		offset := len(buffer)
 		buffer = append(buffer, value...)
 		values = append(values, buffer[offset:])
diff --git a/encoding/delta/binary_packed.go b/encoding/delta/binary_packed.go
index 6e47e84..3228d57 100644
--- a/encoding/delta/binary_packed.go
+++ b/encoding/delta/binary_packed.go
@@ -323,8 +323,8 @@ func decodeInt32(dst, src []byte) ([]byte, []byte, error) {
 					miniBlockData = miniBlockData[:miniBlockSize]
 				}
 				src = src[len(miniBlockData):]
-				if cap(miniBlockData) < miniBlockSize+bitpack.Padding {
-					miniBlockTemp = resize(miniBlockTemp[:0], miniBlockSize+bitpack.Padding)
+				if cap(miniBlockData) < miniBlockSize+bitpack.PaddingInt32 {
+					miniBlockTemp = resize(miniBlockTemp[:0], miniBlockSize+bitpack.PaddingInt32)
 					miniBlockData = miniBlockTemp[:copy(miniBlockTemp, miniBlockData)]
 				}
 				miniBlockData = miniBlockData[:miniBlockSize]
@@ -386,8 +386,8 @@ func decodeInt64(dst, src []byte) ([]byte, []byte, error) {
 					miniBlockData = src[:miniBlockSize]
 				}
 				src = src[len(miniBlockData):]
-				if len(miniBlockData) < miniBlockSize+bitpack.Padding {
-					miniBlockTemp = resize(miniBlockTemp[:0], miniBlockSize+bitpack.Padding)
+				if len(miniBlockData) < miniBlockSize+bitpack.PaddingInt64 {
+					miniBlockTemp = resize(miniBlockTemp[:0], miniBlockSize+bitpack.PaddingInt64)
 					miniBlockData = miniBlockTemp[:copy(miniBlockTemp, miniBlockData)]
 				}
 				miniBlockData = miniBlockData[:miniBlockSize]
diff --git a/encoding/plain/plain.go b/encoding/plain/plain.go
index 43191f1..cd4aeac 100644
--- a/encoding/plain/plain.go
+++ b/encoding/plain/plain.go
@@ -71,7 +71,7 @@ func (e *Encoding) EncodeDouble(dst, src []byte) ([]byte, error) {
 }
 
 func (e *Encoding) EncodeByteArray(dst []byte, src []byte) ([]byte, error) {
-	if err := ValidateByteArrays(src); err != nil {
+	if err := ValidateByteArray(src); err != nil {
 		return dst[:0], encoding.Error(e, err)
 	}
 	return append(dst[:0], src...), nil
@@ -124,7 +124,7 @@ func (e *Encoding) DecodeDouble(dst, src []byte) ([]byte, error) {
 }
 
 func (e *Encoding) DecodeByteArray(dst, src []byte) ([]byte, error) {
-	if err := ValidateByteArrays(src); err != nil {
+	if err := ValidateByteArray(src); err != nil {
 		return dst[:0], encoding.Error(e, err)
 	}
 	return append(dst[:0], src...), nil
@@ -246,8 +246,8 @@ const (
 	errTooLarge
 )
 
-func ValidateByteArrays(b []byte) error {
-	switch validateByteArrays(b) {
+func ValidateByteArray(b []byte) error {
+	switch validateByteArray(b) {
 	case errTooShort:
 		return ErrTooShort(len(b))
 	case errTooLarge:
@@ -257,7 +257,7 @@ func ValidateByteArrays(b []byte) error {
 	}
 }
 
-func RangeByteArrays(b []byte, do func([]byte) error) (err error) {
+func RangeByteArray(b []byte, do func([]byte) error) (err error) {
 	for len(b) > 0 {
 		var v []byte
 		if v, b, err = NextByteArray(b); err != nil {
diff --git a/encoding/plain/plain_amd64.go b/encoding/plain/plain_amd64.go
index 96efec1..b537771 100644
--- a/encoding/plain/plain_amd64.go
+++ b/encoding/plain/plain_amd64.go
@@ -3,4 +3,4 @@
 package plain
 
 //go:noescape
-func validateByteArrays([]byte) status
+func validateByteArray([]byte) status
diff --git a/encoding/plain/plain_amd64.s b/encoding/plain/plain_amd64.s
index 748470c..5f29fd7 100644
--- a/encoding/plain/plain_amd64.s
+++ b/encoding/plain/plain_amd64.s
@@ -2,45 +2,40 @@
 
 #include "textflag.h"
 
-// validateByteArrays is implemented in assembly because it is the bottleneck in
+// validateByteArray is implemented in assembly because it is the bottleneck in
 // many cases when encoding byte arrays. This version of the function yields
 // ~50% better throughput than the code generated by the Go compiler.
 //
-// func validateByteArrays([]byte) status
-TEXT ·validateByteArrays(SB), NOSPLIT, $0-32
+// func validateByteArray([]byte) status
+TEXT ·validateByteArray(SB), NOSPLIT, $0-32
     MOVQ arg_base+0(FP), AX
     MOVQ arg_len+8(FP), BX
 
     CMPQ BX, $0
     JE done
 
-    XORQ SI, SI
-loop:
-    MOVQ BX, CX
-    SUBQ SI, CX
-    CMPQ CX, $4
+    CMPQ BX, $4
     JB errTooShort
 
-    MOVLQZX (AX)(SI*1), DX
-    ADDQ $4, SI
-    SUBQ $4, CX
-    CMPQ DX, CX
-    JA errTooShort
-    CMPQ DX, $0x7FFFFFFF
+    ADDQ AX, BX // end
+loop:
+    MOVL (AX), CX
+
+    CMPL CX, $0x7FFFFFFF
     JA errTooLarge
 
-    ADDQ DX, SI
-    CMPQ SI, BX
-    JNE loop
+    LEAQ 4(AX)(CX*1), AX
+    CMPQ AX, BX
+    JA errTooShort
+    JB loop
 done:
-    MOVQ $0, CX
-    MOVQ CX, ret+24(FP)
+    XORQ AX, AX
+return:
+    MOVQ AX, ret+24(FP)
     RET
 errTooShort:
-    MOVQ $1, CX
-    MOVQ CX, ret+24(FP)
-    RET
+    MOVQ $1, AX
+    JMP return
 errTooLarge:
-    MOVQ $2, CX
-    MOVQ CX, ret+24(FP)
-    RET
+    MOVQ $2, AX
+    JMP return
diff --git a/encoding/plain/plain_purego.go b/encoding/plain/plain_purego.go
index 4b4e7ad..5f9a50d 100644
--- a/encoding/plain/plain_purego.go
+++ b/encoding/plain/plain_purego.go
@@ -2,7 +2,7 @@
 
 package plain
 
-func validateByteArrays(b []byte) status {
+func validateByteArray(b []byte) status {
 	for i := 0; i < len(b); {
 		r := len(b) - i
 		if r < ByteArrayLengthSize {
diff --git a/encoding/rle/rle.go b/encoding/rle/rle.go
index 9fba3df..1aa03c3 100644
--- a/encoding/rle/rle.go
+++ b/encoding/rle/rle.go
@@ -341,7 +341,7 @@ func decodeInt32(dst, src []byte, bitWidth uint) ([]byte, error) {
 		return dst, errDecodeInvalidBitWidth("INT32", bitWidth)
 	}
 
-	buf := make([]byte, 2*bitpack.Padding)
+	buf := make([]byte, 2*bitpack.PaddingInt32)
 
 	for i := 0; i < len(src); {
 		u, n := binary.Uvarint(src[i:])
@@ -367,10 +367,10 @@ func decodeInt32(dst, src []byte, bitWidth uint) ([]byte, error) {
 			// buffer we can use it, otherwise we have to copy it to a larger
 			// location (which should rarely happen).
 			in := src[i : i+length]
-			if (cap(in) - len(in)) >= bitpack.Padding {
+			if (cap(in) - len(in)) >= bitpack.PaddingInt32 {
 				in = in[:cap(in)]
 			} else {
-				buf = resize(buf, len(in)+bitpack.Padding)
+				buf = resize(buf, len(in)+bitpack.PaddingInt32)
 				copy(buf, in)
 				in = buf
 			}
diff --git a/internal/bitpack/unpack.go b/internal/bitpack/unpack.go
index 0e948ea..752a6f7 100644
--- a/internal/bitpack/unpack.go
+++ b/internal/bitpack/unpack.go
@@ -1,15 +1,19 @@
 package bitpack
 
-// Padding is the padding expected to exist after the end of input buffers for
-// the unpacking algorithms to avoid reading beyond the end of the input.
-const Padding = 16
+// PaddingInt32 is the padding expected to exist after the end of input buffers
+// for the UnpackInt32 algorithm to avoid reading beyond the end of the input.
+const PaddingInt32 = 16
+
+// PaddingInt64 is the padding expected to exist after the end of input buffers
+// for the UnpackInt32 algorithm to avoid reading beyond the end of the input.
+const PaddingInt64 = 32
 
 // UnpackInt32 unpacks 32 bit integers from src to dst.
 //
 // The function unpacked len(dst) integers, it panics if src is too short to
 // contain len(dst) values of the given bit width.
 func UnpackInt32(dst []int32, src []byte, bitWidth uint) {
-	assertUnpack(src, len(dst), bitWidth)
+	_ = src[:ByteCount(bitWidth*uint(len(dst))+8*PaddingInt32)]
 	unpackInt32(dst, src, bitWidth)
 }
 
@@ -18,10 +22,6 @@ func UnpackInt32(dst []int32, src []byte, bitWidth uint) {
 // The function unpacked len(dst) integers, it panics if src is too short to
 // contain len(dst) values of the given bit width.
 func UnpackInt64(dst []int64, src []byte, bitWidth uint) {
-	assertUnpack(src, len(dst), bitWidth)
+	_ = src[:ByteCount(bitWidth*uint(len(dst))+8*PaddingInt64)]
 	unpackInt64(dst, src, bitWidth)
 }
-
-func assertUnpack(src []byte, count int, bitWidth uint) {
-	_ = src[:ByteCount(bitWidth*uint(count)+8*Padding)]
-}
diff --git a/internal/bitpack/unpack_int32_amd64.s b/internal/bitpack/unpack_int32_amd64.s
index 94f5d86..e0f2bf4 100644
--- a/internal/bitpack/unpack_int32_amd64.s
+++ b/internal/bitpack/unpack_int32_amd64.s
@@ -3,7 +3,7 @@
 #include "funcdata.h"
 #include "textflag.h"
 
-// func unpackInt32Default(dst []int32, src []uint32, bitWidth uint)
+// func unpackInt32Default(dst []int32, src []byte, bitWidth uint)
 TEXT ·unpackInt32Default(SB), NOSPLIT, $0-56
     MOVQ dst_base+0(FP), AX
     MOVQ dst_len+8(FP), DX
@@ -95,7 +95,7 @@ test:
 // In this version of the algorithm, we can perform a single memory load in each
 // loop iteration since we know that 8 values will fit in a single XMM register.
 //
-// func unpackInt32x1to16bitsAVX2(dst []int32, src []uint32, bitWidth uint)
+// func unpackInt32x1to16bitsAVX2(dst []int32, src []byte, bitWidth uint)
 TEXT ·unpackInt32x1to16bitsAVX2(SB), NOSPLIT, $56-56
     NO_LOCAL_POINTERS
     MOVQ dst_base+0(FP), AX
@@ -115,7 +115,7 @@ TEXT ·unpackInt32x1to16bitsAVX2(SB), NOSPLIT, $56-56
     SHLQ CX, R8
     DECQ R8
     MOVQ R8, X0
-    VPBROADCASTD X0, X0
+    VPBROADCASTD X0, X0 // bitMask = (1 << bitWidth) - 1
 
     MOVQ CX, R9
     DECQ R9
@@ -173,7 +173,7 @@ done:
 // In this version of the algorithm, we need to 32 bytes at each loop iteration
 // because 8 bit-packed values will span across two XMM registers.
 //
-// func unpackInt32x17to26bitsAVX2(dst []int32, src []uint32, bitWidth uint)
+// func unpackInt32x17to26bitsAVX2(dst []int32, src []byte, bitWidth uint)
 TEXT ·unpackInt32x17to26bitsAVX2(SB), NOSPLIT, $56-56
     NO_LOCAL_POINTERS
     MOVQ dst_base+0(FP), AX
@@ -262,7 +262,7 @@ done:
 //
 // The amount of LEFT shifts is always "8 minus the amount of RIGHT shift".
 //
-// func unpackInt32x27to31bitsAVX2(dst []int32, src []uint32, bitWidth uint)
+// func unpackInt32x27to31bitsAVX2(dst []int32, src []byte, bitWidth uint)
 TEXT ·unpackInt32x27to31bitsAVX2(SB), NOSPLIT, $56-56
     NO_LOCAL_POINTERS
     MOVQ dst_base+0(FP), AX
diff --git a/internal/bitpack/unpack_int64_amd64.go b/internal/bitpack/unpack_int64_amd64.go
index ac4b38a..011c0ca 100644
--- a/internal/bitpack/unpack_int64_amd64.go
+++ b/internal/bitpack/unpack_int64_amd64.go
@@ -2,13 +2,22 @@
 
 package bitpack
 
-import "github.com/segmentio/parquet-go/internal/unsafecast"
+import (
+	"github.com/segmentio/parquet-go/internal/unsafecast"
+	"golang.org/x/sys/cpu"
+)
 
 //go:noescape
 func unpackInt64Default(dst []int64, src []byte, bitWidth uint)
 
+//go:noescape
+func unpackInt64x1to32bitsAVX2(dst []int64, src []byte, bitWidth uint)
+
 func unpackInt64(dst []int64, src []byte, bitWidth uint) {
+	hasAVX2 := cpu.X86.HasAVX2
 	switch {
+	case hasAVX2 && bitWidth <= 32:
+		unpackInt64x1to32bitsAVX2(dst, src, bitWidth)
 	case bitWidth == 64:
 		copy(dst, unsafecast.BytesToInt64(src))
 	default:
diff --git a/internal/bitpack/unpack_int64_amd64.s b/internal/bitpack/unpack_int64_amd64.s
index 122ca27..91d28f6 100644
--- a/internal/bitpack/unpack_int64_amd64.s
+++ b/internal/bitpack/unpack_int64_amd64.s
@@ -67,3 +67,132 @@ test:
     CMPQ SI, DX
     JNE loop
     RET
+
+// This bit unpacking function was inspired from the 32 bit version, but
+// adapted to account for the fact that eight 64 bit values span across
+// two YMM registers, and across lanes of YMM registers.
+//
+// Because of the two lanes of YMM registers, we cannot use the VPSHUFB
+// instruction to dispatch bytes of the input to the registers. Instead we use
+// the VPERMD instruction, which has higher latency but supports dispatching
+// bytes across register lanes. Measurable throughput gains remain despite the
+// algorithm running on a few more CPU cycles per loop.
+//
+// The initialization phase of this algorithm generates masks for
+// permutations and shifts used to decode the bit-packed values.
+//
+// The permutation masks are written to Y7 and Y8, and contain the results
+// of this formula:
+//
+//      temp[i] = (bitWidth * i) / 32
+//      mask[i] = temp[i] | ((temp[i] + 1) << 32)
+//
+// Since VPERMQ only supports reading the permutation combination from an
+// immediate value, we use VPERMD and generate permutation for pairs of two
+// consecutive 32 bit words, which is why we have the upper part of each 64
+// bit word set with (x+1)<<32.
+//
+// The masks for right shifts are written to Y5 and Y6, and computed with
+// this formula:
+//
+//      shift[i] = (bitWidth * i) - (32 * ((bitWidth * i) / 32))
+//
+// The amount to shift by is the number of values previously unpacked, offseted
+// by the byte count of 32 bit words that we read from first bits from.
+//
+// Technically the masks could be precomputed and declared in global tables;
+// however, declaring masks for all bit width is tedious and makes code
+// maintenance more costly for no measurable benefits on production workloads.
+//
+// func unpackInt64x1to32bitsAVX2(dst []int64, src []byte, bitWidth uint)
+TEXT ·unpackInt64x1to32bitsAVX2(SB), NOSPLIT, $56-56
+    NO_LOCAL_POINTERS
+    MOVQ dst_base+0(FP), AX
+    MOVQ dst_len+8(FP), DX
+    MOVQ src_base+24(FP), BX
+    MOVQ bitWidth+48(FP), CX
+
+    CMPQ DX, $8
+    JB tail
+
+    MOVQ DX, DI
+    SHRQ $3, DI
+    SHLQ $3, DI
+    XORQ SI, SI
+
+    MOVQ $1, R8
+    SHLQ CX, R8
+    DECQ R8
+    MOVQ R8, X0
+    VPBROADCASTQ X0, Y0 // bitMask = (1 << bitWidth) - 1
+
+    VPCMPEQQ Y1, Y1, Y1
+    VPSRLQ $63, Y1, Y1  // [1,1,1,1]
+
+    MOVQ CX, X2
+    VPBROADCASTQ X2, Y2 // [bitWidth]
+
+    VMOVDQU range0n7<>+0(SB), Y3  // [0,1,2,3]
+    VMOVDQU range0n7<>+32(SB), Y4 // [4,5,6,7]
+
+    VPMULLD Y2, Y3, Y5 // [bitWidth] * [0,1,2,3]
+    VPMULLD Y2, Y4, Y6 // [bitWidth] * [4,5,6,7]
+
+    VPSRLQ $5, Y5, Y7 // ([bitWidth] * [0,1,2,3]) / 32
+    VPSRLQ $5, Y6, Y8 // ([bitWidth] * [4,5,6,7]) / 32
+
+    VPSLLQ $5, Y7, Y9  // (([bitWidth] * [0,1,2,3]) / 32) * 32
+    VPSLLQ $5, Y8, Y10 // (([bitWidth] * [4,5,6,7]) / 32) * 32
+
+    VPADDQ Y1, Y7, Y11
+    VPADDQ Y1, Y8, Y12
+    VPSLLQ $32, Y11, Y11
+    VPSLLQ $32, Y12, Y12
+    VPOR Y11, Y7, Y7 // permutations[i] = [i | ((i + 1) << 32)]
+    VPOR Y12, Y8, Y8 // permutations[i] = [i | ((i + 1) << 32)]
+
+    VPSUBQ Y9, Y5, Y5 // shifts
+    VPSUBQ Y10, Y6, Y6
+loop:
+    VMOVDQU (BX), Y1
+
+    VPERMD Y1, Y7, Y2
+    VPERMD Y1, Y8, Y3
+
+    VPSRLVQ Y5, Y2, Y2
+    VPSRLVQ Y6, Y3, Y3
+
+    VPAND Y0, Y2, Y2
+    VPAND Y0, Y3, Y3
+
+    VMOVDQU Y2, (AX)(SI*8)
+    VMOVDQU Y3, 32(AX)(SI*8)
+
+    ADDQ CX, BX
+    ADDQ $8, SI
+    CMPQ SI, DI
+    JNE loop
+    VZEROUPPER
+
+    CMPQ SI, DX
+    JE done
+    LEAQ (AX)(SI*8), AX
+    SUBQ SI, DX
+tail:
+    MOVQ AX, dst_base-56(SP)
+    MOVQ DX, dst_len-48(SP)
+    MOVQ BX, src_base-32(SP)
+    MOVQ CX, bitWidth-8(SP)
+    CALL ·unpackInt64Default(SB)
+done:
+    RET
+
+GLOBL range0n7<>(SB), RODATA|NOPTR, $64
+DATA range0n7<>+0(SB)/8,  $0
+DATA range0n7<>+8(SB)/8,  $1
+DATA range0n7<>+16(SB)/8, $2
+DATA range0n7<>+24(SB)/8, $3
+DATA range0n7<>+32(SB)/8, $4
+DATA range0n7<>+40(SB)/8, $5
+DATA range0n7<>+48(SB)/8, $6
+DATA range0n7<>+56(SB)/8, $7
diff --git a/internal/quick/quick.go b/internal/quick/quick.go
index bce7ef0..2bb51f2 100644
--- a/internal/quick/quick.go
+++ b/internal/quick/quick.go
@@ -2,6 +2,7 @@ package quick
 
 import (
 	"fmt"
+	"math"
 	"math/rand"
 	"reflect"
 	"strings"
@@ -77,7 +78,7 @@ func MakeValueFuncOf(t reflect.Type) MakeValueFunc {
 
 	case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
 		return func(v reflect.Value, r *rand.Rand) {
-			v.SetInt(r.Int63())
+			v.SetInt(r.Int63n(math.MaxInt32))
 		}
 
 	case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
diff --git a/parquet_go18.go b/parquet_go18.go
new file mode 100644
index 0000000..9ebf72b
--- /dev/null
+++ b/parquet_go18.go
@@ -0,0 +1,87 @@
+//go:build go1.18
+
+package parquet
+
+import (
+	"io"
+	"os"
+)
+
+// Read reads and returns rows from the parquet file in the given reader.
+//
+// The type T defines the type of rows read from r. T must be compatible with
+// the file's schema or an error will be returned. The row type might represent
+// a subset of the full schema, in which case only a subset of the columns will
+// be loaded from r.
+//
+// This function is provided for convenience to facilitate reading of parquet
+// files from arbitrary locations in cases where the data set fit in memory.
+func Read[T any](r io.ReaderAt, size int64, options ...ReaderOption) (rows []T, err error) {
+	config, err := NewReaderConfig(options...)
+	if err != nil {
+		return nil, err
+	}
+	file, err := OpenFile(r, size)
+	if err != nil {
+		return nil, err
+	}
+	rows = make([]T, file.NumRows())
+	reader := NewGenericReader[T](file, config)
+	n, err := reader.Read(rows)
+	if n < len(rows) && err == io.EOF {
+		rows, err = rows[:n], err
+	}
+	reader.Close()
+	return rows, err
+}
+
+// ReadFile reads rows of the parquet file at the given path.
+//
+// The type T defines the type of rows read from r. T must be compatible with
+// the file's schema or an error will be returned. The row type might represent
+// a subset of the full schema, in which case only a subset of the columns will
+// be loaded from the file.
+//
+// This function is provided for convenience to facilitate reading of parquet
+// files from the file system in cases where the data set fit in memory.
+func ReadFile[T any](path string, options ...ReaderOption) (rows []T, err error) {
+	f, err := os.Open(path)
+	if err != nil {
+		return nil, err
+	}
+	defer f.Close()
+	s, err := f.Stat()
+	if err != nil {
+		return nil, err
+	}
+	return Read[T](f, s.Size())
+}
+
+// Write writes the given list of rows to a parquet file written to w.
+//
+// This function is provided for convenience to facilitate the creation of
+// parquet files.
+func Write[T any](w io.Writer, rows []T, options ...WriterOption) error {
+	config, err := NewWriterConfig(options...)
+	if err != nil {
+		return err
+	}
+	writer := NewGenericWriter[T](w, config)
+	if _, err := writer.Write(rows); err != nil {
+		return err
+	}
+	return writer.Close()
+}
+
+// Write writes the given list of rows to a parquet file written to w.
+//
+// This function is provided for convenience to facilitate writing parquet
+// files to the file system.
+func WriteFile[T any](path string, rows []T, options ...WriterOption) error {
+	f, err := os.Create(path)
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	return Write(f, rows, options...)
+}
diff --git a/reader.go b/reader.go
index f87efaa..db68bb8 100644
--- a/reader.go
+++ b/reader.go
@@ -7,7 +7,7 @@ import (
 	"reflect"
 )
 
-// A Reader reads Go values from parquet files.
+// Deprecated: A Reader reads Go values from parquet files.
 //
 // This example showcases a typical use of parquet readers:
 //
@@ -29,6 +29,8 @@ import (
 //	}
 //
 //
+// For programs building with Go 1.18 or later, the GenericReader[T] type
+// supersedes this one.
 type Reader struct {
 	seen     reflect.Type
 	file     reader
diff --git a/writer.go b/writer.go
index aecac13..fbf77bb 100644
--- a/writer.go
+++ b/writer.go
@@ -18,8 +18,8 @@ import (
 	"github.com/segmentio/parquet-go/format"
 )
 
-// A Writer uses a parquet schema and sequence of Go values to produce a parquet
-// file to an io.Writer.
+// Deprecated: A Writer uses a parquet schema and sequence of Go values to
+// produce a parquet file to an io.Writer.
 //
 // This example showcases a typical use of parquet writers:
 //
@@ -41,6 +41,9 @@ import (
 // entire row group. Programs that need to produce parquet files with sorted
 // row groups should use the Buffer type to buffer and sort the rows prior to
 // writing them to a Writer.
+//
+// For programs building with Go 1.18 or later, the GenericWriter[T] type
+// supersedes this one.
 type Writer struct {
 	output io.Writer
 	config *WriterConfig


Test patch:
diff --git a/encoding/delta/length_byte_array_test.go b/encoding/delta/length_byte_array_test.go
index c7c14b6..32f7755 100644
--- a/encoding/delta/length_byte_array_test.go
+++ b/encoding/delta/length_byte_array_test.go
@@ -28,7 +28,7 @@ func TestDecodeLengthByteArray(t *testing.T) {
 	}
 
 	index := 0
-	err = plain.RangeByteArrays(dst, func(got []byte) error {
+	err = plain.RangeByteArray(dst, func(got []byte) error {
 		want := characters[:index%len(characters)]
 
 		if want != string(got) {
diff --git a/encoding/plain/plain_test.go b/encoding/plain/plain_test.go
index fa610dd..84eb7b3 100644
--- a/encoding/plain/plain_test.go
+++ b/encoding/plain/plain_test.go
@@ -33,14 +33,14 @@ func TestAppendBoolean(t *testing.T) {
 	}
 }
 
-func TestValidateByteArrays(t *testing.T) {
+func TestValidateByteArray(t *testing.T) {
 	t.Run("ok", func(t *testing.T) {
 		var b []byte
 		b = plain.AppendByteArrayString(b, "Hello")
 		b = plain.AppendByteArrayString(b, "World")
 		b = plain.AppendByteArrayString(b, "!")
 
-		if err := plain.ValidateByteArrays(b); err != nil {
+		if err := plain.ValidateByteArray(b); err != nil {
 			t.Error(err)
 		}
 	})
@@ -51,7 +51,7 @@ func TestValidateByteArrays(t *testing.T) {
 		b = plain.AppendByteArrayString(b, "World")
 		b = plain.AppendByteArrayString(b, "!")
 
-		if plain.ValidateByteArrays(b[:len(b)-1]) == nil {
+		if plain.ValidateByteArray(b[:len(b)-1]) == nil {
 			t.Error("expected non-nil error")
 		}
 	})
diff --git a/internal/bitpack/unpack_test.go b/internal/bitpack/unpack_test.go
index b906aec..e77bab6 100644
--- a/internal/bitpack/unpack_test.go
+++ b/internal/bitpack/unpack_test.go
@@ -25,7 +25,7 @@ func TestUnpackInt32(t *testing.T) {
 			}
 
 			size := (blockSize * bitWidth) / 8
-			buf := make([]byte, size+bitpack.Padding)
+			buf := make([]byte, size+bitpack.PaddingInt32)
 			bitpack.PackInt32(buf, block[:], bitWidth)
 
 			src := buf[:size]
@@ -39,7 +39,7 @@ func TestUnpackInt32(t *testing.T) {
 				bitpack.UnpackInt32(dst[:n], src, bitWidth)
 
 				if !reflect.DeepEqual(block[:n], dst[:n]) {
-					t.Errorf("values mismatch for length=%d\nwant: %v\ngot:  %v", n, block[:n], dst[:n])
+					t.Fatalf("values mismatch for length=%d\nwant: %v\ngot:  %v", n, block[:n], dst[:n])
 				}
 			}
 		})
@@ -58,7 +58,7 @@ func TestUnpackInt64(t *testing.T) {
 			}
 
 			size := (blockSize * bitWidth) / 8
-			buf := make([]byte, size+bitpack.Padding)
+			buf := make([]byte, size+bitpack.PaddingInt64)
 			bitpack.PackInt64(buf, block[:], bitWidth)
 
 			src := buf[:size]
@@ -72,7 +72,7 @@ func TestUnpackInt64(t *testing.T) {
 				bitpack.UnpackInt64(dst[:n], src, bitWidth)
 
 				if !reflect.DeepEqual(block[:n], dst[:n]) {
-					t.Errorf("values mismatch for length=%d\nwant: %v\ngot:  %v", n, block[:n], dst[:n])
+					t.Fatalf("values mismatch for length=%d\nwant: %v\ngot:  %v", n, block[:n], dst[:n])
 				}
 			}
 		})
@@ -82,7 +82,7 @@ func TestUnpackInt64(t *testing.T) {
 func BenchmarkUnpackInt32(b *testing.B) {
 	for bitWidth := uint(1); bitWidth <= 32; bitWidth++ {
 		block := [blockSize]int32{}
-		buf := [4*blockSize + bitpack.Padding]byte{}
+		buf := [4*blockSize + bitpack.PaddingInt32]byte{}
 		bitpack.PackInt32(buf[:], block[:], bitWidth)
 
 		b.Run(fmt.Sprintf("bitWidth=%d", bitWidth), func(b *testing.B) {
@@ -101,7 +101,7 @@ func BenchmarkUnpackInt32(b *testing.B) {
 func BenchmarkUnpackInt64(b *testing.B) {
 	for bitWidth := uint(1); bitWidth <= 64; bitWidth++ {
 		block := [blockSize]int64{}
-		buf := [8*blockSize + bitpack.Padding]byte{}
+		buf := [8*blockSize + bitpack.PaddingInt64]byte{}
 		bitpack.PackInt64(buf[:], block[:], bitWidth)
 
 		b.Run(fmt.Sprintf("bitWidth=%d", bitWidth), func(b *testing.B) {
diff --git a/parquet_go18_test.go b/parquet_go18_test.go
new file mode 100644
index 0000000..cf658b3
--- /dev/null
+++ b/parquet_go18_test.go
@@ -0,0 +1,50 @@
+//go:build go1.18
+
+package parquet_test
+
+import (
+	"fmt"
+	"log"
+
+	"github.com/segmentio/parquet-go"
+)
+
+func ExampleReadFile() {
+	type Row struct {
+		ID   int64  `parquet:"id"`
+		Name string `parquet:"name,zstd"`
+	}
+
+	ExampleWriteFile()
+
+	rows, err := parquet.ReadFile[Row]("/tmp/file.parquet")
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	for _, row := range rows {
+		fmt.Printf("%d: %q\n", row.ID, row.Name)
+	}
+
+	// Output:
+	// 0: "Bob"
+	// 1: "Alice"
+	// 2: "Franky"
+}
+
+func ExampleWriteFile() {
+	type Row struct {
+		ID   int64  `parquet:"id"`
+		Name string `parquet:"name,zstd"`
+	}
+
+	if err := parquet.WriteFile("/tmp/file.parquet", []Row{
+		{ID: 0, Name: "Bob"},
+		{ID: 1, Name: "Alice"},
+		{ID: 2, Name: "Franky"},
+	}); err != nil {
+		log.Fatal(err)
+	}
+
+	// Output:
+}
diff --git a/parquet_test.go b/parquet_test.go
index c0a977b..88615f0 100644
--- a/parquet_test.go
+++ b/parquet_test.go
@@ -58,7 +58,7 @@ func (row int32Column) generate(prng *rand.Rand) int32Column {
 }
 
 type int64Column struct {
-	Value int64
+	Value int64 `parquet:",delta"`
 }
 
 func (row int64Column) generate(prng *rand.Rand) int64Column {