Repository: ImperialCollegeLondon/pycsvy

Issue:
`pycsvy` won't work with Unicode data on Windows
Currently, `pycsvy`'s various reader functions don't explicitly specify a desired text-encoding to the underlying calls to `open()`, which means that the platform's default will be used.

On Linux and macOS, the default encoding is UTF-8 but for unfortunate legacy reasons the default encoding on Windows is `cp1252`. Even worse, this encoding doesn't support Unicode, so if you try writing data containing Unicode to file, you'll get an error :disappointed:. This actually crops up more often than you'd think, in data sets containing names with accents, for example (looking at you @dalonsoa...) and I've been bitten by it a few times recently.

It looks like UTF-8 will be made the default, but not until Python 3.15: https://peps.python.org/pep-0686/. In the meantime I think it probably makes sense to just make UTF-8 for `pycsvy`, regardless of the platform.

Another consideration is that users can specify the encoding in the YAML header. We have a couple of options here:

1. If the user specifies an encoding other than UTF-8, we raise an error/warning and use UTF-8 anyway
2. We use the user's specified encoding to parse the file (not sure whether we should also read the header with this encoding -- which will require reading it twice -- or just the CSV portion of the file)

Discussion:

Patch:
diff --git a/README.md b/README.md
index 18efd21..7005830 100644
--- a/README.md
+++ b/README.md
@@ -8,8 +8,8 @@
 [![codecov](https://codecov.io/gh/ImperialCollegeLondon/pycsvy/branch/develop/graph/badge.svg?token=N03KYNUD18)](https://codecov.io/gh/ImperialCollegeLondon/pycsvy)
 [![Codacy Badge](https://app.codacy.com/project/badge/Grade/8d1b791b315f4814a128d94483499561)](https://app.codacy.com/gh/ImperialCollegeLondon/pycsvy/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=ImperialCollegeLondon/pycsvy&amp;utm_campaign=Badge_Grade)
 [![pre-commit.ci status](https://results.pre-commit.ci/badge/github/ImperialCollegeLondon/pycsvy/develop.svg)](https://results.pre-commit.ci/latest/github/ImperialCollegeLondon/pycsvy/develop)
-<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
-[![All Contributors](https://img.shields.io/badge/all_contributors-6-orange.svg?style=flat-square)](#contributors-)
+<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
+[![All Contributors](https://img.shields.io/badge/all_contributors-6-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
 
 CSV is a popular format for storing tabular data used in many disciplines. Metadata
@@ -111,31 +111,37 @@ YAML-formatted header, respectively.
 You can also instruct a writer to use line buffering, instead of the usual chunk buffering.
 
 Finally, you can control the character(s) used to indicate comments by setting the
-`comment` keyword when writing a file. By default, there is no character (""). During reading, the comment character is found atomatically.
+`comment` keyword when writing a file. By default, there is no character ("").
+During reading, the comment character is found automatically.
+
+Note that, by default, these reader functions will assume UTF-8 encoding. You can choose a
+different character encoding by setting the `encoding` keyword argument to any of these
+reader or writer functions. For example, on Windows, Windows-1252 encoding is often used,
+which can be specified via `encoding='cp1252'`.
 
 ## Contributors ✨
 
 Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
 
-<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
-<!-- prettier-ignore-start -->
-<!-- markdownlint-disable -->
-<table>
-  <tbody>
-    <tr>
-      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/research-software-engineering/"><img src="https://avatars.githubusercontent.com/u/6095790?v=4?s=100" width="100px;" alt="Diego Alonso Álvarez"/><br /><sub><b>Diego Alonso Álvarez</b></sub></a><br /><a href="#infra-dalonsoa" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="#ideas-dalonsoa" title="Ideas, Planning, & Feedback">🤔</a> <a href="#maintenance-dalonsoa" title="Maintenance">🚧</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dalonsoa" title="Tests">⚠️</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/issues?q=author%3Adalonsoa" title="Bug reports">🐛</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dalonsoa" title="Code">💻</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/research-software-engineering"><img src="https://avatars.githubusercontent.com/u/23149834?v=4?s=100" width="100px;" alt="Alex Dewar"/><br /><sub><b>Alex Dewar</b></sub></a><br /><a href="#ideas-alexdewar" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=alexdewar" title="Tests">⚠️</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=alexdewar" title="Code">💻</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://github.com/AdrianDAlessandro"><img src="https://avatars.githubusercontent.com/u/40875798?v=4?s=100" width="100px;" alt="Adrian D'Alessandro"/><br /><sub><b>Adrian D'Alessandro</b></sub></a><br /><a href="https://github.com/ImperialCollegeLondon/pycsvy/issues?q=author%3AAdrianDAlessandro" title="Bug reports">🐛</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=AdrianDAlessandro" title="Code">💻</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=AdrianDAlessandro" title="Documentation">📖</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/research-software-engineering"><img src="https://avatars.githubusercontent.com/u/6853046?v=4?s=100" width="100px;" alt="James Paul Turner"/><br /><sub><b>James Paul Turner</b></sub></a><br /><a href="#infra-jamesturner246" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=jamesturner246" title="Code">💻</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://dc2917.github.io"><img src="https://avatars.githubusercontent.com/u/45606273?v=4?s=100" width="100px;" alt="Dan Cummins"/><br /><sub><b>Dan Cummins</b></sub></a><br /><a href="#infra-dc2917" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dc2917" title="Code">💻</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://github.com/mikeheyns"><img src="https://avatars.githubusercontent.com/u/123199379?v=4?s=100" width="100px;" alt="mikeheyns"/><br /><sub><b>mikeheyns</b></sub></a><br /><a href="#infra-mikeheyns" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td>
-    </tr>
-  </tbody>
-</table>
-
-<!-- markdownlint-restore -->
-<!-- prettier-ignore-end -->
-
+<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
+<!-- prettier-ignore-start -->
+<!-- markdownlint-disable -->
+<table>
+  <tbody>
+    <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/research-software-engineering/"><img src="https://avatars.githubusercontent.com/u/6095790?v=4?s=100" width="100px;" alt="Diego Alonso Álvarez"/><br /><sub><b>Diego Alonso Álvarez</b></sub></a><br /><a href="#infra-dalonsoa" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="#ideas-dalonsoa" title="Ideas, Planning, & Feedback">🤔</a> <a href="#maintenance-dalonsoa" title="Maintenance">🚧</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dalonsoa" title="Tests">⚠️</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/issues?q=author%3Adalonsoa" title="Bug reports">🐛</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dalonsoa" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/research-software-engineering"><img src="https://avatars.githubusercontent.com/u/23149834?v=4?s=100" width="100px;" alt="Alex Dewar"/><br /><sub><b>Alex Dewar</b></sub></a><br /><a href="#ideas-alexdewar" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=alexdewar" title="Tests">⚠️</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=alexdewar" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/AdrianDAlessandro"><img src="https://avatars.githubusercontent.com/u/40875798?v=4?s=100" width="100px;" alt="Adrian D'Alessandro"/><br /><sub><b>Adrian D'Alessandro</b></sub></a><br /><a href="https://github.com/ImperialCollegeLondon/pycsvy/issues?q=author%3AAdrianDAlessandro" title="Bug reports">🐛</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=AdrianDAlessandro" title="Code">💻</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=AdrianDAlessandro" title="Documentation">📖</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/research-software-engineering"><img src="https://avatars.githubusercontent.com/u/6853046?v=4?s=100" width="100px;" alt="James Paul Turner"/><br /><sub><b>James Paul Turner</b></sub></a><br /><a href="#infra-jamesturner246" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=jamesturner246" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://dc2917.github.io"><img src="https://avatars.githubusercontent.com/u/45606273?v=4?s=100" width="100px;" alt="Dan Cummins"/><br /><sub><b>Dan Cummins</b></sub></a><br /><a href="#infra-dc2917" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dc2917" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/mikeheyns"><img src="https://avatars.githubusercontent.com/u/123199379?v=4?s=100" width="100px;" alt="mikeheyns"/><br /><sub><b>mikeheyns</b></sub></a><br /><a href="#infra-mikeheyns" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td>
+    </tr>
+  </tbody>
+</table>
+
+<!-- markdownlint-restore -->
+<!-- prettier-ignore-end -->
+
 <!-- ALL-CONTRIBUTORS-LIST:END -->
 
 This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
diff --git a/csvy/readers.py b/csvy/readers.py
index ae34cbb..d1cf267 100644
--- a/csvy/readers.py
+++ b/csvy/readers.py
@@ -4,7 +4,7 @@ from __future__ import annotations
 
 import logging
 from pathlib import Path
-from typing import Any
+from typing import Any, Literal
 
 import yaml
 
@@ -61,13 +61,14 @@ def get_comment(line: str, marker: str = "---") -> str:
 
 
 def read_header(
-    filename: Path | str, marker: str = "---", **kwargs: Any
+    filename: Path | str, marker: str = "---", encoding: str = "utf-8", **kwargs: Any
 ) -> tuple[dict[str, Any], int, str]:
     """Read the yaml-formatted header from a file.
 
     Args:
         filename: Name of the file to read the header from.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         **kwargs: Arguments to pass to 'yaml.safe_load'.
 
     Returns:
@@ -78,7 +79,7 @@ def read_header(
     markers = 0
     nlines = 0
     comment = ""
-    with Path(filename).open("r") as f:
+    with Path(filename).open("r", encoding=encoding) as f:
         for line in f:
             if nlines == 0:
                 comment = get_comment(line, marker=marker)
@@ -96,24 +97,26 @@ def read_header(
 
 
 def read_metadata(
-    filename: Path | str, marker: str = "---", **kwargs: Any
+    filename: Path | str, marker: str = "---", encoding: str = "utf-8", **kwargs: Any
 ) -> dict[str, Any]:
     """Read the yaml-formatted metadata from a file.
 
     Args:
         filename: Name of the file to read the header from.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         **kwargs: Arguments to pass to 'yaml.safe_load'.
 
     Returns:
         The metadata stored in the header.
     """
-    return read_header(filename, marker, **kwargs)[0]
+    return read_header(filename, marker, encoding, **kwargs)[0]
 
 
 def read_to_array(
     filename: Path | str,
     marker: str = "---",
+    encoding: str = "utf-8",
     csv_options: dict[str, Any] | None = None,
     yaml_options: dict[str, Any] | None = None,
 ) -> tuple[NDArray, dict[str, Any]]:
@@ -122,6 +125,7 @@ def read_to_array(
     Args:
         filename:  Name of the file to read.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         csv_options: Options to pass to np.loadtxt.
         yaml_options: Options to pass to yaml.safe_load.
 
@@ -138,17 +142,20 @@ def read_to_array(
     import numpy as np
 
     yaml_options = yaml_options if yaml_options is not None else {}
-    header, nlines, comment = read_header(filename, marker=marker, **yaml_options)
+    header, nlines, comment = read_header(
+        filename, marker=marker, encoding=encoding, **yaml_options
+    )
 
     options = csv_options.copy() if csv_options is not None else {}
     options["skiprows"] = nlines + options.get("skiprows", 0)
     options["comments"] = comment[0] if len(comment) >= 1 else "#"
-    return np.loadtxt(filename, **options), header
+    return np.loadtxt(filename, encoding=encoding, **options), header
 
 
 def read_to_dataframe(
     filename: Path | str,
     marker: str = "---",
+    encoding: str = "utf-8",
     csv_options: dict[str, Any] | None = None,
     yaml_options: dict[str, Any] | None = None,
 ) -> tuple[DataFrame, dict[str, Any]]:
@@ -160,6 +167,7 @@ def read_to_dataframe(
     Args:
         filename:  Name of the file to read.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         csv_options: Options to pass to pd.read_csv.
         yaml_options: Options to pass to yaml.safe_load.
 
@@ -176,17 +184,20 @@ def read_to_dataframe(
     import pandas as pd
 
     yaml_options = yaml_options if yaml_options is not None else {}
-    header, nlines, comment = read_header(filename, marker=marker, **yaml_options)
+    header, nlines, comment = read_header(
+        filename, marker=marker, encoding=encoding, **yaml_options
+    )
 
     options = csv_options.copy() if csv_options is not None else {}
     options["skiprows"] = nlines
     options["comment"] = comment[0] if len(comment) >= 1 else None
-    return pd.read_csv(filename, **options), header
+    return pd.read_csv(filename, encoding=encoding, **options), header
 
 
 def read_to_polars(
     filename: Path | str,
     marker: str = "---",
+    encoding: Literal["utf8", "utf8-lossy"] = "utf8",
     csv_options: dict[str, Any] | None = None,
     yaml_options: dict[str, Any] | None = None,
     eager: bool = False,
@@ -203,16 +214,21 @@ def read_to_polars(
     Args:
         filename:  Name of the file to read.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         csv_options: Options to pass to pl.scan_csv.
         yaml_options: Options to pass to yaml.safe_load.
         eager: Whether to load the data into memory.
 
     Raises:
         ModuleNotFoundError: If polars is not found.
+        ValueError: If an invalid character encoding is specified.
 
     Returns:
         Tuple containing: The polars LazyFrame and the header as a dictionary.
     """
+    if encoding not in ("utf8", "utf8-lossy"):
+        raise ValueError("Encoding must be either 'utf8' or 'utf8-lossy'")
+
     if LazyFrame is None:
         raise ModuleNotFoundError(
             "Module polars is not present. Install it to read data into DataFrame."
@@ -220,13 +236,15 @@ def read_to_polars(
     import polars as pl
 
     yaml_options = yaml_options if yaml_options is not None else {}
-    header, nlines, comment = read_header(filename, marker=marker, **yaml_options)
+    header, nlines, comment = read_header(
+        filename, marker=marker, encoding="utf-8", **yaml_options
+    )
 
     options = csv_options.copy() if csv_options is not None else {}
     options["skip_rows"] = nlines
     options["comment_prefix"] = comment[0] if len(comment) >= 1 else None
 
-    lf = pl.scan_csv(filename, **options)
+    lf = pl.scan_csv(filename, encoding=encoding, **options)
     if eager:
         return lf.collect(), header
     return lf, header
@@ -235,6 +253,7 @@ def read_to_polars(
 def read_to_list(
     filename: Path | str,
     marker: str = "---",
+    encoding: str = "utf-8",
     csv_options: dict[str, Any] | None = None,
     yaml_options: dict[str, Any] | None = None,
 ) -> tuple[list[list], dict[str, Any]]:
@@ -243,6 +262,7 @@ def read_to_list(
     Args:
         filename: Name of the file to read.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         csv_options: Options to pass to csv.reader.
         yaml_options: Options to pass to yaml.safe_load.
 
@@ -255,12 +275,14 @@ def read_to_list(
     import csv
 
     yaml_options = yaml_options if yaml_options is not None else {}
-    header, nlines, _ = read_header(filename, marker=marker, **yaml_options)
+    header, nlines, _ = read_header(
+        filename, marker=marker, encoding=encoding, **yaml_options
+    )
 
     options = csv_options.copy() if csv_options is not None else {}
 
     data = []
-    with open(filename, newline="") as csvfile:
+    with open(filename, encoding=encoding, newline="") as csvfile:
         csvreader = csv.reader(csvfile, **options)
 
         for _ in range(nlines):
diff --git a/csvy/writers.py b/csvy/writers.py
index a5a3d17..c311f7f 100644
--- a/csvy/writers.py
+++ b/csvy/writers.py
@@ -35,6 +35,7 @@ def write(
     data: Any,
     header: dict[str, Any],
     comment: str = "",
+    encoding: str = "utf-8",
     csv_options: dict[str, Any] | None = None,
     yaml_options: dict[str, Any] | None = None,
 ) -> None:
@@ -46,6 +47,7 @@ def write(
         data: The data to add to the file.
         header: Dictionary with the header information to save.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         csv_options: Arguments to pass to the CSV writer, being this `savetxt`, panda's
             `to_csv` or something else. Mind that any argument related to the character
             to indicate a comment or header line will be ignored.
@@ -55,8 +57,8 @@ def write(
     csv_options = csv_options if csv_options is not None else {}
     yaml_options = yaml_options if yaml_options is not None else {}
 
-    write_header(filename, header, comment, **yaml_options)
-    write_data(filename, data, comment, **csv_options)
+    write_header(filename, header, comment, encoding, **yaml_options)
+    write_data(filename, data, comment, encoding, **csv_options)
 
 
 class Writer:
@@ -70,6 +72,7 @@ class Writer:
         filename: Path | str,
         header: dict[str, Any],
         comment: str = "",
+        encoding: str = "utf-8",
         csv_options: dict[str, Any] | None = None,
         yaml_options: dict[str, Any] | None = None,
         line_buffering: bool = False,
@@ -80,6 +83,7 @@ class Writer:
             filename: Path to file. If it exists it will be overwritten.
             header: Dictionary with the header information to save.
             comment: String to use to mark the header lines as comments.
+            encoding: The character encoding to use when writing to file.
             csv_options: Arguments to pass to csv.writer()
             yaml_options: Arguments to pass to the 'yaml.safe_dump' function to control
                 writing the header.
@@ -94,8 +98,10 @@ class Writer:
         buffering = 1 if line_buffering else -1
 
         # Newline must be "" as per csv.writer's documentation
-        self._file = Path(filename).open("w", newline="", buffering=buffering)
-        write_header(self._file, header, comment, **yaml_options)
+        self._file = Path(filename).open(
+            "w", encoding=encoding, newline="", buffering=buffering
+        )
+        write_header(self._file, header, comment, encoding, **yaml_options)
 
         self._writer = csv.writer(self._file, **csv_options)
 
@@ -124,6 +130,7 @@ def write_header(
     file: Path | str | TextIOBase,
     header: dict[str, Any],
     comment: str = "",
+    encoding: str = "utf-8",
     **kwargs: Any,
 ) -> None:
     """Writes the header dictionary into the file with lines starting with comment.
@@ -132,12 +139,13 @@ def write_header(
         file: File handle or path to file. Will be overwritten if it exists.
         header: Dictionary with the header information to save.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to pass to 'yaml.safe_dump'. If "sort_keys" is not one of
             arguments, it will be set to sort_keys=False.
     """
     header_ = validate_write(header)
     if not isinstance(file, TextIOBase):
-        with Path(file).open("w") as f:
+        with Path(file).open("w", encoding=encoding) as f:
             write_header(f, header_, comment, **kwargs)
             return
 
@@ -152,7 +160,11 @@ def write_header(
 
 
 def write_data(
-    filename: Path | str, data: Any, comment: str = "", **kwargs: Any
+    filename: Path | str,
+    data: Any,
+    comment: str = "",
+    encoding: str = "utf-8",
+    **kwargs: Any,
 ) -> None:
     """Writes the tabular data to the chosen file, adding it after the header.
 
@@ -164,18 +176,23 @@ def write_data(
             package. If it is a numpy array, the `savetxt` will be used, while if it is
             a pandas Dataframe, the `to_csv` method will be used.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to be passed to the underlaying saving method.
     """
     for fun in KNOWN_WRITERS:
         if fun(filename, data, comment, **kwargs):
             return
 
-    write_csv(filename, data, comment, **kwargs)
+    write_csv(filename, data, comment, encoding, **kwargs)
 
 
 @register_writer
 def write_numpy(
-    filename: Path | str, data: Any, comment: str = "", **kwargs: Any
+    filename: Path | str,
+    data: Any,
+    comment: str = "",
+    encoding: str = "utf-8",
+    **kwargs: Any,
 ) -> bool:
     """Writes the numpy array to the chosen file, adding it after the header.
 
@@ -185,6 +202,7 @@ def write_numpy(
         data: The data. If it is a numpy array, it will be saved, otherwise nothing is
             done.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to be passed to the underlaying saving method.
 
     Return:
@@ -195,7 +213,7 @@ def write_numpy(
 
         kwargs["comments"] = comment
         if isinstance(data, np.ndarray):
-            with open(filename, "a") as f:
+            with open(filename, "a", encoding=encoding) as f:
                 np.savetxt(f, data, **kwargs)
 
             return True
@@ -208,7 +226,11 @@ def write_numpy(
 
 @register_writer
 def write_pandas(
-    filename: Path | str, data: Any, comment: str = "", **kwargs: Any
+    filename: Path | str,
+    data: Any,
+    comment: str = "",
+    encoding: str = "utf-8",
+    **kwargs: Any,
 ) -> bool:
     """Writes the pandas dataframe to the chosen file, adding it after the header.
 
@@ -218,6 +240,7 @@ def write_pandas(
         data: The data. If it is a pandas dataframe, it will be saved, otherwise nothing
             is done.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to be passed to the underlaying saving method.
 
     Returns:
@@ -227,7 +250,7 @@ def write_pandas(
         import pandas as pd
 
         if isinstance(data, pd.DataFrame):
-            with open(filename, "a", newline="") as f:
+            with open(filename, "a", encoding=encoding, newline="") as f:
                 data.to_csv(f, **kwargs)
 
             return True
@@ -240,7 +263,11 @@ def write_pandas(
 
 @register_writer
 def write_polars(
-    filename: Path | str, data: Any, comment: str = "", **kwargs: Any
+    filename: Path | str,
+    data: Any,
+    comment: str = "",
+    encoding: str = "utf-8",
+    **kwargs: Any,
 ) -> bool:
     """Writes the polars dataframe to the chosen file, adding it after the header.
 
@@ -250,6 +277,7 @@ def write_polars(
         data: The data. If it is a polars DataFrame or LazyFrame, it will be saved,
             otherwise nothing is done.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to be passed to the underlaying saving method.
 
     Returns:
@@ -263,7 +291,7 @@ def write_polars(
             # collect the data into a DataFrame first
             data = data.collect()
         if isinstance(data, pl.DataFrame):
-            with open(filename, "a", newline="") as f:
+            with open(filename, "a", encoding=encoding, newline="") as f:
                 data.write_csv(f, **kwargs)
 
             return True
@@ -275,7 +303,11 @@ def write_polars(
 
 
 def write_csv(
-    filename: Path | str, data: Any, comment: str = "", **kwargs: Any
+    filename: Path | str,
+    data: Any,
+    comment: str = "",
+    encoding: str = "utf-8",
+    **kwargs: Any,
 ) -> bool:
     """Writes the tabular to the chosen file, adding it after the header.
 
@@ -285,12 +317,13 @@ def write_csv(
         data: The data. Can have anything that counts as a sequence. Each component of
             the sequence will be saved in a different row.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to be passed to the underlaying saving method.
 
     Returns:
         True if the writer worked, False otherwise.
     """
-    with open(filename, "a", newline="") as f:
+    with open(filename, "a", encoding=encoding, newline="") as f:
         writer = csv.writer(f, **kwargs)
         for row in data:
             writer.writerow(row)


Test patch:
diff --git a/tests/test_read.py b/tests/test_read.py
index f0768bb..c251442 100644
--- a/tests/test_read.py
+++ b/tests/test_read.py
@@ -45,10 +45,14 @@ def test_read_metadata(read_header_mock, data_path):
 
     filename = "test.csv"
     marker = "!!!"
+    encoding = "hieroglyphics"
     kwargs = {"key": "value"}
-    assert read_metadata(filename=filename, marker=marker, **kwargs) == "a"
+    assert (
+        read_metadata(filename=filename, marker=marker, encoding=encoding, **kwargs)
+        == "a"
+    )
 
-    read_header_mock.assert_called_once_with(filename, marker, **kwargs)
+    read_header_mock.assert_called_once_with(filename, marker, encoding, **kwargs)
 
 
 def test_read_to_array(array_data_path):
@@ -114,6 +118,9 @@ def test_read_to_polars(data_path):
     with pytest.raises(ModuleNotFoundError):
         read_to_polars(data_path)
 
+    with pytest.raises(ValueError):
+        read_to_polars(data_path, encoding="utf-9")  # type: ignore [arg-type]
+
 
 def test_read_to_list(array_data_path):
     """Test the read_to_list function."""
diff --git a/tests/test_write.py b/tests/test_write.py
index ecb1c28..34a843d 100644
--- a/tests/test_write.py
+++ b/tests/test_write.py
@@ -145,12 +145,13 @@ def test_writer(mock_write_header, mock_csv_writer, csv_options, yaml_options, t
     filename = tmpdir / "some_file.csv"
     header = {"name": "HAL"}
     comment = "# "
+    encoding = "utf-8"
 
-    writer = Writer(filename, header, comment, csv_options, yaml_options)
+    writer = Writer(filename, header, comment, encoding, csv_options, yaml_options)
     csv_options = csv_options or {}
     yaml_options = yaml_options or {}
     mock_write_header.assert_called_once_with(
-        writer._file, header, comment, **yaml_options
+        writer._file, header, comment, encoding, **yaml_options
     )
 
     mock_csv_writer.assert_called_once_with(writer._file, **csv_options)
@@ -221,6 +222,7 @@ def test_write(mock_write_data, mock_write_header):
     data = [[1, 2], [3, 4]]
     header = {"name": "HAL"}
     comment = "# "
+    encoding = "encoding"
     csv_options = {"delimiter": ","}
     yaml_options = {"sort_keys": False}
 
@@ -229,12 +231,17 @@ def test_write(mock_write_data, mock_write_header):
         data,
         header,
         comment,
+        encoding,
         csv_options=csv_options,
         yaml_options=yaml_options,
     )
 
-    mock_write_header.assert_called_once_with(filename, header, comment, **yaml_options)
-    mock_write_data.assert_called_once_with(filename, data, comment, **csv_options)
+    mock_write_header.assert_called_once_with(
+        filename, header, comment, encoding, **yaml_options
+    )
+    mock_write_data.assert_called_once_with(
+        filename, data, comment, encoding, **csv_options
+    )
 
 
 @patch("csvy.writers.write_csv")
@@ -245,11 +252,12 @@ def test_write_data(mock_write_csv):
     filename = "here.csv"
     data = [[1, 2], [3, 4]]
     comment = "# "
+    encoding = "encoding"
     csv_options = {"delimiter": ","}
 
     KNOWN_WRITERS.clear()
     KNOWN_WRITERS.append(MagicMock(return_value=True))
-    write_data(filename, data, comment, **csv_options)
+    write_data(filename, data, comment, encoding, **csv_options)
     KNOWN_WRITERS[0].assert_called_once_with(  # type: ignore [attr-defined]
         filename, data, comment, **csv_options
     )
@@ -257,8 +265,10 @@ def test_write_data(mock_write_csv):
 
     KNOWN_WRITERS.clear()
     KNOWN_WRITERS.append(MagicMock(return_value=False))
-    write_data(filename, data, comment, **csv_options)
+    write_data(filename, data, comment, encoding, **csv_options)
     KNOWN_WRITERS[0].assert_called_once_with(  # type: ignore [attr-defined]
         filename, data, comment, **csv_options
     )
-    mock_write_csv.assert_called_once_with(filename, data, comment, **csv_options)
+    mock_write_csv.assert_called_once_with(
+        filename, data, comment, encoding, **csv_options
+    )