Repository: ImperialCollegeLondon/pycsvy (language: python)

Issue:
`pycsvy` won't work with Unicode data on Windows
Currently, `pycsvy`'s various reader functions don't explicitly specify a desired text-encoding to the underlying calls to `open()`, which means that the platform's default will be used.

On Linux and macOS, the default encoding is UTF-8 but for unfortunate legacy reasons the default encoding on Windows is `cp1252`. Even worse, this encoding doesn't support Unicode, so if you try writing data containing Unicode to file, you'll get an error :disappointed:. This actually crops up more often than you'd think, in data sets containing names with accents, for example (looking at you @dalonsoa...) and I've been bitten by it a few times recently.

It looks like UTF-8 will be made the default, but not until Python 3.15: https://peps.python.org/pep-0686/. In the meantime I think it probably makes sense to just make UTF-8 for `pycsvy`, regardless of the platform.

Another consideration is that users can specify the encoding in the YAML header. We have a couple of options here:

1. If the user specifies an encoding other than UTF-8, we raise an error/warning and use UTF-8 anyway
2. We use the user's specified encoding to parse the file (not sure whether we should also read the header with this encoding -- which will require reading it twice -- or just the CSV portion of the file)


Discussion / hints:
I can take a look at this. 

I'm not sure if it's a great idea to just enforce utf-8? Perhaps we can pass that as a default (for all OS), and then if a custom encoding is set in the yaml header, we attempt to use that.
Go ahead with it. To be honest, I am not really sure of the implications.

I agree it will be useful to allow the user to specify the encoding. If we do so, it should be a top level parameter to the read/write functions, not included within the header, because if we try to read the header and the file has the wrong encoding, we won't be able to read it! So we need to know it upfront.

Fix patch:
diff --git a/README.md b/README.md
index 18efd21..7005830 100644
--- a/README.md
+++ b/README.md
@@ -8,8 +8,8 @@
 [![codecov](https://codecov.io/gh/ImperialCollegeLondon/pycsvy/branch/develop/graph/badge.svg?token=N03KYNUD18)](https://codecov.io/gh/ImperialCollegeLondon/pycsvy)
 [![Codacy Badge](https://app.codacy.com/project/badge/Grade/8d1b791b315f4814a128d94483499561)](https://app.codacy.com/gh/ImperialCollegeLondon/pycsvy/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=ImperialCollegeLondon/pycsvy&amp;utm_campaign=Badge_Grade)
 [![pre-commit.ci status](https://results.pre-commit.ci/badge/github/ImperialCollegeLondon/pycsvy/develop.svg)](https://results.pre-commit.ci/latest/github/ImperialCollegeLondon/pycsvy/develop)
-<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
-[![All Contributors](https://img.shields.io/badge/all_contributors-6-orange.svg?style=flat-square)](#contributors-)
+<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
+[![All Contributors](https://img.shields.io/badge/all_contributors-6-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
 
 CSV is a popular format for storing tabular data used in many disciplines. Metadata
@@ -111,31 +111,37 @@ YAML-formatted header, respectively.
 You can also instruct a writer to use line buffering, instead of the usual chunk buffering.
 
 Finally, you can control the character(s) used to indicate comments by setting the
-`comment` keyword when writing a file. By default, there is no character (""). During reading, the comment character is found atomatically.
+`comment` keyword when writing a file. By default, there is no character ("").
+During reading, the comment character is found automatically.
+
+Note that, by default, these reader functions will assume UTF-8 encoding. You can choose a
+different character encoding by setting the `encoding` keyword argument to any of these
+reader or writer functions. For example, on Windows, Windows-1252 encoding is often used,
+which can be specified via `encoding='cp1252'`.
 
 ## Contributors ✨
 
 Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
 
-<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
-<!-- prettier-ignore-start -->
-<!-- markdownlint-disable -->
-<table>
-  <tbody>
-    <tr>
-      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/research-software-engineering/"><img src="https://avatars.githubusercontent.com/u/6095790?v=4?s=100" width="100px;" alt="Diego Alonso Álvarez"/><br /><sub><b>Diego Alonso Álvarez</b></sub></a><br /><a href="#infra-dalonsoa" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="#ideas-dalonsoa" title="Ideas, Planning, & Feedback">🤔</a> <a href="#maintenance-dalonsoa" title="Maintenance">🚧</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dalonsoa" title="Tests">⚠️</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/issues?q=author%3Adalonsoa" title="Bug reports">🐛</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dalonsoa" title="Code">💻</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/research-software-engineering"><img src="https://avatars.githubusercontent.com/u/23149834?v=4?s=100" width="100px;" alt="Alex Dewar"/><br /><sub><b>Alex Dewar</b></sub></a><br /><a href="#ideas-alexdewar" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=alexdewar" title="Tests">⚠️</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=alexdewar" title="Code">💻</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://github.com/AdrianDAlessandro"><img src="https://avatars.githubusercontent.com/u/40875798?v=4?s=100" width="100px;" alt="Adrian D'Alessandro"/><br /><sub><b>Adrian D'Alessandro</b></sub></a><br /><a href="https://github.com/ImperialCollegeLondon/pycsvy/issues?q=author%3AAdrianDAlessandro" title="Bug reports">🐛</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=AdrianDAlessandro" title="Code">💻</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=AdrianDAlessandro" title="Documentation">📖</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/research-software-engineering"><img src="https://avatars.githubusercontent.com/u/6853046?v=4?s=100" width="100px;" alt="James Paul Turner"/><br /><sub><b>James Paul Turner</b></sub></a><br /><a href="#infra-jamesturner246" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=jamesturner246" title="Code">💻</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://dc2917.github.io"><img src="https://avatars.githubusercontent.com/u/45606273?v=4?s=100" width="100px;" alt="Dan Cummins"/><br /><sub><b>Dan Cummins</b></sub></a><br /><a href="#infra-dc2917" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dc2917" title="Code">💻</a></td>
-      <td align="center" valign="top" width="14.28%"><a href="https://github.com/mikeheyns"><img src="https://avatars.githubusercontent.com/u/123199379?v=4?s=100" width="100px;" alt="mikeheyns"/><br /><sub><b>mikeheyns</b></sub></a><br /><a href="#infra-mikeheyns" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td>
-    </tr>
-  </tbody>
-</table>
-
-<!-- markdownlint-restore -->
-<!-- prettier-ignore-end -->
-
+<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
+<!-- prettier-ignore-start -->
+<!-- markdownlint-disable -->
+<table>
+  <tbody>
+    <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/admin-services/ict/self-service/research-support/rcs/service-offering/research-software-engineering/"><img src="https://avatars.githubusercontent.com/u/6095790?v=4?s=100" width="100px;" alt="Diego Alonso Álvarez"/><br /><sub><b>Diego Alonso Álvarez</b></sub></a><br /><a href="#infra-dalonsoa" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="#ideas-dalonsoa" title="Ideas, Planning, & Feedback">🤔</a> <a href="#maintenance-dalonsoa" title="Maintenance">🚧</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dalonsoa" title="Tests">⚠️</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/issues?q=author%3Adalonsoa" title="Bug reports">🐛</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dalonsoa" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/research-software-engineering"><img src="https://avatars.githubusercontent.com/u/23149834?v=4?s=100" width="100px;" alt="Alex Dewar"/><br /><sub><b>Alex Dewar</b></sub></a><br /><a href="#ideas-alexdewar" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=alexdewar" title="Tests">⚠️</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=alexdewar" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/AdrianDAlessandro"><img src="https://avatars.githubusercontent.com/u/40875798?v=4?s=100" width="100px;" alt="Adrian D'Alessandro"/><br /><sub><b>Adrian D'Alessandro</b></sub></a><br /><a href="https://github.com/ImperialCollegeLondon/pycsvy/issues?q=author%3AAdrianDAlessandro" title="Bug reports">🐛</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=AdrianDAlessandro" title="Code">💻</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=AdrianDAlessandro" title="Documentation">📖</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://www.imperial.ac.uk/research-software-engineering"><img src="https://avatars.githubusercontent.com/u/6853046?v=4?s=100" width="100px;" alt="James Paul Turner"/><br /><sub><b>James Paul Turner</b></sub></a><br /><a href="#infra-jamesturner246" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=jamesturner246" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://dc2917.github.io"><img src="https://avatars.githubusercontent.com/u/45606273?v=4?s=100" width="100px;" alt="Dan Cummins"/><br /><sub><b>Dan Cummins</b></sub></a><br /><a href="#infra-dc2917" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a> <a href="https://github.com/ImperialCollegeLondon/pycsvy/commits?author=dc2917" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/mikeheyns"><img src="https://avatars.githubusercontent.com/u/123199379?v=4?s=100" width="100px;" alt="mikeheyns"/><br /><sub><b>mikeheyns</b></sub></a><br /><a href="#infra-mikeheyns" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td>
+    </tr>
+  </tbody>
+</table>
+
+<!-- markdownlint-restore -->
+<!-- prettier-ignore-end -->
+
 <!-- ALL-CONTRIBUTORS-LIST:END -->
 
 This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
diff --git a/csvy/readers.py b/csvy/readers.py
index ae34cbb..d1cf267 100644
--- a/csvy/readers.py
+++ b/csvy/readers.py
@@ -4,7 +4,7 @@
 
 import logging
 from pathlib import Path
-from typing import Any
+from typing import Any, Literal
 
 import yaml
 
@@ -61,13 +61,14 @@ def get_comment(line: str, marker: str = "---") -> str:
 
 
 def read_header(
-    filename: Path | str, marker: str = "---", **kwargs: Any
+    filename: Path | str, marker: str = "---", encoding: str = "utf-8", **kwargs: Any
 ) -> tuple[dict[str, Any], int, str]:
     """Read the yaml-formatted header from a file.
 
     Args:
         filename: Name of the file to read the header from.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         **kwargs: Arguments to pass to 'yaml.safe_load'.
 
     Returns:
@@ -78,7 +79,7 @@ def read_header(
     markers = 0
     nlines = 0
     comment = ""
-    with Path(filename).open("r") as f:
+    with Path(filename).open("r", encoding=encoding) as f:
         for line in f:
             if nlines == 0:
                 comment = get_comment(line, marker=marker)
@@ -96,24 +97,26 @@ def read_header(
 
 
 def read_metadata(
-    filename: Path | str, marker: str = "---", **kwargs: Any
+    filename: Path | str, marker: str = "---", encoding: str = "utf-8", **kwargs: Any
 ) -> dict[str, Any]:
     """Read the yaml-formatted metadata from a file.
 
     Args:
         filename: Name of the file to read the header from.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         **kwargs: Arguments to pass to 'yaml.safe_load'.
 
     Returns:
         The metadata stored in the header.
     """
-    return read_header(filename, marker, **kwargs)[0]
+    return read_header(filename, marker, encoding, **kwargs)[0]
 
 
 def read_to_array(
     filename: Path | str,
     marker: str = "---",
+    encoding: str = "utf-8",
     csv_options: dict[str, Any] | None = None,
     yaml_options: dict[str, Any] | None = None,
 ) -> tuple[NDArray, dict[str, Any]]:
@@ -122,6 +125,7 @@ def read_to_array(
     Args:
         filename:  Name of the file to read.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         csv_options: Options to pass to np.loadtxt.
         yaml_options: Options to pass to yaml.safe_load.
 
@@ -138,17 +142,20 @@ def read_to_array(
     import numpy as np
 
     yaml_options = yaml_options if yaml_options is not None else {}
-    header, nlines, comment = read_header(filename, marker=marker, **yaml_options)
+    header, nlines, comment = read_header(
+        filename, marker=marker, encoding=encoding, **yaml_options
+    )
 
     options = csv_options.copy() if csv_options is not None else {}
     options["skiprows"] = nlines + options.get("skiprows", 0)
     options["comments"] = comment[0] if len(comment) >= 1 else "#"
-    return np.loadtxt(filename, **options), header
+    return np.loadtxt(filename, encoding=encoding, **options), header
 
 
 def read_to_dataframe(
     filename: Path | str,
     marker: str = "---",
+    encoding: str = "utf-8",
     csv_options: dict[str, Any] | None = None,
     yaml_options: dict[str, Any] | None = None,
 ) -> tuple[DataFrame, dict[str, Any]]:
@@ -160,6 +167,7 @@ def read_to_dataframe(
     Args:
         filename:  Name of the file to read.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         csv_options: Options to pass to pd.read_csv.
         yaml_options: Options to pass to yaml.safe_load.
 
@@ -176,17 +184,20 @@ def read_to_dataframe(
     import pandas as pd
 
     yaml_options = yaml_options if yaml_options is not None else {}
-    header, nlines, comment = read_header(filename, marker=marker, **yaml_options)
+    header, nlines, comment = read_header(
+        filename, marker=marker, encoding=encoding, **yaml_options
+    )
 
     options = csv_options.copy() if csv_options is not None else {}
     options["skiprows"] = nlines
     options["comment"] = comment[0] if len(comment) >= 1 else None
-    return pd.read_csv(filename, **options), header
+    return pd.read_csv(filename, encoding=encoding, **options), header
 
 
 def read_to_polars(
     filename: Path | str,
     marker: str = "---",
+    encoding: Literal["utf8", "utf8-lossy"] = "utf8",
     csv_options: dict[str, Any] | None = None,
     yaml_options: dict[str, Any] | None = None,
     eager: bool = False,
@@ -203,16 +214,21 @@ def read_to_polars(
     Args:
         filename:  Name of the file to read.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         csv_options: Options to pass to pl.scan_csv.
         yaml_options: Options to pass to yaml.safe_load.
         eager: Whether to load the data into memory.
 
     Raises:
         ModuleNotFoundError: If polars is not found.
+        ValueError: If an invalid character encoding is specified.
 
     Returns:
         Tuple containing: The polars LazyFrame and the header as a dictionary.
     """
+    if encoding not in ("utf8", "utf8-lossy"):
+        raise ValueError("Encoding must be either 'utf8' or 'utf8-lossy'")
+
     if LazyFrame is None:
         raise ModuleNotFoundError(
             "Module polars is not present. Install it to read data into DataFrame."
@@ -220,13 +236,15 @@ def read_to_polars(
     import polars as pl
 
     yaml_options = yaml_options if yaml_options is not None else {}
-    header, nlines, comment = read_header(filename, marker=marker, **yaml_options)
+    header, nlines, comment = read_header(
+        filename, marker=marker, encoding="utf-8", **yaml_options
+    )
 
     options = csv_options.copy() if csv_options is not None else {}
     options["skip_rows"] = nlines
     options["comment_prefix"] = comment[0] if len(comment) >= 1 else None
 
-    lf = pl.scan_csv(filename, **options)
+    lf = pl.scan_csv(filename, encoding=encoding, **options)
     if eager:
         return lf.collect(), header
     return lf, header
@@ -235,6 +253,7 @@ def read_to_polars(
 def read_to_list(
     filename: Path | str,
     marker: str = "---",
+    encoding: str = "utf-8",
     csv_options: dict[str, Any] | None = None,
     yaml_options: dict[str, Any] | None = None,
 ) -> tuple[list[list], dict[str, Any]]:
@@ -243,6 +262,7 @@ def read_to_list(
     Args:
         filename: Name of the file to read.
         marker: The marker characters that indicate the yaml header.
+        encoding: The character encoding in the file to read.
         csv_options: Options to pass to csv.reader.
         yaml_options: Options to pass to yaml.safe_load.
 
@@ -255,12 +275,14 @@ def read_to_list(
     import csv
 
     yaml_options = yaml_options if yaml_options is not None else {}
-    header, nlines, _ = read_header(filename, marker=marker, **yaml_options)
+    header, nlines, _ = read_header(
+        filename, marker=marker, encoding=encoding, **yaml_options
+    )
 
     options = csv_options.copy() if csv_options is not None else {}
 
     data = []
-    with open(filename, newline="") as csvfile:
+    with open(filename, encoding=encoding, newline="") as csvfile:
         csvreader = csv.reader(csvfile, **options)
 
         for _ in range(nlines):
diff --git a/csvy/writers.py b/csvy/writers.py
index a5a3d17..c311f7f 100644
--- a/csvy/writers.py
+++ b/csvy/writers.py
@@ -35,6 +35,7 @@ def write(
     data: Any,
     header: dict[str, Any],
     comment: str = "",
+    encoding: str = "utf-8",
     csv_options: dict[str, Any] | None = None,
     yaml_options: dict[str, Any] | None = None,
 ) -> None:
@@ -46,6 +47,7 @@ def write(
         data: The data to add to the file.
         header: Dictionary with the header information to save.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         csv_options: Arguments to pass to the CSV writer, being this `savetxt`, panda's
             `to_csv` or something else. Mind that any argument related to the character
             to indicate a comment or header line will be ignored.
@@ -55,8 +57,8 @@ def write(
     csv_options = csv_options if csv_options is not None else {}
     yaml_options = yaml_options if yaml_options is not None else {}
 
-    write_header(filename, header, comment, **yaml_options)
-    write_data(filename, data, comment, **csv_options)
+    write_header(filename, header, comment, encoding, **yaml_options)
+    write_data(filename, data, comment, encoding, **csv_options)
 
 
 class Writer:
@@ -70,6 +72,7 @@ def __init__(
         filename: Path | str,
         header: dict[str, Any],
         comment: str = "",
+        encoding: str = "utf-8",
         csv_options: dict[str, Any] | None = None,
         yaml_options: dict[str, Any] | None = None,
         line_buffering: bool = False,
@@ -80,6 +83,7 @@ def __init__(
             filename: Path to file. If it exists it will be overwritten.
             header: Dictionary with the header information to save.
             comment: String to use to mark the header lines as comments.
+            encoding: The character encoding to use when writing to file.
             csv_options: Arguments to pass to csv.writer()
             yaml_options: Arguments to pass to the 'yaml.safe_dump' function to control
                 writing the header.
@@ -94,8 +98,10 @@ def __init__(
         buffering = 1 if line_buffering else -1
 
         # Newline must be "" as per csv.writer's documentation
-        self._file = Path(filename).open("w", newline="", buffering=buffering)
-        write_header(self._file, header, comment, **yaml_options)
+        self._file = Path(filename).open(
+            "w", encoding=encoding, newline="", buffering=buffering
+        )
+        write_header(self._file, header, comment, encoding, **yaml_options)
 
         self._writer = csv.writer(self._file, **csv_options)
 
@@ -124,6 +130,7 @@ def write_header(
     file: Path | str | TextIOBase,
     header: dict[str, Any],
     comment: str = "",
+    encoding: str = "utf-8",
     **kwargs: Any,
 ) -> None:
     """Writes the header dictionary into the file with lines starting with comment.
@@ -132,12 +139,13 @@ def write_header(
         file: File handle or path to file. Will be overwritten if it exists.
         header: Dictionary with the header information to save.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to pass to 'yaml.safe_dump'. If "sort_keys" is not one of
             arguments, it will be set to sort_keys=False.
     """
     header_ = validate_write(header)
     if not isinstance(file, TextIOBase):
-        with Path(file).open("w") as f:
+        with Path(file).open("w", encoding=encoding) as f:
             write_header(f, header_, comment, **kwargs)
             return
 
@@ -152,7 +160,11 @@ def write_header(
 
 
 def write_data(
-    filename: Path | str, data: Any, comment: str = "", **kwargs: Any
+    filename: Path | str,
+    data: Any,
+    comment: str = "",
+    encoding: str = "utf-8",
+    **kwargs: Any,
 ) -> None:
     """Writes the tabular data to the chosen file, adding it after the header.
 
@@ -164,18 +176,23 @@ def write_data(
             package. If it is a numpy array, the `savetxt` will be used, while if it is
             a pandas Dataframe, the `to_csv` method will be used.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to be passed to the underlaying saving method.
     """
     for fun in KNOWN_WRITERS:
         if fun(filename, data, comment, **kwargs):
             return
 
-    write_csv(filename, data, comment, **kwargs)
+    write_csv(filename, data, comment, encoding, **kwargs)
 
 
 @register_writer
 def write_numpy(
-    filename: Path | str, data: Any, comment: str = "", **kwargs: Any
+    filename: Path | str,
+    data: Any,
+    comment: str = "",
+    encoding: str = "utf-8",
+    **kwargs: Any,
 ) -> bool:
     """Writes the numpy array to the chosen file, adding it after the header.
 
@@ -185,6 +202,7 @@ def write_numpy(
         data: The data. If it is a numpy array, it will be saved, otherwise nothing is
             done.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to be passed to the underlaying saving method.
 
     Return:
@@ -195,7 +213,7 @@ def write_numpy(
 
         kwargs["comments"] = comment
         if isinstance(data, np.ndarray):
-            with open(filename, "a") as f:
+            with open(filename, "a", encoding=encoding) as f:
                 np.savetxt(f, data, **kwargs)
 
             return True
@@ -208,7 +226,11 @@ def write_numpy(
 
 @register_writer
 def write_pandas(
-    filename: Path | str, data: Any, comment: str = "", **kwargs: Any
+    filename: Path | str,
+    data: Any,
+    comment: str = "",
+    encoding: str = "utf-8",
+    **kwargs: Any,
 ) -> bool:
     """Writes the pandas dataframe to the chosen file, adding it after the header.
 
@@ -218,6 +240,7 @@ def write_pandas(
         data: The data. If it is a pandas dataframe, it will be saved, otherwise nothing
             is done.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to be passed to the underlaying saving method.
 
     Returns:
@@ -227,7 +250,7 @@ def write_pandas(
         import pandas as pd
 
         if isinstance(data, pd.DataFrame):
-            with open(filename, "a", newline="") as f:
+            with open(filename, "a", encoding=encoding, newline="") as f:
                 data.to_csv(f, **kwargs)
 
             return True
@@ -240,7 +263,11 @@ def write_pandas(
 
 @register_writer
 def write_polars(
-    filename: Path | str, data: Any, comment: str = "", **kwargs: Any
+    filename: Path | str,
+    data: Any,
+    comment: str = "",
+    encoding: str = "utf-8",
+    **kwargs: Any,
 ) -> bool:
     """Writes the polars dataframe to the chosen file, adding it after the header.
 
@@ -250,6 +277,7 @@ def write_polars(
         data: The data. If it is a polars DataFrame or LazyFrame, it will be saved,
             otherwise nothing is done.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to be passed to the underlaying saving method.
 
     Returns:
@@ -263,7 +291,7 @@ def write_polars(
             # collect the data into a DataFrame first
             data = data.collect()
         if isinstance(data, pl.DataFrame):
-            with open(filename, "a", newline="") as f:
+            with open(filename, "a", encoding=encoding, newline="") as f:
                 data.write_csv(f, **kwargs)
 
             return True
@@ -275,7 +303,11 @@ def write_polars(
 
 
 def write_csv(
-    filename: Path | str, data: Any, comment: str = "", **kwargs: Any
+    filename: Path | str,
+    data: Any,
+    comment: str = "",
+    encoding: str = "utf-8",
+    **kwargs: Any,
 ) -> bool:
     """Writes the tabular to the chosen file, adding it after the header.
 
@@ -285,12 +317,13 @@ def write_csv(
         data: The data. Can have anything that counts as a sequence. Each component of
             the sequence will be saved in a different row.
         comment: String to use to mark the header lines as comments.
+        encoding: The character encoding to use in the file to write.
         **kwargs: Arguments to be passed to the underlaying saving method.
 
     Returns:
         True if the writer worked, False otherwise.
     """
-    with open(filename, "a", newline="") as f:
+    with open(filename, "a", encoding=encoding, newline="") as f:
         writer = csv.writer(f, **kwargs)
         for row in data:
             writer.writerow(row)


Test patch:
diff --git a/tests/test_read.py b/tests/test_read.py
index f0768bb..c251442 100644
--- a/tests/test_read.py
+++ b/tests/test_read.py
@@ -45,10 +45,14 @@ def test_read_metadata(read_header_mock, data_path):
 
     filename = "test.csv"
     marker = "!!!"
+    encoding = "hieroglyphics"
     kwargs = {"key": "value"}
-    assert read_metadata(filename=filename, marker=marker, **kwargs) == "a"
+    assert (
+        read_metadata(filename=filename, marker=marker, encoding=encoding, **kwargs)
+        == "a"
+    )
 
-    read_header_mock.assert_called_once_with(filename, marker, **kwargs)
+    read_header_mock.assert_called_once_with(filename, marker, encoding, **kwargs)
 
 
 def test_read_to_array(array_data_path):
@@ -114,6 +118,9 @@ def test_read_to_polars(data_path):
     with pytest.raises(ModuleNotFoundError):
         read_to_polars(data_path)
 
+    with pytest.raises(ValueError):
+        read_to_polars(data_path, encoding="utf-9")  # type: ignore [arg-type]
+
 
 def test_read_to_list(array_data_path):
     """Test the read_to_list function."""
diff --git a/tests/test_write.py b/tests/test_write.py
index ecb1c28..34a843d 100644
--- a/tests/test_write.py
+++ b/tests/test_write.py
@@ -145,12 +145,13 @@ def test_writer(mock_write_header, mock_csv_writer, csv_options, yaml_options, t
     filename = tmpdir / "some_file.csv"
     header = {"name": "HAL"}
     comment = "# "
+    encoding = "utf-8"
 
-    writer = Writer(filename, header, comment, csv_options, yaml_options)
+    writer = Writer(filename, header, comment, encoding, csv_options, yaml_options)
     csv_options = csv_options or {}
     yaml_options = yaml_options or {}
     mock_write_header.assert_called_once_with(
-        writer._file, header, comment, **yaml_options
+        writer._file, header, comment, encoding, **yaml_options
     )
 
     mock_csv_writer.assert_called_once_with(writer._file, **csv_options)
@@ -221,6 +222,7 @@ def test_write(mock_write_data, mock_write_header):
     data = [[1, 2], [3, 4]]
     header = {"name": "HAL"}
     comment = "# "
+    encoding = "encoding"
     csv_options = {"delimiter": ","}
     yaml_options = {"sort_keys": False}
 
@@ -229,12 +231,17 @@ def test_write(mock_write_data, mock_write_header):
         data,
         header,
         comment,
+        encoding,
         csv_options=csv_options,
         yaml_options=yaml_options,
     )
 
-    mock_write_header.assert_called_once_with(filename, header, comment, **yaml_options)
-    mock_write_data.assert_called_once_with(filename, data, comment, **csv_options)
+    mock_write_header.assert_called_once_with(
+        filename, header, comment, encoding, **yaml_options
+    )
+    mock_write_data.assert_called_once_with(
+        filename, data, comment, encoding, **csv_options
+    )
 
 
 @patch("csvy.writers.write_csv")
@@ -245,11 +252,12 @@ def test_write_data(mock_write_csv):
     filename = "here.csv"
     data = [[1, 2], [3, 4]]
     comment = "# "
+    encoding = "encoding"
     csv_options = {"delimiter": ","}
 
     KNOWN_WRITERS.clear()
     KNOWN_WRITERS.append(MagicMock(return_value=True))
-    write_data(filename, data, comment, **csv_options)
+    write_data(filename, data, comment, encoding, **csv_options)
     KNOWN_WRITERS[0].assert_called_once_with(  # type: ignore [attr-defined]
         filename, data, comment, **csv_options
     )
@@ -257,8 +265,10 @@ def test_write_data(mock_write_csv):
 
     KNOWN_WRITERS.clear()
     KNOWN_WRITERS.append(MagicMock(return_value=False))
-    write_data(filename, data, comment, **csv_options)
+    write_data(filename, data, comment, encoding, **csv_options)
     KNOWN_WRITERS[0].assert_called_once_with(  # type: ignore [attr-defined]
         filename, data, comment, **csv_options
     )
-    mock_write_csv.assert_called_once_with(filename, data, comment, **csv_options)
+    mock_write_csv.assert_called_once_with(
+        filename, data, comment, encoding, **csv_options
+    )