Repository: mesonbuild/meson

Issue:
Stripping non-ASCII characters
**Describe the bug**
We have some Fortran files that contain non-ASCII characters in comments. They compile fine but when used with meson, I get errors in `depscan.py`:
```
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 3329: invalid continuation byte
```

Instead of writing a custom script that strips out these non-ASCII characters, I wonder if there's a way to do it from within meson. Alternatively, is it possible to just bypass this check or make it skip comments?



**To Reproduce**
I'm working off of this pyOptSparse build at mdolab/pyoptsparse#300. The offending Fortran source file is unfortunately not publicly available, but any Fortran file with a non-ASCII character in a comment line should do.

**Expected behavior**
First, I think better debugging info would be helpful. It's not clear from the message which file was the issue, and I had to check with `chardetect` to figure it out.

I think the best thing would be for meson to allow these characters to exist, since the compiler has no problems with it. If that's not possible, then I was wondering if meson provides any utilities in stripping characters from files, as that seems to be fairly useful.

**system parameters**
* Is this a [cross build](https://mesonbuild.com/Cross-compilation.html) or just a plain native build (for the same computer)? Native build
* what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.) Manjaro 21.2.6
* what Python version are you using? Python 3.9
* what `meson --version` 0.63.0
* what `ninja --version` if it's a Ninja build  `1.10.2.git.kitware.jobserver-1`



Discussion:
Is this truly about non-ASCII? The error talks about utf-8.

In depscan.py, we open the fortran files and parse them to try looking for module dependencies. Meson doesn't care what their contents are and isn't specifically looking for or avoiding comments, but it does use Python's file reader with the utf-8 codec in order to open those files.

This may not work if the file is not compatible with the utf-8 codec, but I'm not sure what the solution should be! considering that the error comes from opening the file for reading, not from parsing any specific lines.

Can you verify whether the file is unicode or some other non-ASCII charset?
The offending characters are Å and Ö which are definitely in UTF8. However they might not be encoded correctly in the file for whatever reason, because sometimes they appear as � and chardetect also reports the file as `ISO-8859-1 with confidence 0.73`.

I don't know much about unicode encodings, so if you have any suggestions I'm happy to dig deeper.
If chardetect says it's probably latin1, then it's probably latin1. I don't know what editor you use, but it probably has a mode to select the file encoding, so you could try manually changing the encoding, making sure the offending characters do not currently display as �, and re-saving the file.
From what I can tell all the "active" characters we care about in Fortran are ASCII. Thus we could just always open them in the "ignore encoding errors" mode without breaking anything.
That would be cool if possible, because while Obviously Everyone Should Use UTF-8™, if we can guarantee robustness of representation then being forgiving of errors would be nice.

But I have no idea how the depscan code works, and particularly, do not know fortran :D otoh you're the author of that file so hopefully you do know.

So there's my +1.
Of course, I converted those characters from latin1 to UTF8 and now everything works fine. Thanks for the help!

It would be great if meson is able to ignore those characters since this shouldn't affect anything, but I'm also happy to close this issue if you guys don't plan to work on it.

Resolving patch:
diff --git a/mesonbuild/scripts/depscan.py b/mesonbuild/scripts/depscan.py
--- a/mesonbuild/scripts/depscan.py
+++ b/mesonbuild/scripts/depscan.py
@@ -62,7 +62,7 @@ def scan_file(self, fname: str) -> None:
     def scan_fortran_file(self, fname: str) -> None:
         fpath = pathlib.Path(fname)
         modules_in_this_file = set()
-        for line in fpath.read_text(encoding='utf-8').split('\n'):
+        for line in fpath.read_text(encoding='utf-8', errors='ignore').split('\n'):
             import_match = FORTRAN_USE_RE.match(line)
             export_match = FORTRAN_MODULE_RE.match(line)
             submodule_export_match = FORTRAN_SUBMOD_RE.match(line)
@@ -112,7 +112,7 @@ def scan_fortran_file(self, fname: str) -> None:
 
     def scan_cpp_file(self, fname: str) -> None:
         fpath = pathlib.Path(fname)
-        for line in fpath.read_text(encoding='utf-8').split('\n'):
+        for line in fpath.read_text(encoding='utf-8', errors='ignore').split('\n'):
             import_match = CPP_IMPORT_RE.match(line)
             export_match = CPP_EXPORT_RE.match(line)
             if import_match:


Test patch: