Repository: sharkdp/bat (language: rust)

Issue:
A space appears at the beginning of the file (Byte order mark)
**Describe the bug you encountered:**

If you use bat on C# source files (.cs, .xaml and others), a space appears in the first line. This is due to byte order mark (BOM)
Maybe reproduced for others files on Windows systems 
https://en.wikipedia.org/wiki/Byte_order_mark#Byte_order_marks_by_encoding
**Sample file with BOM:**
[Program.cs.txt](https://github.com/sharkdp/bat/files/7420061/Program.cs.txt)

![IWpnW3OyGgg](https://user-images.githubusercontent.com/76967160/138926587-467bfae1-7d97-4be1-a99c-77e9ea651848.jpg)

![Screenshot_57](https://user-images.githubusercontent.com/76967160/138926853-811f802d-5aaf-4f4c-be7f-c06a26127099.png)

**What did you expect to happen instead?**
If I delete these bytes: 
![Screenshot_58](https://user-images.githubusercontent.com/76967160/138927622-410d7842-0e34-41f3-ab1b-266b52ad1e29.png)

bat works correctly :

![Screenshot_59](https://user-images.githubusercontent.com/76967160/138927884-b111b2b4-d976-499e-8150-efddfd789a55.png)


**How did you install `bat`?**

GitHub release:
**bat-v0.18.3-x86_64-pc-windows-gnu.zip
bat-v0.18.3-x86_64-pc-windows-msvc.zip**

---

**bat version and environment**

#### Software version

bat 0.18.3 (b146958)

#### Operating system

Windows 6.2.9200

#### Command-line

```bash
C:\WINDOWS\system32\bat.exe --diagnostic
```

#### Environment variables

```bash
SHELL=<not set>
PAGER=<not set>
LESS=<not set>
BAT_PAGER=<not set>
BAT_CACHE_PATH=<not set>
BAT_CONFIG_PATH=<not set>
BAT_OPTS=<not set>
BAT_STYLE=<not set>
BAT_TABS=<not set>
BAT_THEME=<not set>
XDG_CONFIG_HOME=<not set>
XDG_CACHE_HOME=<not set>
COLORTERM=<not set>
NO_COLOR=<not set>
MANPAGER=<not set>
```

#### Config file

Could not read contents of 'C:\Users\timoxa\AppData\Roaming\bat\config': Системе не удается найти указанный путь. (os error 3).

#### Compile time information

- Profile: release
- Target triple: x86_64-pc-windows-gnu
- Family: windows
- OS: windows
- Architecture: x86_64
- Pointer width: 64
- Endian: little
- CPU features: fxsr,sse,sse2
- Host: x86_64-pc-windows-msvc


Discussion / hints:
Thank you very much for the detailed bug report.

There a few things to consider here.

- `bat` uses a library called [`content_inspector`](https://github.com/sharkdp/content_inspector), which I created some time ago to distinguish "text" files from "binary" files (see https://dev.to/sharkdp/what-is-a-binary-file-2cf5 if you want to know more). `content_inspector` can also properly detect Byte order marks and correctly classifies your file as "UTF-8-BOM", i.e. UTF-8 encoded text with a Byte order mark.
- `bat` uses this classification (UTF-8, UTF-8-BOM, UTF-16LE, UTF-16BE) to choose a proper decoder for the text file. This is why `bat` (in contrast to `cat`) can properly show the contents of a UTF-16-encoded file (which is often used on Windows). For UTF-8, we don't do anything, as we assume the terminal to be configured as UTF-8. UTF-32 (BE/LE) is currently not supported, but could easily be.
- What `bat` does not do is to strip the BOM from the output. This is (arguably? maybe?) the right thing to do when we are in "plain" mode. And definitely the right thing to do when we are in non-interactive/loopthrough mode, i.e. if we are piping the output to a file or to another program. However, when we are showing the contents of a file on an interactive terminal, we should consider stripping the BOM.
- Since the BOM is not stripped, it will be part of `bat`s output. What this means is that the terminal (which we assume to be on UTF-8 encoding) has to interpret and display the BOM. The people who designed Unicode are obviously quite smart because they mapped the byte sequence `EF BB BF` to the Unicode code point `U+FEFF`, which is a *"Zero Width No-Break Space"*. On my terminal emulator (terminator), this Unicode character does not show up:
  ![image](https://user-images.githubusercontent.com/4209276/138944447-1bce738a-f8d3-4a7b-bc29-3c7ea72fe3c6.png)
  Similarly, calling `bat` on your file looks good:
  ![image](https://user-images.githubusercontent.com/4209276/138944651-431c2243-736d-4717-afa2-d6d88d6fd8d5.png)
  This does not mean that the character is not there. If I select the first line with the mouse and copy the text, the clipboard does actually contain the `U+FEFF` character, which we can confirm with `bat`:
  ![image](https://user-images.githubusercontent.com/4209276/138945114-10a0fb77-8567-45ad-bc59-68fdc52ad682.png)
  By the way: do not confuse `U+FEFF` with the byte sequence `FE FF`, which is actually a UTF-16 (Big Endian) Byte order mark! This is not a coincidence. The Big Endian encoding of `U+FEFF` *is* `FE FF`.

So I guess for now this is a question of whether or not a particular terminal emulator prints the `U+FEFF` zero width character with an actual width of zero. We could probably still do better and simply strip BOMs from `bat`s interactive output.

To be honest, I have never seen a UTF-8 BOM "in the wild". At least on Linux, every program seems to use the BOM-less version when writing UTF-8. That doesn't make this bug less relevant though, because UTF-16 files should suffer from the same problem.

@Enselic: IMO, this is not a Windows-specific bug. Files with UTF-8 BOMs might appear on non-Windows systems as well.

Further reading: Unicode standard, https://www.unicode.org/versions/Unicode6.1.0/ch16.pdf page 562
Great analysis! I confess to not having done that deep of an analysis before putting the windows label on 😊 . Turns out it was overhasty, because there is a similar problem on **macOS 11.6** with **Terminal.app Version 2.11 (440)**:

<img width="347" alt="Screen Shot 2021-10-27 at 07 54 03" src="https://user-images.githubusercontent.com/115040/139007981-c4ea13e3-2049-48ff-bf84-aaf09e41bd9d.png">

The problem persist even in `--plain` mode when the BOM is in the first bytes of the output:

<img width="350" alt="Screen Shot 2021-10-27 at 07 55 43" src="https://user-images.githubusercontent.com/115040/139008002-7397ae77-e749-4d95-81ba-98ddceea29d1.png">

Interestingly, if we bypass the pager, the output is correct:

<img width="351" alt="Screen Shot 2021-10-27 at 07 56 29" src="https://user-images.githubusercontent.com/115040/139008157-6a8dd167-515e-4e50-9d8f-eb1928176678.png">

Turns out the output is correct even if the BOM is not first in the output, as long as the pager is bypassed:

<img width="355" alt="Screen Shot 2021-10-27 at 07 59 00" src="https://user-images.githubusercontent.com/115040/139008264-a78bd227-5550-4e3b-a344-c6bfcaef4bb4.png">

This is with the current vanilla `less` on macOS which is somewhat old:
```
% less --version
less 487 (POSIX regular expressions)
```

So what if we try a later version of less? That seems to solve the problem on macOS:
```
% less --version
less 563 (PCRE regular expressions)
```

<img width="353" alt="Screen Shot 2021-10-27 at 08 06 38" src="https://user-images.githubusercontent.com/115040/139009043-e75c1553-9368-4ecd-a098-aece2f338cdd.png">

@v-timofeev What pager and version are you using? 
@Enselic Sorry for not answering for a long time!

> What pager and version are you using?

**`PAGER=<not set>`**

I noticed that the behavior depends on the terminal emulator:

For PowerShell 7.1.5:
![Screenshot_4](https://user-images.githubusercontent.com/76967160/139296911-e446bfac-5803-49f5-838b-6826b0ab82f7.png)

For Cygwin64 Terminal:
![Screenshot_5](https://user-images.githubusercontent.com/76967160/139297360-68ba8c31-e84d-43ff-9804-dffaf0af6b98.png)

For VNC (Centos 8)
![Screenshot_6](https://user-images.githubusercontent.com/76967160/139298003-30a014e2-a764-4d31-969d-b2244d8fe176.png)
You can see, that first symbol (#) with broken color scheme:
![Screenshot_7](https://user-images.githubusercontent.com/76967160/139298417-cc7cbbe4-caf6-4bc2-a663-3af4345000a5.png)



If no pager is specified, `less` is used.

* What is your output for `less --version`?

* What is your output in PowerShell with `bat --pager=never Program.cs`?

Powershell on windows:
![Screenshot_8](https://user-images.githubusercontent.com/76967160/139302966-ba1dae75-463c-4936-8447-6ae45cd31453.png)

Powershell on Centos 8 (via ssh)
![Screenshot_9](https://user-images.githubusercontent.com/76967160/139303240-78d73ceb-0938-4cdf-a24c-a8854023c6fa.png)



Does it work if the BOM are the first bytes of the output? Try both `bat --plain Program.cs` and `bat --plain --pager=never Program.cs` just to be sure, even though a pager might not even be used in your case.
In PowerShell same behavior:
![image](https://user-images.githubusercontent.com/76967160/139705450-14d57b44-45f4-498c-842e-c5069a34a715.png)
![image](https://user-images.githubusercontent.com/76967160/139705511-cfe222d3-5279-4335-9e4c-f88475c116eb.png)

In cmder:
![image](https://user-images.githubusercontent.com/76967160/139705864-9658f027-9b8f-4045-8567-1dd9f90efb7a.png)

You can see, that BOM breaks the syntax highlighting on first line of file:
with BOM:
![image](https://user-images.githubusercontent.com/76967160/139706432-e9410345-a43c-4df5-a92c-8f532386747a.png)
without BOM:
![image](https://user-images.githubusercontent.com/76967160/139706636-d6bd76f5-cd60-4126-b50b-aab3abfa16bf.png)

I suspect the highlighting error is because most syntax regex patterns do not work with a BOM. So even if the terminal displays it properly (i.e. not at all), we still need to strip it whenever we want syntax highlighting to work. Even with `--plain`, as long as we highlight. Not on loop-through mode though; I agree.

A nice side effect of that is that it we also "fix" when the pager and/or terminal in question do not display the BOM properly.

I'm setting good-first-issue on this because it shouldn't be very hard to do.

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1b5c3c1d7c..2ae4744843 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,7 @@
 - Prevent fork nightmare with `PAGER=batcat`. See #2235 (@johnmatthiggins)
 - Make `--no-paging`/`-P` override `--paging=...` if passed as a later arg, see #2201 (@themkat)
 - `--map-syntax` and `--ignored-suffix` now works together, see #2093 (@czzrr)
+- Strips byte order mark from output when in non-loop-through mode. See #1922 (@dag-h)
 
 ## Other
 
diff --git a/src/printer.rs b/src/printer.rs
index 3f7f1e0920..4f962df9eb 100644
--- a/src/printer.rs
+++ b/src/printer.rs
@@ -419,7 +419,7 @@ impl<'a> Printer for InteractivePrinter<'a> {
         let line = if self.config.show_nonprintable {
             replace_nonprintable(line_buffer, self.config.tab_width)
         } else {
-            match self.content_type {
+            let line = match self.content_type {
                 Some(ContentType::BINARY) | None => {
                     return Ok(());
                 }
@@ -430,6 +430,15 @@ impl<'a> Printer for InteractivePrinter<'a> {
                     .decode(line_buffer, DecoderTrap::Replace)
                     .map_err(|_| "Invalid UTF-16BE")?,
                 _ => String::from_utf8_lossy(line_buffer).to_string(),
+            };
+            // Remove byte order mark from the first line if it exists
+            if line_number == 1 {
+                match line.strip_prefix('\u{feff}') {
+                    Some(stripped) => stripped.to_string(),
+                    None => line,
+                }
+            } else {
+                line
             }
         };
 


Test patch:
diff --git a/tests/examples/test_BOM.txt b/tests/examples/test_BOM.txt
new file mode 100644
index 0000000000..c638ce6117
--- /dev/null
+++ b/tests/examples/test_BOM.txt
@@ -0,0 +1,1 @@
+﻿hello world
diff --git a/tests/integration_tests.rs b/tests/integration_tests.rs
index 6a6263270d..02e60a7581 100644
--- a/tests/integration_tests.rs
+++ b/tests/integration_tests.rs
@@ -758,14 +758,66 @@ fn config_read_arguments_from_file() {
 
 #[test]
 fn utf16() {
-    // The output will be converted to UTF-8 with a leading UTF-8 BOM
+    // The output will be converted to UTF-8 with the leading UTF-16
+    // BOM removed. This behavior is wanted in interactive mode as
+    // some terminals seem to display the BOM character as a space,
+    // and it also breaks syntax highlighting.
     bat()
         .arg("--plain")
         .arg("--decorations=always")
         .arg("test_UTF-16LE.txt")
         .assert()
         .success()
-        .stdout(std::str::from_utf8(b"\xEF\xBB\xBFhello world\n").unwrap());
+        .stdout("hello world\n");
+}
+
+// Regression test for https://github.com/sharkdp/bat/issues/1922
+#[test]
+fn bom_not_stripped_in_loop_through_mode() {
+    bat()
+        .arg("--plain")
+        .arg("--decorations=never")
+        .arg("--color=never")
+        .arg("test_BOM.txt")
+        .assert()
+        .success()
+        .stdout("\u{feff}hello world\n");
+}
+
+// Regression test for https://github.com/sharkdp/bat/issues/1922
+#[test]
+fn bom_stripped_when_colored_output() {
+    bat()
+        .arg("--color=always")
+        .arg("--decorations=never")
+        .arg("test_BOM.txt")
+        .assert()
+        .success()
+        .stdout(
+            predicate::str::is_match("\u{1b}\\[38;5;[0-9]{3}mhello world\u{1b}\\[0m\n").unwrap(),
+        );
+}
+
+// Regression test for https://github.com/sharkdp/bat/issues/1922
+#[test]
+fn bom_stripped_when_no_color_and_not_loop_through() {
+    bat()
+        .arg("--color=never")
+        .arg("--decorations=always")
+        .arg("--style=numbers,grid,header")
+        .arg("--terminal-width=80")
+        .arg("test_BOM.txt")
+        .assert()
+        .success()
+        .stdout(
+            "\
+─────┬──────────────────────────────────────────────────────────────────────────
+     │ File: test_BOM.txt
+─────┼──────────────────────────────────────────────────────────────────────────
+   1 │ hello world
+─────┴──────────────────────────────────────────────────────────────────────────
+",
+        );
 }
 
 #[test]
diff --git a/tests/syntax-tests/highlighted/PowerShell/test.ps1 b/tests/syntax-tests/highlighted/PowerShell/test.ps1
index aae615d5a9..97e8e73848 100644
--- a/tests/syntax-tests/highlighted/PowerShell/test.ps1
+++ b/tests/syntax-tests/highlighted/PowerShell/test.ps1
@@ -1,4 +1,4 @@
-[38;2;248;248;242m﻿[0m[38;2;117;113;94m#[0m[38;2;117;113;94m PowerShell script for testing syntax highlighting[0m
+[38;2;117;113;94m#[0m[38;2;117;113;94m PowerShell script for testing syntax highlighting[0m
 
 [3;38;2;102;217;239mfunction[0m[38;2;248;248;242m [0m[38;2;166;226;46mGet-FutureTime[0m[38;2;248;248;242m [0m[38;2;248;248;242m{[0m
 [38;2;248;248;242m    [0m[38;2;249;38;114mparam[0m[38;2;248;248;242m [0m[38;2;248;248;242m([0m