Repository: dtolnay/cargo-expand (language: rust)

Issue:
UnicodeDecodeError
cmd: `cargo expand --color never`
version: v0.3.10

Error:

```shell
** Error while highlighting:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1832403: ordinal not in range(128)
   (file "/usr/local/lib/python2.7/site-packages/Pygments-1.6-py2.7.egg/pygments/lexer.py", line 168, in get_tokens)
```

I don't understand why pygments is invoked at all when using `--color never`. I can also reproduce without the option.


Discussion / hints:
@dtolnay I get the same error when I do `cargo expand` (without `--color never`). 
Can you share the code of the crate you are expanding?

Do you get the same error piping to pygmentize manually?

```bash
cargo rustc --color=always -- -Zunstable-options --pretty=expanded | pygmentize -l rust
```

Can you tell what the line containing position 1832403 (I assume a byte offset from the beginning of the expanded output) looks like?
> Can you share the code of the crate you are expanding?

I am expanding the `stdsimd` crate. 

> Do you get the same error piping to pygmentize manually?

Yes.

> Can you tell what the line containing position 1832403

This line:

```rust
        /// # '❤'.encode_utf16(&mut some_utf16_words);
```
Thanks! I have not been able to reproduce the error but we are on the right track. Does it succeed with `-O encoding=utf8`?

```bash
cargo rustc --color=always -- -Zunstable-options --pretty=expanded | pygmentize -l rust -O encoding=utf8
```
> Does it succeed with -O encoding=utf8?

Yes! That works!
I am on MacOS X btw, maybe it has to do with the version of python installed here.

Fix patch:
diff --git a/src/main.rs b/src/main.rs
index 3b9b58b..56ccd21 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -99,7 +99,7 @@ fn cargo_expand() -> io::Result<i32> {
 
     // Pipe to pygmentize
     let _wait = match which_pygmentize {
-        Some(pyg) => Some(cmd.pipe_to(shell!(pyg "-l" "rust"), None)?),
+        Some(pyg) => Some(cmd.pipe_to(shell!(pyg "-l" "rust" "-O" "encoding=utf8"), None)?),
         None => None,
     };
 


Test patch: