Repository: simonw/strip-tags (language: python)

Issue:
Issue with encoding while attempting to strip tags
While attempting to strip tags from the NY Times front page I received the following error:
Traceback (most recent call last):
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\Scripts\strip-tags.exe\__main__.py", line 7, in <module>
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\strip_tags\cli.py", line 119, in cli
    click.echo(final)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\click\utils.py", line 299, in echo
    file.write(out)  # type: ignore
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2192' in position 9634: character maps to <undefined>

I don't know if this is an issue with strip-tags or a more core issue with encodings in general.

Issue with encoding while attempting to strip tags
While attempting to strip tags from the NY Times front page I received the following error:
Traceback (most recent call last):
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\Scripts\strip-tags.exe\__main__.py", line 7, in <module>
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\strip_tags\cli.py", line 119, in cli
    click.echo(final)
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\site-packages\click\utils.py", line 299, in echo
    file.write(out)  # type: ignore
  File "C:\Users\Bowlingb\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2192' in position 9634: character maps to <undefined>

I don't know if this is an issue with strip-tags or a more core issue with encodings in general.



Discussion / hints:
Could you grab a copy of an example of HTML that this broke on? I'd like to investigate further.
I got a similar issue with HTML from this document (https://www.sec.gov/Archives/edgar/data/1046179/000114554905001363/u99743a1fv3za.htm)

Error for me is:
``` bash
Traceback (most recent call last):
  File "/Users/localuser/.local/bin/strip-tags", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/strip_tags/cli.py", line 91, in cli
    soup = BeautifulSoup(input, "html5lib")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/bs4/__init__.py", line 314, in __init__
    markup = markup.read()
             ^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 2997: invalid start byte
```

Looks like a pretty classic encoding error when parsing a file, will look into this.
Could you grab a copy of an example of HTML that this broke on? I'd like to investigate further.
I got a similar issue with HTML from this document (https://www.sec.gov/Archives/edgar/data/1046179/000114554905001363/u99743a1fv3za.htm)

Error for me is:
``` bash
Traceback (most recent call last):
  File "/Users/localuser/.local/bin/strip-tags", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/strip_tags/cli.py", line 91, in cli
    soup = BeautifulSoup(input, "html5lib")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/localuser/.local/pipx/venvs/strip-tags/lib/python3.11/site-packages/bs4/__init__.py", line 314, in __init__
    markup = markup.read()
             ^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 2997: invalid start byte
```

Looks like a pretty classic encoding error when parsing a file, will look into this.

Fix patch:
diff --git a/strip_tags/cli.py b/strip_tags/cli.py
index a44f19b..4387a5f 100644
--- a/strip_tags/cli.py
+++ b/strip_tags/cli.py
@@ -13,7 +13,7 @@
     multiple=True,
     help="Remove content in these selectors",
 )
-@click.option("-i", "--input", type=click.File("r"), default="-", help="Input file")
+@click.option("-i", "--input", type=click.File("rb"), default="-", help="Input file")
 @click.option("-m", "--minify", is_flag=True, help="Minify whitespace")
 @click.option("keep_tags", "-t", "--keep-tag", multiple=True, help="Keep these <tags>")
 @click.option("--all-attrs", is_flag=True, help="Include all attributes on kept tags")


Test patch:
diff --git a/tests/test_strip_tags.py b/tests/test_strip_tags.py
index cc4cdcd..a04abdb 100644
--- a/tests/test_strip_tags.py
+++ b/tests/test_strip_tags.py
@@ -79,3 +79,13 @@ def test_strip_lib(input, args, expected):
         all_attrs=all_attrs,
     )
     assert result == expected
+
+
+def test_not_ut8_encoding():
+    runner = CliRunner()
+    with runner.isolated_filesystem():
+        with open("input.html", "wb") as fp:
+            fp.write("<h1>hello</h1>".encode("utf-16"))
+        result = runner.invoke(cli, ["-i", "input.html"])
+    assert result.exit_code == 0
+    assert result.output == "hello\n"