Repository: bskinn/sphobjinv

Issue:
UnicodeDecodeError
**Brief description**
<!-- Clear and concise description of the bug. -->
Not clear if this is a bug/limitation in sphobjinv or if the objects.inv in question is messed up, but I get a UnicodeDecodeError for 
```
sphobjinv suggest -u https://fonttools.readthedocs.io/en/latest/objects.inv TTFont
``` 
(and related versions of the call)

**Expected behavior**
<!-- Description of what you expected to happen. -->
A suggestion.

**Actual behavior**
<!-- Description of what actually happened. -->
```
No inventory at provided URL.
Attempting "https://fonttools.readthedocs.io/en/latest/objects.inv" ...
Remote inventory found.
 
Traceback (most recent call last):
  File "/local/data1/miniconda3/bin/sphobjinv", line 8, in <module>
    sys.exit(main())
  File "/local/data1/miniconda3/lib/python3.8/site-packages/sphobjinv/cli/core.py", line 214, in main
    do_suggest(inv, params)
  File "/local/data1/miniconda3/lib/python3.8/site-packages/sphobjinv/cli/core.py", line 109, in do_suggest
    results = inv.suggest(
  File "/local/data1/miniconda3/lib/python3.8/site-packages/sphobjinv/inventory.py", line 530, in suggest
    for match, score in fwp.extract(name, srch_list, limit=None)
  File "/local/data1/miniconda3/lib/python3.8/site-packages/sphobjinv/_vendored/fuzzywuzzy/process.py", line 60, in extract
    processed = processor(choice)
  File "/local/data1/miniconda3/lib/python3.8/site-packages/sphobjinv/_vendored/fuzzywuzzy/process.py", line 51, in <lambda>
    processor = lambda x: utils.asciidammit(x)
  File "/local/data1/miniconda3/lib/python3.8/site-packages/sphobjinv/_vendored/fuzzywuzzy/utils.py", line 17, in asciidammit
    return asciionly(s)
  File "/local/data1/miniconda3/lib/python3.8/site-packages/sphobjinv/_vendored/fuzzywuzzy/utils.py", line 12, in asciionly
    return s.encode().translate(None, bad_chars).decode()  # B Skinn 2021-12-11
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 21: invalid continuation byte
```

**To reproduce**
<!--
  Steps to reproduce the behavior:
  1. Open '...'
  2. Type '...'
  3. See error
-->
```
sphobjinv suggest -u https://fonttools.readthedocs.io/en/latest/objects.inv TTFont
``` 

**Attachments**
<!--
  If possible, please paste in an inventory file that
  demonstrates the bug, or provide a link to a relevant
  inventory on web.

  If relevant, paste in screenshot(s) demonstrating
  the bug.
-->

**System information**
 - Device: 
 - OS: CentOS 7

**Python environment**

*Python*
<!--
  Please paste the output of `$ {python} --version --version`
  executed in the relvant environment.
-->

Python 3.8.12 | packaged by conda-forge | (default, Jan 30 2022, 23:42:07) 
[GCC 9.4.0]

*Libraries*
<!--
  Please paste the output of `$ pip list` or `$ pip freeze`
  executed in the relevant environment.
--> 
Skipping this, let me know if required.

**Additional information**
<!-- Add any other context about the problem here. -->


Discussion:
bskinn: That looks like a great solution! The `replace` will have some effect on the search/suggest behavior for any objects where a decode error occurs, but I can't imagine it'll noticeably affect user experience. 

I would like to add a smoke test for this inventory -- please download a copy of the misbehaving fonttools `objects.inv`, rename to `objects_fonttools.inv`, and place in the `tests/resource` directory. It'll get picked up by the CI.
oscargus: (Clearly I didn't look much into adding a test, more thinking about the options.)

Very nice test setup!
bskinn: Whoa, well that blew up. Not sure why.
bskinn: OH.  Because it does do a comparison against the inventory housed on the `main` branch of this repo.

And, that file doesn't exist yet.  One sec.
bskinn: Oh, interesting -- this issue has revealed a gap in my test suite: I don't have a `suggest` test that touches all of the inventories in `tests/resource`.
bskinn: Ok! I think `main` is now ready to accommodate this PR.

Please rebase your working branch for the PR onto my upstream `main` and push back up to Github. (If you're not sure how to do this, let me know and I can provide steps.)

At this point, the test suite should FAIL.

Then, if you remove the special case for `fonttools` [here](https://github.com/bskinn/sphobjinv/blob/a35742667319c3f0bda51000cb211e85d9ab8e10/tests/test_api_good.py#L477-L483) should 🤞 then make the test suite pass.

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c092f6e..f4349e4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project strives to adhere to
 [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+  * UnicodeDecodeErrors are ignored when converting inventories to ASCII.
 
 ### [2.2.1] - 2022-02-05
 
diff --git a/src/sphobjinv/_vendored/fuzzywuzzy/utils.py b/src/sphobjinv/_vendored/fuzzywuzzy/utils.py
index ab756b4..28ce02f 100644
--- a/src/sphobjinv/_vendored/fuzzywuzzy/utils.py
+++ b/src/sphobjinv/_vendored/fuzzywuzzy/utils.py
@@ -9,7 +9,7 @@ trans_table=bytes.maketrans(table_from.encode(), table_to.encode())  # B Skinn 2
 
 
 def asciionly(s):
-    return s.encode().translate(None, bad_chars).decode()  # B Skinn 2021-12-11
+    return s.encode().translate(None, bad_chars).decode(errors='replace')  # B Skinn 2021-12-11
 
 # remove non-ASCII characters from strings
 def asciidammit(s):
@@ -32,8 +32,4 @@ def validate_string(s):
 def full_process(s):
     s = asciidammit(s)
     # B Skinn 2021-12-11
-    return s.encode().translate(trans_table, bad_chars).decode().strip()
-
-
-
-
+    return s.encode().translate(trans_table, bad_chars).decode(errors='replace').strip()


Test patch:
diff --git a/tests/test_api_good.py b/tests/test_api_good.py
index 8bb8910..c05e3b1 100644
--- a/tests/test_api_good.py
+++ b/tests/test_api_good.py
@@ -474,14 +474,6 @@ class TestInventory:
         """Confirm that a suggest operation works on all smoke-test inventories."""
         inv = soi.Inventory(testall_inv_path)
 
-        if "fonttools" in inv.project.lower():
-            try:
-                inv.suggest("class")
-            except UnicodeDecodeError:
-                pytest.xfail("Known unhandled bad character in decode operation")
-            else:  # pragma: no cover
-                pytest.fail("'fonttools' was expected to fail, but didn't")
-
         inv.suggest("class")
 
     @pytest.mark.testall