Repository: streamlit/streamlit

Issue:
UnhashableTypeError: Cannot hash object of type _json.Scanner...
### Summary

When attempting to use st.cache decorator (with streamlit 1.10.0) on a json data loader method, streamlit returns an unhashable error for a _json.Scanner. This behavior does not exist in the previous version (1.9.2).  

### Steps to reproduce

1. Update streamlit to version 1.10.0
2. Use st.cache on any method that loads data from a json file

For example:

```python
@st.cache(allow_output_mutation=True)
def fetch_data_json(filename: str) -> Dict[str, Any]:
    with open(filename, "r") as f:
        config = json.load(f)
    return config
```

Returns: 

```
UnhashableTypeError: Cannot hash object of type _json.Scanner, found in the body of fetch_data_json().

While caching the body of fetch_data_json(), Streamlit encountered an object of type _json.Scanner, which it does not know how to hash.

To address this, please try helping Streamlit understand how to hash that type by passing the hash_funcs argument into @st.cache. For example:


@st.cache(hash_funcs={_json.Scanner: my_hash_func})
def my_func(...):
  
If you don't know where the object of type _json.Scanner is coming from, try looking at the hash chain below for an object that you do recognize, then pass that to hash_funcs instead:


Object of type _json.Scanner: <_json.Scanner object at 0x7fa22470b160>
Object of type builtins.method: <bound method JSONDecoder.raw_decode of <json.decoder.JSONDecoder object at 0x7fa2247b0760>>
Object of type builtins.method: <bound method JSONDecoder.decode of <json.decoder.JSONDecoder object at 0x7fa2247b0760>>
Object of type builtins.function: <function loads at 0x7fa22470e3a0>
Object of type builtins.function: <function load at 0x7fa22470e310>
Object of type builtins.function: <function fetch_data_json at 0x7fa22128cdc0>
Please see the hash_funcs [documentation] (https://docs.streamlit.io/library/advanced-features/caching#the-hash_funcs-parameter) for more details.

Traceback:
File "app.py", line 23, in <module>
    json_file = fetch_data_json(TEST_CONFIG_FOLDER + "/test_config.json")
File "/usr/lib/python3.8/copyreg.py", line 66, in _reduce_ex
    raise TypeError(f"cannot pickle {cls.__name__!r} object")
```

**Expected behavior:**

In version 1.9.2 the same code would produce valid parsed json data without @st.cache attempting to has an unhashable object. This image below is the method outlined above implemented in 1.10.0 and 1.9.2:

![bug](https://user-images.githubusercontent.com/73495852/174457986-a2912a56-2849-4ee2-a5f7-a00754298898.png)


### Debug info

- Streamlit version: 1.10.0
- Python version: 3.8.12
- Using Conda? PipEnv? PyEnv? Pex? Pip via Poetry
- OS version: Ubuntu 20.04 (via Windows 11)
- Browser version: Chrome 102.0.5005.115





Discussion:
Hi @xec-abailey , Thanks for submitting this bug!
I just tried reproducing this error with the following code:

```
import streamlit as st
import json
@st.cache(allow_output_mutation=True)
def fetch_data_json(filename: str):
    with open(filename, "r") as f:
        config = json.load(f)
    return config

st.write(fetch_data_json('random_file.json'))
```

where random_file.json contains 

`
{"name":"John", "age":30, "car":null}
`

I couldn't quite reproduce the error. Could you possibly give me more of the actual code that you're running and the data or anything else to help me reproduce the error?
Hello.

I'm having exactly the same issue here. I have updated Streamlit from 1.7.0 to 1.10.0 and I'm now having an UnhashableTypeError without changing anything in my code.

Weird thing is that sometimes it randomly work (let's say 1 time out of 10).

I was able to find a temporary fix by adding `hash_funcs={'_json.Scanner': hash}` to each `@st.cache` decorator that triggers this error, as stated in an old issue -> https://github.com/streamlit/streamlit/issues/1627#issuecomment-779293099. So far it seems to work.

@willhuang1997, I can reproduce (a similar error) with the below code. I first noticed this behaviour when upgrading a larger app from streamlit 1.8 to 1.10.

Possibly explaining the "1 time out of 10" behaviour seen by @LexABzH, I can make the error message go away by keeping a browser client open while restarting server. More concretely:

 1. Run `streamlit run app.py`
 2. Click network url in terminal output...
   - Result: New browser tab **with error**: `UnhashableTypeError: Cannot hash object of type _thread.RLock, found in the body of repeat_string_with_logging().`
 3. _While keeping browser tab with error open_, stop app (ctrl+C), then restart it with `streamlit run app.py`
   - Result: Browser tab reloads automatically **without error** (but with pop up saying `page that you have requested does not seem to exist. Running the app's main page.`). At this point reloading tab (and opening app in new tabs) will work without error and without pop up.

app.py:
```python
import logging
import streamlit as st

logger = logging.Logger(__name__)

@st.cache
def repeat_string_with_logging(x: str):
    logger.info(f"Here is x: {x}")
    return x * 3

st.write(repeat_string_with_logging("a"))
```

Using:
- Python 3.10
- Streamlit 1.10 installed in fresh Pipenv
- Ubuntu 1804 in WSL2
- Firefox 102.0.1

Terminal output when error happens (username and venv removed):
<details>

````python traceback
2022-07-13 09:59:01.306 Uncaught app exception
Traceback (most recent call last):
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 684, in _to_bytes
    reduce_data = obj.__reduce__()
  File "/usr/lib/python3.10/copyreg.py", line 76, in _reduce_ex
    raise TypeError(f"cannot pickle {cls.__name__!r} object")
TypeError: cannot pickle 'RLock' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/scriptrunner/script_runner.py", line 554, in _run_script
    exec(code, module.__dict__)
  File "app.py", line 11, in <module>
    st.write(repeat_string_with_logging("a"))
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/caching.py", line 573, in wrapped_func
    return get_or_create_cached_value()
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/caching.py", line 498, in get_or_create_cached_value
    cache_key = _hash_func(func, hash_funcs)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/caching.py", line 624, in _hash_func
    update_hash(
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 116, in update_hash
    ch.update(hasher, val, context)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 393, in update
    b = self.to_bytes(obj, context)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 368, in to_bytes
    b = b"%s:%s" % (tname, self._to_bytes(obj, context))
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 639, in _to_bytes
    h.update(self._code_to_bytes(code, context, func=obj))
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 708, in _code_to_bytes
    self.update(h, ref, context)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 393, in update
    b = self.to_bytes(obj, context)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 368, in to_bytes
    b = b"%s:%s" % (tname, self._to_bytes(obj, context))
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 639, in _to_bytes
    h.update(self._code_to_bytes(code, context, func=obj))
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 708, in _code_to_bytes
    self.update(h, ref, context)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 393, in update
    b = self.to_bytes(obj, context)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 368, in to_bytes
    b = b"%s:%s" % (tname, self._to_bytes(obj, context))
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 639, in _to_bytes
    h.update(self._code_to_bytes(code, context, func=obj))
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 708, in _code_to_bytes
    self.update(h, ref, context)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 393, in update
    b = self.to_bytes(obj, context)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 368, in to_bytes
    b = b"%s:%s" % (tname, self._to_bytes(obj, context))
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 639, in _to_bytes
    h.update(self._code_to_bytes(code, context, func=obj))
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 708, in _code_to_bytes
    self.update(h, ref, context)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 393, in update
    b = self.to_bytes(obj, context)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 368, in to_bytes
    b = b"%s:%s" % (tname, self._to_bytes(obj, context))
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 686, in _to_bytes
    raise UnhashableTypeError(e, obj)
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 684, in _to_bytes
    reduce_data = obj.__reduce__()
  File "/usr/lib/python3.10/copyreg.py", line 76, in _reduce_ex
    raise TypeError(f"cannot pickle {cls.__name__!r} object")
streamlit.legacy_caching.hashing.UnhashableTypeError: Cannot hash object of type `_thread.RLock`, found in the body of
`repeat_string_with_logging()`.

While caching the body of `repeat_string_with_logging()`, Streamlit encountered an
object of type `_thread.RLock`, which it does not know how to hash.

To address this, please try helping Streamlit understand how to hash that type
by passing the `hash_funcs` argument into `@st.cache`. For example:

```
@st.cache(hash_funcs={_thread.RLock: my_hash_func})
def my_func(...):
    ...
```

If you don't know where the object of type `_thread.RLock` is coming
from, try looking at the hash chain below for an object that you do recognize,
then pass that to `hash_funcs` instead:

```
Object of type _thread.RLock: <unlocked _thread.RLock object owner=0 count=0 at 0x7f2f8cff04c0>
Object of type builtins.function: <function _acquireLock at 0x7f2f8d046ef0>
Object of type builtins.method: <bound method Logger.isEnabledFor of <Logger __main__ (NOTSET)>>
Object of type builtins.method: <bound method Logger.info of <Logger __main__ (NOTSET)>>
Object of type builtins.function: <function repeat_string_with_logging at 0x7f2f5ca381f0>
```

Please see the `hash_funcs` [documentation]
(https://docs.streamlit.io/library/advanced-features/caching#the-hash_funcs-parameter)
for more details.
````

With steps in https://github.com/streamlit/streamlit/issues/4876#issuecomment-1182933353 I can also reproduce the exact _json.Scanner error (including the keep-browser-open way of making it go away) using this code:

```python
import json
import streamlit as st

fname = "dummy.json"
with open(fname, "w") as f:
    json.dump({}, f)

@st.cache(allow_output_mutation=True)
def fetch_data_json(filename: str):
    with open(filename, "r") as f:
        config = json.load(f)
    return config

st.write(fetch_data_json(fname))
```

In this case traceback also points to streamlit/legacy_caching/hashing.py line 684:

```python traceback
Traceback (most recent call last):
  File "/home/<user>/.local/share/virtualenvs/<venv>/lib/python3.10/site-packages/streamlit/legacy_caching/hashing.py", line 684, in _to_bytes
    reduce_data = obj.__reduce__()
  File "/usr/lib/python3.10/copyreg.py", line 76, in _reduce_ex
    raise TypeError(f"cannot pickle {cls.__name__!r} object")
TypeError: cannot pickle 'Scanner' object```
I am facing the exact same issue and was able to find a work around it "locally" by keeping a browser client open while restarting server as @amhrasmussen mentioned. However, there's no work around it when I deploy it to the AKS remote server. I think this has to do with the newest version of streamlit 1.10. I'd have used older version if I were not using the multipage function that comes with v1.10. 

I'd really appreciate any follow ups on this issue by the streamlit team :)  
It seems that the errors are caused by streamlit erroneously(?) trying to hash code outside the current app directory (for example from Python standard modules like json, logging, ...).

This happens when [_get_main_script_directory()](https://github.com/streamlit/streamlit/blob/develop/lib/streamlit/legacy_caching/hashing.py#L714) returns '' (empty string), instead of the absolute path to folder containing the script. 

A superficial fix would be to have that function return `str(Path(main_path).absolute())` instead of `str(os.path.dirname(main_path))`, but maybe a deeper fix of the `__main__.__file__` magic in `ScriptRunnner` would be better...

(I would be happy to contribute a pull request, but have so far been unable to set up a local dev environment...)
I've poked the code a bit more and I've found that:
- `_get_main_script_directory()` returns an empty string because `__main__.__file__` is filename only ('app.py') instead of full path ('/path/to/app.py')

The setting of `__main__.__file__` happens in `ScriptRunner._run_script()` through these steps (in the failing case):
1. `main_page_info` is derived from `source_util.get_pages(main_script_path)`. Here `main_script_path` has full path, but `main_page_info["script_path"]` ends up with only script filename. Possibly due to caching (not keyed on input) inside get_pages() remembering previous calls with filename only as input (`app.py`).
1. `current_page_info = main_page_info`
1. `script_path = current_page_info["script_path"]`
1. `module.__dict__["__file__"] = script_path`

Hello, thanks for reporting this issue!
@xec-abailey, @amhrasmussen we developed new cache primitives that have fewer responsibilities and are designed for more specific cases. 

Could you please try to use `st.experimental_memo` or `st.experimental_singleton` for exact same purpose (reading config file)depends on what works better with your business logic

We currently preparing them for productional usage, and are interested in any feedback or use cases

Resolving patch:
diff --git a/lib/streamlit/runtime/legacy_caching/hashing.py b/lib/streamlit/runtime/legacy_caching/hashing.py
--- a/lib/streamlit/runtime/legacy_caching/hashing.py
+++ b/lib/streamlit/runtime/legacy_caching/hashing.py
@@ -712,14 +712,14 @@ def _code_to_bytes(self, code, context: Context, func=None) -> bytes:
 
     @staticmethod
     def _get_main_script_directory() -> str:
-        """Get the directory of the main script."""
+        """Get the absolute path to directory of the main script."""
         import __main__
-        import os
+        import pathlib
 
         # This works because we set __main__.__file__ to the
         # script path in ScriptRunner.
-        main_path = __main__.__file__
-        return str(os.path.dirname(main_path))
+        abs_main_path = pathlib.Path(__main__.__file__).resolve()
+        return str(abs_main_path.parent)
 
 
 def get_referenced_objects(code, context: Context) -> List[Any]:


Test patch:
diff --git a/lib/tests/streamlit/runtime/legacy_caching/hashing_test.py b/lib/tests/streamlit/runtime/legacy_caching/hashing_test.py
--- a/lib/tests/streamlit/runtime/legacy_caching/hashing_test.py
+++ b/lib/tests/streamlit/runtime/legacy_caching/hashing_test.py
@@ -1118,3 +1118,36 @@ def h(a, b):
         self.assertNotEqual(get_hash(np.remainder), get_hash(np.logical_and))
         self.assertEqual(get_hash(f), get_hash(g))
         self.assertNotEqual(get_hash(f), get_hash(h))
+
+
+class MainScriptDirectoryDetectionTest(unittest.TestCase):
+
+    relative_path = "app.py"
+    abs_path = "/path/to/app.py"
+
+    @patch("__main__.__file__", relative_path)
+    def test_relative_main_file(self):
+        """Test that we get abs dir path when __main__.__file__ is script file name only."""
+
+        ch = _CodeHasher()
+
+        # We don't want empty string returned:
+        self.assertNotEqual(ch._get_main_script_directory(), "")
+
+        # During testing, __main__.__file__ has not modified so we expect current
+        # working dir:
+        self.assertEqual(ch._get_main_script_directory(), os.getcwd())
+
+    @patch("__main__.__file__", abs_path)
+    def test_absolute_main_file(self):
+        """Test that we get abs dir path when __main__.__file__ relative path."""
+
+        import pathlib
+
+        ch = _CodeHasher()
+
+        # When __main__.__file__ is absolute path to script, we expect parent dir to be
+        # returned:
+        self.assertEqual(
+            ch._get_main_script_directory(), str(pathlib.Path(self.abs_path).parent)
+        )