Repository: modin-project/modin

Issue:
AttributeError: type object 'Callable' has no attribute '_abc_registry'
Modin seems to be working but generates an AttributeError on import.

System Info: 
```
Ubuntu 18.04
CPython 3.7.2
IPython 6.5.0

compiler   : GCC 7.3.0
system     : Linux
release    : 4.19.11-041911-generic
machine    : x86_64
```
Package Versions:
```
numpy       1.16.2
pandas      0.24.2
modin        0.4.0                    pypi_0    pypi
ray             0.7.0.dev2               pypi_0    pypi
```
Command: `import modin.pandas as pd`
Error:
```
2019-04-10 13:56:46,230	INFO node.py:469 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-04-10_13-56-46_16743/logs.
2019-04-10 13:56:46,338	INFO services.py:407 -- Waiting for redis server at 127.0.0.1:35351 to respond...
2019-04-10 13:56:46,452	INFO services.py:407 -- Waiting for redis server at 127.0.0.1:58663 to respond...
2019-04-10 13:56:46,457	INFO services.py:804 -- Starting Redis shard with 10.0 GB max memory.
2019-04-10 13:56:46,479	INFO node.py:483 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-04-10_13-56-46_16743/logs.
2019-04-10 13:56:46,482	WARNING services.py:1304 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 33493577728 bytes available. This may slow down performance! You may be able to free up space by deleting files in /dev/shm or terminating any running plasma_store_server processes. If you are inside a Docker container, you may need to pass an argument with the flag '--shm-size' to 'docker run'.
2019-04-10 13:56:46,484	INFO services.py:1427 -- Starting the Plasma object store with 40.0 GB memory using /tmp.
2019-04-10 13:56:47,951	ERROR worker.py:1672 -- Failed to unpickle the remote function 'modin.engines.ray.pandas_on_ray.io._read_parquet_columns' with function ID c0b4968ae548eb29942a3720be6d262ba97bd664. Traceback:
Traceback (most recent call last):
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/ray/function_manager.py", line 416, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/modin/engines/ray/pandas_on_ray/io.py", line 5, in <module>
    import pandas
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/typing.py", line 1356, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/typing.py", line 1004, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-04-10 13:56:47,953	ERROR worker.py:1672 -- Failed to unpickle the remote function 'modin.engines.ray.pandas_on_ray.io._read_parquet_columns' with function ID c0b4968ae548eb29942a3720be6d262ba97bd664. Traceback:
Traceback (most recent call last):
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/ray/function_manager.py", line 416, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/modin/engines/ray/pandas_on_ray/io.py", line 5, in <module>
    import pandas
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/typing.py", line 1356, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/typing.py", line 1004, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-04-10 13:56:47,964	ERROR worker.py:1672 -- Failed to unpickle the remote function 'modin.engines.ray.pandas_on_ray.io._read_parquet_columns' with function ID c0b4968ae548eb29942a3720be6d262ba97bd664. Traceback:
Traceback (most recent call last):
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/ray/function_manager.py", line 416, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/modin/engines/ray/pandas_on_ray/io.py", line 5, in <module>
    import pandas
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/typing.py", line 1356, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/typing.py", line 1004, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

...(keeps repeating)
```



Discussion:
Thanks for posting @BlackArbsCEO! 

If you run `pip install -U typing` does that fix your error? For some reason the typing used by pandas seems to be having some internal error. Relevant part of the log here:
```
File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/typing.py", line 1356, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/media/bcr/HDD/anaconda3/envs/py37/lib/python3.7/site-packages/typing.py", line 1004, in __new__
    self._abc_registry = extra._abc_registry
```

Let me know if that fixes it. You mentioned that it is working even though this error messeage is happening? It may be that the workers are not properly starting because of this and the performance may be impacted, so let's try to find the source of the issue if we can.
Hi @devin-petersohn, turns out this is a package conflict in Python 3.7 between the standard library typing and a pypi package called `typing 3.6.6`. I removed the pypi package and that error stopped however I have another warning message that may also be impacting performance. 
```
2019-04-11 08:33:15,528	INFO node.py:469 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-04-11_08-33-15_7964/logs.
2019-04-11 08:33:15,640	INFO services.py:407 -- Waiting for redis server at 127.0.0.1:16484 to respond...
2019-04-11 08:33:15,757	INFO services.py:407 -- Waiting for redis server at 127.0.0.1:40049 to respond...
2019-04-11 08:33:15,762	INFO services.py:804 -- Starting Redis shard with 10.0 GB max memory.
2019-04-11 08:33:15,786	INFO node.py:483 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-04-11_08-33-15_7964/logs.
2019-04-11 08:33:15,790	WARNING services.py:1304 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 33439985664 bytes available. This may slow down performance! You may be able to free up space by deleting files in /dev/shm or terminating any running plasma_store_server processes. If you are inside a Docker container, you may need to pass an argument with the flag '--shm-size' to 'docker run'.
2019-04-11 08:33:15,791	INFO services.py:1427 -- Starting the Plasma object store with 40.0 GB memory using /tmp.
```
If this is addressed in a different issue I would be happy to discuss over there.
@BlackArbsCEO Are you talking about this warning:
```
2019-04-11 08:33:15,790	WARNING services.py:1304 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 33439985664 bytes available. This may slow down performance! You may be able to free up space by deleting files in /dev/shm or terminating any running plasma_store_server processes. If you are inside a Docker container, you may need to pass an argument with the flag '--shm-size' to 'docker run'.
```

This may be affecting performance a bit, but we commonly get this warning when testing on larger machines and it doesn't seem to affect things too much. This warning is coming from the initialization of the plasma store.

If you'd like to create a new issue feel free!
I got the same issue for python 3.7, package "typing" conflict. Will the issue be fixed in next release?
Since it seems to be causing problems for a number of people, I will look into removing the dependency altogether. It should be possible because it's mostly used for development with Type Hints.

Resolving patch:
diff --git a/modin/engines/base/frame/partition_manager.py b/modin/engines/base/frame/partition_manager.py
--- a/modin/engines/base/frame/partition_manager.py
+++ b/modin/engines/base/frame/partition_manager.py
@@ -2,8 +2,6 @@
 from __future__ import division
 from __future__ import print_function
 
-from typing import Tuple
-
 import numpy as np
 import pandas
 
@@ -178,7 +176,7 @@ def _remove_empty_blocks(self):
             )
 
     @property
-    def shape(self) -> Tuple[int, int]:
+    def shape(self):
         return int(np.sum(self.block_lengths)), int(np.sum(self.block_widths))
 
     def full_reduce(self, map_func, reduce_func, axis):
diff --git a/modin/pandas/indexing.py b/modin/pandas/indexing.py
--- a/modin/pandas/indexing.py
+++ b/modin/pandas/indexing.py
@@ -7,7 +7,6 @@
 from pandas.api.types import is_scalar, is_list_like, is_bool
 from pandas.core.dtypes.common import is_integer
 from pandas.core.indexing import IndexingError
-from typing import Tuple
 from warnings import warn
 
 from .dataframe import DataFrame
@@ -315,7 +314,7 @@ def _compute_enlarge_labels(self, locator, base_index):
             )
         return nan_labels
 
-    def _compute_lookup(self, row_loc, col_loc) -> Tuple[pandas.Index, pandas.Index]:
+    def _compute_lookup(self, row_loc, col_loc):
         if is_list_like(row_loc) and len(row_loc) == 1:
             if (
                 isinstance(self.qc.index.values[0], np.datetime64)
@@ -362,7 +361,7 @@ def __setitem__(self, key, item):
         row_lookup, col_lookup = self._compute_lookup(row_loc, col_loc)
         super(_iLocIndexer, self).__setitem__(row_lookup, col_lookup, item)
 
-    def _compute_lookup(self, row_loc, col_loc) -> Tuple[pandas.Index, pandas.Index]:
+    def _compute_lookup(self, row_loc, col_loc):
         row_lookup = (
             pandas.RangeIndex(len(self.qc.index)).to_series().iloc[row_loc].index
         )
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -15,7 +15,7 @@
     url="https://github.com/modin-project/modin",
     long_description=long_description,
     long_description_content_type="text/markdown",
-    install_requires=["pandas==0.24.2", "ray==0.6.2", "typing"],
+    install_requires=["pandas==0.24.2", "ray==0.6.2"],
     extras_require={
         # can be installed by pip install modin[dask]
         "dask": ["dask==1.1.0", "distributed==1.25.0"]


Test patch: