internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ray-project/ray

Issue:
[DataFrame] Error checking on Pandas version
We need better reporting for issues with the Pandas version on a user's system. 


Discussion:
How about in `dataframe.py`, we add a line like

```python
# Pandas must be at least version 0.22.0.
pd_major = int(pd.__version__.split(".")[0])
pd_minor = int(pd.__version__.split(".")[1])
if (pd_major == 0) and (pd_minor < 22):
    raise Exception("Your Pandas version is less than 0.22.0. Please "
                    "update Pandas to the latest version.")
```


I was thinking it belongs in `dataframe/__init__.py`  before we start the Ray cluster.
That's probably better.

Resolving patch:
diff --git a/python/ray/dataframe/__init__.py b/python/ray/dataframe/__init__.py
--- a/python/ray/dataframe/__init__.py
+++ b/python/ray/dataframe/__init__.py
@@ -1,8 +1,18 @@
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
+
+import pandas as pd
 import threading
 
+pd_version = pd.__version__
+pd_major = int(pd_version.split(".")[0])
+pd_minor = int(pd_version.split(".")[1])
+
+if pd_major == 0 and pd_minor < 22:
+    raise Exception("In order to use Pandas on Ray, please upgrade your Pandas"
+                    " version to >= 0.22.")
+
 DEFAULT_NPARTITIONS = 4
 
 


Test patch:
Repository: ray-project/ray

Issue:
"This should be unreachable" error message hard to interpret
I've seen this when forgetting to call `ray.init()`, maybe it should say `..Unreachable...Did you call ray.init()?`

```
    return func(*args, **kwargs)
  File "/home/ubuntu/Dropbox/git0/tensorpack/tensorpack/train/base.py", line 253, in main_loop
    self.run_step()  # implemented by subclass
  File "resnet_numpy_concat_ray_test.py", line 184, in run_step
    ps = ParameterServer.remote(grad_values_flat.size)
  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/ray/actor.py", line 761, in remote
    ray.worker.global_worker)
  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/ray/actor.py", line 387, in export_actor_class
    assert False, "This should be unreachable."
```


Discussion:

Resolving patch:
diff --git a/python/ray/actor.py b/python/ray/actor.py
--- a/python/ray/actor.py
+++ b/python/ray/actor.py
@@ -706,6 +706,10 @@ class ActorHandle(actor_handle_class):
 
         @classmethod
         def remote(cls, *args, **kwargs):
+            if ray.worker.global_worker.mode is None:
+                raise Exception("Actors cannot be created before ray.init() "
+                                "has been called.")
+
             actor_id = random_actor_id()
             # The ID for this instance of ActorHandle. These should be unique
             # across instances with the same _ray_actor_id.


Test patch:
Repository: ray-project/ray

Issue:
SyntaxError: Non-ASCII character '\xe2' in file ray/dataframe/dataframe.py
Hi, 

I built the ray from source in Ubuntu 16.04 (python 2.7) and tried to import ray.dataframe:
`import ray.dataframe as pd`

and received the following error:
```
SyntaxError: Non-ASCII character '\xe2' in file ray/dataframe/dataframe.py on line 745, but no encoding dttp://python.org/dev/peps/pep-0263/ for details
```

https://github.com/ray-project/ray/blob/2b747ba46cee4445c8ff1640a2a5d7c88e2d1723/python/ray/dataframe/dataframe.py#L744

Could you please change all the non-ASCII single-quote (’) in the docstring to the standard quote (') in your repo? It works after I did that. Thanks!



Discussion:

Resolving patch:
diff --git a/python/ray/dataframe/dataframe.py b/python/ray/dataframe/dataframe.py
--- a/python/ray/dataframe/dataframe.py
+++ b/python/ray/dataframe/dataframe.py
@@ -741,8 +741,8 @@ def drop(self, labels=None, axis=0, index=None, columns=None, level=None,
         Args:
             labels: Index or column labels to drop.
 
-            axis: Whether to drop labels from the index (0 / ‘index’) or
-                columns (1 / ‘columns’).
+            axis: Whether to drop labels from the index (0 / 'index') or
+                columns (1 / 'columns').
 
             index, columns: Alternative to specifying axis (labels, axis=1 is
                 equivalent to columns=labels).
@@ -751,7 +751,7 @@ def drop(self, labels=None, axis=0, index=None, columns=None, level=None,
 
             inplace: If True, do operation inplace and return None.
 
-            errors: If ‘ignore’, suppress error and existing labels are
+            errors: If 'ignore', suppress error and existing labels are
                 dropped.
         Returns:
             dropped : type of caller
@@ -972,7 +972,7 @@ def fillna(self, value=None, method=None, axis=None, inplace=False,
                 backfill.
                 bfill: use NEXT valid observation to fill gap.
 
-            axis: 0 or ‘index’, 1 or ‘columns’.
+            axis: 0 or 'index', 1 or 'columns'.
 
             inplace: If True, fill in place. Note: this will modify any other
                 views on this object.
@@ -986,7 +986,7 @@ def fillna(self, value=None, method=None, axis=None, inplace=False,
                 than 0 if not None.
 
             downcast: A dict of item->dtype of what to downcast if possible,
-                or the string ‘infer’ which will try to downcast to an
+                or the string 'infer' which will try to downcast to an
                 appropriate equal type.
 
         Returns:


Test patch:
Repository: ray-project/ray

Issue:
Speed up actor ID (and other random ID generation).
This function is really slow (on the order of 5-10ms). It is used in a bunch of places and should be replaced with something faster (e.g., a deterministic hash). This is an issue e.g., for actor creation.

https://github.com/ray-project/ray/blob/cae108d019a51bd7bd35df7f93b68ff07a84d9cd/python/ray/utils.py#L84-L107


Discussion:

Resolving patch:
diff --git a/python/ray/actor.py b/python/ray/actor.py
--- a/python/ray/actor.py
+++ b/python/ray/actor.py
@@ -12,18 +12,10 @@
 import ray.local_scheduler
 import ray.signature as signature
 import ray.worker
-from ray.utils import (FunctionProperties, random_string, is_cython,
+from ray.utils import (FunctionProperties, _random_string, is_cython,
                        push_error_to_driver)
 
 
-def random_actor_id():
-    return ray.local_scheduler.ObjectID(random_string())
-
-
-def random_actor_class_id():
-    return random_string()
-
-
 def compute_actor_handle_id(actor_handle_id, num_forks):
     """Deterministically comopute an actor handle ID.
 
@@ -750,7 +742,7 @@ def remote(cls, *args, **kwargs):
                 raise Exception("Actors cannot be created before ray.init() "
                                 "has been called.")
 
-            actor_id = random_actor_id()
+            actor_id = ray.local_scheduler.ObjectID(_random_string())
             # The ID for this instance of ActorHandle. These should be unique
             # across instances with the same _ray_actor_id.
             actor_handle_id = ray.local_scheduler.ObjectID(
@@ -930,7 +922,7 @@ def __ray_checkpoint_restore__(self):
     Class.__module__ = cls.__module__
     Class.__name__ = cls.__name__
 
-    class_id = random_actor_class_id()
+    class_id = _random_string()
 
     return actor_handle_from_class(Class, class_id, resources,
                                    checkpoint_interval, actor_method_cpus)
diff --git a/python/ray/utils.py b/python/ray/utils.py
--- a/python/ray/utils.py
+++ b/python/ray/utils.py
@@ -4,9 +4,11 @@
 
 import binascii
 import collections
+import hashlib
 import numpy as np
 import os
 import sys
+import uuid
 
 import ray.local_scheduler
 
@@ -15,7 +17,11 @@
 
 
 def _random_string():
-    return np.random.bytes(20)
+    id_hash = hashlib.sha1()
+    id_hash.update(uuid.uuid4().bytes)
+    id_bytes = id_hash.digest()
+    assert len(id_bytes) == 20
+    return id_bytes
 
 
 def format_error_message(exception_message, task_exception=False):


Test patch:
Repository: ray-project/ray

Issue:
[webui] Connecting to webui doesn't work when Ray is started with `ray.init()`.
Running the webui gives me an error message like "Some processes on this node may not have started yet".


Discussion:

Resolving patch:
diff --git a/python/ray/worker.py b/python/ray/worker.py
--- a/python/ray/worker.py
+++ b/python/ray/worker.py
@@ -1174,7 +1174,10 @@ def get_address_info_from_redis_helper(redis_address, node_ip_address):
         assert b"ray_client_id" in info
         assert b"node_ip_address" in info
         assert b"client_type" in info
-        if info[b"node_ip_address"].decode("ascii") == node_ip_address:
+        client_node_ip_address = info[b"node_ip_address"].decode("ascii")
+        if (client_node_ip_address == node_ip_address or
+                (client_node_ip_address == "127.0.0.1" and
+                 redis_ip_address == ray.services.get_node_ip_address())):
             if info[b"client_type"].decode("ascii") == "plasma_manager":
                 plasma_managers.append(info)
             elif info[b"client_type"].decode("ascii") == "local_scheduler":


Test patch:
diff --git a/test/multi_node_test.py b/test/multi_node_test.py
--- a/test/multi_node_test.py
+++ b/test/multi_node_test.py
@@ -283,5 +283,24 @@ def f():
         subprocess.Popen(["ray", "stop"]).wait()
 
 
+class MiscellaneousTest(unittest.TestCase):
+    def tearDown(self):
+        ray.worker.cleanup()
+
+    def testConnectingInLocalCase(self):
+        address_info = ray.init(num_cpus=0)
+
+        # Define a driver that just connects to Redis.
+        driver_script = """
+import ray
+ray.init(redis_address="{}")
+print("success")
+""".format(address_info["redis_address"])
+
+        out = run_string_as_driver(driver_script)
+        # Make sure the other driver succeeded.
+        self.assertIn("success", out)
+
+
 if __name__ == "__main__":
     unittest.main(verbosity=2)
Repository: ray-project/ray

Issue:
[rllib] Deadlock error when running ES.
When running 

```
python ray/python/ray/rllib/train.py --redis-address=172.31.7.72:6379 --env=Humanoid-v1 --run=ES --config='{"episodes_per_batch": 1000, "timesteps_per_batch": 10000, "num_workers": 400}'
```

on a cluster (100 machines), I see

```
Traceback (most recent call last):
  File "/home/ubuntu/ray/python/ray/worker.py", line 1720, in fetch_and_execute_function_to_run
    function = pickle.loads(serialized_function)
  File "/home/ubuntu/ray/python/ray/rllib/__init__.py", line 17, in <module>
    _register_all()
  File "/home/ubuntu/ray/python/ray/rllib/__init__.py", line 14, in _register_all
    register_trainable(key, get_agent_class(key))
  File "/home/ubuntu/ray/python/ray/rllib/agent.py", line 229, in get_agent_class
    from ray.rllib import es
  File "/home/ubuntu/ray/python/ray/rllib/es/__init__.py", line 1, in <module>
    from ray.rllib.es.es import (ESAgent, DEFAULT_CONFIG)
  File "/home/ubuntu/ray/python/ray/rllib/es/es.py", line 19, in <module>
    from ray.rllib.es import policies
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 168, in __enter__
  File "<frozen importlib._bootstrap>", line 110, in acquire
_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('ray.rllib.es.policies') at 139937598221224
```

This likely has to do with recursive imports in rllib, probably related to #1716.


Discussion:
The same happens with PPO:

```
  File "/opt/conda/lib/python3.6/site-packages/ray/actor.py", line 284, in fetch_and_register_actor
    unpickled_class = pickle.loads(pickled_class)
  File "/opt/conda/lib/python3.6/site-packages/ray/rllib/ppo/__init__.py", line 1, in <module>
    from ray.rllib.ppo.ppo import (PPOAgent, DEFAULT_CONFIG)
  File "/opt/conda/lib/python3.6/site-packages/ray/rllib/ppo/ppo.py", line 17, in <module>
    from ray.rllib.ppo.ppo_evaluator import PPOEvaluator
  File "/opt/conda/lib/python3.6/site-packages/ray/rllib/ppo/ppo_evaluator.py", line 14, in <module>
    from ray.rllib.optimizers import PolicyEvaluator, SampleBatch
  File "<frozen importlib._bootstrap>", line 960, in _find_and_load
  File "<frozen importlib._bootstrap>", line 151, in __enter__
  File "<frozen importlib._bootstrap>", line 93, in acquire
_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('ray.rllib.optimizers') at 140202634781472```

Resolving patch:
diff --git a/python/ray/worker.py b/python/ray/worker.py
--- a/python/ray/worker.py
+++ b/python/ray/worker.py
@@ -881,7 +881,8 @@ def _become_actor(self, task):
         while key not in self.imported_actor_classes:
             time.sleep(0.001)
 
-        self.fetch_and_register_actor(key, task.required_resources(), self)
+        with self.lock:
+            self.fetch_and_register_actor(key, task.required_resources(), self)
 
     def _wait_for_and_process_task(self, task):
         """Wait for a task to be ready and process the task.


Test patch:
Repository: ray-project/ray

Issue:
[XRay] GPU label support
Start two raylets:

```
ray start --head --use-raylet --num-gpus 1
```

```
ray start --redis-address 172.31.23.99:28629 --use-raylet --num-gpus 1
```

Execute this script:

```
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import ray

@ray.remote(num_gpus=1)
class Worker(object):

    def __init__(self):
        pass

    def get_node_ip(self):
        return ray.services.get_node_ip_address()

ray.init(redis_address="172.31.23.99:28629", use_raylet=True)
ws = [Worker.remote() for i in range(10)]

print("XXX", [ray.get(w.get_node_ip.remote()) for w in ws])
```

Output:
```
XXX ['172.31.23.99', '172.31.23.99', '172.31.23.99', '172.31.23.99', '172.31.23.99', '172.31.23.99', '172.31.23.99', '172.31.23.99', '172.31.23.99', '172.31.23.99']
```


Discussion:

Resolving patch:
diff --git a/python/ray/services.py b/python/ray/services.py
--- a/python/ray/services.py
+++ b/python/ray/services.py
@@ -1551,7 +1551,8 @@ def start_ray_node(node_ip_address,
         redirect_output=redirect_output,
         resources=resources,
         plasma_directory=plasma_directory,
-        huge_pages=huge_pages)
+        huge_pages=huge_pages,
+        use_raylet=use_raylet)
 
 
 def start_ray_head(address_info=None,


Test patch:
Repository: ray-project/ray

Issue:
[DataFrame] _Index_MetaData cannot be instantiated with an OID.
This was a functionality reversion merged in #1965. Since we're caching the Index now, we need an additional layer of abstraction to allow `_Index_Metadata` objects to be instantiated with indexes created from remote tasks. We use this in `merge`.

PR coming shortly to fix it.

cc @Veryku 


Discussion:

Resolving patch:
diff --git a/python/ray/dataframe/index_metadata.py b/python/ray/dataframe/index_metadata.py
--- a/python/ray/dataframe/index_metadata.py
+++ b/python/ray/dataframe/index_metadata.py
@@ -103,8 +103,6 @@ def _get_index(self):
         _IndexMetadata constructor for more details)
         """
         if isinstance(self._coord_df_cache, ray.local_scheduler.ObjectID):
-            if self._index_cache is None:
-                self._index_cache = pd.RangeIndex(len(self))
             return self._index_cache
         else:
             return self._coord_df_cache.index
@@ -128,6 +126,37 @@ def _set_index(self, new_index):
 
     index = property(_get_index, _set_index)
 
+    def _get_index_cache(self):
+        """Get the cached Index object, which may sometimes be an OID.
+
+        This will ray.get the Index object out of the Ray store lazily, such
+        that it is not grabbed until it is needed in the driver. This layer of
+        abstraction is important for allowing this object to be instantiated
+        with a remote Index object.
+
+        Returns:
+            The Index object in _index_cache.
+        """
+        if self._index_cache_validator is None:
+            self._index_cache_validator = pd.RangeIndex(len(self))
+        elif isinstance(self._index_cache_validator,
+                        ray.local_scheduler.ObjectID):
+            self._index_cache_validator = ray.get(self._index_cache_validator)
+
+        return self._index_cache_validator
+
+    def _set_index_cache(self, new_index):
+        """Sets the new index cache.
+
+        Args:
+            new_index: The Index to set the _index_cache to.
+        """
+        self._index_cache_validator = new_index
+
+    # _index_cache_validator is an extra layer of abstraction to allow the
+    # cache to accept ObjectIDs and ray.get them when needed.
+    _index_cache = property(_get_index_cache, _set_index_cache)
+
     def coords_of(self, key):
         """Returns the coordinates (partition, index_within_partition) of the
         provided key in the index. Can be called on its own or implicitly
diff --git a/python/ray/dataframe/utils.py b/python/ray/dataframe/utils.py
--- a/python/ray/dataframe/utils.py
+++ b/python/ray/dataframe/utils.py
@@ -112,7 +112,6 @@ def to_pandas(df):
     else:
         pd_df = pd.concat(ray.get(df._col_partitions),
                           axis=1)
-    print(df.columns)
     pd_df.index = df.index
     pd_df.columns = df.columns
     return pd_df


Test patch:
Repository: ray-project/ray

Issue:
[xray] Users get no warning for infeasible tasks.
Start Ray with 

```
RAY_USE_XRAY=1 ray start --head --redis-port=6379 --num-gpus=0
```

Then start `RAY_USE_XRAY=1 ipython` and run

```python
import ray

ray.init(redis_address='localhost:6379')

@ray.remote(num_gpus=1)
def f():
    return 1

f.remote()
```

`f` will never execute because it is infeasible, and yet the user will get no warning.


Discussion:
what about this : https://github.com/ray-project/ray/blob/master/src/ray/raylet/scheduling_policy.cc#L110

That's good, but it goes to a log file (assuming output is redirected, which is the default in the cluster setting). It'd be best for it to be pushed to the driver.

Resolving patch:
diff --git a/python/ray/ray_constants.py b/python/ray/ray_constants.py
--- a/python/ray/ray_constants.py
+++ b/python/ray/ray_constants.py
@@ -42,6 +42,7 @@ def env_integer(key, default):
 WORKER_DIED_PUSH_ERROR = "worker_died"
 PUT_RECONSTRUCTION_PUSH_ERROR = "put_reconstruction"
 HASH_MISMATCH_PUSH_ERROR = "object_hash_mismatch"
+INFEASIBLE_TASK_ERROR = "infeasible_task"
 
 # Abort autoscaling if more than this number of errors are encountered. This
 # is a safety feature to prevent e.g. runaway node launches.


Test patch:
diff --git a/test/failure_test.py b/test/failure_test.py
--- a/test/failure_test.py
+++ b/test/failure_test.py
@@ -480,3 +480,26 @@ def __init__(self):
 
     # Make sure that a warning is generated.
     wait_for_errors(ray_constants.PICKLING_LARGE_OBJECT_PUSH_ERROR, 2)
+
+
+@pytest.mark.skipif(
+    os.environ.get("RAY_USE_XRAY") != "1",
+    reason="This test only works with xray.")
+def test_warning_for_infeasible_tasks(ray_start_regular):
+    # Check that we get warning messages for infeasible tasks.
+
+    @ray.remote(num_gpus=1)
+    def f():
+        pass
+
+    @ray.remote(resources={"Custom": 1})
+    class Foo(object):
+        pass
+
+    # This task is infeasible.
+    f.remote()
+    wait_for_errors(ray_constants.INFEASIBLE_TASK_ERROR, 1)
+
+    # This actor placement task is infeasible.
+    Foo.remote()
+    wait_for_errors(ray_constants.INFEASIBLE_TASK_ERROR, 2)
Repository: ray-project/ray

Issue:
Ship Modin with Ray
### Describe the problem
<!-- Describe the problem clearly here. -->
I think it makes sense to ship Modin with Ray. I suggest doing this similar to how pyarrow is shipped with Ray.

We don't need to rely on the dependencies of Modin, but some of the Modin source will have to be updated to make sure that the pandas version is correct.



Discussion:

Resolving patch:
diff --git a/python/ray/__init__.py b/python/ray/__init__.py
--- a/python/ray/__init__.py
+++ b/python/ray/__init__.py
@@ -46,6 +46,9 @@
             e.args += (helpful_message, )
     raise
 
+modin_path = os.path.join(os.path.abspath(os.path.dirname(__file__)), "modin")
+sys.path.insert(0, modin_path)
+
 from ray.raylet import ObjectID, _config  # noqa: E402
 from ray.profiling import profile  # noqa: E402
 from ray.worker import (error_info, init, connect, disconnect, get, put, wait,


Test patch:
diff --git a/python/ray/test/test_modin.py b/python/ray/test/test_modin.py
new file mode 100644
--- /dev/null
+++ b/python/ray/test/test_modin.py
@@ -0,0 +1,12 @@
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import ray  # noqa F401
+
+
+def test_modin_import():
+    import modin.pandas as pd
+    frame_data = [1, 2, 3, 4, 5, 6, 7, 8]
+    frame = pd.DataFrame(frame_data)
+    assert frame.sum().squeeze() == sum(frame_data)
Repository: ray-project/ray

Issue:
[tune] run_experiment should allow upload_dir to be None
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 18.04
- **Ray installed from (source or binary)**: source
- **Ray version**: 0.5.3
- **Python version**: 3.6

### Describe the problem
`run_experiment` definition does currently not allow `None` to be passed as the `upload_dir`. This happens because the `to_argv` called [here](https://github.com/ray-project/ray/blob/4182b856118a38295e76db37c752e0a5c68352d8/python/ray/tune/config_parser.py#L182) converts `None` to `'null'`. Noting that and empty string works fine.

### Source code / logs
```python
import time
import ray
from ray import tune


def my_func(config, reporter):  # add the reporter parameter
    i = 0
    while True:
        reporter(timesteps_total=i, mean_accuracy=i ** config['alpha'])
        i += config['beta']
        time.sleep(0.01)


tune.register_trainable('my_func', my_func)
ray.init()


tune.run_experiments({
    'my_experiment': {
        'run': 'my_func',
        'trial_resources': {'cpu': 8},
        'stop': {'mean_accuracy': 100},
        'config': {
            'alpha': tune.grid_search([0.0, 0.2, 0.4, 0.6, 0.8, 1.0]),
            'beta': tune.grid_search([2]),
        },
        'local_dir': '~/ray_results',
        'upload_dir': None,
    }
})
```


Discussion:

Resolving patch:
diff --git a/python/ray/tune/experiment.py b/python/ray/tune/experiment.py
--- a/python/ray/tune/experiment.py
+++ b/python/ray/tune/experiment.py
@@ -85,7 +85,7 @@ def __init__(self,
                  repeat=1,
                  num_samples=1,
                  local_dir=None,
-                 upload_dir="",
+                 upload_dir=None,
                  checkpoint_freq=0,
                  checkpoint_at_end=False,
                  max_failures=3,
@@ -97,7 +97,7 @@ def __init__(self,
             "trial_resources": trial_resources,
             "num_samples": num_samples,
             "local_dir": local_dir or DEFAULT_RESULTS_DIR,
-            "upload_dir": upload_dir,
+            "upload_dir": upload_dir or "",  # argparse converts None to "null"
             "checkpoint_freq": checkpoint_freq,
             "checkpoint_at_end": checkpoint_at_end,
             "max_failures": max_failures,


Test patch:
diff --git a/python/ray/tune/test/trial_runner_test.py b/python/ray/tune/test/trial_runner_test.py
--- a/python/ray/tune/test/trial_runner_test.py
+++ b/python/ray/tune/test/trial_runner_test.py
@@ -185,6 +185,21 @@ def train(config, reporter):
             }
         })
 
+    def testUploadDirNone(self):
+        def train(config, reporter):
+            reporter(timesteps_total=1)
+
+        [trial] = run_experiments({
+            "foo": {
+                "run": train,
+                "upload_dir": None,
+                "config": {
+                    "a": "b"
+                },
+            }
+        })
+        self.assertFalse(trial.upload_dir)
+
     def testLogdirStartingWithTilde(self):
         local_dir = '~/ray_results/local_dir'