internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ray-project/ray

Issue:
[tune] Missing keep_checkpoints_num arg in tune
Hi everybody,

Thanks a lot for this lib, I've been using for the past few weeks with TF2 and it works pretty well!

In my current use case, I'm checkpointing regularly to ensure a minimal loss in the face of OOM. I do control TF checkpoints manually to avoid filling my disk space but I still need to dump some ray metadata (iterations count, etc.).
I've noticed that ray create folders like `checkpoint_{iter}` to store those but from `tune` there is no way to tell him that I don't need to keep dozens of folder.
I also noticed that `experiment`, and `trial` classes have the perfect argument: `keep_checkpoints_num`.

Sadly this argument is missing from the `tune` config. 
Is there any reasons why it is missing from `tune` arguments?

Have a nice day.


Discussion:
No particular reason, we just missed it - would you be willing to make a PR?

Resolving patch:
diff --git a/python/ray/tune/tune.py b/python/ray/tune/tune.py
--- a/python/ray/tune/tune.py
+++ b/python/ray/tune/tune.py
@@ -49,6 +49,8 @@ def run(run_or_experiment,
         sync_to_driver=None,
         checkpoint_freq=0,
         checkpoint_at_end=False,
+        keep_checkpoints_num=None,
+        checkpoint_score_attr=None,
         global_checkpoint_period=10,
         export_formats=None,
         max_failures=3,
@@ -114,6 +116,13 @@ def run(run_or_experiment,
             checkpoints. A value of 0 (default) disables checkpointing.
         checkpoint_at_end (bool): Whether to checkpoint at the end of the
             experiment regardless of the checkpoint_freq. Default is False.
+        keep_checkpoints_num (int): Number of checkpoints to keep. A value of
+            `None` keeps all checkpoints. Defaults to `None`. If set, need
+            to provide `checkpoint_score_attr`.
+        checkpoint_score_attr (str): Specifies by which attribute to rank the
+            best checkpoint. Default is increasing order. If attribute starts
+            with `min-` it will rank attribute in decreasing order, i.e.
+            `min-validation_loss`.
         global_checkpoint_period (int): Seconds between global checkpointing.
             This does not affect `checkpoint_freq`, which specifies frequency
             for individual trials.
@@ -199,6 +208,8 @@ def run(run_or_experiment,
             loggers=loggers,
             checkpoint_freq=checkpoint_freq,
             checkpoint_at_end=checkpoint_at_end,
+            keep_checkpoints_num=keep_checkpoints_num,
+            checkpoint_score_attr=checkpoint_score_attr,
             export_formats=export_formats,
             max_failures=max_failures,
             restore=restore,


Test patch:
Repository: ray-project/ray

Issue:
[Tune] The logdir string of Trial is always truncated
For now, the logdir string of a trial is created by `Trial.create_logdir`: https://github.com/ray-project/ray/blob/6f737e6a500dc9f500d4cf7ba7b31f979922a18b/python/ray/tune/trial.py#L373-L389

The `identifier` is always be truncated to a length of `MAX_LEN_IDENTIFIER`. This should be configurable since the max length of file names could be 256 in some systems.
@richardliaw 


Discussion:
Can you make a PR? Maybe just set it to an environment variable?
Yeah, I'll try.

Resolving patch:
diff --git a/python/ray/tune/trial.py b/python/ray/tune/trial.py
--- a/python/ray/tune/trial.py
+++ b/python/ray/tune/trial.py
@@ -30,7 +30,7 @@
 from ray.utils import binary_to_hex, hex_to_binary
 
 DEBUG_PRINT_INTERVAL = 5
-MAX_LEN_IDENTIFIER = 130
+MAX_LEN_IDENTIFIER = int(os.environ.get("MAX_LEN_IDENTIFIER", 130))
 logger = logging.getLogger(__name__)
 
 


Test patch:
Repository: ray-project/ray

Issue:
[Tune] Experiment Analysis get_best behaviour
Hi,
In their current version, the [`get_best_config`](https://github.com/ray-project/ray/blob/ed89897a311fbe63afdd5fa05a4ef8b7576ca6a4/python/ray/tune/analysis/experiment_analysis.py#L56) and [`get_best_logdir`](https://github.com/ray-project/ray/blob/ed89897a311fbe63afdd5fa05a4ef8b7576ca6a4/python/ray/tune/analysis/experiment_analysis.py#L70) methods of the `Analysis` object may consider different Trials as the best one:
- `get_best_config` will first retrieve the best row of each trial dataframe and then select the best trial from these rows.
- `get_best_logdir` will first retrieve the last row of each trial and then selects the best one.

Is it the expected behaviour ? 
If it isn't, I think that the correct way of doing it is the first one. This could be done by simply passing the `metric` and `max` arguments to the [`self.dataframe`](https://github.com/ray-project/ray/blob/ed89897a311fbe63afdd5fa05a4ef8b7576ca6a4/python/ray/tune/analysis/experiment_analysis.py#L78) call in `get_best_dir`.




Discussion:
Hi @TomVeniat, that's a good point, and I think your proposal is right. Would you be willing to push a fix?

Resolving patch:
diff --git a/python/ray/tune/analysis/experiment_analysis.py b/python/ray/tune/analysis/experiment_analysis.py
--- a/python/ray/tune/analysis/experiment_analysis.py
+++ b/python/ray/tune/analysis/experiment_analysis.py
@@ -75,7 +75,7 @@ def get_best_logdir(self, metric, mode="max"):
             mode (str): One of [min, max].
 
         """
-        df = self.dataframe()
+        df = self.dataframe(metric=metric, mode=mode)
         if mode == "max":
             return df.iloc[df[metric].idxmax()].logdir
         elif mode == "min":


Test patch:
diff --git a/python/ray/tune/tests/test_experiment_analysis.py b/python/ray/tune/tests/test_experiment_analysis.py
--- a/python/ray/tune/tests/test_experiment_analysis.py
+++ b/python/ray/tune/tests/test_experiment_analysis.py
@@ -141,6 +141,13 @@ def testBestLogdir(self):
         self.assertTrue(logdir2.startswith(self.test_dir))
         self.assertNotEquals(logdir, logdir2)
 
+    def testBestConfigIsLogdir(self):
+        analysis = Analysis(self.test_dir)
+        for metric, mode in [(self.metric, "min"), (self.metric, "max")]:
+            logdir = analysis.get_best_logdir(metric, mode=mode)
+            best_config = analysis.get_best_config(metric, mode=mode)
+            self.assertEquals(analysis.get_all_configs()[logdir], best_config)
+
 
 if __name__ == "__main__":
     unittest.main(verbosity=2)
Repository: ray-project/ray

Issue:
hang on ray.get when redis_password has a space
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 18.04.3 LTS
- **Ray installed from (source or binary)**: pip install ray
- **Ray version**: 0.7.3
- **Python version**: 3.6.8
- **Exact command to reproduce**: See below

### Describe the problem
Calling ray.init with redis_password containing no spaces works fine. Using a redis_password with a space causes the code to hang between the two print statements.

### Source code / logs
```
import ray 

ray.init(redis_password="pw 1")

@ray.remote
def increment(num):
    return num + 1 

x = increment.remote(2)
print(x)
print(ray.get(x))
```


Discussion:

Resolving patch:
diff --git a/python/ray/services.py b/python/ray/services.py
--- a/python/ray/services.py
+++ b/python/ray/services.py
@@ -772,6 +772,8 @@ def _start_redis_instance(executable,
         # Construct the command to start the Redis server.
         command = [executable]
         if password:
+            if " " in password:
+                raise ValueError("Spaces not permitted in redis password.")
             command += ["--requirepass", password]
         command += (
             ["--port", str(port), "--loglevel", "warning"] + load_module_args)


Test patch:
Repository: ray-project/ray

Issue:
Killing `ray start --block` should stop all child processes
It would be useful for deployment if `ray start --block` (or some other equivalent command) would start up all the Ray subprocesses in such a way that killing the initial command would stop all child processes. This makes it easier for Ray to be deployed using existing deployment tools which assume killing the started process fully cleans things up.

See https://github.com/ray-project/ray/issues/2214#issuecomment-411659135.


Discussion:
See also #2587.
On Linux, the best way to assure this is by setting `pdeathsig` on the parent process. Then if the parent process dies, Linux kills also all children. In Python you can do this by adding to `subprocess` `Popen` call arguments:

```
preexec_fn=lambda: prctl.set_pdeathsig(signal.SIGKILL)
```

You [need this package](https://pythonhosted.org/python-prctl/).

The issue is that this is not graceful, but that is OK. You should gracefully try to kill stuff from the parent process while it is running, but if it dies, then the rest dies with it.
I would say that it might be better to just reuse some existing process supervisor here instead of reimplementing all this and figuring our all the details on how to manage subprocesses on various systems.
Isn't this also related to #2005?
One related comment about this. `ray start --block` should probably exit if the child processes have died. It could just check periodically to see if the raylet that it started is still up and running and if not then exit.

Resolving patch:
diff --git a/python/ray/node.py b/python/ray/node.py
--- a/python/ray/node.py
+++ b/python/ray/node.py
@@ -192,10 +192,10 @@ def atexit_handler():
 
         atexit.register(atexit_handler)
 
-        # Register the a handler to be called if we get a SIGTERM.
+        # Register the handler to be called if we get a SIGTERM.
         # In this case, we want to exit with an error code (1) after
         # cleaning up child processes.
-        def sigterm_handler():
+        def sigterm_handler(signum, frame):
             return clean_up_children(lambda *args, **kwargs: sys.exit(1))
 
         signal.signal(signal.SIGTERM, sigterm_handler)


Test patch:
Repository: ray-project/ray

Issue:
Kubernetes Docker Container untagged
The container used by the kubernetes manifests (rayproject/autoscaler) doesn't have any useful tags. Furthermore the contents of the container are actually built using github.com/edoakes/ray and the container is two months old.

It might be better to properly tag the containers with versions and ensure they are built from the main repo (by travis) so that there aren't small differences in the code and it stays up to date.





Discussion:

Resolving patch:
diff --git a/python/setup.py b/python/setup.py
--- a/python/setup.py
+++ b/python/setup.py
@@ -3,6 +3,7 @@
 from __future__ import print_function
 
 import glob
+from itertools import chain
 import os
 import re
 import shutil
@@ -82,6 +83,8 @@
     "tune": ["tabulate"],
 }
 
+extras["all"] = list(set(chain.from_iterable(extras.values())))
+
 
 class build_ext(_build_ext.build_ext):
     def run(self):


Test patch:
Repository: ray-project/ray

Issue:
Inconsistency in Diagonal Gaussian between log probability and entropy
In the DiagonalGaussian action distribution the logp is calculated like so:

`        return (-0.5 * tf.reduce_sum(
            tf.square((x - self.mean) / self.std), reduction_indices=[1]) -
                0.5 * np.log(2.0 * np.pi) * tf.to_float(tf.shape(x)[1]) -
                tf.reduce_sum(self.log_std, reduction_indices=[1]))`

Here the term  tf.reduce_sum(self.log_std, reduction_indices=[1])) is supposed to compute the part of the logprobability due to the determinant of the covariance matrix. The diagonals of the covariance matrix are the squared standard deviations, so the absence of the 0.5 in front of the term makes sense. But, then, in that case the entropy

`tf.reduce_sum(
            .5 * self.log_std + .5 * np.log(2.0 * np.pi * np.e),
            reduction_indices=[1])`

<img width="310" alt="image" src="https://user-images.githubusercontent.com/7660397/70380361-ed418a80-18ee-11ea-917f-0f266b48ada5.png">

should also not have a 0.5 in front of the log-std term and should be
`tf.reduce_sum(
            self.log_std + .5 * np.log(2.0 * np.pi * np.e),
            reduction_indices=[1])`


Discussion:
I see, should we be reverting https://github.com/ray-project/ray/pull/2968 then?
cc @pcmoritz 
cc @michaelzhiluo, as he was the one who mentioned this change I think?
I agree that `0.5` should be removed because of using `log_std` not the `log_variance`.
I also agree that `0.5` should not be in front of `self.log_std` in entropy since there is a 2x term factoring out of the variance terms in the covariance matrix.
@eugenevinitsky do you mind pushing a fix?
Yeah, one sec!

Resolving patch:
diff --git a/rllib/models/tf/tf_action_dist.py b/rllib/models/tf/tf_action_dist.py
--- a/rllib/models/tf/tf_action_dist.py
+++ b/rllib/models/tf/tf_action_dist.py
@@ -166,7 +166,7 @@ def kl(self, other):
     @override(ActionDistribution)
     def entropy(self):
         return tf.reduce_sum(
-            .5 * self.log_std + .5 * np.log(2.0 * np.pi * np.e),
+            self.log_std + .5 * np.log(2.0 * np.pi * np.e),
             reduction_indices=[1])
 
     @override(TFActionDistribution)


Test patch:
Repository: ray-project/ray

Issue:
Fix simple typo: verion -> version
# Issue Type

[x] Bug (Typo)

# Steps to Replicate

1. Examine python/ray/scripts/scripts.py.
2. Search for `verion`.

# Expected Behaviour

1. Should read `version`.




Discussion:

Resolving patch:
diff --git a/python/ray/scripts/scripts.py b/python/ray/scripts/scripts.py
--- a/python/ray/scripts/scripts.py
+++ b/python/ray/scripts/scripts.py
@@ -381,7 +381,7 @@ def start(node_ip_address, redis_address, address, redis_port,
         redis_client = services.create_redis_client(
             redis_address, password=redis_password)
 
-        # Check that the verion information on this node matches the version
+        # Check that the version information on this node matches the version
         # information that the cluster was started with.
         services.check_version_info(redis_client)
 


Test patch:
Repository: ray-project/ray

Issue:
[projects] ray session stop does not kill pods
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

Using ray 0.9.0dev0

Ray projects does not delete pods associated with the project when running `ray session stop`

### Reproduction (REQUIRED)

When launching the cluster with `ray session start <command>` the name of the cluster supplied in the `cluster.yml` is overwritten with the name of the project. However the same replacment is not made when running `ray session stop`. 

A temporary workaround is possible by running `ray session stop --name project_name`


refer to the stop code here
https://github.com/ray-project/ray/blob/master/python/ray/projects/scripts.py#L327

refer to the start code here
https://github.com/ray-project/ray/blob/master/python/ray/projects/scripts.py#L352

### Potential fix

Alter the stop function to the following

```
def stop(name):

   if not name:
        name = project_definition.config["name"]

    project_definition = load_project_or_throw()
    teardown_cluster(
        project_definition.cluster_yaml(),
        yes=True,
        workers_only=False,
        override_cluster_name=name)
```


I think however it would be neater if the config loader did the replacement i.e here https://github.com/ray-project/ray/blob/master/python/ray/projects/projects.py#L49


Discussion:
Thanks for attention, I will reproduce and fix it if it's a bug.
@Nintorac @Qstar Thanks for pointing this out and offering to fix it, let me know if you need help! The proposed fix looks great to me :)

Resolving patch:
diff --git a/python/ray/projects/scripts.py b/python/ray/projects/scripts.py
--- a/python/ray/projects/scripts.py
+++ b/python/ray/projects/scripts.py
@@ -326,6 +326,10 @@ def attach(screen, tmux):
 @click.option("--name", help="Name of the session to stop", default=None)
 def stop(name):
     project_definition = load_project_or_throw()
+
+    if not name:
+        name = project_definition.config["name"]
+
     teardown_cluster(
         project_definition.cluster_yaml(),
         yes=True,


Test patch:
Repository: ray-project/ray

Issue:
Turn off OMP_NUM_THREADS warnings? 
Can we just turn off the warnings on each ray.init? We can't force everyone to set the environment variable. 

https://github.com/ray-project/ray/blob/3f99be8dad5e0e1abfaede1f25753a0af74f1648/python/ray/__init__.py#L16-L21


Discussion:
I promoted this to release blockers because many users may find this warning annoying. (cc @pcmoritz @robertnishihara )
How about we change the log level to INFO?

Resolving patch:
diff --git a/python/ray/__init__.py b/python/ray/__init__.py
--- a/python/ray/__init__.py
+++ b/python/ray/__init__.py
@@ -14,9 +14,9 @@
                       "packaged along with Ray).")
 
 if "OMP_NUM_THREADS" not in os.environ:
-    logger.warning("[ray] Forcing OMP_NUM_THREADS=1 to avoid performance "
-                   "degradation with many workers (issue #6998). You can "
-                   "override this by explicitly setting OMP_NUM_THREADS.")
+    logger.debug("[ray] Forcing OMP_NUM_THREADS=1 to avoid performance "
+                 "degradation with many workers (issue #6998). You can "
+                 "override this by explicitly setting OMP_NUM_THREADS.")
     os.environ["OMP_NUM_THREADS"] = "1"
 
 # Add the directory containing pickle5 to the Python path so that we find the


Test patch:
Repository: ray-project/ray

Issue:
[tune] GPU utilization getting checked for cluster with no gpu
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

Trivial issue, which does not actually impact performance as far as I can tell, but it seems if you launch a cluster with a configuration that sets gpus = 0, then there might be a check which avoids this error message...


```
(pid=4628) Exception in thread Thread-2: 
(pid=4628) Traceback (most recent call last):
(pid=4628)   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
(pid=4628)     self.run()
(pid=4628)   File "/home/ubuntu/algo/lib/python3.6/site-packages/ray/tune/utils/util.py", line 89, in run
(pid=4628)     self._read_utilization()
(pid=4628)   File "/home/ubuntu/algo/lib/python3.6/site-packages/ray/tune/utils/util.py", line 65, in _read_utilization
(pid=4628)     for gpu in GPUtil.getGPUs():
(pid=4628)   File "/home/ubuntu/algo/lib/python3.6/site-packages/GPUtil/GPUtil.py", line 102, in getGPUs
(pid=4628)     deviceIds = int(vals[i])
(pid=4628) ValueError: invalid literal for int() with base 10: "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make  sure that the latest NVIDIA driver is installed and running."
```



*Ray version and other system information (Python version, TensorFlow version, OS):*
Ray 0.8.1, TF 2.1, RHEL 7.7

### Reproduction (REQUIRED)
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):

If we cannot run your script, we cannot fix your issue.

- [ ] I have verified my script runs in a clean environment and reproduces the issue.
- [ ] I have verified the issue also occurs with the [latest wheels](https://ray.readthedocs.io/en/latest/installation.html).



Discussion:

Resolving patch:
diff --git a/python/ray/tune/utils/util.py b/python/ray/tune/utils/util.py
--- a/python/ray/tune/utils/util.py
+++ b/python/ray/tune/utils/util.py
@@ -58,7 +58,12 @@ def _read_utilization(self):
                 self.values["ram_util_percent"].append(
                     float(getattr(psutil.virtual_memory(), "percent")))
             if GPUtil is not None:
-                for gpu in GPUtil.getGPUs():
+                gpu_list = []
+                try:
+                    gpu_list = GPUtil.getGPUs()
+                except Exception:
+                    logger.debug("GPUtil failed to retrieve GPUs.")
+                for gpu in gpu_list:
                     self.values["gpu_util_percent" + str(gpu.id)].append(
                         float(gpu.load))
                     self.values["vram_util_percent" + str(gpu.id)].append(


Test patch:
Repository: ray-project/ray

Issue:
[Python] jsonschema included twice in setup.py requires list.
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

`jsonschema` is included twice in the Python package [setup.py `requires` list](https://github.com/ray-project/ray/blob/master/python/setup.py#L176-L183).  This is causing the usage of the Ray Python library within Bazel to fail during the analysis phase due to label duplication in the generated `py_library` target's `'deps'`:

```
ERROR: .../external/requirements_py3_pypi__ray_0_9_0_dev0/BUILD:6:1: Label '@requirements_py3_pypi__jsonschema_3_2_0//:pkg' is duplicated in the 'deps' attribute of rule 'pkg'
```

This bug was introduced in the [cluster json schema validator PR](https://github.com/ray-project/ray/pull/7261/files#diff-8cf6167d58ce775a08acafcfe6f40966).

*Ray version and other system information (Python version, TensorFlow version, OS):*

Ray master commit 90b553ed058a546e036374cd0919e00604892514 (most recent commit as of this issue filing)


### Reproduction (REQUIRED)



- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [x] I have verified the issue also occurs with the [latest wheels](https://ray.readthedocs.io/en/latest/installation.html).



Discussion:

Resolving patch:
diff --git a/python/setup.py b/python/setup.py
--- a/python/setup.py
+++ b/python/setup.py
@@ -180,7 +180,6 @@ def find_version(*filepath):
     "packaging",
     "pytest",
     "pyyaml",
-    "jsonschema",
     "redis>=3.3.2",
     # NOTE: Don't upgrade the version of six! Doing so causes installation
     # problems. See https://github.com/ray-project/ray/issues/4169.


Test patch:
Repository: ray-project/ray

Issue:
AttributeError: 'Worker' object has no attribute 'lock'
the following code
```python
import ray

# ray.init()

class Foo(object):
    def __init__(self, value):
        self.value = value

def custom_serializer(obj):
    return obj.value

def custom_deserializer(value):
    object = Foo(value)
    return object

ray.register_custom_serializer(
    Foo, serializer=custom_serializer, deserializer=custom_deserializer)

object_id = ray.put(Foo(100))
```
trigger an exception when calling `ray.register_custom_serializer`,
```
Traceback (most recent call last):
  File "ray_err.py", line 20, in <module>
    Foo, serializer=custom_serializer, deserializer=custom_deserializer)
  File "/home/cloudhan/ray/python/ray/worker.py", line 1405, in register_custom_serializer
    context = global_worker.get_serialization_context()
  File "/home/cloudhan/ray/python/ray/worker.py", line 198, in get_serialization_context
    with self.lock:
AttributeError: 'Worker' object has no attribute 'lock'
```
inspecting into the code you will find the lock is missing,
https://github.com/ray-project/ray/blob/ca6eabc9cb728a829d5305a4dd19216ed925f2e4/python/ray/worker.py#L182-L202

the reason is simply not initialize ray, but the error is confusing.


Discussion:

Resolving patch:
diff --git a/python/ray/worker.py b/python/ray/worker.py
--- a/python/ray/worker.py
+++ b/python/ray/worker.py
@@ -1391,6 +1391,9 @@ def register_custom_serializer(cls,
         use_dict: Deprecated.
         class_id (str): Unique ID of the class. Autogenerated if None.
     """
+    worker = global_worker
+    worker.check_connected()
+
     if use_pickle:
         raise DeprecationWarning(
             "`use_pickle` is no longer a valid parameter and will be removed "


Test patch: