Repository: ray-project/ray

Issue:
[tune] sync_on_checkpoint ignored when list of experiments passed to tune.run
### What is the problem?
When passing a list of predefined Experiments to tune via the `run` function, the `sync_on_checkpoint` flag is ignored.

**Reason for this behaviour:**
The `make_parser` function takes in `no_sync_on_checkpoint`; it's set to False by default.
https://github.com/ray-project/ray/blob/981f67bfb01725f1e153c881f81082fcba0ac223/python/ray/tune/config_parser.py#L75-L77

The create_trial_from_spec function gets a `sync_on_checkpoint` from the Experiment spec, but that key is never used by the parser. So the following line makes the `sync_on_checkpoint` of the returned `Trial` object `True` by default, thus ignoring the value of `sync_on_checkpoint` in the incoming Experiment spec.
https://github.com/ray-project/ray/blob/981f67bfb01725f1e153c881f81082fcba0ac223/python/ray/tune/config_parser.py#L185

*Ray Version: All versions starting from 0.7.7 when the feature was released.*

### Reproduction (REQUIRED)
```python3
import ray
from ray import tune
from ray.tune import Experiment

def func(config):
    pass

sync_true_obj = Experiment("test_sync", func, sync_on_checkpoint=True)
sync_false_obj = Experiment("test_sync", func, sync_on_checkpoint=False)

ls = [sync_true_obj, sync_false_obj]

ray.init(local_mode=True)
analysis = tune.run(ls)
```
To check the error, add the following lines just before the `return Trial(` line in `config_parser.py`
```python3
print(args)
print(args.no_sync_on_checkpoint)
```
https://github.com/ray-project/ray/blob/981f67bfb01725f1e153c881f81082fcba0ac223/python/ray/tune/config_parser.py#L173


- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [x] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:

Resolving patch:
diff --git a/python/ray/tune/config_parser.py b/python/ray/tune/config_parser.py
--- a/python/ray/tune/config_parser.py
+++ b/python/ray/tune/config_parser.py
@@ -73,13 +73,13 @@ def make_parser(parser_creator=None, **kwargs):
         help="Whether to checkpoint at the end of the experiment. "
         "Default is False.")
     parser.add_argument(
-        "--no-sync-on-checkpoint",
+        "--sync-on-checkpoint",
         action="store_true",
-        help="Disable sync-down of trial checkpoint, which is enabled by "
-        "default to guarantee recoverability. If set, checkpoint syncing from "
-        "worker to driver is asynchronous. Set this only if synchronous "
-        "checkpointing is too slow and trial restoration failures can be "
-        "tolerated")
+        help="Enable sync-down of trial checkpoint to guarantee "
+        "recoverability. If unset, checkpoint syncing from worker "
+        "to driver is asynchronous, so unset this only if synchronous "
+        "checkpointing is too slow and trial restoration failures "
+        "can be tolerated.")
     parser.add_argument(
         "--keep-checkpoints-num",
         default=None,
@@ -182,7 +182,7 @@ def create_trial_from_spec(spec, output_path, parser, **trial_kwargs):
         remote_checkpoint_dir=spec.get("remote_checkpoint_dir"),
         checkpoint_freq=args.checkpoint_freq,
         checkpoint_at_end=args.checkpoint_at_end,
-        sync_on_checkpoint=not args.no_sync_on_checkpoint,
+        sync_on_checkpoint=args.sync_on_checkpoint,
         keep_checkpoints_num=args.keep_checkpoints_num,
         checkpoint_score_attr=args.checkpoint_score_attr,
         export_formats=spec.get("export_formats", []),


Test patch: