Repository: EleutherAI/gpt-neox

Issue:
The `hostfile` NeoX Argument Is No Longer Working
**Describe the bug**

The `hostile` NeoX configuration argument is no longer taking effect. Regardless of the value set, the hostile path is always `/job/hostfile`.

![image](https://user-images.githubusercontent.com/17308542/188505112-10484da7-1e62-45f0-929e-158d59161bde.png)

The hostile updates if you use the `--hostfile` parameter. 

![image](https://user-images.githubusercontent.com/17308542/188505334-20a53f9a-fb2d-4ee5-a396-b1a101e67703.png)


**To Reproduce**
Steps to reproduce the behavior:
1. On the latest main build, run `pytest tests -m cpu` and observe that the `test_neoxargs_consume_deepy_args_without_yml_suffix` unit test is failing.
2. (Alt) Try to set a custom value for the `hostfile` config and observe that the program nonetheless uses the `/job/hostfile` path.

**Expected behavior**

When the `hostfile` argument is set, the value used in the program (even if it is a dummy path) matches the config value.

![image](https://user-images.githubusercontent.com/17308542/188504842-7a3ca361-d4bb-4ee2-a4fc-ef29278e4089.png)

The `test_neoxargs_consume_deepy_args_without_yml_suffix` unit test passes.

![image](https://user-images.githubusercontent.com/17308542/188504459-23a98cd2-3a5d-47dc-a36a-e34a147b4e8e.png)

**Proposed solution**

I suspect that this bug was introduced in 5e0d614744983cc87cb33eb127c60f13562399bc . When I comment out the CLI parameter, the test passes. 

![image](https://user-images.githubusercontent.com/17308542/188506125-5bd2c005-17d7-4e76-8564-c0fff5b01917.png)

The default value for this parameter is likely overwriting any values read from the config. 

**Environment (please complete the following information):**
 - GPUs: 8x NVIDIA A40
- Configs: Default
- Environment: Ubuntu in EleutherAI Cluster 



Discussion:
Removing the default value argument in the CLI argument parses logic seems to fix the overwriting issue.

![image](https://user-images.githubusercontent.com/17308542/188507511-318fdb98-aa06-40ea-8d7e-85cdae0ec68f.png)

That default value logic also seems redundant. The default value for the `hostfile` argument is already covered in the `deepspeed` package.

![image](https://user-images.githubusercontent.com/17308542/188507756-9a03b69f-8f93-419c-878b-239cff75c408.png)

![image](https://user-images.githubusercontent.com/17308542/188507620-424eac24-8c82-4a6f-bc87-cb4ec2691fe1.png)


Resolving patch:
diff --git a/megatron/neox_arguments/arguments.py b/megatron/neox_arguments/arguments.py
--- a/megatron/neox_arguments/arguments.py
+++ b/megatron/neox_arguments/arguments.py
@@ -288,7 +288,6 @@ def consume_deepy_args(cls):
             "-H",
             "--hostfile",
             type=str,
-            default=DLTS_HOSTFILE,
             help="Hostfile path (in MPI style) that defines the "
                  "resource pool available to the job (e.g., "
                  "worker-0 slots=4)"


Test patch:
diff --git a/tests/neox_args/test_neoxargs_commandline.py b/tests/neox_args/test_neoxargs_commandline.py
--- a/tests/neox_args/test_neoxargs_commandline.py
+++ b/tests/neox_args/test_neoxargs_commandline.py
@@ -63,6 +63,34 @@ def test_neoxargs_consume_deepy_args_without_yml_suffix():
 
     assert args_loaded_yamls == args_loaded_consume
 
+@pytest.mark.cpu
+def test_neoxargs_consume_deepy_args_with_hostfile_param():
+    """
+    Verify consume_deepy_args processes command line arguments without yaml suffix.
+    Also test the hostfile CLI arg
+    """
+
+    from megatron.neox_arguments import NeoXArgs
+
+    # load neox args with command line
+    with patch(
+        "sys.argv",
+        [str(get_root_directory() / "deepy.py"), "train.py"]
+        + get_configs_with_path(["small", "local_setup"])
+        + ["--hostfile=/mock_path"]
+    ):
+        args_loaded_consume = NeoXArgs.consume_deepy_args()
+
+    # load neox args directly from yaml files
+    args_loaded_yamls = NeoXArgs.from_ymls(
+        get_configs_with_path(["small.yml", "local_setup.yml"])
+    )
+
+    # update values from yaml files that cannot otherwise be matched
+    args_loaded_yamls.update_value("user_script", "train.py")
+    args_loaded_yamls.wandb_group = args_loaded_consume.wandb_group
+
+    assert args_loaded_yamls == args_loaded_consume
 
 @pytest.mark.cpu
 def test_neoxargs_consume_deepy_args_with_config_dir():