Repository: Lightning-Universe/lightning-transformers (language: python)

Issue:
Create scripts directory
## 🚀 Feature

Move the train.py script:

`train.py -> lightning_transformers/cli/train.py`

This way we can import it and test it easily. And users can subclass it.

We can also add it as an entrypoint in setup.py:

```
setuptools.setup(
    entry_points={
        "console_scripts": [
            "pl-transformers-train=lightning_transformers.cli.train:hydra_entry",
        ],
    }
)
```

so users can do:

```console
$ pl-transformers-train +task=huggingface/text_classification +dataset=text_classification/emotion 
```


Discussion / hints:
Hi! thanks for your contribution!, great first issue!
Not sure how I feel about this, personally a main script within the root of the repo is clear for people to use imo. We can discuss next about the idea, still a good proposal!
I agree it is nice and common to be able to see it from the root

However, that's more for research projects and other showcase repositories. This is a library so we want to provide these scripts as part of it.

It is not about being clearer but about gaining test-ability and letting users subclass our scripts to add functionality without duplicating it.
Makes sense :) It's important not to lose sight that we can approach both providing extensibility in research projects and easy to use scripts for people who just want to train models on their own data/inference off pre-trained models with all the PL features that we have.

Your proposal seems similar to what Fairseq has decided on and I'm happy with it on balance. Fairseq provide a CLI backed by Hydra: https://github.com/pytorch/fairseq/blob/master/fairseq_cli/hydra_train.py

and the typical expose: https://github.com/pytorch/fairseq/blob/master/setup.py#L225



Fix patch:
diff --git a/README.md b/README.md
index 884bbe50..e4f285a2 100644
--- a/README.md
+++ b/README.md
@@ -29,14 +29,14 @@ pip install lightning-transformers["extra"]
 
 Train bert-base-cased on CARER emotion dataset using text classification task.
 ```bash
-python train.py \
+python pl-transformers-train \
     +task=nlp/huggingface/text_classification \
     +dataset=nlp/text_classification/emotion
 ```
 
 Train roberta-base backbone, on SWAG dataset multiple choice task.
 ```bash
-python train.py \
+python pl-transformers-train \
     +task=nlp/huggingface/multiple_choice \
     +dataset=nlp/multiple_choice/swag \
     backbone.pretrained_model_name_or_path=roberta-base
@@ -44,7 +44,7 @@ python train.py \
 
 Inference with pre-trained bert-base-cased on SQuAD using question-answering task with 2 GPUs.
 ```bash
-python train.py \
+python pl-transformers-train \
     +task=nlp/huggingface/question_answering \
     +dataset=nlp/question_answering/squad \
     trainer.gpus=2 \
@@ -53,7 +53,7 @@ python train.py \
 
 Enable Sharded Training.
 ```bash
-python train.py \
+python pl-transformers-train \
     +task=nlp/huggingface/text_classification \
     +dataset=nlp/text_classification/emotion \
     trainer=sharded
@@ -61,7 +61,7 @@ python train.py \
 
 Enable DeepSpeed ZeRO-Offload Training.
 ```bash
-python train.py \
+python pl-transformers-train \
     +task=nlp/huggingface/text_classification \
     +dataset=nlp/text_classification/emotion \
     trainer=zero_offload
diff --git a/cli/__init__.py b/cli/__init__.py
deleted file mode 100644
index bc138c18..00000000
--- a/cli/__init__.py
+++ /dev/null
@@ -1,1 +0,0 @@
-from cli.train import main, run  # noqa: F401
diff --git a/conf/__init__.py b/conf/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/lightning_transformers/cli/__init__.py b/lightning_transformers/cli/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/cli/train.py b/lightning_transformers/cli/train.py
similarity index 86%
rename from cli/train.py
rename to lightning_transformers/cli/train.py
index 59fd7a27..a6c7079a 100644
--- a/cli/train.py
+++ b/lightning_transformers/cli/train.py
@@ -1,5 +1,6 @@
 from typing import Any, Optional
 
+import hydra
 from omegaconf import DictConfig, OmegaConf
 from pytorch_lightning.utilities.distributed import rank_zero_info
 
@@ -19,7 +20,7 @@ def run(
     trainer: TrainerConfig = TrainerConfig(),
     tokenizer: Optional[TokenizerConfig] = None,
     logger: Optional[Any] = None,
-):
+) -> None:
     if ignore_warnings:
         set_ignore_warnings()
 
@@ -38,7 +39,7 @@ def run(
     trainer.test(model, datamodule=data_module)
 
 
-def main(cfg: DictConfig):
+def main(cfg: DictConfig) -> None:
     rank_zero_info(OmegaConf.to_yaml(cfg))
     instantiator = HydraInstantiator()
     logger = instantiator.logger(cfg)
@@ -47,8 +48,17 @@ def main(cfg: DictConfig):
         ignore_warnings=cfg.ignore_warnings,
         do_train=cfg.training.do_train,
         dataset=cfg.dataset,
-        tokenizer=cfg.tokenizer if "tokenizer" in cfg else None,
+        tokenizer=cfg.get("tokenizer", None),
         task=cfg.task,
         trainer=cfg.trainer,
         logger=logger,
     )
+
+
+@hydra.main(config_path="../../conf", config_name="config")
+def hydra_entry(cfg: DictConfig) -> None:
+    main(cfg)
+
+
+if __name__ == "__main__":
+    hydra_entry()
diff --git a/requirements.txt b/requirements.txt
index 225ad182..67b296fc 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -10,6 +10,7 @@ datasets
 
 # hydra
 hydra-core>=1.1.0.dev2 # todo: swap to 1.1 version once out
+omegaconf==2.1.0.dev18 # issues with `throw_on_missing_key`
 
 # task-specific
 rouge-score
diff --git a/setup.py b/setup.py
index e5c43def..39a5a68a 100755
--- a/setup.py
+++ b/setup.py
@@ -49,6 +49,11 @@ def load_long_description():
     python_requires=">=3.6",
     setup_requires=[],
     install_requires=load_requirements(PATH_ROOT),
+    entry_points={
+        "console_scripts": [
+            "pl-transformers-train=lightning_transformers.cli.train:hydra_entry",
+        ],
+    },
     project_urls={
         "Bug Tracker": "https://github.com/PyTorchLightning/lightning-transformers/issues",
         # "Documentation": "TODO",
diff --git a/train.py b/train.py
deleted file mode 100644
index 0f306cc3..00000000
--- a/train.py
+++ /dev/null
@@ -1,13 +0,0 @@
-import hydra
-from omegaconf import DictConfig
-
-from cli import main
-
-
-@hydra.main(config_path="conf", config_name="config")
-def hydra_main(cfg: DictConfig):
-    main(cfg)
-
-
-if __name__ == "__main__":
-    hydra_main()


Test patch:
diff --git a/.github/workflows/ci_testing.yml b/.github/workflows/ci_testing.yml
index 39efc6ab..481164e7 100644
--- a/.github/workflows/ci_testing.yml
+++ b/.github/workflows/ci_testing.yml
@@ -58,6 +58,7 @@ jobs:
       run: |
         pip install --requirement requirements.txt --upgrade --quiet --find-links https://download.pytorch.org/whl/cpu/torch_stable.html
         pip install --requirement requirements/test.txt --quiet
+        pip install .
         python --version
         pip --version
         pip list
diff --git a/tests/cli/test_train.py b/tests/cli/test_train.py
new file mode 100644
index 00000000..015ef5ca
--- /dev/null
+++ b/tests/cli/test_train.py
@@ -0,0 +1,34 @@
+import shutil
+import subprocess
+from unittest import mock
+from unittest.mock import ANY, MagicMock
+
+from omegaconf import OmegaConf
+
+import lightning_transformers.cli.train as cli
+
+
+def test_train_run():
+    trainer_mock = MagicMock()
+    instantiator = MagicMock()
+    instantiator.trainer.return_value = trainer_mock
+
+    cli.run(instantiator)
+
+    trainer_mock.fit.assert_called()
+    trainer_mock.test.assert_called()
+
+
+@mock.patch("lightning_transformers.cli.train.run")
+def test_train_main(run_mock):
+    cfg = OmegaConf.create({"training": {"do_train": True}})
+    cli.main(cfg)
+    run_mock.assert_called_with(
+        ANY, ignore_warnings=ANY, do_train=True, dataset=ANY, task=ANY, trainer=ANY, logger=None, tokenizer=ANY
+    )
+
+
+def test_train_entry_point():
+    proc = subprocess.run([shutil.which("pl-transformers-train"), "--help"], stdout=subprocess.PIPE)
+    help = proc.stdout.decode().strip()
+    assert help.startswith("train is powered by Hydra")
diff --git a/tests/conftest.py b/tests/conftest.py
index f5d7337f..23cb85de 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -5,7 +5,7 @@
 from hydra.experimental import compose, initialize
 from hydra.test_utils.test_utils import find_parent_dir_containing
 
-from cli import main
+from lightning_transformers.cli.train import main as train_main
 
 # GitHub Actions use this path to cache datasets.
 # Use `datadir` fixture where possible and use `DATASETS_PATH` in
@@ -41,7 +41,7 @@ def run(task: str, dataset: str, suffix: str = ""):
         relative_conf_dir = find_hydra_conf_dir()
         with initialize(config_path=relative_conf_dir, job_name="test_app"):
             cfg = compose(config_name="config", overrides=cmd_line.split(" "))
-            main(cfg)
+            train_main(cfg)
 
     return run