Repository: azavea/raster-vision

Issue:
Ignored tmpdir option, missing predict command after native installation on Ubuntu 16.04
## 🐛 Bug

After following the native installation steps as described by the Ubuntu 16.04 [Dockerfile](https://github.com/azavea/raster-vision/blob/4eaa709ea4beeddded048fcbf670306d42262d9d/Dockerfile#L8), Raster Vision exhibits a couple of problems: (i) it appears to ignore the `--tmpdir` option; (ii) the `predict` command is missing.

## To Reproduce

Steps to reproduce the behavior:

Native installation on Ubuntu 16.04 can be achieved using the following commands:
```bash
sudo apt install software-properties-common python-software-properties
sudo apt install wget=1.*
sudo apt install git=1:2.*
sudo apt install python-protobuf=2.*
sudo apt install python3-tk=3.*
sudo apt install jq=1.5*
sudo apt install build-essential
sudo apt install libsqlite3-dev=3.11.*
sudo apt install zlib1g-dev=1:1.2.*
sudo apt install libopencv-dev=2.4.*
sudo apt install python-opencv=2.4.*
sudo apt install unzip
sudo apt install curl
if [ ! -d "$HOME/Downloads" ]
then
  mkdir $HOME/Downloads
fi
cd $HOME/Downloads
wget https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh
bash Miniconda3-4.7.12.1-Linux-x86_64.sh -b
source $HOME/miniconda3/bin/activate
conda init
conda config --set auto_activate_base false
conda update -n base -c defaults -y conda
conda deactivate
sudo shutdown -r now
conda create -n "rv" python=3.6.13 -y
conda activate rv
conda install -c conda-forge gdal=3.0.4 -y
python3 -m pip install rastervision==0.13
```

Both problems can be observed by running the following commands:
```bash
(rv) $ mkdir $HOME/tmp
(rv) $ rastervision --tmpdir $HOME/tmp/ --help
```

Output:
```
YYYY-MM-DD HH:MM:SS:rastervision.pipeline.rv_config: WARNING - Root temporary directory cannot be used: /opt/data/tmp. Using root: /tmp/tmp3brelzyl
Usage: rastervision [OPTIONS] COMMAND [ARGS]...

  The main click command.

  Sets the profile, verbosity, and tmp_dir in RVConfig.

Options:
  -p, --profile TEXT  Sets the configuration profile name to use.
  -v, --verbose       Increment the verbosity level.
  --tmpdir TEXT       Root of temporary directories to use.
  --help              Show this message and exit.

Commands:
  run          Run sequence of commands within pipeline(s).
  run_command  Run an individual command within a pipeline.
```

The `--tmpdir` option has been ignored and the `predict` command is not listed.

## Expected behavior

The expected behavior is (i) no warning regarding the temporary directory, and (ii) the predict command is listed among the commands.

## Environment

 - How you installed and are running Raster Vision: pip install
 - Raster Vision version or commit: 0.13
 - OS (e.g., Linux): Ubuntu 16.04
 - Python version: 3.6.13
 - CUDA/cuDNN version if running on GPU: CPU only

## Additional context

Raster Vision looks like a very promising and exciting project! Being able to use Raster Vision natively and outside of Docker is key to our use case. Currently, we can make limited use of Raster Vision on Ubuntu 16.04 after natively executing the relevant commands from the [Dockerfile](https://github.com/azavea/raster-vision/blob/4eaa709ea4beeddded048fcbf670306d42262d9d/Dockerfile). However, the issues above seem to indicate problems with the native installation process. Crucially, the predict command is not available.



Discussion:
Thanks for the detailed report!

> (i) no warning regarding the temporary directory

The warning that you are seeing is actually being generated before your `--tmpdir` is processed. I believe it is due to the instantiation of `RVConfig()` here https://github.com/azavea/raster-vision/blob/4eaa709ea4beeddded048fcbf670306d42262d9d/rastervision_pipeline/rastervision/pipeline/__init__.py#L17
which attempts to set the temporary directory to the default `/opt/data/tmp`: https://github.com/azavea/raster-vision/blob/4eaa709ea4beeddded048fcbf670306d42262d9d/rastervision_pipeline/rastervision/pipeline/rv_config.py#L52. The `--tmpdir` directory is processed here where we use it to replace the default temporary directory https://github.com/azavea/raster-vision/blob/4eaa709ea4beeddded048fcbf670306d42262d9d/rastervision_pipeline/rastervision/pipeline/cli.py#L103.

So you will always see this warning unless you're running inside the container (where `/opt/data/` exists). The fact that you don't see a second warning suggests that your provided temporary directory works fine. I agree that this is confusing and not ideal.

---

> (ii) the predict command is listed among the commands

The absence of the `predict` command suggests that RV is not able to find the `rastervision_core` package, either because it is not installed or it is not discoverable for some reason.

Can you provide the outputs of the following commands?
1. `pip freeze | grep -i rastervision`, and
2. ```python
    from pkgutil import iter_modules
    import rastervision
    print('1', rastervision.__path__)
    print('2', [name for _, name, _ in iter_modules(rastervision.__path__, prefix='rastervision.')])
    ```
Thanks! I will ignore the tmpdir warning. For the predict command issue:

1. Output from `pip freeze | grep -i rastervision` (with the Raster Vision conda environment activated):
    ```
    rastervision==0.13
    rastervision-aws-batch==0.13
    rastervision-aws-s3==0.13
    rastervision-core==0.13
    rastervision-gdal-vsi==0.13
    rastervision-pipeline==0.13
    rastervision-pytorch-backend==0.13
    rastervision-pytorch-learner==0.13
    ````
    <details>
    <summary>Full output:</summary>
    <p>

    ```
    absl-py==0.13.0
    affine==2.3.0
    albumentations==0.5.2
    attrs==21.2.0
    awscli==1.15.85
    boto3==1.7.84
    botocore==1.10.84
    certifi==2021.5.30
    click==6.7
    click-plugins==1.1.1
    cligj==0.7.2
    colorama==0.3.9
    configobj==5.0.6
    cycler==0.10.0
    Cython==0.28.6
    dataclasses==0.8
    decorator==4.4.2
    docutils==0.17.1
    everett==0.9
    future==0.18.2
    GDAL==3.0.4
    geojson==2.5.0
    grpcio==1.39.0
    imageio==2.3.0
    imgaug==0.4.0
    importlib-metadata==4.6.4
    jmespath==0.10.0
    kiwisolver==1.3.1
    Markdown==3.3.4
    mask-to-polygons==0.0.2
    matplotlib==3.2.2
    mercantile==1.2.1
    networkx==2.5.1
    numpy==1.16.6
    opencv-python==4.1.2.30
    opencv-python-headless==4.5.3.56
    Pillow==5.0.0
    protobuf==3.17.3
    psutil==5.8.0
    pyasn1==0.4.8
    pycocotools==2.0.2
    pydantic==1.2
    pyparsing==2.4.7
    pyproj==2.6.1.post1
    pystac==0.5.2
    python-dateutil==2.8.2
    PyWavelets==1.1.1
    PyYAML==3.13
    rasterio==1.0.7
    rastervision==0.13
    rastervision-aws-batch==0.13
    rastervision-aws-s3==0.13
    rastervision-core==0.13
    rastervision-gdal-vsi==0.13
    rastervision-pipeline==0.13
    rastervision-pytorch-backend==0.13
    rastervision-pytorch-learner==0.13
    rsa==3.4.2
    s3transfer==0.1.13
    scikit-image==0.17.2
    scikit-learn==0.19.2
    scipy==1.5.4
    Shapely==1.6.4.post2
    six==1.11.0
    snuggs==1.4.7
    supermercado==0.0.5
    tensorboard==1.15.0
    tifffile==2020.9.3
    torch==1.7.1
    torchvision==0.8.2
    typing-extensions==3.7.4.1
    Werkzeug==2.0.1
    zipp==3.5.0
    ```

    </p>
    </details>
2. ```
    1 _NamespacePath(['/home/username/miniconda3/envs/rv/lib/python3.6/site-packages/rastervision'])
    2 ['rastervision.aws_batch', 'rastervision.aws_s3', 'rastervision.core', 'rastervision.gdal_vsi', 'rastervision.pipeline', 'rastervision.pipeline_example_plugin1', 'rastervision.pipeline_example_plugin2', 'rastervision.pytorch_backend', 'rastervision.pytorch_learner']
    ```

Resolving patch:
diff --git a/rastervision_pipeline/rastervision/pipeline/cli.py b/rastervision_pipeline/rastervision/pipeline/cli.py
--- a/rastervision_pipeline/rastervision/pipeline/cli.py
+++ b/rastervision_pipeline/rastervision/pipeline/cli.py
@@ -240,9 +240,11 @@ def run_command(cfg_json_uri: str, command: str, split_ind: Optional[int],
         runner=runner)
 
 
-# This has to go here and nowhere else :)
-for pc in registry.get_plugin_commands():
-    main.add_command(pc)
+def _main():
+    for pc in registry.get_plugin_commands():
+        main.add_command(pc)
+    main()
+
 
 if __name__ == '__main__':
-    main()
+    _main()
diff --git a/rastervision_pipeline/setup.py b/rastervision_pipeline/setup.py
--- a/rastervision_pipeline/setup.py
+++ b/rastervision_pipeline/setup.py
@@ -34,6 +34,6 @@
     zip_safe=False,
     entry_points='''
         [console_scripts]
-        rastervision=rastervision.pipeline.cli:main
+        rastervision=rastervision.pipeline.cli:_main
     ''',
 )


Test patch: