Repository: azavea/raster-vision

Issue:
Runner runs commands needlessly
I ran a full workflow, then deleted the predict and eval output, then ran again (without `-r ...`) and it ran predict, eval, and then... train. Using the verbose flag, I saw that the train command expected three files which all exist, so I'm surprised it wanted to run it again.


Discussion:
> I ran a full workflow

Please be more explicit.
> > I ran a full workflow
> 
> Please be more explicit.

This problem surfaced when running an experiment in a private project (I can provide details on Slack) but there's a good chance that you could replicate this with any experiment in the examples repo. I'm going to try doing that now.
I've replicated this problem using the Vegas Spacenet example. https://github.com/azavea/raster-vision-examples#spacenet-vegas running locally but with S3 storage. (I ran it with local storage and could not replicate the issue). First, I ran the full experiment (should take a minute or two).
```
root@b79cc38c07b5:/opt/src# export ROOT_URI=s3://raster-vision-lf-dev/rv-output/vegas-spacenet/
root@b79cc38c07b5:/opt/src# rastervision run local -e spacenet.vegas \
>     -a test True \
>     -a use_remote_data True \
>     -a root_uri ${ROOT_URI} \
>     -a target buildings \
>     -a task_type semantic_segmentation
```

Then I deleted the predict and eval output directories on S3. Then I did a dry run of the same commands as above to re-run the experiment, and instead of just planning to run `predict` and `eval` it also planned to run `train` again.

```
root@b79cc38c07b5:/opt/src# export ROOT_URI=s3://raster-vision-lf-dev/rv-output/vegas-spacenet/
root@b79cc38c07b5:/opt/src# rastervision run local -e spacenet.vegas \
>     -a test True \
>     -a use_remote_data True \
>     -a root_uri ${ROOT_URI} \
>     -a target buildings \
>     -a task_type semantic_segmentation
None
Ensuring input files exist    [############------------------------]   34%  0d 00:00:09^C

Aborted!
root@b79cc38c07b5:/opt/src# rastervision run local -e spacenet.vegas     -a test True     -a use_remote_data True     -a root_uri ${ROOT_URI}     -a target buildings     -a task_type semantic_segmentation -n
None
Ensuring input files exist    [####################################]  100%
Checking for existing output  [####################################]  100%

Commands skipped because output already exists:
ANALYZE from buildings_semantic_segmentation
CHIP from buildings_semantic_segmentation
BUNDLE from buildings_semantic_segmentation


Commands to be run in this order:
PREDICT from buildings_semantic_segmentation

EVAL from buildings_semantic_segmentation
  DEPENDS ON: PREDICT from buildings_semantic_segmentation

TRAIN from buildings_semantic_segmentation

Shell commands to be issued:
  python -m rastervision  run_command s3://raster-vision-lf-dev/rv-output/vegas-spacenet/predict/buildings_semantic_segmentation/command-config.json
  python -m rastervision  run_command s3://raster-vision-lf-dev/rv-output/vegas-spacenet/eval/buildings_semantic_segmentation/command-config.json
  python -m rastervision  run_command s3://raster-vision-lf-dev/rv-output/vegas-spacenet/train/buildings_semantic_segmentation/command-config.json
```
I ran the `rastervision` command with the `--verbose` flag and got this output which may be helpful.

```
TRAIN from buildings_semantic_segmentation
  INPUTS:
    s3://raster-vision-lf-dev/rv-output/vegas-spacenet/chip/buildings_semantic_segmentation/train-0.record
    s3://raster-vision-lf-dev/rv-output/vegas-spacenet/chip/buildings_semantic_segmentation/validation-0.record
    http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_train_aug_2018_01_29.tar.gz
  OUTPUTS:
    s3://raster-vision-lf-dev/rv-output/vegas-spacenet/train/buildings_semantic_segmentation/model
    buildings_semantic_segmentation
```

Resolving patch:
diff --git a/rastervision/backend/tf_deeplab_config.py b/rastervision/backend/tf_deeplab_config.py
--- a/rastervision/backend/tf_deeplab_config.py
+++ b/rastervision/backend/tf_deeplab_config.py
@@ -121,7 +121,10 @@ def update_for_command(self,
             # Set the fine tune checkpoint name to the experiment id
             if not self.fine_tune_checkpoint_name:
                 self.fine_tune_checkpoint_name = experiment_config.id
-            io_def.add_output(self.fine_tune_checkpoint_name)
+            full_checkpoint_path = '{}.tar.gz'.format(
+                os.path.join(self.training_output_uri,
+                             self.fine_tune_checkpoint_name))
+            io_def.add_output(full_checkpoint_path)
         if command_type in [rv.PREDICT, rv.BUNDLE]:
             if not self.model_uri:
                 io_def.add_missing('Missing model_uri.')


Test patch: