Repository: ibi-group/datatools-server
Issue: During OTP ELB deployment, a condition exists where no running EC2 servers remain, but the job succeeds

## Observed behavior

If during deployment to ELB, a different graph build config is used than the final servers to run, there is a case where the graph build server succeeds (and shut downs as per its instructions), but the graph loading server(s) fail but the deploy job still succeeds.

## Expected behavior

If the number of running servers at the end of deployment is zero, the job should fail (and the previous servers should not be terminated).

## Steps to reproduce the problem

1. Set up an ELB server to deploy to with different instance sizes for graph build (i.e., normal size for graph build, micro for the other).
2. Kick off deploy job to server with expected graph size that is too large for graph loading server.
3. Job completes successfully with zero servers running.

## Any special notes on configuration used

N/A

## Version of datatools-server and datatools-ui if applicable (exact commit hash or branch name)

latest dev

Changed files:
src/main/java/com/conveyal/datatools/manager/jobs/DeployJob.java

diff --git a/src/main/java/com/conveyal/datatools/manager/jobs/DeployJob.java b/src/main/java/com/conveyal/datatools/manager/jobs/DeployJob.java
index cc59b1de..4f87a150 100644
--- a/src/main/java/com/conveyal/datatools/manager/jobs/DeployJob.java
+++ b/src/main/java/com/conveyal/datatools/manager/jobs/DeployJob.java
@@ -496,8 +496,8 @@ public class DeployJob extends MonitorableJob {
             // Track any previous instances running for the server we're deploying to in order to de-register and
             // terminate them later.
             List<EC2InstanceSummary> previousInstances = otpServer.retrieveEC2InstanceSummaries();
-            // Track new instances added.
-            List<Instance> instances = new ArrayList<>();
+            // Track new instances that should be added to target group once the deploy job is completed.
+            List<Instance> newInstancesForTargetGroup = new ArrayList<>();
             // First start graph-building instance and wait for graph to successfully build.
             if (!deployType.equals(DeployType.USE_PREBUILT_GRAPH)) {
                 status.message = "Starting up graph building EC2 instance";
@@ -567,7 +567,7 @@ public class DeployJob extends MonitorableJob {
                     status.numServersRemaining = Math.max(otpServer.ec2Info.instanceCount, 1);
                 } else {
                     // same configuration exists, so keep instance on and add to list of running instances
-                    instances.addAll(graphBuildingInstances);
+                    newInstancesForTargetGroup.addAll(graphBuildingInstances);
                     status.numServersRemaining = otpServer.ec2Info.instanceCount <= 0
                         ? 0
                         : otpServer.ec2Info.instanceCount - 1;
@@ -608,7 +608,13 @@ public class DeployJob extends MonitorableJob {
                 }
             }
             // Add all servers that did not encounter issues to list for registration with ELB.
-            instances.addAll(remainingInstances);
+            newInstancesForTargetGroup.addAll(remainingInstances);
+            // Fail deploy job if no instances are running at this point (i.e., graph builder instance has shut down
+            // and the graph loading instance(s) failed to load graph successfully).
+            if (newInstancesForTargetGroup.size() == 0) {
+                status.fail("Job failed because no running instances remain.");
+                return;
+            }
             String finalMessage = "Server setup is complete!";
             // Get EC2 servers running that are associated with this server.
             if (deployType.equals(DeployType.REPLACE)) {