Repository: sql-machine-learning/elasticdl

Issue:
Master pod should not be created when docker image failed to push
Docker image failed to push:
```
===== Pushing Docker Image =====
{'status': 'The push refers to repository [reg.docker.alibaba-inc.com/yuan.tang/elasticdl]'}
{'status': 'Preparing', 'progressDetail': {}, 'id': 'a29e21fbecd7'}
...
{'status': 'Waiting', 'progressDetail': {}, 'id': '2ee61facf36a'}
{'status': 'Waiting', 'progressDetail': {}, 'id': '05ea944dce26'}
{'errorDetail': {'message': 'denied: requested access to the resource is denied'}, 'error': 'denied: requested access to the resource is denied'}
```
But the master pod is created. We should not start the master pod in this case since it cannot find the image anyways. Otherwise users would have to delete the pod themselves before submitting a new elasticdl job.
```
Events:
  Type     Reason             Age               From                  Message
  ----     ------             ----              ----                  -------
  Normal   Scheduled          1m                default-scheduler     Successfully assigned elasticdl-test-mnist-terry-master to ckas389w00k
  Normal   Pulling            1m                kubelet, ckas389w00k  pulling image "reg.docker.alibaba-inc.com/yuan.tang/elasticdl:01baa62fcc9d4d8e89a5fc8c735fcb0f"
  Warning  Failed             1m                kubelet, ckas389w00k  Failed to pull image "reg.docker.alibaba-inc.com/yuan.tang/elasticdl:01baa62fcc9d4d8e89a5fc8c735fcb0f": [rpc error: code = Unknown desc = Error response from daemon: manifest unknown: manifest unknown, rpc error: code = Unknown desc = Error response from daemon: manifest unknown: manifest unknown]
  Warning  Failed             1m                kubelet, ckas389w00k  Error: ErrImagePull
  Warning  MissingClusterDNS  1m (x12 over 1m)  kubelet, ckas389w00k  pod: "elasticdl-test-mnist-terry-master_kubemaker(c14c88d4-a4c0-11e9-bf35-02420ba6e969)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
  Normal   BackOff            1m (x5 over 1m)   kubelet, ckas389w00k  Back-off pulling image "reg.docker.alibaba-inc.com/yuan.tang/elasticdl:01baa62fcc9d4d8e89a5fc8c735fcb0f"
  Warning  Failed             1m (x5 over 1m)   kubelet, ckas389w00k  Error: ImagePullBackOff
```


Discussion:

Resolving patch:
diff --git a/elasticdl/python/elasticdl/image_builder.py b/elasticdl/python/elasticdl/image_builder.py
--- a/elasticdl/python/elasticdl/image_builder.py
+++ b/elasticdl/python/elasticdl/image_builder.py
@@ -143,6 +143,15 @@ def _generate_unique_image_name(prefix):
     )
 
 
+def _print_docker_progress(line):
+    error = line.get("error", None)
+    if error:
+        raise RuntimeError("Docker image build: " + error)
+    text = line.get("stream", None)
+    if text:
+        print(text)
+
+
 def _build_docker_image(client, ctx_dir, dockerfile, image_name):
     print("===== Building Docker Image =====")
     for line in client.build(
@@ -152,15 +161,10 @@ def _build_docker_image(client, ctx_dir, dockerfile, image_name):
         tag=image_name,
         decode=True,
     ):
-        error = line.get("error", None)
-        if error:
-            raise RuntimeError("Docker image build: " + error)
-        text = line.get("stream", None)
-        if text:
-            print(text)
+        _print_docker_progress(line)
 
 
 def _push_docker_image(client, image_name):
     print("===== Pushing Docker Image =====")
     for line in client.push(image_name, stream=True, decode=True):
-        print(line)
+        _print_docker_progress(line)


Test patch: