Repository: sql-machine-learning/elasticdl

Issue:
Benchmark of elastic AllReduce using Horovod.
Experiments of the benchmark to show ElasticDL can improve the GPU utilization and don't affect the convergence.

Experiment 1:
Two jobs are running on a cluster, and one job starts after another. The number of GPUs required by the two jobs exceeds the number of GPUs on the cluster.

Total GPUs: 6
GPUs of Job A: 4
GPUs of Job B: 4

Experiment 2:
We change the number of workers during training and check the convergence. We can use Resnet20 to train CIFAR10 dataset and check the convergence by the accuracy of the validation dataset.

Model: Resnet20
Dataset: CIFAR10
minibatch size:32
epoch: 100
learning rate scheduler:
```python
def callbacks():
    def _schedule(model_version):
        LR = float(os.getenv("LEARING_RATE", 0.001))
        STEPS_PER_EPOCH = int(os.getenv("STEPS_PER_EPOCH", 400))
        epoch = model_version / STEPS_PER_EPOCH
        if epoch > 85:
            lr = LR * 0.01
        elif epoch > 70:
            lr = LR * 0.1
        elif epoch > 55:
            lr = LR * 0.2
        elif epoch > 40:
            lr = LR * 0.5
        else:
            lr = LR
        return lr * hvd.size()
    return [LearningRateScheduler(_schedule)]
```


Discussion:

Resolving patch:
diff --git a/docs/benchmark/allreduce/scripts/acc_plot.py b/docs/benchmark/allreduce/scripts/acc_plot.py
new file mode 100644
--- /dev/null
+++ b/docs/benchmark/allreduce/scripts/acc_plot.py
@@ -0,0 +1,48 @@
+# Copyright 2020 The ElasticDL Authors. All rights reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pandas as pd
+from matplotlib import pyplot as plt
+
+if __name__ == "__main__":
+    df = pd.read_csv("../data/cifar10_resnet20_acc.csv")
+
+    f = plt.figure(figsize=(10, 8))
+
+    x = df["epoch_index"]
+    y = df["baseline_4workers_0"]
+    plt.plot(x, y, "--", label="baseline_4workers")
+
+    y = df["baseline_4workers_1"]
+    plt.plot(x, y, "--", label="baseline_4workers")
+
+    y = df["baseline_2workers_0"]
+    plt.plot(x, y, "-.", label="baseline_2workers")
+
+    y = df["baseline_2workers_1"]
+    plt.plot(x, y, "-.", label="baseline_2workers")
+
+    y = df["elastic_2_4workers_0"]
+    plt.plot(x, y, label="elastic_2_4workers")
+
+    y = df["elastic_2_4workers_1"]
+    plt.plot(x, y, label="elastic_2_4workers")
+
+    plt.title("The Accuracy of Resnet20 on cifar10 test dataset")
+    plt.xlabel(
+        xlabel="Iteration epoches", fontsize=18, fontfamily="Times New Roman"
+    )
+    plt.ylabel(ylabel="Accuracy", fontsize=18, fontfamily="Times New Roman")
+    plt.legend(loc="upper left", bbox_to_anchor=(0.02, 0.98))
+    plt.ylim((0.3, 1))
+    plt.show()
diff --git a/docs/benchmark/allreduce/scripts/gpu_uilization_plot.py b/docs/benchmark/allreduce/scripts/gpu_uilization_plot.py
new file mode 100644
--- /dev/null
+++ b/docs/benchmark/allreduce/scripts/gpu_uilization_plot.py
@@ -0,0 +1,48 @@
+# Copyright 2020 The ElasticDL Authors. All rights reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pandas as pd
+from matplotlib import pyplot as plt
+
+gang_gpu_df = pd.read_csv("../data/gang_allreduce_gpu.csv")
+elastic_gpu_df = pd.read_csv("../data/elastic_allreduce_gpu.csv")
+
+# fig = plt.figure(figsize=(15, 4))
+fig = plt.figure(figsize=(13, 6))
+ax = fig.add_subplot(211)
+ax.plot(gang_gpu_df["time"], gang_gpu_df["gpu"])
+
+plt.title(
+    "Gang scheduling -- two jobs one after another",
+    fontsize=14,
+    fontfamily="Times New Roman",
+)
+plt.xlabel(xlabel="time(sec)", fontsize=14, fontfamily="Times New Roman")
+plt.ylabel(ylabel="Utilized GPUs", fontsize=14, fontfamily="Times New Roman")
+plt.xlim(0, 1000)
+plt.ylim(0, 7)
+
+# plt.subplot(2, 1, 2)
+ax = fig.add_subplot(212)
+ax.plot(elastic_gpu_df["time"], elastic_gpu_df["gpu"])
+plt.title(
+    "Elastic scheduling -- two jobs overlap and fully use the cluster",
+    fontsize=14,
+    fontfamily="Times New Roman",
+)
+plt.xlabel(xlabel="time(sec)", fontsize=14, fontfamily="Times New Roman")
+plt.ylabel(ylabel="Utilized GPU", fontsize=14, fontfamily="Times New Roman")
+plt.xlim(0, 1000)
+plt.ylim(0, 7)
+plt.subplots_adjust(hspace=0.5)
+plt.show()


Test patch: