Repository: sql-machine-learning/elasticdl

Issue:
Only one task is created for each epoch for ODPS data source
There are more than 100 observations in the ODPS table and `--minibatch_size=10`, `--records_per_task=10`, and `--num_epochs=5` are used. However, only one task is created for each epoch. This only happens for ODPS data source though. RecordIO data source is working correctly as usual.

Here are the logs from master pod for a job with :
```
[2019-09-25 14:20:31,653] [INFO] [task_dispatcher.py:80:create_tasks] Creating a new set of training tasks for model version -1
[2019-09-25 14:20:31,654] [INFO] [task_dispatcher.py:118:get] Starting epoch 1
[2019-09-25 14:20:32,201] [INFO] [task_dispatcher.py:151:report] Task:1 completed, 1 remaining tasks
[2019-09-25 14:20:32,563] [INFO] [task_dispatcher.py:80:create_tasks] Creating a new set of training tasks for model version -1
[2019-09-25 14:20:32,563] [INFO] [task_dispatcher.py:118:get] Starting epoch 2
[2019-09-25 14:20:32,575] [INFO] [task_dispatcher.py:151:report] Task:2 completed, 1 remaining tasks
[2019-09-25 14:20:33,320] [INFO] [task_dispatcher.py:80:create_tasks] Creating a new set of training tasks for model version -1
[2019-09-25 14:20:33,320] [INFO] [task_dispatcher.py:118:get] Starting epoch 3
[2019-09-25 14:20:33,325] [INFO] [task_dispatcher.py:151:report] Task:3 completed, 1 remaining tasks
[2019-09-25 14:20:34,163] [INFO] [task_dispatcher.py:80:create_tasks] Creating a new set of training tasks for model version -1
[2019-09-25 14:20:34,163] [INFO] [task_dispatcher.py:118:get] Starting epoch 4
[2019-09-25 14:20:34,169] [INFO] [task_dispatcher.py:151:report] Task:4 completed, 1 remaining tasks
[2019-09-25 14:20:34,910] [INFO] [task_dispatcher.py:151:report] Task:5 completed, 0 remaining tasks
[2019-09-25 14:20:35,762] [INFO] [k8s_worker_manager.py:120:_event_cb] Got event MODIFIED, phase Succeeded for pod: elasticdl-test-iris-custom-model-odps-worker-0
[2019-09-25 14:20:35,784] [INFO] [k8s_worker_manager.py:120:_event_cb] Got event MODIFIED, phase Succeeded for pod: elasticdl-test-iris-custom-model-odps-worker-0
[2019-09-25 14:20:35,951] [INFO] [k8s_worker_manager.py:120:_event_cb] Got event MODIFIED, phase Succeeded for pod: elasticdl-test-iris-custom-model-odps-worker-0
[2019-09-25 14:20:35,983] [INFO] [k8s_worker_manager.py:120:_event_cb] Got event MODIFIED, phase Succeeded for pod: elasticdl-test-iris-custom-model-odps-worker-0
[2019-09-25 14:20:46,186] [INFO] [k8s_worker_manager.py:120:_event_cb] Got event MODIFIED, phase Running for pod: elasticdl-test-iris-custom-model-odps-master
[2019-09-25 14:20:46,188] [INFO] [main.py:294:main] Stopping RPC server
[2019-09-25 14:20:46,188] [INFO] [main.py:311:main] Master stopped
```


Discussion:

Resolving patch:
diff --git a/elasticdl/python/master/task_dispatcher.py b/elasticdl/python/master/task_dispatcher.py
--- a/elasticdl/python/master/task_dispatcher.py
+++ b/elasticdl/python/master/task_dispatcher.py
@@ -87,12 +87,24 @@ def create_tasks(self, task_type, model_version=-1):
             shards = self._prediction_shards
         tasks = []
         for shard_name, (start_ind, num_records) in shards.items():
-            for start in range(start_ind, num_records, self._records_per_task):
+            for start in range(
+                start_ind, start_ind + num_records, self._records_per_task
+            ):
+                end_ind = min(start + self._records_per_task, num_records)
+                # If the start index is not smaller than end index,
+                # we need to take the start index into account
+                if start >= end_ind:
+                    end_ind = min(
+                        start + self._records_per_task, start + num_records
+                    )
+                # Note that only records in [start, end) of this task
+                # will be consumed later in the worker that handles
+                # this task.
                 tasks.append(
                     _Task(
                         shard_name=shard_name,
                         start=start,
-                        end=min(start + self._records_per_task, num_records),
+                        end=end_ind,
                         type=task_type,
                         model_version=model_version,
                     )


Test patch:
diff --git a/elasticdl/python/tests/task_dispatcher_test.py b/elasticdl/python/tests/task_dispatcher_test.py
--- a/elasticdl/python/tests/task_dispatcher_test.py
+++ b/elasticdl/python/tests/task_dispatcher_test.py
@@ -5,7 +5,7 @@
 
 
 class TaskQueueTest(unittest.TestCase):
-    def test_create_get(self):
+    def test_create_get_zero_start_ind(self):
         task_d = _TaskDispatcher({"f1": (0, 10), "f2": (0, 10)}, {}, {}, 3, 1)
 
         all_tasks = [
@@ -55,6 +55,29 @@ def test_create_get(self):
 
         self.assertTrue(task_d.finished())
 
+    def test_create_get_none_zero_start_ind(self):
+        task_d = _TaskDispatcher({"f1": (0, 10), "f2": (10, 10)}, {}, {}, 3, 1)
+
+        all_tasks = [
+            ("f1", 0, 3, elasticdl_pb2.TRAINING, -1),
+            ("f1", 3, 6, elasticdl_pb2.TRAINING, -1),
+            ("f1", 6, 9, elasticdl_pb2.TRAINING, -1),
+            ("f1", 9, 10, elasticdl_pb2.TRAINING, -1),
+            ("f2", 10, 13, elasticdl_pb2.TRAINING, -1),
+            ("f2", 13, 16, elasticdl_pb2.TRAINING, -1),
+            ("f2", 16, 19, elasticdl_pb2.TRAINING, -1),
+            ("f2", 19, 22, elasticdl_pb2.TRAINING, -1),
+        ]
+
+        # get all tasks out, each worker is assigned 2 tasks.
+        got_tasks = [task_d.get(i // 2) for i in range(8)]
+
+        # verify ids ranges from 1 to 8
+        self.assertEqual(list(range(1, 9)), [k for k, _ in got_tasks])
+
+        # verify tasks
+        self.assertEqual(sorted([v._info() for _, v in got_tasks]), all_tasks)
+
     def test_epoch(self):
         task_d = _TaskDispatcher({"f1": (0, 10), "f2": (0, 10)}, {}, {}, 3, 2)