Repository: Project-MONAI/MONAI

Issue:
CI/CD Pipeline setup



Discussion:
Yan, I'm taking the liberty of copying your initial email here as it is the best place for discussion

Hi Ben,

This item was assigned to you and me. If you don’t mind, I can do some research on possible solutions and put together a proposal.

But we need to first define some requirements. Here are what I can think of:

When PR merge request is received:
Check and make sure the PR has been reviewed and approved.
Run code style check and email result to author
Run all unit tests and make sure nothing is broken
Run integration tests and make sure nothing is broken
Notify team if the PR is accepted
Nightly build?

Please feel free to add/change.

Depending on the solution options we have, we may need to adjust these.
Let me know what you think.
Thanks,
Yan
We have been using 'GitLab CI/CD for GitHub' as it comes with full GPU test agent support. Shall we just go with that?

Let's protect master so that it can only be updated via PR.

Github provides 'auto-squash' option on PRs so that they are always a single commit on master. This is good because it keeps master history clean. I'd go further and suggest rewriting commit message for the PR so that it summarises the feature.

Re checks, we should be able to configure CI so that testing happens automatically on PR submission (i.e. before it has been passed/merged). All unit and integration tests should be run on a CI basis, ideally.

We can have a nightly build. We may wish to have longer running training tests that execute on it. It is a bit tricky given the unstable nature of some networks, however.

Notification of the team; I guess you mean automatic emails; don't we get them from the issues, updates to the issues? If not, I guess that there is a hook of some kind.
Thanks @atbenmurray, just to confirm that now the master can only be updated via PR, squash merging is enabled for PR.

Resolving patch:
diff --git a/monai/utils/generateddata.py b/monai/utils/generateddata.py
new file mode 100644
--- /dev/null
+++ b/monai/utils/generateddata.py
@@ -0,0 +1,44 @@
+# Copyright 2020 MONAI Consortium
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import numpy as np
+
+from monai.utils.arrayutils import rescale_array
+
+
+def create_test_image_2d(width, height, num_objs=12, rad_max=30, noise_max=0.0, num_seg_classes=5):
+    """
+    Return a noisy 2D image with `numObj' circles and a 2D mask image. The maximum radius of the circles is given as
+    `radMax'. The mask will have `numSegClasses' number of classes for segmentations labeled sequentially from 1, plus a
+    background class represented as 0. If `noiseMax' is greater than 0 then noise will be added to the image taken from
+    the uniform distribution on range [0,noiseMax).
+    """
+    image = np.zeros((width, height))
+
+    for i in range(num_objs):
+        x = np.random.randint(rad_max, width - rad_max)
+        y = np.random.randint(rad_max, height - rad_max)
+        rad = np.random.randint(5, rad_max)
+        spy, spx = np.ogrid[-x : width - x, -y : height - y]
+        circle = (spx * spx + spy * spy) <= rad * rad
+
+        if num_seg_classes > 1:
+            image[circle] = np.ceil(np.random.random() * num_seg_classes)
+        else:
+            image[circle] = np.random.random() * 0.5 + 0.5
+
+    labels = np.ceil(image).astype(np.int32)
+
+    norm = np.random.uniform(0, num_seg_classes * noise_max, size=image.shape)
+    noisyimage = rescale_array(np.maximum(image, norm))
+
+    return noisyimage, labels


Test patch:
diff --git a/tests/integration_unet2d.py b/tests/integration_unet2d.py
new file mode 100644
--- /dev/null
+++ b/tests/integration_unet2d.py
@@ -0,0 +1,59 @@
+# Copyright 2020 MONAI Consortium
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+
+import numpy as np
+import torch
+from ignite.engine import create_supervised_trainer
+from torch.utils.data import DataLoader, Dataset
+
+from monai import networks, utils
+
+
+def run_test(batch_size=64, train_steps=100, device=torch.device("cuda:0")):
+
+    class _TestBatch(Dataset):
+
+        def __getitem__(self, _unused_id):
+            im, seg = utils.generateddata.create_test_image_2d(128, 128, noise_max=1, num_objs=4, num_seg_classes=1)
+            return im[None], seg[None].astype(np.float32)
+
+        def __len__(self):
+            return train_steps
+
+    net = networks.nets.UNet(
+        dimensions=2,
+        in_channels=1,
+        num_classes=1,
+        channels=(4, 8, 16, 32),
+        strides=(2, 2, 2),
+        num_res_units=2,
+    )
+
+    loss = networks.losses.DiceLoss()
+    opt = torch.optim.Adam(net.parameters(), 1e-4)
+    src = DataLoader(_TestBatch(), batch_size=batch_size)
+
+    def loss_fn(pred, grnd):
+        return loss(pred[0], grnd)
+
+    trainer = create_supervised_trainer(net, opt, loss_fn, device, False)
+
+    trainer.run(src, 1)
+
+    return trainer.state.output
+
+
+if __name__ == "__main__":
+    result = run_test()
+
+    sys.exit(0 if result < 1 else 1)
diff --git a/tests/utils.py b/tests/utils.py
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -15,7 +15,7 @@
 import numpy as np
 import torch
 
-from monai.utils.arrayutils import rescale_array
+from monai.utils.generateddata import create_test_image_2d
 
 quick_test_var = "QUICKTEST"
 
@@ -26,35 +26,6 @@ def skip_if_quick(obj):
     return unittest.skipIf(is_quick, "Skipping slow tests")(obj)
 
 
-def create_test_image(width, height, num_objs=12, rad_max=30, noise_max=0.0, num_seg_classes=5):
-    """
-    Return a noisy 2D image with `numObj' circles and a 2D mask image. The maximum radius of the circles is given as
-    `radMax'. The mask will have `numSegClasses' number of classes for segmentations labeled sequentially from 1, plus a
-    background class represented as 0. If `noiseMax' is greater than 0 then noise will be added to the image taken from
-    the uniform distribution on range [0,noiseMax).
-    """
-    image = np.zeros((width, height))
-
-    for i in range(num_objs):
-        x = np.random.randint(rad_max, width - rad_max)
-        y = np.random.randint(rad_max, height - rad_max)
-        rad = np.random.randint(5, rad_max)
-        spy, spx = np.ogrid[-x : width - x, -y : height - y]
-        circle = (spx * spx + spy * spy) <= rad * rad
-
-        if num_seg_classes > 1:
-            image[circle] = np.ceil(np.random.random() * num_seg_classes)
-        else:
-            image[circle] = np.random.random() * 0.5 + 0.5
-
-    labels = np.ceil(image).astype(np.int32)
-
-    norm = np.random.uniform(0, num_seg_classes * noise_max, size=image.shape)
-    noisyimage = rescale_array(np.maximum(image, norm))
-
-    return noisyimage, labels
-
-
 class NumpyImageTestCase2D(unittest.TestCase):
     im_shape = (128, 128)
     input_channels = 1
@@ -62,7 +33,7 @@ class NumpyImageTestCase2D(unittest.TestCase):
     num_classes = 3
 
     def setUp(self):
-        im, msk = create_test_image(self.im_shape[0], self.im_shape[1], 4, 20, 0, self.num_classes)
+        im, msk = create_test_image_2d(self.im_shape[0], self.im_shape[1], 4, 20, 0, self.num_classes)
 
         self.imt = im[None, None]
         self.seg1 = (msk[None, None] > 0).astype(np.float32)