Repository: HumanCompatibleAI/imitation (language: python)

Issue:
Speed up Mac OS CI
The Mac OS CI executor is currently taking 4x longer than our Linux executor. This seems to be for two main reasons: 
  1. The `brew install` step alone takes 9m, to install just a handful of packages. We might be able to cache this to speed things up. I also question whether some of the packages are necessary.
  2. It's a smaller executor than the others (medium and not x-large). Unfortunately CircleCI only goes up to large for MacOS and that's not available to us without upgrading to an annual plan. I think the best option is to run multiple MacOS executors in parallel and shard the tests across them using e.g. [pytest-shard](https://pypi.org/project/pytest-shard/)


Discussion / hints:
@Rocamonde @taufeeque9 one of you should probably take this on? Juan you seemed excited/annoyed by Mac OS so feel free to take this if it scratches an itch, but otherwise probably better for @taufeeque9 to look into this as you'll be busy with the EPIC project soon.
@AdamGleave I agree that this is an issue, right now macOS checks are absurdly slow. I spent a bit of time looking into this issue as I use macOS for all my work and found this to be quite surprising. Unfortunately, I did not manage to make much progress. I'll look into it more and contribute it asynchronously if I come up with anything. However, given my other project commitments it might work out best if someone else takes the lead, unless no active contributor uses macOS, as that would make it a bit hard for them to test stuff locally (I'd be surprised, but it would also be interesting to know).

Fix patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index 09fe9d67b..4b8f1a4fb 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -74,7 +74,7 @@ commands:
     steps:
       - run:
           name: install macOS packages
-          command: brew install coreutils ffmpeg gnu-getopt
+          command: HOMEBREW_NO_AUTO_UPDATE=1 brew install coreutils parallel gnu-getopt
 
       - checkout
 
@@ -283,7 +283,7 @@ jobs:
   unit-test-macos:
     macos:
       xcode: 13.4.1
-
+    parallelism: 2
     steps:
       - dependencies-macos
       - restore-pytest-cache
@@ -292,7 +292,8 @@ jobs:
           name: run tests
           command: |
             source ~/venv/bin/activate
-            pytest -n auto --junitxml=/tmp/test-reports/junit.xml -vv tests/
+            TESTFILES=$(circleci tests glob tests/**/test*.py | circleci tests split --split-by=timings)
+            pytest -n auto --junitxml=/tmp/test-reports/junit.xml -vv $TESTFILES
 
       - save-pytest-cache
       - store-test-output-unix
@@ -332,7 +333,6 @@ jobs:
 
       - store_test_results:
           path: \tmp\test-reports
-          unit-test-windows:
 
       - store_artifacts:
           path: \tmp\resource-usage


Test patch:
diff --git a/tests/scripts/test_scripts.py b/tests/scripts/test_scripts.py
index 6876cda68..6a959b1e9 100644
--- a/tests/scripts/test_scripts.py
+++ b/tests/scripts/test_scripts.py
@@ -10,11 +10,12 @@
 import os
 import pathlib
 import pickle
+import platform
 import shutil
 import sys
 import tempfile
 from collections import Counter
-from typing import List, Mapping, Optional
+from typing import Dict, List, Mapping, Optional
 from unittest import mock
 
 import numpy as np
@@ -413,13 +414,21 @@ def test_train_rl_sac(tmpdir):
     assert isinstance(run.result, dict)
 
 
-EVAL_POLICY_CONFIGS = [
-    {"videos": True},
-    {"videos": True, "video_kwargs": {"single_video": False}},
+# check if platform is macos
+
+EVAL_POLICY_CONFIGS: List[Dict] = [
     {"reward_type": "zero", "reward_path": "foobar"},
     {"rollout_save_path": "{log_dir}/rollouts.pkl"},
 ]
 
+if platform.system() != "Darwin":
+    EVAL_POLICY_CONFIGS.extend(
+        [
+            {"videos": True},
+            {"videos": True, "video_kwargs": {"single_video": False}},
+        ],
+    )
+
 
 @pytest.mark.parametrize("config", EVAL_POLICY_CONFIGS)
 def test_eval_policy(config, tmpdir):