Repository: 0xSpaceShard/starknet-devnet (language: python)

Issue:
Alternative testing parallelization on CircleCI using Large Docker resource class
Currently we have four parallel runs where each has to do the whole setup ([example](https://app.circleci.com/pipelines/github/Shard-Labs/starknet-devnet/1972/workflows/2c4cdd12-eb91-4058-ad00-f13389c39714/jobs/4273)). Each of these runs has to:
1. Install dev tools (5 min)
2. Compile contracts (1 min)
3. Lint + check imports (1 min)
4. Run its designated set of tests (22 min)

The first three steps are the same for all runs (7 min).

Currently we are using a Medium Docker (2 CPU, 4 GB RAM, 10 credits/min).

I don't know how CircleCI parallelizes this currently, i.e. if it uses just 1 CPU per each of the four parallel runs.

If we switched to a Large Docker (4 CPU, 8 GB RAM, 20 credits/min), we could simply rely on our `scripts/test.sh` (`poetry run pytest -n auto --dist loadscope -v`) which parallelizes the tests using all available CPUs), we could end up with:
- potentially cheaper runs
- potentially faster runs
- simpler config.yml


One thing that the current parallelization does well is the splitting of tests into batches (based on previous timing). I don't know if we could preserve this time-based splitting with the approach in `scripts/test.sh`.

The best way to find out how this works is to run this on a side branch with modified config.yml

CircleCI specifications and pricing can be found [here](https://circleci.com/product/features/resource-classes/).


Discussion / hints:
Also consider the newly introduced M1 executor on CircleCI

Fix patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index a8075f770..75394245f 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -20,15 +20,9 @@ jobs:
   test_py_max: &test_template
     docker:
       - image: cimg/python:3.9-node
-    resource_class: medium
-    parallelism: 4
+    resource_class: large
     steps:
       - checkout
-      - run:
-          # if this was done later (e.g. just before the test) and the file wasn't present, it would fail after all the installation
-          name: Check out previous test metadata
-          command: |
-            jq . "${CIRCLE_INTERNAL_TASK_DATA}/circle-test-results/results.json"
       - run:
           name: Install development tools
           command: ./scripts/install_dev_tools.sh
@@ -46,21 +40,7 @@ jobs:
           command: ./scripts/check_import_sort.sh
       - run:
           name: Run pytest tests
-          parallel: true
-          command: |
-            mkdir -p test-results
-            shopt -s globstar
-            SPLIT_BY_CLASS=$(
-              circleci tests glob test/**/test*.py |
-              sed  "s/\.py//g; s/\//./g" |
-              circleci tests split --split-by=timings --timings-type=classname
-            )
-            TEST_FILES=$(sed "s/\./\//g; s/ /.py /g; s/$/.py/g;" \<<< $SPLIT_BY_CLASS)
-            poetry run pytest -s -v --junitxml=test-results/junit.xml $TEST_FILES
-      - store_test_results:
-          path: test-results
-      - store_artifacts:
-          path: test-results
+          command: ./scripts/test.sh
 
   # Since cairo-lang 0.11.0, supported py_min and py_max are the same.
   # Uncomment and modify these lines if py_min and py_max stop being the same.


Test patch:
diff --git a/scripts/test.sh b/scripts/test.sh
index ae81e9882..7a5631791 100755
--- a/scripts/test.sh
+++ b/scripts/test.sh
@@ -3,6 +3,8 @@ set -eu
 
 ./scripts/check_starknet_artifacts_version.sh
 
-CMD="poetry run pytest -n auto --dist loadscope -v ${1:-test/}"
+# Using dist=loadfile because currently some tests might have certain collisions
+# namely dump tests which do a clean up and potentially remove dumps created by other tests
+CMD="poetry run pytest --numprocesses=auto --maxprocesses=8 --dist=loadfile -vv ${1:-test/}"
 echo $CMD
 $CMD