Repository: AlexsLemonade/refinebio

Issue:
Cache Docker Images by Branch
### Context

We want to be able to cache docker image layers that are created locally as testing artfacts locally to be used by github actions.
The current prepare_images.sh does this but there was an issue with the definition for branch_name.
We also don't want to remove support non-ccdl members developing locally.


![image](https://github.com/AlexsLemonade/refinebio/assets/1075609/e984ea3b-658e-41f4-ba61-7d652841e89e)


### Solution or next step

- After #3285 is merged, we should set sensible defaults that can be overridden for external contributors.
- Get current branch name or tag to be set when pushing images to ccdl(staging) repo.

Determine:
- If they don't have access to the docker repo should we just build locally and not push?
- How long can docker tags be / are they compatible with our longer branch names.



Discussion:

Resolving patch:
diff --git a/common/setup.py b/common/setup.py
--- a/common/setup.py
+++ b/common/setup.py
@@ -1,4 +1,6 @@
 import os
+import re
+from datetime import datetime
 
 from setuptools import find_packages, setup
 
@@ -11,11 +13,21 @@
         version_string = version_file.read().strip().split("-")[0]
 except OSError:
     print(
-        "Cannot read version to determine System Version."
-        " Please create a file common/version containing an up to date System Version."
+        "Cannot read version file to determine system version. "
+        "Please create a file common/version containing an up to date system version."
     )
     raise
 
+version_re = re.compile(
+    r"^([1-9][0-9]*!)?(0|[1-9][0-9]*)"
+    "(\.(0|[1-9][0-9]*))*((a|b|rc)(0|[1-9][0-9]*))"
+    "?(\.post(0|[1-9][0-9]*))?(\.dev(0|[1-9][0-9]*))?$"
+)
+if not version_re.match(version_string):
+    # Generate version based on the datetime.now(): e.g., 2023.5.17.dev1684352560.
+    now = datetime.now()
+    version_string = f"{now.strftime('%Y.%-m.%-d.dev')}{int(datetime.timestamp(now))}"
+
 setup(
     name="data-refinery-common",
     version=version_string,


Test patch:
diff --git a/common/dockerfiles/Dockerfile.common_tests b/common/dockerfiles/Dockerfile.common_tests
--- a/common/dockerfiles/Dockerfile.common_tests
+++ b/common/dockerfiles/Dockerfile.common_tests
@@ -1,5 +1,6 @@
 ARG DOCKERHUB_REPO
-FROM $DOCKERHUB_REPO/dr_base:latest
+ARG SYSTEM_VERSION
+FROM $DOCKERHUB_REPO/dr_base:$SYSTEM_VERSION
 
 # Fail in case of an error at any stage in the pipe.
 SHELL ["/bin/bash", "-o", "pipefail", "-c"]
@@ -16,7 +17,6 @@ RUN pip3 install --ignore-installed --no-cache-dir -r requirements.txt
 COPY common/ .
 COPY config config
 
-ARG SYSTEM_VERSION
 ENV SYSTEM_VERSION=$SYSTEM_VERSION
 
 USER user
diff --git a/workers/run_tests.sh b/workers/run_tests.sh
--- a/workers/run_tests.sh
+++ b/workers/run_tests.sh
@@ -527,20 +527,20 @@ if [ -z "$tag" ] || [ "$tag" = "compendia" ]; then
     fi
 fi
 
-. scripts/common.sh
+. ./scripts/common.sh
+
 DB_HOST_IP=$(get_docker_db_ip_address)
 
 # Ensure permissions are set for everything within the test data directory.
 chmod -R a+rwX "$volume_directory"
 
-worker_images="salmon transcriptome no_op downloaders smasher illumina agilent affymetrix qn affymetrix_local janitor compendia"
+worker_images="salmon transcriptome no_op downloaders smasher illumina agilent affymetrix qn janitor compendia"
+
 for image in $worker_images; do
     if [ -z "$tag" ] || [ "$tag" = "$image" ]; then
         if [ "$image" = "agilent" ] || [ "$image" = "affymetrix" ]; then
             # Agilent uses the same docker image as Affymetrix.
-            ./scripts/prepare_image.sh -d -i affymetrix -s workers
-            ./scripts/prepare_image.sh -i affymetrix_local -r ccdlstaging
-            docker tag "$DOCKERHUB_REPO/dr_affymetrix_local:latest" "$DOCKERHUB_REPO/dr_affymetrix:latest"
+            ./scripts/prepare_image.sh -i affymetrix -s workers
             image_name="$DOCKERHUB_REPO/dr_affymetrix"
         elif [ "$tag" = "qn" ]; then
             ./scripts/prepare_image.sh -i smasher -s workers