Repository: ietf-tools/datatracker

Issue:
Provide `rsync`'ed assets via a container image
### Description

I recently started `rsync`ing the extra assets via `docker/scripts/app-rsync-extras.sh`, which places a lot of files into `data/developers`. That caused `parcel` to run extremely slowly inside the container, due to https://github.com/parcel-bundler/parcel/issues/8128.

Would it be possible to provide those extra assets via a CI-built container image that we hook into the `docker-compose` setup? That way, they wouldn't need to be under `/workspace` and hence would not slow `parcel` down.

An alternative would be to go somehow re-enable running `yarn` and `parcel` on the host rather than in the container, which is what my original Dockerfile did, but I think @NGPixel found some downsides with that?

### Code of Conduct

- [X] I agree to follow the [IETF's Code of Conduct](https://github.com/ietf-tools/.github/blob/main/CODE_OF_CONDUCT.md)


Discussion:
most developers wouldn't have parcel/yarn on their host.
We already have an issue for building better resources for "all the things" or "enough of the things". A mount that can be attached to a pre-built container makes sense, but we should offer lighterweight containers than "everything" for people on less capable development boxes or behind slow networks.
We should also address having the full, or a lighterweight, development database image.
I'm not sure I understand the extra container image for assets? The assets can be stored outside the /workspace directory. That location can be mounted from the host as well. (I don't think Github would appreciate us generating multi-gigabytes images)

You can run yarn / parcel on the host (it was never disabled). As long as you have node + yarn installed on your machine. Then run `yarn rebuild`, followed by `yarn dev`.
They can be, but by default they are not. We should maybe change the default.

Thanks for the tip about yarn/parcel!
The database image is already multiple gigabytes.

The raw data for all these artifacts is completely dominated by the id-archive which is 12GB of mostly sparse text. (Don't images take advantage of compression?)

If this really turned into an issue for github, we could look at storing images elsewhere.

The goal is to help bootstrap developers. If we do that by changing the mountpoints and having them run rsync on their own, I'm ok with that. But I still think having the ability for them to get 'reasonably enough' vs 'everything' is worth pursuing, both for this _and_ the developer database.

GitHub doesn't have storage limits on public repos and our current image size is still very acceptable IMO. My issue was with having 20GB images generated nightly.

I'm going to add a volume at `/assets` and change the settings / scripts to point to that location.
Where is the threshold for the nightly generation image size concern? We're building the db image nightly...

Resolving patch:
diff --git a/docker/configs/settings_local.py b/docker/configs/settings_local.py
--- a/docker/configs/settings_local.py
+++ b/docker/configs/settings_local.py
@@ -31,25 +31,25 @@
 INTERNET_ALL_DRAFTS_ARCHIVE_DIR = "test/archive/"
 RFC_PATH = "test/rfc/"
 
-AGENDA_PATH = 'data/developers/www6s/proceedings/'
+AGENDA_PATH = '/assets/www6s/proceedings/'
 MEETINGHOST_LOGO_PATH = AGENDA_PATH
 
 USING_DEBUG_EMAIL_SERVER=True
 EMAIL_HOST='localhost'
 EMAIL_PORT=2025
 
-MEDIA_BASE_DIR = 'data/developers'
+MEDIA_BASE_DIR = '/assets'
 MEDIA_ROOT = MEDIA_BASE_DIR + '/media/'
 MEDIA_URL = '/media/'
 
 PHOTOS_DIRNAME = 'photo'
 PHOTOS_DIR = MEDIA_ROOT + PHOTOS_DIRNAME
 
-SUBMIT_YANG_CATALOG_MODEL_DIR = 'data/developers/ietf-ftp/yang/catalogmod/'
-SUBMIT_YANG_DRAFT_MODEL_DIR = 'data/developers/ietf-ftp/yang/draftmod/'
-SUBMIT_YANG_INVAL_MODEL_DIR = 'data/developers/ietf-ftp/yang/invalmod/'
-SUBMIT_YANG_IANA_MODEL_DIR = 'data/developers/ietf-ftp/yang/ianamod/'
-SUBMIT_YANG_RFC_MODEL_DIR   = 'data/developers/ietf-ftp/yang/rfcmod/'
+SUBMIT_YANG_CATALOG_MODEL_DIR = '/assets/ietf-ftp/yang/catalogmod/'
+SUBMIT_YANG_DRAFT_MODEL_DIR = '/assets/ietf-ftp/yang/draftmod/'
+SUBMIT_YANG_INVAL_MODEL_DIR = '/assets/ietf-ftp/yang/invalmod/'
+SUBMIT_YANG_IANA_MODEL_DIR = '/assets/ietf-ftp/yang/ianamod/'
+SUBMIT_YANG_RFC_MODEL_DIR   = '/assets/ietf-ftp/yang/rfcmod/'
 
 # Set INTERNAL_IPS for use within Docker. See https://knasmueller.net/fix-djangos-debug-toolbar-not-showing-inside-docker
 import socket
@@ -60,15 +60,15 @@
 #    'ietf.context_processors.sql_debug',
 # ]
 
-DOCUMENT_PATH_PATTERN = 'data/developers/ietf-ftp/{doc.type_id}/'
-INTERNET_DRAFT_PATH = 'data/developers/ietf-ftp/internet-drafts/'
-RFC_PATH = 'data/developers/ietf-ftp/rfc/'
-CHARTER_PATH = 'data/developers/ietf-ftp/charter/'
-BOFREQ_PATH = 'data/developers/ietf-ftp/bofreq/'
-CONFLICT_REVIEW_PATH = 'data/developers/ietf-ftp/conflict-reviews/'
-STATUS_CHANGE_PATH = 'data/developers/ietf-ftp/status-changes/'
-INTERNET_DRAFT_ARCHIVE_DIR = 'data/developers/ietf-ftp/internet-drafts/'
-INTERNET_ALL_DRAFTS_ARCHIVE_DIR = 'data/developers/ietf-ftp/internet-drafts/'
+DOCUMENT_PATH_PATTERN = '/assets/ietf-ftp/{doc.type_id}/'
+INTERNET_DRAFT_PATH = '/assets/ietf-ftp/internet-drafts/'
+RFC_PATH = '/assets/ietf-ftp/rfc/'
+CHARTER_PATH = '/assets/ietf-ftp/charter/'
+BOFREQ_PATH = '/assets/ietf-ftp/bofreq/'
+CONFLICT_REVIEW_PATH = '/assets/ietf-ftp/conflict-reviews/'
+STATUS_CHANGE_PATH = '/assets/ietf-ftp/status-changes/'
+INTERNET_DRAFT_ARCHIVE_DIR = '/assets/ietf-ftp/internet-drafts/'
+INTERNET_ALL_DRAFTS_ARCHIVE_DIR = '/assets/ietf-ftp/internet-drafts/'
 
 NOMCOM_PUBLIC_KEYS_DIR = 'data/nomcom_keys/public_keys/'
 SLIDE_STAGING_PATH = 'test/staging/'
diff --git a/docker/configs/settings_local_sqlitetest.py b/docker/configs/settings_local_sqlitetest.py
--- a/docker/configs/settings_local_sqlitetest.py
+++ b/docker/configs/settings_local_sqlitetest.py
@@ -56,7 +56,7 @@ def __getitem__(self, item):
 INTERNET_ALL_DRAFTS_ARCHIVE_DIR = "test/archive/"
 RFC_PATH = "test/rfc/"
 
-AGENDA_PATH = 'data/developers/www6s/proceedings/'
+AGENDA_PATH = '/assets/www6s/proceedings/'
 MEETINGHOST_LOGO_PATH = AGENDA_PATH
 
 USING_DEBUG_EMAIL_SERVER=True
@@ -70,12 +70,12 @@ def __getitem__(self, item):
 PHOTOS_DIRNAME = 'photo'
 PHOTOS_DIR = MEDIA_ROOT + PHOTOS_DIRNAME
 
-DOCUMENT_PATH_PATTERN = 'data/developers/ietf-ftp/{doc.type_id}/'
+DOCUMENT_PATH_PATTERN = '/assets/ietf-ftp/{doc.type_id}/'
 
-SUBMIT_YANG_CATALOG_MODEL_DIR = 'data/developers/ietf-ftp/yang/catalogmod/'
-SUBMIT_YANG_DRAFT_MODEL_DIR = 'data/developers/ietf-ftp/yang/draftmod/'
-SUBMIT_YANG_INVAL_MODEL_DIR = 'data/developers/ietf-ftp/yang/invalmod/'
-SUBMIT_YANG_IANA_MODEL_DIR = 'data/developers/ietf-ftp/yang/ianamod/'
-SUBMIT_YANG_RFC_MODEL_DIR   = 'data/developers/ietf-ftp/yang/rfcmod/'
+SUBMIT_YANG_CATALOG_MODEL_DIR = '/assets/ietf-ftp/yang/catalogmod/'
+SUBMIT_YANG_DRAFT_MODEL_DIR = '/assets/ietf-ftp/yang/draftmod/'
+SUBMIT_YANG_INVAL_MODEL_DIR = '/assets/ietf-ftp/yang/invalmod/'
+SUBMIT_YANG_IANA_MODEL_DIR = '/assets/ietf-ftp/yang/ianamod/'
+SUBMIT_YANG_RFC_MODEL_DIR   = '/assets/ietf-ftp/yang/rfcmod/'
 
 DE_GFM_BINARY = '/usr/local/bin/de-gfm'


Test patch: