Repository: sissbruecker/linkding

Issue:
Background process to archive link times out
I recently discovered that using the docker image, supervisord has changed and the included supervisord.conf includes `user=root` which fails to de-escalate privileges and stops processing any further commands. I have a little more work to figure things out completely there, the jobs stdout isn't being sent to stdout for some reason, when/if I get it figured out I'll open a new ticket or a PR. However it is running and now every background process is failing on timing out (and I get rate limited by GitHub).

I've tried issuing the `single-file` commands stand alone, and it appears the issue is being caused by the `--browser-arg="--load-extension=uBlock0.chromium"`. If I remove the parameter, the archive succeeds. My thinking was that something with the extension is blocking the browser from processing.

I found this issue https://github.com/gildas-lormeau/single-file-cli/issues/83 that says the frame issue is fixed in 2.0.33, so I tried installing that, and it fails with:

```
/tmp/node_modules/single-file-cli/single-file --browser-arg="--headless=new" --browser-arg="--user-data-dir=/etc/linkding/chromium-profile" --browser-arg="--load-extension=uBlock0.chromium" https://google.com /etc/linkding/data/assets/test.html.gz
fetch failed [2024-09-28T01:06:34.771Z] URL: https://google.com
```

I've reached the end of my debugging abilities with chromium and single-file.


Discussion:
uBlock uses an extension model that has been deprecated for some time in Chrome and was scheduled to be removed soon. Maybe that time has come, in that the latest Chromium version installed in the Docker image doesn't support the extension anymore? Would be good to have some logs from the browser itself, though single-file might swallow those.

Also interesting that you are the first one to report this. Are you using the alpine based Docker images? Those might use a newer Chromium version than the Debian based ones.

Apart from that I'm not sure what you are referring to regarding supervisor, I don't think there has been any change to that since introducing the HTML snapshot feature.
Looks like only the `latest-plus-alpine` is affected. That one now uses Chromium 128, which seems to include some changes that might cause the extension not to load properly. I guess as a quick fix I'll remove uBlock, and then later see if I can integrate uBlock lite as a replacement.
supervisord in the alpine image is at version 4.2.5, in 3.3.4 they changed the warning for `Can't drop user privileges` to an error, as such supervisord no longer starts with the setting `user=root` in the configuration file as per https://github.com/Supervisor/supervisor/issues/1218 when the container process runs as the `www-data` user.

I am using the alpine image and I too found it interesting that I'm the only one reporting this problem. I could try the debian image and see if that makes a difference, but it sounds like you've already found the issue.
Mhh, supervisor seemed to work fine when I tested the alpine image. The container should actually run as `root`, it's only the uwsgi and supervisor background task that run as `www-data`. Did you configure your Docker container to run as `www-data` instead?
Yeah, I am running as `www-data`, a friend gave me their deployment manifest and it had a comment about using the `www-data` user with a link to https://github.com/sissbruecker/linkding/issues/660 so that's the way I left it.
OK, sounds like that will not work for now. I'm not sure why I configured supervisor to run as `root`. Maybe that part can be removed.
Thank you for taking care of this so quickly. I'm waiting on a new image push to test it out.



Patch:
diff --git a/.gitignore b/.gitignore
index 5850eb5d..68fad2d9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -192,7 +192,7 @@ typings/
 # Database file
 /data
 # ublock + chromium
-/uBlock0.chromium
+/uBOLite.chromium.mv3
 /chromium-profile
 # direnv
 /.direnv
diff --git a/docker/alpine.Dockerfile b/docker/alpine.Dockerfile
index e8c2210b..74e2aa86 100644
--- a/docker/alpine.Dockerfile
+++ b/docker/alpine.Dockerfile
@@ -88,18 +88,20 @@ CMD ["./bootstrap.sh"]
 FROM node:18-alpine AS ublock-build
 WORKDIR /etc/linkding
 # Install necessary tools
-RUN apk add --no-cache curl jq unzip
-# Fetch the latest release tag
-# Download the library
-# Unzip the library
-RUN TAG=$(curl -sL https://api.github.com/repos/gorhill/uBlock/releases/latest | jq -r '.tag_name') && \
-    DOWNLOAD_URL=https://github.com/gorhill/uBlock/releases/download/$TAG/uBlock0_$TAG.chromium.zip && \
-    curl -L -o uBlock0.zip $DOWNLOAD_URL && \
-    unzip uBlock0.zip
-# Patch assets.json to enable easylist-cookies by default
-RUN curl -L -o ./uBlock0.chromium/assets/thirdparties/easylist/easylist-cookies.txt https://ublockorigin.github.io/uAssets/thirdparties/easylist-cookies.txt
-RUN jq '."assets.json" |= del(.cdnURLs) | ."assets.json".contentURL = ["assets/assets.json"] | ."fanboy-cookiemonster" |= del(.off) | ."fanboy-cookiemonster".contentURL += ["assets/thirdparties/easylist/easylist-cookies.txt"]' ./uBlock0.chromium/assets/assets.json > temp.json && \
-    mv temp.json ./uBlock0.chromium/assets/assets.json
+# Download and unzip the latest uBlock Origin Lite release
+# Patch manifest to enable annoyances by default
+# Patch ruleset-manager.js to use rulesets enabled in manifest by default
+RUN apk add --no-cache curl jq unzip && \
+    TAG=$(curl -sL https://api.github.com/repos/uBlockOrigin/uBOL-home/releases/latest | jq -r '.tag_name') && \
+    DOWNLOAD_URL=https://github.com/uBlockOrigin/uBOL-home/releases/download/$TAG/$TAG.chromium.mv3.zip && \
+    echo "Downloading $DOWNLOAD_URL" && \
+    curl -L -o uBOLite.zip $DOWNLOAD_URL && \
+    unzip uBOLite.zip -d uBOLite.chromium.mv3 && \
+    rm uBOLite.zip && \
+    jq '.declarative_net_request.rule_resources |= map(if .id == "annoyances-overlays" or .id == "annoyances-cookies" or .id == "annoyances-social" or .id == "annoyances-widgets" or .id == "annoyances-others" then .enabled = true else . end)' \
+        uBOLite.chromium.mv3/manifest.json > temp.json && \
+    mv temp.json uBOLite.chromium.mv3/manifest.json && \
+    sed -i 's/const out = \[ '\''default'\'' \];/const out = await dnr.getEnabledRulesets();/' uBOLite.chromium.mv3/js/ruleset-manager.js
 
 
 FROM linkding AS linkding-plus
@@ -107,8 +109,8 @@ FROM linkding AS linkding-plus
 RUN apk update && apk add nodejs npm chromium
 # install single-file from fork for now, which contains several hotfixes
 RUN npm install -g https://github.com/sissbruecker/single-file-cli/tarball/4c54b3bc704cfb3e96cec2d24854caca3df0b3b6
-# copy uBlock0
-COPY --from=ublock-build /etc/linkding/uBlock0.chromium uBlock0.chromium/
+# copy uBlock
+COPY --from=ublock-build /etc/linkding/uBOLite.chromium.mv3 uBOLite.chromium.mv3/
 # create chromium profile folder for user running background tasks
 RUN mkdir -p chromium-profile && chown -R www-data:www-data chromium-profile
 # enable snapshot support
diff --git a/docker/default.Dockerfile b/docker/default.Dockerfile
index aaa048f1..629bbdd8 100644
--- a/docker/default.Dockerfile
+++ b/docker/default.Dockerfile
@@ -86,18 +86,20 @@ CMD ["./bootstrap.sh"]
 FROM node:18-alpine AS ublock-build
 WORKDIR /etc/linkding
 # Install necessary tools
-RUN apk add --no-cache curl jq unzip
-# Fetch the latest release tag
-# Download the library
-# Unzip the library
-RUN TAG=$(curl -sL https://api.github.com/repos/gorhill/uBlock/releases/latest | jq -r '.tag_name') && \
-    DOWNLOAD_URL=https://github.com/gorhill/uBlock/releases/download/$TAG/uBlock0_$TAG.chromium.zip && \
-    curl -L -o uBlock0.zip $DOWNLOAD_URL && \
-    unzip uBlock0.zip
-# Patch assets.json to enable easylist-cookies by default
-RUN curl -L -o ./uBlock0.chromium/assets/thirdparties/easylist/easylist-cookies.txt https://ublockorigin.github.io/uAssets/thirdparties/easylist-cookies.txt
-RUN jq '."assets.json" |= del(.cdnURLs) | ."assets.json".contentURL = ["assets/assets.json"] | ."fanboy-cookiemonster" |= del(.off) | ."fanboy-cookiemonster".contentURL += ["assets/thirdparties/easylist/easylist-cookies.txt"]' ./uBlock0.chromium/assets/assets.json > temp.json && \
-    mv temp.json ./uBlock0.chromium/assets/assets.json
+# Download and unzip the latest uBlock Origin Lite release
+# Patch manifest to enable annoyances by default
+# Patch ruleset-manager.js to use rulesets enabled in manifest by default
+RUN apk add --no-cache curl jq unzip && \
+    TAG=$(curl -sL https://api.github.com/repos/uBlockOrigin/uBOL-home/releases/latest | jq -r '.tag_name') && \
+    DOWNLOAD_URL=https://github.com/uBlockOrigin/uBOL-home/releases/download/$TAG/$TAG.chromium.mv3.zip && \
+    echo "Downloading $DOWNLOAD_URL" && \
+    curl -L -o uBOLite.zip $DOWNLOAD_URL && \
+    unzip uBOLite.zip -d uBOLite.chromium.mv3 && \
+    rm uBOLite.zip && \
+    jq '.declarative_net_request.rule_resources |= map(if .id == "annoyances-overlays" or .id == "annoyances-cookies" or .id == "annoyances-social" or .id == "annoyances-widgets" or .id == "annoyances-others" then .enabled = true else . end)' \
+        uBOLite.chromium.mv3/manifest.json > temp.json && \
+    mv temp.json uBOLite.chromium.mv3/manifest.json && \
+    sed -i 's/const out = \[ '\''default'\'' \];/const out = await dnr.getEnabledRulesets();/' uBOLite.chromium.mv3/js/ruleset-manager.js
 
 
 FROM linkding AS linkding-plus
@@ -111,9 +113,9 @@ RUN apt-get install -y gnupg2 apt-transport-https ca-certificates && \
     apt-get update && apt-get install -y nodejs
 # install single-file from fork for now, which contains several hotfixes
 RUN npm install -g https://github.com/sissbruecker/single-file-cli/tarball/4c54b3bc704cfb3e96cec2d24854caca3df0b3b6
+# copy uBlock
+COPY --from=ublock-build /etc/linkding/uBOLite.chromium.mv3 uBOLite.chromium.mv3/
 # create chromium profile folder for user running background tasks
 RUN mkdir -p chromium-profile && chown -R www-data:www-data chromium-profile
-# copy uBlock0
-COPY --from=ublock-build /etc/linkding/uBlock0.chromium uBlock0.chromium/
 # enable snapshot support
 ENV LD_ENABLE_SNAPSHOTS=True
diff --git a/scripts/setup-ublock.sh b/scripts/setup-ublock.sh
index 415f0d65..568ea8a4 100755
--- a/scripts/setup-ublock.sh
+++ b/scripts/setup-ublock.sh
@@ -1,13 +1,18 @@
-rm -rf ublock0.chromium
+rm -rf uBOLite.chromium.mv3
 
-TAG=$(curl -sL https://api.github.com/repos/gorhill/uBlock/releases/latest | jq -r '.tag_name')
-DOWNLOAD_URL=https://github.com/gorhill/uBlock/releases/download/$TAG/uBlock0_$TAG.chromium.zip
-curl -L -o uBlock0.zip $DOWNLOAD_URL
-unzip uBlock0.zip
-rm uBlock0.zip
+# Download uBlock Origin Lite
+TAG=$(curl -sL https://api.github.com/repos/uBlockOrigin/uBOL-home/releases/latest | jq -r '.tag_name')
+DOWNLOAD_URL=https://github.com/uBlockOrigin/uBOL-home/releases/download/$TAG/$TAG.chromium.mv3.zip
+echo "Downloading $DOWNLOAD_URL"
+curl -L -o uBOLite.zip $DOWNLOAD_URL
+unzip uBOLite.zip -d uBOLite.chromium.mv3
+rm uBOLite.zip
 
-curl -L -o ./uBlock0.chromium/assets/thirdparties/easylist/easylist-cookies.txt https://ublockorigin.github.io/uAssets/thirdparties/easylist-cookies.txt
-jq '."assets.json" |= del(.cdnURLs) | ."assets.json".contentURL = ["assets/assets.json"] | ."fanboy-cookiemonster" |= del(.off) | ."fanboy-cookiemonster".contentURL += ["assets/thirdparties/easylist/easylist-cookies.txt"]' ./uBlock0.chromium/assets/assets.json > temp.json
-mv temp.json ./uBlock0.chromium/assets/assets.json
+# Patch uBlock Origin Lite to respect rulesets enabled in manifest.json
+sed -i '' "s/const out = \[ 'default' \];/const out = await dnr.getEnabledRulesets();/" uBOLite.chromium.mv3/js/ruleset-manager.js
+
+# Enable annoyances rulesets in manifest.json
+jq '.declarative_net_request.rule_resources |= map(if .id == "annoyances-overlays" or .id == "annoyances-cookies" or .id == "annoyances-social" or .id == "annoyances-widgets" or .id == "annoyances-others" then .enabled = true else . end)' uBOLite.chromium.mv3/manifest.json > temp.json
+mv temp.json uBOLite.chromium.mv3/manifest.json
 
 mkdir -p chromium-profile
diff --git a/siteroot/settings/base.py b/siteroot/settings/base.py
index f7678e60..f8220751 100644
--- a/siteroot/settings/base.py
+++ b/siteroot/settings/base.py
@@ -296,7 +296,7 @@
             '--browser-arg="--headless=new"',
             '--browser-arg="--user-data-dir=./chromium-profile"',
             '--browser-arg="--no-sandbox"',
-            '--browser-arg="--load-extension=uBlock0.chromium"',
+            '--browser-arg="--load-extension=uBOLite.chromium.mv3"',
         ]
     ),
 )


Test patch:
diff --git a/bookmarks/tests/test_singlefile_service.py b/bookmarks/tests/test_singlefile_service.py
index b9b0f7a7..13ca3bb7 100644
--- a/bookmarks/tests/test_singlefile_service.py
+++ b/bookmarks/tests/test_singlefile_service.py
@@ -66,7 +66,7 @@ def test_create_snapshot_empty_options(self):
                 '--browser-arg="--headless=new"',
                 '--browser-arg="--user-data-dir=./chromium-profile"',
                 '--browser-arg="--no-sandbox"',
-                '--browser-arg="--load-extension=uBlock0.chromium"',
+                '--browser-arg="--load-extension=uBOLite.chromium.mv3"',
                 "http://example.com",
                 self.html_filepath + ".tmp",
             ]
@@ -88,7 +88,7 @@ def test_create_snapshot_custom_options(self):
                 '--browser-arg="--headless=new"',
                 '--browser-arg="--user-data-dir=./chromium-profile"',
                 '--browser-arg="--no-sandbox"',
-                '--browser-arg="--load-extension=uBlock0.chromium"',
+                '--browser-arg="--load-extension=uBOLite.chromium.mv3"',
                 "--some-option",
                 "some value",
                 "--another-option",