Repository: chaincrucial/guild-operators
Pull request #6: Sidecar healthchecks updates
Author: login: adamsthws
Created: 2025-05-10T16:10:20Z | Merged: 2025-05-28T20:45:00Z

- Added additional healthcheck functions for pooltool send slots and logmonitor entrypoints.

- Added Koios api error handling and fixed an issue with parsing the api key in the curl header.

- Minor improvements and typo fixes in log lines and code comments.

(Please see commit comments for more detailed comments).
Thankyou!


Diff:
diff --git a/files/docker/node/addons/healthcheck.sh b/files/docker/node/addons/healthcheck.sh
index 66b41b485..e29690c74 100755
--- a/files/docker/node/addons/healthcheck.sh
+++ b/files/docker/node/addons/healthcheck.sh
@@ -7,13 +7,13 @@
 ######################################
 
 ENTRYPOINT_PROCESS="${ENTRYPOINT_PROCESS:-cnode.sh}"            # Get the script from ENTRYPOINT_PROCESS or default to "cnode.sh" if not set
-HEALTHCHECK_CPU_THRESHOLD="${HEALTHCHECK_CPU_THRESHOLD:-80}"    # The CPU threshold to warn about if the sidecar process exceeds this for more than 60 seconds, defaults to 80%.
+HEALTHCHECK_CPU_THRESHOLD="${HEALTHCHECK_CPU_THRESHOLD:-80}"    # The CPU threshold to warn about if the sidecar process exceeds this for more than 60 seconds, defaults to 80%
 HEALTHCHECK_RETRIES="${HEALTHCHECK_RETRIES:-20}"                # The number of retries if tip is not incrementing, or cpu usage is over the threshold
 HEALTHCHECK_RETRY_WAIT="${HEALTHCHECK_RETRY_WAIT:-3}"           # The time (in seconds) to wait between retries
 NODE_ALLOWED_DRIFT="${NODE_ALLOWED_DRIFT:-6}"                   # The allowed drift in blocks for the node to be considered in sync
 DB_SYNC_ALLOWED_DRIFT="${DB_SYNC_ALLOWED_DRIFT:-3600}"          # The allowed drift in seconds for the DB to be considered in sync
 CNCLI_DB_ALLOWED_DRIFT="${CNCLI_DB_ALLOWED_DRIFT:-300}"         # The allowed drift in slots for the CNCLI DB to be considered in sync
-CNCLI_SENDTIP_LOG_TIMEOUT="${CNCLI_SENDTIP_LOG_TIMEOUT:-119}"   # log capturing timeout (should one second be lower than container healthcheck '--timeout', which defaults to 120)
+CNCLI_SENDTIP_LOG_TIMEOUT="${CNCLI_SENDTIP_LOG_TIMEOUT:-119}"   # log capture timeout (should be one second lower than container healthcheck '--timeout', which defaults to 120)
 CNCLI_SENDTIP_ALLOWED_DRIFT="${CNCLI_SENDTIP_ALLOWED_DRIFT:-3}" # The allowable difference of the tip moving before it's sent to Pooltool. (Not every tip progression is sent to Pooltool)
 
 ######################################
@@ -25,44 +25,36 @@ CNCLI_SENDTIP_ALLOWED_DRIFT="${CNCLI_SENDTIP_ALLOWED_DRIFT:-3}" # The allowable
 [[ ! -f "${PARENT}"/env ]] && echo -e "\nCommon env file missing in \"${PARENT}\", please ensure latest guild-deploy.sh was run and this script is being run from ${CNODE_HOME}/scripts folder! \n" && exit 1
 . "${PARENT}"/env offline
 
-if [[ -z "${KOIOS_API_HEADERS[*]}" ]] ; then
-    if [[ -n "${KOIOS_API_TOKEN}" ]] ; then
-        KOIOS_API_HEADERS=(-H "'Authorization: Bearer ${KOIOS_API_TOKEN}'")
-    else
-        KOIOS_API_HEADERS=()
-    fi
-fi
-
-
 # Define a mapping of scripts to their corresponding health check functions
 declare -A PROCESS_TO_HEALTHCHECK
 PROCESS_TO_HEALTHCHECK=(
     ["dbsync.sh"]="check_db_sync"
     ["cnode.sh"]="check_node"
     ["cncli.sh"]="check_cncli"
+    ["logMonitor.sh"]="check_logmonitor"
 )
 
 # FUNCTIONS
+
+# Function to start the relevant health check based on the subcommand of cncli.sh
 check_cncli() {
     cncli_pid=$(pgrep -f "${ENTRYPOINT_PROCESS}")
     cncli_subcmd=$(ps -p "${cncli_pid}" -o cmd= | awk '{print $NF}')
 
-    if [[ "${cncli_subcmd}" != "ptsendtip" ]]; then
-        if check_cncli_db ; then
-            return 0
-        else
-            return 1
-        fi
-    else
-        if check_cncli_sendtip; then
-            return 0
-        else
-            return 1
-        fi
-    fi
+    case "${cncli_subcmd}" in
+    ptsendtip)
+        check_cncli_sendtip
+        ;;
+    ptsendslots)
+        check_cncli_sendslots
+        ;;
+    *)
+        check_cncli_db
+        ;;
+    esac
 }
 
-
+# Function to check if the cncli sqlite db is in sync with the node
 check_cncli_db() {
     CCLI=$(which cardano-cli)
     SQLITE=$(which sqlite3)
@@ -78,7 +70,6 @@ check_cncli_db() {
     fi
 }
 
-
 # Function to check if the tip is successfully being sent to Pooltool
 check_cncli_sendtip() {
     # Get the process ID of cncli
@@ -125,9 +116,87 @@ check_cncli_sendtip() {
     fi
 }
 
+# Function to check if the cncli status is ok. (Stdout is "ok" if status="ok", error message if not)
+check_cncli_status() {
+    CNCLI=$(which cncli)
+    SQLITE=$(which sqlite3)
+    # Get the required genesis file paths from the config file
+    local -A genesis_files
+    for genesis_file in "ByronGenesisFile" "ShelleyGenesisFile"; do
+        if [[ "${CONFIG##*.}" = "yaml" ]]; then
+            [[ $(grep "${genesis_file}.*\.json" "${CONFIG}") =~ ${genesis_file}:.\"(.+\.json)\" ]] && genesis_files["${genesis_file}"]="${BASH_REMATCH[1]}"
+        elif [[ "${CONFIG##*.}" = "json" ]]; then
+            genesis_files["${genesis_file}"]=$(jq -r ".${genesis_file}" "${CONFIG}")
+        fi
+    done
+    # Check if the genesis path has been retrieved from the config file and that the files exist
+    for genesis_type in "Byron" "Shelley"; do
+        if [[ -n "${genesis_files[${genesis_type}GenesisFile]}" ]] && [[ ! -f "${genesis_files[${genesis_type}GenesisFile]}" ]]; then
+            genesis_error_message="\"${genesis_type}\" genesis file is either not defined in the config file or the genesis file does not exist"
+            echo " \"${genesis_error_message}\": ${genesis_files[${genesis_type}GenesisFile]}"
+            return 1
+        fi
+    done
+    # Check that CNCLI_DB file is available and readable
+    if [[ -z "${CNCLI_DB}" ]]; then
+        CNCLI_DB="${CNODE_HOME}/guild-db/cncli/cncli.db"
+    fi
+    if [[ ! -f "${CNCLI_DB}" || ! -r "${CNCLI_DB}" ]]; then
+        echo "Error: CNCLI_DB file does not exist or is not readable: ${CNCLI_DB}"
+        return 1
+    fi
+    # Verify database can be queried with SQLite
+    if ! timeout 10 "${SQLITE}" "${CNCLI_DB}" ".exit"; then
+        echo "Error: Cannot query CNCLI_DB: ${CNCLI_DB}"
+        return 1
+    fi
+    # Check cncli status
+    local cncli_status
+    cncli_status=$(${CNCLI} status \
+        --byron-genesis "${genesis_files[ByronGenesisFile]}" \
+        --shelley-genesis "${genesis_files[ShelleyGenesisFile]}" \
+        --db "${CNCLI_DB}" |
+        jq -r .status)
+    if [[ $? -eq 0 ]]; then
+        echo "$cncli_status"
+        return 0
+    else
+        echo "Error: Status check failed when running \"cncli status\" command"
+        return 1
+    fi
+}
+
+# Function to check if the cncli sendslots process is running
+check_cncli_sendslots() {
+    # Check that the "cncli.sh ptsendslots" process is running
+    if ! pgrep -f "cncli.sh ptsendslots" >/dev/null; then
+        echo "Error: sendslots process is not running"
+        return 1
+    fi
+    # Check $PT_API_KEY is available and valid
+    if [[ -z "${PT_API_KEY}" ]] || [[ ! "${PT_API_KEY}" =~ ^[a-z0-9-]+$ ]]; then
+        echo "Error: PT_API_KEY is not set or not valid (must be lowercase alphanumeric and hyphen chars only)"
+        return 1
+    fi
+    # Check $POOL_TICKER is available and valid
+    if [[ -z "${POOL_TICKER}" ]] || [[ ! "${POOL_TICKER}" =~ ^[A-Z0-9]{3,5}$ ]]; then
+        echo "Error: POOL_TICKER is not set or not valid (must be 3-5 characters in length, A-Z and 0-9 only)"
+        return 1
+    fi
+    # Check cncli status
+    local cncli_status
+    cncli_status="$(check_cncli_status)"
+    if [[ "${cncli_status}" != "ok" ]]; then
+        echo "Error when checking cncli status: \"${cncli_status}\""
+        return 1
+    else
+        echo "Ready to send slots to PoolTool at the beginning of next epoch"
+        return 0
+    fi
+}
 
+# Check if the Postgres DB is in sync
 check_db_sync() {
-    # Check if the DB is in sync
     [[ -z "${PGPASSFILE}" ]] && PGPASSFILE="${CNODE_HOME}/priv/.pgpass"
     if [[ ! -f "${PGPASSFILE}" ]]; then
         echo "ERROR: The PGPASSFILE (${PGPASSFILE}) not found, please ensure you've followed the instructions on guild-operators website!" && exit 1
@@ -149,7 +218,6 @@ check_db_sync() {
     fi
 }
 
-
 # Function to check if the node is running and is on tip
 check_node() {
     CCLI=$(which cardano-cli)
@@ -174,7 +242,18 @@ check_node() {
             return 1
         fi
     else
-        SECOND=$($CURL -s "${KOIOS_API_HEADERS[@]}" "${URL}" | $JQ '.[0].block_no')
+    # Query the Koios API and check if the response is valid
+    KOIOS_RESPONSE=$(${CURL} -s "${KOIOS_API_HEADERS[@]}" "${URL}")
+    if ! echo "${KOIOS_RESPONSE}" | $JQ . &>/dev/null; then
+        echo "Error. Koios API query output: \"${KOIOS_RESPONSE}\""
+        return 1
+    else
+        SECOND=$(echo "${KOIOS_RESPONSE}" | $JQ '.[0].block_no')
+        if ! [[ "$SECOND" =~ ^[0-9]+$ ]]; then
+            echo "Error. Failed to get valid block height from Koios API"
+            return 1
+        fi
+    fi
 
         for (( CHECK=0; CHECK<=HEALTHCHECK_RETRIES; CHECK++ )); do
             # Set BIDIRECTIONAL_DRIFT to 1 since using an API call
@@ -217,7 +296,6 @@ check_process() {
     return 1  # Return 1 if retries are exhausted
 }
 
-
 # Function to check if the tip is progressing or within the allowed drift.
 # FIRST_TIP: The first tip to compare
 # SECOND_TIP: The second tip to compare
@@ -226,7 +304,6 @@ check_process() {
 #                      tips. This is useful when the tip can move backwards, like with API calls.
 # Returns 0 if the diff of the second tip minus the first tip is between 0 and the allowed drift.
 # Returns 1 if the diff is outside the allowed drift.
-
 check_tip() {
     FIRST_TIP=$1
     SECOND_TIP=$2
@@ -254,6 +331,50 @@ check_tip() {
     fi
 }
 
+# Function to check if logMonitor.sh is running and properly processing logs
+check_logmonitor() {
+    SQLITE=$(which sqlite3)
+    # Check if logMonitor process is running
+    if ! pgrep -f "logMonitor.sh" >/dev/null; then
+        echo "ERROR: logMonitor.sh process is not running"
+        return 1
+    fi
+    # Get the log file that logMonitor is watching from the config file
+    local logfile=""
+    if [[ "${CONFIG##*.}" = "yaml" ]]; then
+        [[ $(grep "scName.*\.json" "${CONFIG}") =~ scName:.\"(.+\.json)\" ]] && logfile="${BASH_REMATCH[1]}"
+    elif [[ "${CONFIG##*.}" = "json" ]]; then
+        logfile=$(jq -r '.setupScribes[] | select (.scFormat == "ScJson") | .scName' "${CONFIG}")
+    fi
+    # Check if the path to the log file is present
+    if [[ -z "${logfile}" ]]; then
+        echo "ERROR: Failed to locate json log file."
+        return 1
+    fi
+    # Check if the node log file is present
+    if [[ ! -f "${logfile}" ]]; then
+        echo "ERROR: Node log file does not exist. ${logfile} "
+        return 1
+    fi
+    # Check if the blocklog database is present
+    if [[ ! -f "${BLOCKLOG_DB}" ]]; then
+        echo "ERROR: Block log database ${BLOCKLOG_DB} does not exist"
+        return 1
+    fi
+    # Verify write access to the blocklog database file
+    if [[ ! -w "${BLOCKLOG_DB}" ]]; then
+        echo "ERROR: No write permission to block log database ${BLOCKLOG_DB}"
+        return 1
+    fi
+    # Verify the database is readable with SQLite
+    if ! timeout 10 "${SQLITE}" "${BLOCKLOG_DB}" ".exit"; then
+        echo "ERROR: Cannot connect to block log database ${BLOCKLOG_DB}"
+        return 1
+    fi
+    # If all checks pass
+    echo "Healthy: logMonitor.sh is running and able to process logs"
+    return 0
+}
 
 # MAIN
 if [[ -n "${PROCESS_TO_HEALTHCHECK[$ENTRYPOINT_PROCESS]}" ]]; then
@@ -261,12 +382,11 @@ if [[ -n "${PROCESS_TO_HEALTHCHECK[$ENTRYPOINT_PROCESS]}" ]]; then
     eval "${PROCESS_TO_HEALTHCHECK[$ENTRYPOINT_PROCESS]}"
     exit $?
 else
-    # When 
     # Determine the process name or script to check health
     if [[ -n "${SCRIPT_TO_BINARY_MAP[$ENTRYPOINT_PROCESS]}" ]]; then
         process="${SCRIPT_TO_BINARY_MAP[$ENTRYPOINT_PROCESS]}"
     fi
-    echo "Checking health for process: $process"
+    echo "Checking health for process: $process."
     check_process "$process" "$HEALTHCHECK_CPU_THRESHOLD"
     exit $?
 fi
diff --git a/scripts/cnode-helper-scripts/env b/scripts/cnode-helper-scripts/env
index 04c56f959..28338da3e 100644
--- a/scripts/cnode-helper-scripts/env
+++ b/scripts/cnode-helper-scripts/env
@@ -1021,7 +1021,7 @@ set_default_vars() {
   [[ -z ${STRICT_VERSION_CHECK} ]] && STRICT_VERSION_CHECK="Y"
   if [[ -z "${KOIOS_API_HEADERS[*]}" ]] ; then
     if [[ -n "${KOIOS_API_TOKEN}" ]] ; then
-      KOIOS_API_HEADERS=(-H "'Authorization: Bearer ${KOIOS_API_TOKEN}'")
+      KOIOS_API_HEADERS=(-H "Authorization: Bearer ${KOIOS_API_TOKEN}")
     else
       KOIOS_API_HEADERS=()
     fi


Reviews:
user: login: TrevorBenson
body: Mostly, looks good. Just a question about the headers to resolve first.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T14:23:11Z
user: login: adamsthws
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-12T15:35:16Z
user: login: adamsthws
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-13T10:08:03Z

Review comments:
diff_hunk: @@ -174,7 +239,18 @@ check_node() {
             return 1
         fi
     else
-        SECOND=$($CURL -s "${KOIOS_API_HEADERS[@]}" "${URL}" | $JQ '.[0].block_no')
+    # Query the Koios API and check if the response is valid
+    KOIOS_RESPONSE=$(${CURL} -s ${KOIOS_API_TOKEN:+-H "Authorization: Bearer ${KOIOS_API_TOKEN}"} ${URL})
path: files/docker/node/addons/healthcheck.sh
user: login: TrevorBenson
body: This is usually an array set in env, hence the original form including all possible headers from the array. This change would drop any additional headers.

What issue did you observe with KOIOS_API_HEADERS in its original form that led to this change?
created_at: 2025-05-12T14:06:09Z
author_association: COLLABORATOR
diff_hunk: @@ -174,7 +239,18 @@ check_node() {
             return 1
         fi
     else
-        SECOND=$($CURL -s "${KOIOS_API_HEADERS[@]}" "${URL}" | $JQ '.[0].block_no')
+    # Query the Koios API and check if the response is valid
+    KOIOS_RESPONSE=$(${CURL} -s ${KOIOS_API_TOKEN:+-H "Authorization: Bearer ${KOIOS_API_TOKEN}"} ${URL})
path: files/docker/node/addons/healthcheck.sh
user: login: adamsthws
body: **Koios API key - not working:**
```
KOIOS_API_TOKEN="<valid_koios_api_token>"
URL="https://api.koios.rest/api/v1/tip"
CURL=$(which curl)

if [[ -z "${KOIOS_API_HEADERS[*]}" ]] ; then
    if [[ -n "${KOIOS_API_TOKEN}" ]] ; then
        KOIOS_API_HEADERS=(-H "'Authorization: Bearer ${KOIOS_API_TOKEN}'")
    else
        KOIOS_API_HEADERS=()
    fi
fi

$CURL -s "${KOIOS_API_HEADERS[@]}" "${URL}"
```
When using the api headers as set in the env file, I was encountering the koios rate limit of 5k request per day which is the 'public' (unauthenticated) tier, even though a valid koios api key was defined (with the $KOIOS_API_TOKEN var).

Possible quoting issue?
When you echo "${KOIOS_API_HEADERS[@]}", the header that is returned looks correct and includes the api key. I think it may be a quoting issue but I didn't dive into it more than this.

**Koios API key - working:**
```
KOIOS_API_TOKEN="<valid_koios_api_token>"
URL="https://api.koios.rest/api/v1/tip"
CURL=$(which curl)

${CURL} -s -H "Authorization: Bearer ${KOIOS_API_TOKEN}" ${URL}
```
When switching to the header above, I no longer encountered the koios rate limit (using the same $KOIOS_API_TOKEN var).



**Proposed solution**
Inspecting how the headers are created in env, the only header it creates is the koios authorization header:
```  
#env

if [[ -z "${KOIOS_API_HEADERS[*]}" ]] ; then
    if [[ -n "${KOIOS_API_TOKEN}" ]] ; then
        KOIOS_API_HEADERS=(-H "'Authorization: Bearer ${KOIOS_API_TOKEN}'")
    else
        KOIOS_API_HEADERS=()
    fi
fi
```
The newly proposed method below fixes the issue and uses less code for the same result. It behaves the a similar way as the headers code in env (only setting an authorization header if $KOIOS_API_TOKEN has been defined):
```
${CURL} -s ${KOIOS_API_TOKEN:+-H "Authorization: Bearer ${KOIOS_API_TOKEN}"} ${URL}
```

I would ask; why would the healthcheck ever need additional headers?
Happy to change it if you'd prefer a different approach?

created_at: 2025-05-12T15:35:16Z
author_association: NONE
diff_hunk: @@ -174,7 +239,18 @@ check_node() {
             return 1
         fi
     else
-        SECOND=$($CURL -s "${KOIOS_API_HEADERS[@]}" "${URL}" | $JQ '.[0].block_no')
+    # Query the Koios API and check if the response is valid
+    KOIOS_RESPONSE=$(${CURL} -s ${KOIOS_API_TOKEN:+-H "Authorization: Bearer ${KOIOS_API_TOKEN}"} ${URL})
path: files/docker/node/addons/healthcheck.sh
user: login: adamsthws
body: > Possible quoting issue?

A quick glance at the quoting by the env headers shows it contains single quotes surrounded by double quotes:
```-H "'Authorization: Bearer ${KOIOS_API_TOKEN}'"```

The single quotes inside the string mean the actual string will include the single quotes literally, which might not be what we want when passing to curl:
```'Authorization: Bearer abc123'```

created_at: 2025-05-13T10:08:03Z
author_association: NONE

PR comments:
user: login: TrevorBenson
created_at: 2025-05-13T16:50:20Z
body: That's by design. The header is formed so the CLI ends up `curl -H
'Authorization: Bearer xxxxxxxxxxx'`. Otherwise, without single quotes, the
CLI gets multiple words `curl -H Authorization: Bearer xxxxxxxxxxxxxxx`and
you will see a name resolution errors because everything past
`Authorization:` is presumed to be a hostname for a url:
curl: (6) Could not resolve host: Bearer
curl: (6) Could not resolve host: XXXXXXXX


Trevor Benson | Staff Engineer
LinkedIn: @TrevorBenson <https://www.linkedin.com/in/trevorbenson/>
<https://www.linkedin.com/company/scality/> | Blog: solved.scality.com
<https://solved.scality.com/solved/>


On Tue, May 13, 2025 at 3:08 AM Adam Matthews ***@***.***>
wrote:

> ***@***.**** commented on this pull request.
> ------------------------------
>
> In files/docker/node/addons/healthcheck.sh
> <https://github.com/chaincrucial/guild-operators/pull/6#discussion_r2086438782>
> :
>
> > -        SECOND=$($CURL -s "${KOIOS_API_HEADERS[@]}" "${URL}" | $JQ '.[0].block_no')
> +    # Query the Koios API and check if the response is valid
> +    KOIOS_RESPONSE=$(${CURL} -s ${KOIOS_API_TOKEN:+-H "Authorization: Bearer ${KOIOS_API_TOKEN}"} ${URL})
>
> A quick glance at the quoting by the env headers shows it contains single
> quotes surrounded by double quotes:
> -H "'Authorization: Bearer ${KOIOS_API_TOKEN}'"
>
> The single quotes inside the string mean the actual string will include
> the single quotes literally, which might not be what we want when passing
> to curl:
> 'Authorization: Bearer abc123'
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/chaincrucial/guild-operators/pull/6#discussion_r2086438782>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACF6ID5N5U264DZ4QEHC4X326HABRAVCNFSM6AAAAAB427Q7JOVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDQMZWGEYDAMJTGI>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>

author_association: COLLABORATOR
user: login: adamsthws
created_at: 2025-05-14T10:43:27Z
body: @TrevorBenson 
I've created a script which aims to validate my findings and allow others to reproduce:
https://gist.github.com/adamsthws/dfbb18525598a2a3396b8acb3770b24e
Please could I ask you to test at your end too?
author_association: NONE
user: login: TrevorBenson
created_at: 2025-05-16T23:03:12Z
body: > @TrevorBenson I've created a script which aims to validate my findings and allow others to reproduce: https://gist.github.com/adamsthws/dfbb18525598a2a3396b8acb3770b24e Please could I ask you to test at your end too?

So `broken` is working for me, but with slight modifications because there was redundant quoting in the test and it was overwriting my token from the env:

```
--- koios_api_header_test_orig.sh	2025-05-16 22:37:06.000000000 +0000
+++ koios_api_header_test.sh	2025-05-16 22:41:38.119999850 +0000
@@ -14,7 +14,7 @@
 # Working headers are the ones as defined in the proposed changes to the new heathcheck:
 # KOIOS_API_HEADERS=$(${CURL} -s ${KOIOS_API_TOKEN:+-H "Authorization: Bearer ${KOIOS_API_TOKEN}"} ${URL})
 
-KOIOS_API_TOKEN="<your_token_here>"
+[[ -z ${KOIOS_API_TOKEN} ]] && KOIOS_API_TOKEN="<your_token_here>"
 MAX_CONCURRENT=10 # Define the maximum number of concurrent processes
 
 URL="https://api.koios.rest/api/v1/tip" # Mainnet
@@ -61,7 +61,7 @@
     local koios_response_command
 
     if [[ "$test_type" == "broken" ]]; then
-        koios_response_command="$CURL -s \"${KOIOS_API_HEADERS[*]}\" \"${URL}\""
+        koios_response_command="$CURL -s ${KOIOS_API_HEADERS[*]} ${URL}"
     elif [[ "$test_type" == "working" ]]; then
         koios_response_command="$CURL -s ${KOIOS_API_TOKEN:+-H \"Authorization: Bearer ${KOIOS_API_TOKEN}\"} \"${URL}\""
     else
```


As this is actually redundant quoting and leads to passing a single string to curl like `"-H 'Authorization Bearer: XXXXXXXX' -H 'Other Headers if they exist: BLAH'"`. The strange part is I see upstream instances (apparently not related to `echo` or `printf`) which seem to have quoting included. Thats not the way I recall it being implemented, but I will have to test it more thoroughly with `bash -x` again and see if everything is still working. I would suspect people would be complaining if koios was failing, but maybe it's rare that anyone injects the API token and just relies on free tier usage.


Here are the results of the broken test, after I inherit my API token from the env and remove the redundant quoting (API token redacted):
```
----------------------------------------

Test 1 passed

Height reported by Koios API:
11873610

Koios output:
"[{"hash":"fee12d2e200da81fa33d4e868a2b76ed5147419ff026ada6f866316d3ed0dae4","epoch_no":558,"abs_slot":155868988,"epoch_slot":176188,"block_height":11873610,"block_no":11873610,"block_time":1747435279}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

----------------------------------------

Test 2 passed

Height reported by Koios API:
11873610

Koios output:
"[{"hash":"fee12d2e200da81fa33d4e868a2b76ed5147419ff026ada6f866316d3ed0dae4","epoch_no":558,"abs_slot":155868988,"epoch_slot":176188,"block_height":11873610,"block_no":11873610,"block_time":1747435279}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

----------------------------------------

Test 3 passed

Height reported by Koios API:
11873610

Koios output:
"[{"hash":"fee12d2e200da81fa33d4e868a2b76ed5147419ff026ada6f866316d3ed0dae4","epoch_no":558,"abs_slot":155868988,"epoch_slot":176188,"block_height":11873610,"block_no":11873610,"block_time":1747435279}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

----------------------------------------

Test 4 passed

Height reported by Koios API:
11873610

Koios output:
"[{"hash":"fee12d2e200da81fa33d4e868a2b76ed5147419ff026ada6f866316d3ed0dae4","epoch_no":558,"abs_slot":155868988,"epoch_slot":176188,"block_height":11873610,"block_no":11873610,"block_time":1747435279}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

----------------------------------------

Test 5 passed

Height reported by Koios API:
11873610

Koios output:
"[{"hash":"fee12d2e200da81fa33d4e868a2b76ed5147419ff026ada6f866316d3ed0dae4","epoch_no":558,"abs_slot":155868988,"epoch_slot":176188,"block_height":11873610,"block_no":11873610,"block_time":1747435279}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

----------------------------------------

Test 6 passed

Height reported by Koios API:
11873610

Koios output:
"[{"hash":"fee12d2e200da81fa33d4e868a2b76ed5147419ff026ada6f866316d3ed0dae4","epoch_no":558,"abs_slot":155868988,"epoch_slot":176188,"block_height":11873610,"block_no":11873610,"block_time":1747435279}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

----------------------------------------

Test 7 passed

Height reported by Koios API:
11873610

Koios output:
"[{"hash":"fee12d2e200da81fa33d4e868a2b76ed5147419ff026ada6f866316d3ed0dae4","epoch_no":558,"abs_slot":155868988,"epoch_slot":176188,"block_height":11873610,"block_no":11873610,"block_time":1747435279}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

----------------------------------------

Test 8 passed

Height reported by Koios API:
11873611

Koios output:
"[{"hash":"bc3136f75e317f62b5d3439e4f5b7e3ba3bd77749d4f9a64339cececc6188f39","epoch_no":558,"abs_slot":155869013,"epoch_slot":176213,"block_height":11873611,"block_no":11873611,"block_time":1747435304}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

----------------------------------------

Test 9 passed

Height reported by Koios API:
11873611

Koios output:
"[{"hash":"bc3136f75e317f62b5d3439e4f5b7e3ba3bd77749d4f9a64339cececc6188f39","epoch_no":558,"abs_slot":155869013,"epoch_slot":176213,"block_height":11873611,"block_no":11873611,"block_time":1747435304}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

----------------------------------------

Test 10 passed

Height reported by Koios API:
11873612

Koios output:
"[{"hash":"77408f65faf87c59efbb8180c7fe0fd524b277c33e0f9922a0e6c28f34cbe931","epoch_no":558,"abs_slot":155869015,"epoch_slot":176215,"block_height":11873612,"block_no":11873612,"block_time":1747435306}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

----------------------------------------

Test 11 passed

Height reported by Koios API:
11873612

Koios output:
"[{"hash":"77408f65faf87c59efbb8180c7fe0fd524b277c33e0f9922a0e6c28f34cbe931","epoch_no":558,"abs_slot":155869015,"epoch_slot":176215,"block_height":11873612,"block_no":11873612,"block_time":1747435306}]"

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip
^C
guild@preview-node:/opt/cardano/cnode/scripts$ 
----------------------------------------
Error. Test 12 failed

Koios output:
""

Command used:
/bin/curl -s -H 'Authorization: Bearer XXXXXXXXXXXXXXXX' https://api.koios.rest/api/v1/tip

```

author_association: COLLABORATOR
user: login: TrevorBenson
created_at: 2025-05-16T23:54:19Z
body: OK, my additional tests just confused me even more until I noticed the differences in array access methods between `env`, `cncli.sh` and your test case script. My comments about earlier quoting probably overlooked a nuance about `@` vs `*` array access methods with additional quoting that I must have tested thoroughly 8 months ago, but since forgot 😆.

----


I decided to try the test_koios function manually and examine the results, then swap the method of access and quoting.

# Method `@` with quotes
Seems to work just fine and return 200's:
```
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 "${KOIOS_API_HEADERS[@]}" ${KOIOS_API}/tip ; echo ''
200
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 "${KOIOS_API_HEADERS[@]}" ${KOIOS_API}/tip ; echo ''
200
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 "${KOIOS_API_HEADERS[@]}" ${KOIOS_API}/tip ; echo ''
200
```

# Method `@` without quotes
Seems to return the response even though its routed to /dev/null, but then a 200 response code is shown 🤔:
```
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 ${KOIOS_API_HEADERS[@]} ${KOIOS_API}/tip ; echo ''
000000[{"hash":"0bf4124e2f903d640ca5dcccbe1148c800eefe0ac76876cfbe9b93d18b5f54dc","epoch_no":934,"abs_slot":80781871,"epoch_slot":84271,"block_height":3241824,"block_no":3241824,"block_time":1747437871}]200
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 ${KOIOS_API_HEADERS[@]} ${KOIOS_API}/tip ; echo ''
000000[{"hash":"0bf4124e2f903d640ca5dcccbe1148c800eefe0ac76876cfbe9b93d18b5f54dc","epoch_no":934,"abs_slot":80781871,"epoch_slot":84271,"block_height":3241824,"block_no":3241824,"block_time":1747437871}]200
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 ${KOIOS_API_HEADERS[@]} ${KOIOS_API}/tip ; echo ''
000000[{"hash":"0bf4124e2f903d640ca5dcccbe1148c800eefe0ac76876cfbe9b93d18b5f54dc","epoch_no":934,"abs_slot":80781871,"epoch_slot":84271,"block_height":3241824,"block_no":3241824,"block_time":1747437871}]200
```

# Method `*` with quotes
Seems to cause a failure, as it only returns 000 and no response code of 200:
```
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 "${KOIOS_API_HEADERS[*]}" ${KOIOS_API}/tip ; echo ''
000
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 "${KOIOS_API_HEADERS[*]}" ${KOIOS_API}/tip ; echo ''
000
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 "${KOIOS_API_HEADERS[*]}" ${KOIOS_API}/tip ; echo ''
000
```

# Method `*` without quotes
Seems to work, but still returns the response, as well as a response code of 200:
```
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 ${KOIOS_API_HEADERS[*]} ${KOIOS_API}/tip ; echo ''
000000[{"hash":"cf4a0b25726a184c3d6fa3c1823e75b7a8e3c8b17a4fe362b02c8e709d8fcf09","epoch_no":934,"abs_slot":80781903,"epoch_slot":84303,"block_height":3241825,"block_no":3241825,"block_time":1747437903}]200
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 ${KOIOS_API_HEADERS[*]} ${KOIOS_API}/tip ; echo ''
000000[{"hash":"cf4a0b25726a184c3d6fa3c1823e75b7a8e3c8b17a4fe362b02c8e709d8fcf09","epoch_no":934,"abs_slot":80781903,"epoch_slot":84303,"block_height":3241825,"block_no":3241825,"block_time":1747437903}]200
guild@preview-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 ${KOIOS_API_HEADERS[*]} ${KOIOS_API}/tip ; echo ''
000000[{"hash":"cf4a0b25726a184c3d6fa3c1823e75b7a8e3c8b17a4fe362b02c8e709d8fcf09","epoch_no":934,"abs_slot":80781903,"epoch_slot":84303,"block_height":3241825,"block_no":3241825,"block_time":1747437903}]200
```

I suspect in the `koios_api_header_test.sh` test cases the method of array access being `*` **WITH** quotes is partly to blame for the issues. There may be some nuances we need to compare the `healthcheck.sh` to `env` or `cncli.sh`, but we should be able to work out relying on the env KOIOS_API_HEADERS array and use it to full effect. We may want to add comments above each use describing our the reasoning for access method and quoting vs. no quoting in the healthcheck (plus maybe some linter disable rules if it causes linting format errors).

If I had added comments in the original PR above each KOIOS_API_HEADERS use it probably would have made isolating this issue much simpler to begin with.
author_association: COLLABORATOR
user: login: adamsthws
created_at: 2025-05-17T12:36:56Z
body: I ran the same 4 variations as above, both testnet and mainnet...

- Below the 5k daily limit, results are the same as yours.
- Above the 5k daily limit the results differ.

Results (Above the 5k daily limit)...

Method `@` with quotes:
```
guild@mainnet-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 "${KOIOS_API_HEADERS[@]}" ${KOIOS_API}/tip ; echo ''
429
```
Method `@` without quotes:
```
guild@mainnet-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 ${KOIOS_API_HEADERS[@]} ${KOIOS_API}/tip ; echo ''
000000429
```
Method `*` with quotes:
```
guild@mainnet-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 "${KOIOS_API_HEADERS[*]}" ${KOIOS_API}/tip ; echo ''
000
```
Method `*` without quotes:
```
guild@mainnet-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 ${KOIOS_API_HEADERS[*]} ${KOIOS_API}/tip ; echo ''
000000429
```
Replacing the header array with ```-H "Authorization: Bearer $KOIOS_API_TOKEN"```:
```
guild@mainnet-node:/opt/cardano/cnode/scripts$ curl -sfk -o /dev/null -w "%{http_code}" -m 5 -H "Authorization: Bearer $KOIOS_API_TOKEN" ${KOIOS_API}/tip ; echo ''
200
```

Note on the ```koios_api_header_test.sh``` [gist](https://gist.github.com/adamsthws/dfbb18525598a2a3396b8acb3770b24e)
The script uses the mainnet api url ```URL="https://api.koios.rest/api/v1/tip"```. 
To use the script to get above the 5k limit on testnet, update the url. eg: ```URL="https://preview.koios.rest/api/v1/tip"```
I should've made the network selectable with a scipt arg but it's a throwaway script so didn't want to spend much time on it.
author_association: NONE
user: login: TrevorBenson
created_at: 2025-05-17T14:47:35Z
body: I've got some suspicions a last minute lint/format was applied with auto fixes. At least that's what comes to mind for how the 429's wouldn't have been caught in testing last September, the tested code being altered last second.

I'll give it some more runs this weekend and compare working versions with headers array and see if I can isolate why it would be sent but fail to be applied. I'll probably spin up a dedicated web service to simulate the API and examine the headers being passed in each version to see what is going on.
author_association: COLLABORATOR
user: login: TrevorBenson
created_at: 2025-05-17T15:46:15Z
body: After some more thought I recalled the nuances of quoting on bash arrays and how IFS handling differs. "$*" is a single string quoted, based on concatenating IFS. "$@" is a list of elements quoted separately, that honors word boundaries and quoting.
```
$ ARRAY=(one two 'three four' "'five six'")
$ for elem in "${ARRAY[*]}" ; do echo $elem ; done
one two three four 'five six'
$ for elem in "${ARRAY[@]}" ; do echo $elem ; done
one
two
three four
'five six'
```

Without the quoting then both styles end up with iteration splitting on IFS, regardless of internal quoting:
```
$ for elem in ${ARRAY[*]} ; do echo $elem ; done
one
two
three
four
'five
six'
$ for elem in ${ARRAY[@]} ; do echo $elem ; done
one
two
three
four
'five
six'
```

Putting together a test service now to double check handoff via curl and then will rerun the tests past 5000 looking to avoid the 429. Today will be busy though, so I may not have a response until tomorrow.
author_association: COLLABORATOR
user: login: TrevorBenson
created_at: 2025-05-17T16:47:05Z
body: OK, your initial inclination is correct. With the existing format in upstream of `"${KOIOS_API_HEADERS[@]}"` being used, the internal single quotes are resulting in sending an redundant quoting per element, vs. `"${KOIOS_API_HEADERS[*]}"` redundantly quoting all elements as a single string.

The fastest working solution I found was setting the KOIOS_API_HEADERS based on your initial evaluation of single quotes causing issues:
```
if [[ -z "${KOIOS_API_HEADERS[*]}" ]]; then
    if [[ -n "${KOIOS_API_TOKEN}" ]]; then
        KOIOS_API_HEADERS=(-H "Authorization: Bearer ${KOIOS_API_TOKEN}")
    else
        KOIOS_API_HEADERS=()
    fi
fi
```

Although to use it with your test required putting the 5000 loop iteration inside broken/working blocks, as trying to embed KOIOS_API_HEADERS via koios_response_command created some additional complexity, which took a couple of minutes to iron out. I just moved the loop into both broken and working blocks w/o using this var for execution, just for printing results. I didnt spend time looking at local commits or figuring out what the original tests must have used, and since one of my testnet servers had `ENABLE_KOIOS=N` I may have only hit 429's for a few minutes at the end of each 24 hour period, making it easy to overlook.

When I get back today I'll post a sample python script as well as a modified version of your test. 
author_association: COLLABORATOR
user: login: adamsthws
created_at: 2025-05-23T10:41:10Z
body: > When I get back today I'll post a sample python script as well as a modified version of your test.

Hey Trevor, quick check-in, is there anything I can do to help or test at this stage?

I'll be getting into nature on a camping trip this weekend but happy to pickup again next week.

Have a good weekend!

author_association: NONE
user: login: TrevorBenson
created_at: 2025-05-27T17:54:34Z
body: > > When I get back today I'll post a sample python script as well as a modified version of your test.
> 
> Hey Trevor, quick check-in, is there anything I can do to help or test at this stage?
> 
> I'll be getting into nature on a camping trip this weekend but happy to pickup again next week.
> 
> Have a good weekend!

Had a long weekend and holiday, just getting back myself. I've attached the python script I used to review the tests to your gist. 

I think there are only a few steps from here before merge:

1. Modifying `env` to remove the single quotes when generating KOIOS_API_HEADERS.
2. Testing healthcheck script while using KOIOS_API_HEADERS (and `@` as the array expansion method)
3. Finally, testing cntools, cncli, gLiveView (any script using a function which would use KOIOS_API_HEADERS) both with and without KOIOS_API_TOKEN defined.

Step 3 is primarily as I recall at least one spot that modified IFS but did not reset it after. Since a modified IFS could impact array expansion testing is simply to be sure removal of the single quotes when generating KOIOS_API_HEADERS does not result in a breaking change to other tools using KOIOS_API_HEADERS if IFS is not set as expected.

----

If time permits then tonight or tomorrow I plan to make the adjustments to my testnet servers, modify your branches current healthcheck script to use KOIOS_API_HEADERS from `env`, ensure `@` is used for array access methods, then test.

author_association: COLLABORATOR
user: login: adamsthws
created_at: 2025-05-28T13:55:30Z
body: I have done a little testing on testnet, here was my methodology...

1- Modified env to remove the single quotes when generating KOIOS_API_HEADERS

2- Used [this script](https://gist.github.com/adamsthws/1a1bd1eece8f40b93820bfa91f6bbea2) to get above the Koios 5000 daily api rate limit

3- Updated healthcheck.sh to use the KOIOS_API_HEADERS from env like so:
```KOIOS_RESPONSE=$($CURL -s "${KOIOS_API_HEADERS[@]}" "${URL}")```

I'm pleased to report that removing the single quotes in env has resolved the issue for the healthcheck script.

...

> Finally, testing cntools, cncli, gLiveView (any script using a function which would use KOIOS_API_HEADERS) both with and without KOIOS_API_TOKEN defined.

I've yet to get to this, will report back soon.





author_association: NONE
user: login: adamsthws
created_at: 2025-05-28T15:46:39Z
body: > Finally, testing cntools, cncli, gLiveView (any script using a function which would use KOIOS_API_HEADERS) both with and without KOIOS_API_TOKEN defined.

The following cntools tests were performed...
- With cntools in light mode to ensure it's using the koios api: ```CNTOOLS_MODE=light```
- After using [this script](https://gist.github.com/adamsthws/1a1bd1eece8f40b93820bfa91f6bbea2) to get above the Koios 5000 daily api rate limit

With KOIOS_API_TOKEN unset AND you are above the 5k api limit, cntools fails to start in light mode with the following error:
```ERROR: Koios query test failed, unable to launch CNTools in light mode utilizing Koios query layer```

Where ```KOIOS_API_HEADERS``` is used and how I tested it...

cncli.sh:

-    getKoiosData() 
      Test: ```cncli.sh leaderlog``` - working

cntools.library:

-    isPoolRegistered()
      Test: ```cntools.sh -l > Pool > Show > Registered: Yes```  - working
-    getAssetInfo()
      Test: ```cntools.sh -la > Advanced > Asset> Show Asset``` - working
-    verifyTx()
      Test: ```cntools.sh -l > Funds > Send > Tx put on chain !!```  - working
-    getDRepStatus()
      Test: ```cntools.sh -l > Vote > Governance > Info & Status > Own DRep Status```  - working
-    getDRepVotePower()
      Test: ```cntools.sh -l > Vote > Governance > Info & Status > Active Vote power```  - working
-    getAllGovActions()
      Test: ```cntools.sh -l > Vote > Governance > List Proposals```  - working
-    getGovAction()
      Test: ```cntools.sh -l > Vote > Governance > List Proposals > Details```  - working
-    getCurrentCommittee()
      Test: ```cntools.sh -l > Vote > Governance > Cast Vote > Committee member```  - working
-    getBalanceKoios()
      Test: ```cntools.sh -l > Wallet > Show```  - working
-    getRewardInfoKoios()
      Test: ```cntools.sh -l > Wallet > Show > Rewards Available```  - working
-    rotatePoolKeys()
      Test: ```cntools.sh -l > Pool > Rotate```  - working
-    submitTxKoiosOgmios()
      Test: ```cntools.sh -l > Transaction > Submit > Tx put on chain !!```  - working

   
env:

-    getProtocolParams()
      Test: ```source /opt/cardano/cnode/scripts/env && getProtocolParams; echo "Exit code: $?"```  - working
-    test_koios()
      Test: ```source /opt/cardano/cnode/scripts/env && test_koios; echo "Exit code: $?"``` - working
     

It seems as though the new env headers are working properly in my tests. 
author_association: NONE
user: login: TrevorBenson
created_at: 2025-05-28T17:26:24Z
body: Great! I just did a quick review to refresh my memory. I don't see any other reason not to merge this.

I'll keep an eye out for the next commit which restores the KOIOS_API_HEADERS, and removes the single quotes. After that I'll approve/squash/merge into the current branch. Once complete I think were ready to transition the upstream PR from Draft to ready to review and get it into alpha.

----

After it merges upstream I'll generate the alpha/test containers in ghcr.io and update all my testnet nodes. I'll hit you up on TG with the image to pull for testing if you run preview/preprod or guild network pools/relays.

🚀 
author_association: COLLABORATOR
user: login: adamsthws
created_at: 2025-05-28T22:10:00Z
body:  
> I'll hit you up on TG with the image to pull for testing if you run preview/preprod or guild network pools/relays.

Fantastic. Looking forward to getting this over the line! I really appreciate all your time and support! 🙏
author_association: NONE
user: login: adamsthws
created_at: 2025-05-28T22:29:23Z
body: > After that I'll approve/squash/merge into the current branch. 

Quick request, I’d like it if my attributed contributions were retained. If squashing the commits, will my committer attribution be removed? If so, perhaps you might add me as a co-author to the squashed commit?

Cheers Trevor 🙏


author_association: NONE
user: login: TrevorBenson
created_at: 2025-05-29T21:07:17Z
body: > > After that I'll approve/squash/merge into the current branch.
> 
> Quick request, I’d like it if my attributed contributions were retained. If squashing the commits, will my committer attribution be removed? If so, perhaps you might add me as a co-author to the squashed commit?
> 
> Cheers Trevor 🙏

Squashing just combines them, but the squashed commit should still be linked back to you in the end. The upstream PR shows PR #6 merged and you retain the attribution:
![image](https://github.com/user-attachments/assets/3eac384f-b7e0-47bf-a254-c9c6214ea167)

Although I am not sure if anything will wake up the cexplorer folks to update the SPO list, but if they finally merge you into it then I may open my own PR to be added 😉.
author_association: COLLABORATOR
user: login: TrevorBenson
created_at: 2025-05-29T21:10:48Z
body: > If so, perhaps you might add me as a co-author to the squashed commit?

You end up being the sole author as the primary (of your PR #6 at least), and I don't believe I made any suggestions you could have clicked merge on. If I did I believe it **might** add me as the co-author when you click merge into your branch from a suggestion I make, then it gets squashed later. Or at least that's how I recall it happening. Although I do not recall if co-author attribution really shows up under github contribution timelines under your profile or in commit searches, or if its just "being honest" that its not a sole author.

EDIT: After reading the github page [Creating a commit with multiple authors](https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors) when the upstream PR gets squashed you should get full attribution as co-author of the work based on it including your email address, or no-reply email address, automatically.
author_association: COLLABORATOR
user: login: adamsthws
created_at: 2025-05-30T15:02:20Z
body: > the cexplorer folks to update the SPO list

I don't think we should hold our breath... Here was thier resoponse when I asked them about getting added to the list:

"The person who was managing lists in the past does not do so anymore - for a few months, maybe even years it hasn't been maintained. 
We've also decided to not take this feature into the upcoming V2 version (of cexplorer), at least not initially and in the same form it exists now, since it's not properly managed and thus not always recent reflecting the reality."

[Link to the question on Discord](https://discord.com/channels/959493368591826954/962284494281392188/1354003296116932661)
author_association: NONE
user: login: adamsthws
created_at: 2025-05-30T15:19:18Z
body: > When the upstream PR gets squashed you should get full attribution as co-author of the work automatically

This was an old github [issue ](https://github.com/isaacs/github/issues/1303)I was looking at recently

It looks like it may be fixed as mentioned in this equally old [article](https://github.blog/changelog/2019-12-19-improved-attribution-when-squashing-commits/)

Gotta get them magic internet points!… Nothing like watching your beautiful attributions vanish into the void, sacrificed at the altar of the upstream squashy! 😅

author_association: NONE
user: login: TrevorBenson
created_at: 2025-05-30T16:27:30Z
body: > Gotta get them magic internet points!… Nothing like watching your beautiful attributions vanish into the void, sacrificed at the altar of the upstream squashy! 😅

On that note, (totally tangential to this PR topic) if you ever have two GitHub accounts with separate attributions and decide to merge them, beware! 

**TL;DR**
My work account had nearly reached 1000 for a given year (I think in 2023?), while my personal was only a few hundred. Together they would have easily broken 1000. What I didn't know from reading the merging GitHub accounts documentation is there is a number of things that don't get merged. Tons of items which were not commits (I believe Pull Requests, Issues, and Comments) all were attributed to https://github.com/ghost after the merge. In the end my merged account had less total items than my work account had before the merge! 

All those internet points lost forever 😢 
author_association: COLLABORATOR