Repository: gdcc/dataverse-recipes
Pull request #17: Download script
Author: login: qqmyers
Created: 2025-05-02T17:15:29Z | Merged: 2025-05-07T15:56:41Z

By default the script should allow download of all public (non-restricted, non-embargoed, non-expired) files from the latest published version of a dataset, bypassing the zipping step. It retains file names and dirPaths. It works by retrieving the /dirindex page to start. 

The script also allows download of any files a user has access to from any version of the dataset and has restart capabilities. Instead of using wget's recursive capabilities (the initial design) it handles files and subdirs separately, keeping track if which files have been successfully downloaded. If the script is interrupted or starts getting download errors, e.g. from rate limiting, it quits. It can be restarted later to continue downloading files. The readme covers how to use the script and it's optional settings, etc.

The script has a way to throttle requests - wget allows a wait-time between file retrievals, so setting a wait time of 0.75 seconds should guarantee that there are less than 400 requests/5 minutes (Harvard's current limit afaik, though would delay unnecessarily for large files.).

Note: Jeremy Y @ Harvard reported errors in an earlier version when used with zsh. I made some modifications to help with that but don't have a way to test zsh/Homebrew easily.


Diff:
diff --git a/README.md b/README.md
index 2d12b6b..9dba7ba 100644
--- a/README.md
+++ b/README.md
@@ -35,6 +35,7 @@ In the following sections, you can find a list of available recipes for each lan
 ### Shell 🐚
 
 - [Upload files using S3 direct upload](shell/s3_direct_upload)
+- [Download files from a dataset](shell/download)
 
 ## 🤝 Contributing
 
diff --git a/shell/download/README.md b/shell/download/README.md
new file mode 100644
index 0000000..8d37a29
--- /dev/null
+++ b/shell/download/README.md
@@ -0,0 +1,62 @@
+# Dataverse Dataset Downloader
+
+## Purpose
+
+The Dataverse Dataset Downloader is a shell script that automates the process of downloading files from a Dataverse dataset and organizing them into their respective folders.
+It's particularly useful when the dataset has many files and/or a directory structure that needs to be preserved.
+
+
+## Features
+
+- Downloads files from the specified Dataverse dataset
+- Names and organizes files into their respective folders as in the specified dataset version
+- Provides progress feedback during the download
+- Tracks successful file downloads and can be restarted after interuption/failure where it left off
+- Supports downloading specific versions of a dataset including drafts
+- Allows access to restricted files with an API key
+- Option to continue downloading even if some files are forbidden, i.e. if you have access to some but not all files in the dataset
+- Configurable wait time between downloads to respect server rate limits
+
+## Prerequisites
+
+- Bash shell (Linux or macOS)
+- `wget`
+- `grep`
+- `sed`
+
+The script checks for these dependencies and will notify you if any are missing.
+
+## Usage
+./dv_downloader.sh <server> <persistentId> [--wait=<wait_time>] [--apikey=<api_key>] [--version=<version>] [--ignoreForbidden]
+
+### Parameters:
+
+- `<server>`: The base URL of the Dataverse server
+- `<persistentId>`: The persistent identifier of the dataset
+
+### Optional Parameters:
+
+- `--wait=<wait_time>`: Time in seconds to wait between file downloads (can be a fraction)
+- `--apikey=<api_key>`: API key for accessing restricted or embargoed files
+- `--version=<version>`: Specific version to download (e.g., '1.2' or ':draft')
+- `--ignoreForbidden`: Continue downloading even if some files return a 403 Forbidden error - required if you do not have access to all files in the dataset
+
+### Example:
+
+./dv_downloader.sh https://demo.dataverse.org doi:10.5072/F2ABCDEF --wait=0.75 --apikey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx --version=1.2 --ignoreForbidden
+
+## Output
+
+The script creates a new directory named after the persistent identifier (with forward slashes replaced by underscores) and organizes all downloaded files within this directory. If a specific version is requested, it's appended to the directory name.
+
+## Error Handling
+
+- The script will exit if it fails to download the main directory index
+- By default, it will exit if it encounters a 403 Forbidden error, unless the `--ignoreForbidden` flag is used
+- It provides informative error messages and suggests retrying in case of temporary network issues or rate limiting
+
+## Notes
+
+- The script resumes interrupted downloads, so you can safely re-run it if it gets interrupted. You may need to wait if the issue is rate limiting at the server, or you may need to add the ignoreForbidden flag if you do not have access to all files.
+- It keeps track of downloaded files to avoid unnecessary re-downloads. Deleting the directory for the dataset will also clear progress tracking.
+- Use the `--wait` option to add a delay between downloads if you're concerned about server load or rate limiting. <tracking period in minutes * 60)/<API calls allowed>, e.g. 0.75 seconds for a rate limit of 400 api calls over 5 minutes should be a conservative estimate of what's needed with small files.
diff --git a/shell/download/dv_downloader.sh b/shell/download/dv_downloader.sh
new file mode 100755
index 0000000..083b73a
--- /dev/null
+++ b/shell/download/dv_downloader.sh
@@ -0,0 +1,200 @@
+#!/bin/bash
+
+# Function to check if a command is available
+command_exists() {
+    command -v "$1" >/dev/null 2>&1
+}
+
+# Check for required commands
+required_commands=("wget" "grep" "sed")
+missing_commands=()
+
+for cmd in "${required_commands[@]}"; do
+    if ! command_exists "$cmd"; then
+        missing_commands+=("$cmd")
+    fi
+done
+
+if [ ${#missing_commands[@]} -ne 0 ]; then
+    echo "Error: The following required commands are not available:"
+    for cmd in "${missing_commands[@]}"; do
+        echo "  - $cmd"
+    done
+    echo "Please install these commands and try again."
+    exit 1
+fi
+
+print_usage() {
+    echo "Download the files from a Dataverse dataset"
+    echo "Usage: $0 <server> <persistentId> [--wait=<wait_time>] [--apikey=<api_key>] [--version=<version>] [--ignoreForbidden]"
+    echo "Example: $0 https://demo.dataverse.org doi:10.5072/F2ABCDEF --wait=0.75 --apikey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx --version=1.2 --ignoreForbidden"
+    echo ""
+    echo "Optional parameters:"
+    echo "  --wait=<wait_time>: Time in seconds to wait between file downloads (can be a fraction)."
+    echo "  --apikey=<api_key>: API key for accessing restricted or embargoed files"
+    echo "  --version=<version>: Specific version to download (e.g., '1.2' or ':draft')"
+    echo "  --ignoreForbidden: Continue downloading even if some files return a 403 Forbidden error"
+}
+
+# Check if the required parameters are provided
+if [ $# -lt 2 ]; then
+    print_usage
+    exit 1
+fi
+
+dvserver="$1"
+persistentId="$2"
+wait_time=0
+api_key=""
+version=""
+ignoreForbidden=false
+
+# Parse optional parameters
+shift 2
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --wait=*)
+        wait_time="${1#*=}"
+        shift
+        ;;
+        --apikey=*)
+        api_key="${1#*=}"
+        shift
+        ;;
+        --version=*)
+        version="${1#*=}"
+        shift
+        ;;
+        --ignoreForbidden)
+        ignoreForbidden=true
+        shift
+        ;;
+        *)
+        echo "Unknown parameter: $1"
+        print_usage
+        exit 1
+        ;;
+    esac
+done
+
+# Escape forward slashes in persistentId with underscores
+escaped_persistentId=$(echo "$persistentId" | tr '/' '_')
+
+# If version is set, append it to escaped_persistentId
+if [ -n "$version" ]; then
+    escaped_persistentId="${escaped_persistentId}_${version}"
+fi
+
+# Function to URL decode a string
+urldecode() {
+    printf '%b' "${1//%/\\x}"
+}
+
+# Function to download a file
+download_file() {
+    local url="$1"
+    local filename="$2"
+    
+    if [ -f "$filename" ] && grep -q "$filename" downloaded_files.txt; then
+        echo "Skipping already downloaded file: $filename"
+    else
+        local wget_cmd="wget -c --wait=$wait_time"
+        if [ -n "$api_key" ]; then
+            wget_cmd+=" --header=\"X-Dataverse-key: $api_key\""
+        fi
+        if eval $wget_cmd \"$url\" -O \"$filename\"; then
+            echo "$filename" >> downloaded_files.txt
+            echo "Successfully downloaded: $filename"
+        else
+            local exit_code=$?
+            if [ $exit_code -eq 8 ] && $ignoreForbidden; then
+                echo "Warning: Failed to download (403 Forbidden): $filename"
+                echo "Continuing due to --ignoreForbidden flag/assuming the file is restricted/embargoed/expired."
+            else
+                echo "Failed to download: $filename"
+                echo "Exiting. If the problem is a temporary network error or rate limiting, you can try this script in a few minutes."
+                echo "Do not delete the contents of the $escaped_persistentId directory and the script will continue where it left off."
+                exit 1
+            fi
+        fi
+    fi
+}
+
+# Change to the escaped_persistentId directory
+mkdir -p "$escaped_persistentId"
+cd "$escaped_persistentId" || exit 1
+
+# Execute wget command to get the main directory index
+if [ ! -f "dirindex" ]; then
+    wget_cmd="wget --wait=$wait_time"
+    if [ -n "$api_key" ]; then
+        wget_cmd+=" --header=\"X-Dataverse-key: $api_key\""
+    fi
+    url="$dvserver/api/datasets/:persistentId/dirindex?persistentId=$persistentId"
+    if [ -n "$version" ]; then
+        url+="&version=$version"
+    fi
+    if ! eval $wget_cmd \"$url\" -O dirindex; then
+        echo "Error: Failed to download the main directory index."
+        echo "Exiting script due to download failure."
+        exit 1
+    fi
+fi
+
+# Create or load the list of downloaded files
+touch downloaded_files.txt
+
+# Function to process an index file
+process_index() {
+    local index_file="$1"
+    local base_folder="$2"
+
+    while IFS= read -r line; do
+        if echo "$line" | grep -q 'href="/api/access/datafile/'; then
+            # Process files (this part remains largely unchanged)
+            local href=$(echo "$line" | sed -n 's/.*href="\([^"]*\)".*/\1/p')
+            local name=$(echo "$line" | sed -n 's/.*>\(.*\)<\/a>.*/\1/p')
+            local old_name=$(basename "$href")
+            local new_name=$(echo "$name" | tr -cd '[:alnum:]._-')
+            local folder="$base_folder"
+
+            if [ -n "$folder" ]; then
+                mkdir -p "$folder"
+            fi
+
+            download_file "$dvserver$href" "${folder:+$folder/}$new_name"
+
+            if [ -f "${folder:+$folder/}$new_name" ]; then
+                echo "Successfully downloaded: ${folder:+$folder/}$new_name"
+            else
+                echo "Warning: File '${folder:+$folder/}$new_name' not found."
+            fi
+        elif echo "$line" | grep -q 'href="/api/datasets/[0-9]*/dirindex/?.*folder='; then
+            # Process subdirectories
+            local subfolder=$(echo "$line" | sed -n 's/.*folder=\([^"]*\)".*/\1/p')
+            subfolder=$(urldecode "$subfolder")
+            local full_path="${base_folder:+$base_folder/}$subfolder"
+            
+            echo "Processing subdirectory: $full_path"
+            
+            # Download the subdirectory index
+            local subindex_file="index_${subfolder//\//_}.html"
+            download_file "$dvserver/api/datasets/:persistentId/dirindex?persistentId=$persistentId&folder=$subfolder" "$subindex_file"
+            
+            # Process the subdirectory
+            process_index "$subindex_file" "$full_path"
+            
+            # Clean up the subdirectory index file
+            rm "$subindex_file"
+        fi
+    done < "$index_file"
+}
+
+# Process the main dirindex file
+process_index "dirindex" ""
+
+# Clean up the main dirindex file
+rm "dirindex"
+rm "downloaded_files.txt"
+
+echo "File organization complete."
\ No newline at end of file


Reviews:
user: login: pdurbin
body: I didn't run the script but here's some quick feedback.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T13:51:19Z
user: login: qqmyers
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T14:40:59Z
user: login: qqmyers
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T14:41:51Z

Review comments:
path: shell/download/DVdownloader.sh
user: login: pdurbin
body: In #12 @JR-1991 and I developed some naming conventions. Can we rename this script to have underscores, like this:

dv_downloader.sh
created_at: 2025-05-07T13:45:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,61 @@
+# Dataverse Dataset Downloader
+
+## Purpose
+
+The Dataverse Dataset Downloader is a shell script that automates the process of downloading files from a Dataverse dataset and organizing them into their respective folders.
+It's particularly useful when the dataset has many files and/or a directory structure than needs to be preserved.
+
+## Prerequisites
+
+- Bash shell (Linux or macOS)
+- `wget`
+- `grep`
+- `sed`
+
+The script checks for these dependencies and will notify you if any are missing.
+
+## Usage
+./DVDownloader.sh <server> <persistentId> [--wait=<wait_time>] [--apikey=<api_key>] [--version=<version>] [--ignoreForbidden]
path: shell/download/README.md
user: login: pdurbin
body: This should match the name of the script.
created_at: 2025-05-07T13:46:22Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,61 @@
+# Dataverse Dataset Downloader
+
+## Purpose
+
+The Dataverse Dataset Downloader is a shell script that automates the process of downloading files from a Dataverse dataset and organizing them into their respective folders.
+It's particularly useful when the dataset has many files and/or a directory structure than needs to be preserved.
+
+## Prerequisites
+
+- Bash shell (Linux or macOS)
+- `wget`
+- `grep`
+- `sed`
+
+The script checks for these dependencies and will notify you if any are missing.
+
+## Usage
+./DVDownloader.sh <server> <persistentId> [--wait=<wait_time>] [--apikey=<api_key>] [--version=<version>] [--ignoreForbidden]
+
+### Parameters:
+
+- `<server>`: The base URL of the Dataverse server
+- `<persistentId>`: The persistent identifier of the dataset
+
+### Optional Parameters:
+
+- `--wait=<wait_time>`: Time in seconds to wait between file downloads (can be a fraction)
+- `--apikey=<api_key>`: API key for accessing restricted or embargoed files
+- `--version=<version>`: Specific version to download (e.g., '1.2' or ':draft')
+- `--ignoreForbidden`: Continue downloading even if some files return a 403 Forbidden error - required if you do not have access to all files in the dataset
+
+### Example:
+
+./DVDownload.sh https://demo.dataverse.org doi:10.5072/F2ABCDEF --wait=0.75 --apikey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx --version=1.2 --ignoreForbidden
path: shell/download/README.md
user: login: pdurbin
body: Same. The name should match.
created_at: 2025-05-07T13:46:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,61 @@
+# Dataverse Dataset Downloader
+
+## Purpose
+
+The Dataverse Dataset Downloader is a shell script that automates the process of downloading files from a Dataverse dataset and organizing them into their respective folders.
+It's particularly useful when the dataset has many files and/or a directory structure than needs to be preserved.
path: shell/download/README.md
user: login: pdurbin
body: ```suggestion
It's particularly useful when the dataset has many files and/or a directory structure that needs to be preserved.

```
created_at: 2025-05-07T13:48:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,61 @@
+# Dataverse Dataset Downloader
+
+## Purpose
+
+The Dataverse Dataset Downloader is a shell script that automates the process of downloading files from a Dataverse dataset and organizing them into their respective folders.
+It's particularly useful when the dataset has many files and/or a directory structure than needs to be preserved.
+
+## Prerequisites
+
+- Bash shell (Linux or macOS)
+- `wget`
+- `grep`
+- `sed`
path: shell/download/README.md
user: login: pdurbin
body: Does this work well on Mac? (I think @landreev may have tested it there.) I ask because it uses BSD sed which can be a bit different than GNU sed.

created_at: 2025-05-07T13:49:03Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,61 @@
+# Dataverse Dataset Downloader
+
+## Purpose
+
+The Dataverse Dataset Downloader is a shell script that automates the process of downloading files from a Dataverse dataset and organizing them into their respective folders.
+It's particularly useful when the dataset has many files and/or a directory structure than needs to be preserved.
+
+## Prerequisites
+
+- Bash shell (Linux or macOS)
+- `wget`
+- `grep`
+- `sed`
+
+The script checks for these dependencies and will notify you if any are missing.
+
+## Usage
+./DVDownloader.sh <server> <persistentId> [--wait=<wait_time>] [--apikey=<api_key>] [--version=<version>] [--ignoreForbidden]
+
+### Parameters:
+
+- `<server>`: The base URL of the Dataverse server
+- `<persistentId>`: The persistent identifier of the dataset
+
+### Optional Parameters:
+
+- `--wait=<wait_time>`: Time in seconds to wait between file downloads (can be a fraction)
+- `--apikey=<api_key>`: API key for accessing restricted or embargoed files
+- `--version=<version>`: Specific version to download (e.g., '1.2' or ':draft')
+- `--ignoreForbidden`: Continue downloading even if some files return a 403 Forbidden error - required if you do not have access to all files in the dataset
+
+### Example:
+
+./DVDownload.sh https://demo.dataverse.org doi:10.5072/F2ABCDEF --wait=0.75 --apikey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx --version=1.2 --ignoreForbidden
+
+## Features
+
+- Downloads files from the specified Dataverse dataset
+- Organizes files into their respective folders as structured in the original dataset
+- Renames files to their original names
+- Provides progress feedback during the download and organization process
+- Supports downloading specific versions of a dataset
+- Allows access to restricted or embargoed files with an API key
+- Option to continue downloading even if some files are forbidden, i.e. if you have access to some but not all files in the dataset
+- Configurable wait time between downloads to respect server rate limits
+
+## Output
+
+The script creates a new directory named after the persistent identifier (with forward slashes replaced by underscores) and organizes all downloaded files within this directory. If a specific version is requested, it's appended to the directory name.
+
+## Error Handling
+
+- The script will exit if it fails to download the main directory index
+- By default, it will exit if it encounters a 403 Forbidden error, unless the `--ignoreForbidden` flag is used
+- It provides informative error messages and suggests retrying in case of temporary network issues or rate limiting
path: shell/download/README.md
user: login: pdurbin
body: If resuming the download is a core feature (great!), should it be mentioned higher up?
created_at: 2025-05-07T13:50:52Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,61 @@
+# Dataverse Dataset Downloader
+
+## Purpose
+
+The Dataverse Dataset Downloader is a shell script that automates the process of downloading files from a Dataverse dataset and organizing them into their respective folders.
+It's particularly useful when the dataset has many files and/or a directory structure than needs to be preserved.
+
+## Prerequisites
+
+- Bash shell (Linux or macOS)
+- `wget`
+- `grep`
+- `sed`
+
+The script checks for these dependencies and will notify you if any are missing.
+
+## Usage
+./DVDownloader.sh <server> <persistentId> [--wait=<wait_time>] [--apikey=<api_key>] [--version=<version>] [--ignoreForbidden]
+
+### Parameters:
+
+- `<server>`: The base URL of the Dataverse server
+- `<persistentId>`: The persistent identifier of the dataset
+
+### Optional Parameters:
+
+- `--wait=<wait_time>`: Time in seconds to wait between file downloads (can be a fraction)
+- `--apikey=<api_key>`: API key for accessing restricted or embargoed files
+- `--version=<version>`: Specific version to download (e.g., '1.2' or ':draft')
+- `--ignoreForbidden`: Continue downloading even if some files return a 403 Forbidden error - required if you do not have access to all files in the dataset
+
+### Example:
+
+./DVDownload.sh https://demo.dataverse.org doi:10.5072/F2ABCDEF --wait=0.75 --apikey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx --version=1.2 --ignoreForbidden
+
+## Features
+
+- Downloads files from the specified Dataverse dataset
+- Organizes files into their respective folders as structured in the original dataset
+- Renames files to their original names
+- Provides progress feedback during the download and organization process
+- Supports downloading specific versions of a dataset
+- Allows access to restricted or embargoed files with an API key
+- Option to continue downloading even if some files are forbidden, i.e. if you have access to some but not all files in the dataset
+- Configurable wait time between downloads to respect server rate limits
+
+## Output
+
+The script creates a new directory named after the persistent identifier (with forward slashes replaced by underscores) and organizes all downloaded files within this directory. If a specific version is requested, it's appended to the directory name.
+
+## Error Handling
+
+- The script will exit if it fails to download the main directory index
+- By default, it will exit if it encounters a 403 Forbidden error, unless the `--ignoreForbidden` flag is used
+- It provides informative error messages and suggests retrying in case of temporary network issues or rate limiting
path: shell/download/README.md
user: login: qqmyers
body: moved features up and added a restart line
created_at: 2025-05-07T14:40:59Z
author_association: MEMBER
path: shell/download/DVdownloader.sh
user: login: qqmyers
body: renamed.
created_at: 2025-05-07T14:41:51Z
author_association: MEMBER

PR comments:
user: login: landreev
created_at: 2025-05-06T23:39:18Z
body: On MacOS: The only reason it did not work for me out of the box were the DOS-style, 2-character (`\r\n`) newlines: 

```
% ./DVdownloader.sh
zsh: ./DVdownloader.sh: bad interpreter: /bin/bash^M: no such file or directory
```
converted to unix-style newlines it works just fine: 

```
perl -npe 's/\r//g;' < DVdownloader.sh > DVdownloader-nodos.sh
chmod +x DVdownloader-nodos.sh
./DVdownloader-nodos.sh http://localhost:8080 doi:10.70122/FK2/CGW9P1
...
```
(I downloaded the script from github via ```wget https://raw.githubusercontent.com/gdcc/dataverse-recipes/download/shell/download/DVdownloader.sh``` - but I can check whether the extra newline characters are still there if I clone the repo and check out the branch locally)
This unhappiness with the extra `\r` in the interpreter line does not seem to be unique to zsh.
author_association: NONE
user: login: landreev
created_at: 2025-05-07T00:17:55Z
body: OK, I see, it's not relying on `wget --content-disposition`, because an extra mv command would still be needed to move the downloaded file into the proper folder. 
... But, why not save it under the real file name right away - as in, `wget -O "${folder:+$folder/}$new_name"`? - no need to rename it then. 

(or are the file ids better somehow for keeping track of which files have already been downloaded?)
author_association: NONE
user: login: landreev
created_at: 2025-05-08T16:34:20Z
body: (also wanted to mention that the --wait option is most appreciated)
author_association: NONE
user: login: pdurbin
created_at: 2025-05-08T18:25:40Z
body: For the record, in the end I did run the script and it worked fine. This is what I did:

```
./dv_downloader.sh https://dataverse.harvard.edu doi:10.7910/DVN/TJCLKP
```
author_association: MEMBER