Repository: The-Strategy-Unit/data_science
Pull request #291: Added addendum to storing data blog to add reading from Sharepoint
Author: login: DCEW
Created: 2025-04-03T13:06:26Z | Merged: 2025-04-04T10:06:28Z



Diff:
diff --git a/blogs/posts/2024-05-22_storing-data-safely/Follow.jpg b/blogs/posts/2024-05-22_storing-data-safely/Follow.jpg
new file mode 100644
index 00000000..93af97e0
Binary files /dev/null and b/blogs/posts/2024-05-22_storing-data-safely/Follow.jpg differ
diff --git a/blogs/posts/2024-05-22_storing-data-safely/index.qmd b/blogs/posts/2024-05-22_storing-data-safely/index.qmd
index 540295b9..141332d9 100644
--- a/blogs/posts/2024-05-22_storing-data-safely/index.qmd
+++ b/blogs/posts/2024-05-22_storing-data-safely/index.qmd
@@ -1,234 +1,276 @@
----
-title: "Storing data safely"
-author: 
-  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
-  - "[Matt Dray](mailto:matt.dray@nhs.net)"
-date: "2024-05-22"
-categories: [learning, R, Python]
-notebook-view:
-  - notebook: azure_python.ipynb
-    title: "Interacting with Azure Storage with Python"
-
----
-
-```{r lockfile}
-#| include: FALSE
-renv::use(lockfile = "renv.lock")
-```
-
-
-## Coffee & Coding
-
-In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
-
-In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
-
-## Posit Connect Pins
-
-
-```{r}
-#| eval: false
-
-
-# A brief intro to using {pins} to store, version, share and protect a dataset
-# on Posit Connect. Documentation: https://pins.rstudio.com/
-
-
-# Setup -------------------------------------------------------------------
-
-
-install.packages(c("pins", "dplyr")) # if not yet installed
-
-suppressPackageStartupMessages({
-  library(pins)
-  library(dplyr) # for wrangling and the 'starwars' demo dataset
-})
-
-board <- board_connect() # will error if you haven't authenticated before
-# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
-# • `server` and `key` not provided for `auth = 'manual'`
-# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
-# • rsconnect package not installed for `auth = 'rsconnect'`
-# Run `rlang::last_trace()` to see where the error occurred.
-
-# To authenticate
-# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
-# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
-# Your browser will open to the Posit Connect web page and you're prompted to
-# for your password. Enter it and you'll be authenticated.
-
-# Once authenticated
-board <- board_connect()
-# Connecting to Posit Connect 2024.03.0 at
-# <https://connect.strategyunitwm.nhs.uk>
-
-board |> pin_list() # see all the pins on that board
-
-
-# Create a pin ------------------------------------------------------------
-
-
-# Write a dataset to the board as a pin
-board |> pin_write(
-  x = starwars,
-  name = "starwars_demo"
-)
-# Guessing `type = 'rds'`
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_exists("starwars_demo")
-# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
-# not "starwars_demo".
-# [1] TRUE
-
-pin_name <- "matt.dray/starwars_demo"
-
-board |> pin_exists(pin_name) # logical, TRUE/FALSE
-board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
-board |> pin_browse(pin_name) # view the pin in the browser
-
-
-# Permissions -------------------------------------------------------------
-
-
-# You can let people see and edit a pin. Log into Posit Connect and select the
-# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
-# the 'sharing' options in the 'Access' tab.
-
-
-# Overwrite and version ---------------------------------------------------
-
-
-starwars_droids <- starwars |>
-  filter(species == "Droid") # beep boop
-
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_versions(pin_name) # see version history
-board |> pin_versions_prune(pin_name, n = 1) # remove history
-board |> pin_versions(pin_name)
-
-# What if you try to overwrite the data but it hasn't changed?
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# ! The hash of pin "matt.dray/starwars_demo" has not changed.
-# • Your pin will not be stored.
-
-
-# Use the pin -------------------------------------------------------------
-
-
-# You can read a pin to your local machine, or access it from a Quarto file
-# or Shiny app hosted on Connect, for example. If the output and the pin are
-# both on Connect, no authentication is required; the board is defaulted to
-# the Posit Connect instance where they're both hosted.
-
-board |>
-  pin_read(pin_name) |> # like you would use e.g. read_csv
-  with(data = _, plot(mass, height)) # wow!
-
-
-# Delete pin --------------------------------------------------------------
-
-
-board |> pin_exists(pin_name) # logical, good function for error handling
-board |> pin_delete(pin_name)
-board |> pin_exists(pin_name)
-
-```
-
-## Azure Storage in R
-
-You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
-
-Below is a sample .Renviron file
-
-
-```
-AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
-AZ_STORAGE_CONTAINER=container-name
-AZ_TENANT_ID=long-sequence-of-numbers-and-letters
-AZ_APP_ID=another-long-sequence-of-numbers-and-letters
-```
-
-```{r}
-#| eval: false
-
-install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
-
-# Load all environment variables
-ep_uri <- Sys.getenv("AZ_STORAGE_EP")
-app_id <- Sys.getenv("AZ_APP_ID")
-container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
-tenant <- Sys.getenv("AZ_TENANT_ID")
-
-# Authenticate
-token <- AzureAuth::get_azure_token(
-  "https://storage.azure.com",
-  tenant = tenant,
-  app = app_id,
-  auth_type = "device_code",
-)
-
-# If you have not authenticated before, you will be taken to an external page to
-# authenticate!Use your mlcsu.nhs.uk account.
-
-# Connect to container
-endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
-container <- AzureStor::storage_container(endpoint, container_name)
-
-# List files in container
-blob_list <- AzureStor::list_blobs(container)
-
-# If you get a 403 error when trying to interact with the container, you may
-# have to clear your Azure token and re-authenticate using a different browser.
-# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
-# AzureAuth::get_azure_token() step above.
-
-# Upload specific file to container
-AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
-
-# Upload contents of a local directory to container
-AzureStor::storage_multiupload(container, "data/*", "newdir")
-
-# Check files have uploaded
-blob_list <- AzureStor::list_blobs(container)
-
-# Load file directly from Azure container
-df_from_azure <- AzureStor::storage_read_csv(
-  container,
-  "newdir/cats.csv",
-  show_col_types = FALSE
-)
-
-# Load file directly from Azure container (by temporarily downloading file
-# and storing it in memory)
-parquet_in_memory <- AzureStor::storage_download(
-  container,
-  src = "newdir/cats.parquet", dest = NULL
-)
-parq_df <- arrow::read_parquet(parquet_in_memory)
-
-# Delete from Azure container (!!!)
-for (blobfile in blob_list$name) {
-  AzureStor::delete_storage_file(container, blobfile)
-}
-```
-
-## Azure Storage in Python
-
-This will use the same environment variables as the R version, just stored in a .env file instead. 
-
-We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
-
-{{< embed azure_python.ipynb echo=true   >}}
-
-
+---
+title: "Storing data safely"
+author: 
+  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
+  - "[Matt Dray](mailto:matt.dray@nhs.net)"
+  - "[Claire Welsh](mailto:claire.welsh8@nhs.net)"
+date: "2024-05-22"
+categories: [learning, R, Python]
+notebook-view:
+  - notebook: azure_python.ipynb
+    title: "Interacting with Azure Storage with Python"
+---
+
+```{r lockfile}
+#| include: FALSE
+renv::use(lockfile = "renv.lock")
+```
+
+::: callout-note
+**UPDATED**: Please see the Addendum to this blog, added 2025-04-03 regarding accessing data from SharePoint
+:::
+
+## Coffee & Coding
+
+In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
+
+In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
+
+## Posit Connect Pins
+
+```{r}
+#| eval: false
+
+
+# A brief intro to using {pins} to store, version, share and protect a dataset
+# on Posit Connect. Documentation: https://pins.rstudio.com/
+
+
+# Setup -------------------------------------------------------------------
+
+
+install.packages(c("pins", "dplyr")) # if not yet installed
+
+suppressPackageStartupMessages({
+  library(pins)
+  library(dplyr) # for wrangling and the 'starwars' demo dataset
+})
+
+board <- board_connect() # will error if you haven't authenticated before
+# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
+# • `server` and `key` not provided for `auth = 'manual'`
+# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
+# • rsconnect package not installed for `auth = 'rsconnect'`
+# Run `rlang::last_trace()` to see where the error occurred.
+
+# To authenticate
+# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
+# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
+# Your browser will open to the Posit Connect web page and you're prompted to
+# for your password. Enter it and you'll be authenticated.
+
+# Once authenticated
+board <- board_connect()
+# Connecting to Posit Connect 2024.03.0 at
+# <https://connect.strategyunitwm.nhs.uk>
+
+board |> pin_list() # see all the pins on that board
+
+
+# Create a pin ------------------------------------------------------------
+
+
+# Write a dataset to the board as a pin
+board |> pin_write(
+  x = starwars,
+  name = "starwars_demo"
+)
+# Guessing `type = 'rds'`
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_exists("starwars_demo")
+# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
+# not "starwars_demo".
+# [1] TRUE
+
+pin_name <- "matt.dray/starwars_demo"
+
+board |> pin_exists(pin_name) # logical, TRUE/FALSE
+board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
+board |> pin_browse(pin_name) # view the pin in the browser
+
+
+# Permissions -------------------------------------------------------------
+
+
+# You can let people see and edit a pin. Log into Posit Connect and select the
+# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
+# the 'sharing' options in the 'Access' tab.
+
+
+# Overwrite and version ---------------------------------------------------
+
+
+starwars_droids <- starwars |>
+  filter(species == "Droid") # beep boop
+
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_versions(pin_name) # see version history
+board |> pin_versions_prune(pin_name, n = 1) # remove history
+board |> pin_versions(pin_name)
+
+# What if you try to overwrite the data but it hasn't changed?
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# ! The hash of pin "matt.dray/starwars_demo" has not changed.
+# • Your pin will not be stored.
+
+
+# Use the pin -------------------------------------------------------------
+
+
+# You can read a pin to your local machine, or access it from a Quarto file
+# or Shiny app hosted on Connect, for example. If the output and the pin are
+# both on Connect, no authentication is required; the board is defaulted to
+# the Posit Connect instance where they're both hosted.
+
+board |>
+  pin_read(pin_name) |> # like you would use e.g. read_csv
+  with(data = _, plot(mass, height)) # wow!
+
+
+# Delete pin --------------------------------------------------------------
+
+
+board |> pin_exists(pin_name) # logical, good function for error handling
+board |> pin_delete(pin_name)
+board |> pin_exists(pin_name)
+
+```
+
+## Azure Storage in R
+
+You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
+
+Below is a sample .Renviron file
+
+```         
+AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
+AZ_STORAGE_CONTAINER=container-name
+AZ_TENANT_ID=long-sequence-of-numbers-and-letters
+AZ_APP_ID=another-long-sequence-of-numbers-and-letters
+```
+
+```{r}
+#| eval: false
+
+install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
+
+# Load all environment variables
+ep_uri <- Sys.getenv("AZ_STORAGE_EP")
+app_id <- Sys.getenv("AZ_APP_ID")
+container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
+tenant <- Sys.getenv("AZ_TENANT_ID")
+
+# Authenticate
+token <- AzureAuth::get_azure_token(
+  "https://storage.azure.com",
+  tenant = tenant,
+  app = app_id,
+  auth_type = "device_code",
+)
+
+# If you have not authenticated before, you will be taken to an external page to
+# authenticate!Use your mlcsu.nhs.uk account.
+
+# Connect to container
+endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
+container <- AzureStor::storage_container(endpoint, container_name)
+
+# List files in container
+blob_list <- AzureStor::list_blobs(container)
+
+# If you get a 403 error when trying to interact with the container, you may
+# have to clear your Azure token and re-authenticate using a different browser.
+# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
+# AzureAuth::get_azure_token() step above.
+
+# Upload specific file to container
+AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
+
+# Upload contents of a local directory to container
+AzureStor::storage_multiupload(container, "data/*", "newdir")
+
+# Check files have uploaded
+blob_list <- AzureStor::list_blobs(container)
+
+# Load file directly from Azure container
+df_from_azure <- AzureStor::storage_read_csv(
+  container,
+  "newdir/cats.csv",
+  show_col_types = FALSE
+)
+
+# Load file directly from Azure container (by temporarily downloading file
+# and storing it in memory)
+parquet_in_memory <- AzureStor::storage_download(
+  container,
+  src = "newdir/cats.parquet", dest = NULL
+)
+parq_df <- arrow::read_parquet(parquet_in_memory)
+
+# Delete from Azure container (!!!)
+for (blobfile in blob_list$name) {
+  AzureStor::delete_storage_file(container, blobfile)
+}
+```
+
+## Azure Storage in Python
+
+This will use the same environment variables as the R version, just stored in a .env file instead.
+
+We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
+
+{{< embed azure_python.ipynb echo=true >}}
+
+## ADDENDUM: Accessing data from SharePoint
+
+SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.
+
+Accessing SharePoint requires user authentication, which you'll be prompted for in the browser when you try to access SharePoint from R. Note that you must 'follow' a SharePoint site before you can fetch its content.
+
+To access data on SharePoint, follow these steps:
+
+1. Navigate to the SharePoint page that has the file of interest in it, using your browser.
+
+2. Click the small star in the top right corner of the window, labelled 'Follow'.
+
+![](Follow.jpg){fig-alt="A screenshot of the upper-right corner of a SharePoint site in a web browser, highlighting a star-shaped toggle labelled 'follow'."}
+
+3. Open (or create) your project's `.Renviron` file, used for [storing environmental variables](https://docs.posit.co/ide/user/ide/guide/environments/r/managing-r.html#renviron). You can do this by running `r usethis::edit_r_environ()` from the R console.
+
+4. Save two new environment variables to the `.Renviron` file:
+    a. `SP_SITE_NAME`: the name of the site (i.e. the name at the top of the browser screen)
+    b. `SP_FILE_PATH`: the full file path from below the `Documents/` folder to the file itself, including the file type extension. 
+5. Save and close the `.Renviron` file then either restart your R session (CTRL + Shift + F10 on Windows machines) or run `r readRenviron(".Renviron")` to make the new variables available in your session.
+
+5. Use the code below to read in an Excel file (xlsx) into R memory, or adapt it to read other file types.
+
+```{r}
+#| eval: false
+#| echo: true
+
+# Read in the securely saved path and site variables
+sharepoint_site <- Sys.getenv("SP_SITE_NAME")
+template_path <- Sys.getenv("SP_FILE_PATH")
+
+# access the dataset and save it into a temporary file
+site <- Microsoft365R::get_sharepoint_site(sharepoint_site)
+drv <- site$get_drive()
+tmp_file <- tempfile(fileext = ".xlsx")
+drv$download_file(template_path, dest = tmp_file)
+
+data <- readxl::read_xlsx(tmp_file)
+
+#tidy up
+unlink(tmp_file)
+```


Reviews:
user: login: matt-dray
body: Thanks for adding this; it's definitely good to write this down given the confusion we had earlier in the week re 'following' a SharePoint site!

Thought I'd review, given this was originally a C&C session by @yiwen-h (who may have suggestions) and me. I've tweaked a few things for clarity, basically (some more subtle than others).
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T09:43:05Z
user: login: ChrisBeeley
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-04T09:57:20Z
user: login: matt-dray
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T10:05:13Z

Review comments:
diff_hunk: @@ -1,234 +1,274 @@
----
-title: "Storing data safely"
-author: 
-  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
-  - "[Matt Dray](mailto:matt.dray@nhs.net)"
-date: "2024-05-22"
-categories: [learning, R, Python]
-notebook-view:
-  - notebook: azure_python.ipynb
-    title: "Interacting with Azure Storage with Python"
-
----
-
-```{r lockfile}
-#| include: FALSE
-renv::use(lockfile = "renv.lock")
-```
-
-
-## Coffee & Coding
-
-In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
-
-In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
-
-## Posit Connect Pins
-
-
-```{r}
-#| eval: false
-
-
-# A brief intro to using {pins} to store, version, share and protect a dataset
-# on Posit Connect. Documentation: https://pins.rstudio.com/
-
-
-# Setup -------------------------------------------------------------------
-
-
-install.packages(c("pins", "dplyr")) # if not yet installed
-
-suppressPackageStartupMessages({
-  library(pins)
-  library(dplyr) # for wrangling and the 'starwars' demo dataset
-})
-
-board <- board_connect() # will error if you haven't authenticated before
-# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
-# • `server` and `key` not provided for `auth = 'manual'`
-# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
-# • rsconnect package not installed for `auth = 'rsconnect'`
-# Run `rlang::last_trace()` to see where the error occurred.
-
-# To authenticate
-# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
-# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
-# Your browser will open to the Posit Connect web page and you're prompted to
-# for your password. Enter it and you'll be authenticated.
-
-# Once authenticated
-board <- board_connect()
-# Connecting to Posit Connect 2024.03.0 at
-# <https://connect.strategyunitwm.nhs.uk>
-
-board |> pin_list() # see all the pins on that board
-
-
-# Create a pin ------------------------------------------------------------
-
-
-# Write a dataset to the board as a pin
-board |> pin_write(
-  x = starwars,
-  name = "starwars_demo"
-)
-# Guessing `type = 'rds'`
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_exists("starwars_demo")
-# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
-# not "starwars_demo".
-# [1] TRUE
-
-pin_name <- "matt.dray/starwars_demo"
-
-board |> pin_exists(pin_name) # logical, TRUE/FALSE
-board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
-board |> pin_browse(pin_name) # view the pin in the browser
-
-
-# Permissions -------------------------------------------------------------
-
-
-# You can let people see and edit a pin. Log into Posit Connect and select the
-# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
-# the 'sharing' options in the 'Access' tab.
-
-
-# Overwrite and version ---------------------------------------------------
-
-
-starwars_droids <- starwars |>
-  filter(species == "Droid") # beep boop
-
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_versions(pin_name) # see version history
-board |> pin_versions_prune(pin_name, n = 1) # remove history
-board |> pin_versions(pin_name)
-
-# What if you try to overwrite the data but it hasn't changed?
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# ! The hash of pin "matt.dray/starwars_demo" has not changed.
-# • Your pin will not be stored.
-
-
-# Use the pin -------------------------------------------------------------
-
-
-# You can read a pin to your local machine, or access it from a Quarto file
-# or Shiny app hosted on Connect, for example. If the output and the pin are
-# both on Connect, no authentication is required; the board is defaulted to
-# the Posit Connect instance where they're both hosted.
-
-board |>
-  pin_read(pin_name) |> # like you would use e.g. read_csv
-  with(data = _, plot(mass, height)) # wow!
-
-
-# Delete pin --------------------------------------------------------------
-
-
-board |> pin_exists(pin_name) # logical, good function for error handling
-board |> pin_delete(pin_name)
-board |> pin_exists(pin_name)
-
-```
-
-## Azure Storage in R
-
-You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
-
-Below is a sample .Renviron file
-
-
-```
-AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
-AZ_STORAGE_CONTAINER=container-name
-AZ_TENANT_ID=long-sequence-of-numbers-and-letters
-AZ_APP_ID=another-long-sequence-of-numbers-and-letters
-```
-
-```{r}
-#| eval: false
-
-install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
-
-# Load all environment variables
-ep_uri <- Sys.getenv("AZ_STORAGE_EP")
-app_id <- Sys.getenv("AZ_APP_ID")
-container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
-tenant <- Sys.getenv("AZ_TENANT_ID")
-
-# Authenticate
-token <- AzureAuth::get_azure_token(
-  "https://storage.azure.com",
-  tenant = tenant,
-  app = app_id,
-  auth_type = "device_code",
-)
-
-# If you have not authenticated before, you will be taken to an external page to
-# authenticate!Use your mlcsu.nhs.uk account.
-
-# Connect to container
-endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
-container <- AzureStor::storage_container(endpoint, container_name)
-
-# List files in container
-blob_list <- AzureStor::list_blobs(container)
-
-# If you get a 403 error when trying to interact with the container, you may
-# have to clear your Azure token and re-authenticate using a different browser.
-# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
-# AzureAuth::get_azure_token() step above.
-
-# Upload specific file to container
-AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
-
-# Upload contents of a local directory to container
-AzureStor::storage_multiupload(container, "data/*", "newdir")
-
-# Check files have uploaded
-blob_list <- AzureStor::list_blobs(container)
-
-# Load file directly from Azure container
-df_from_azure <- AzureStor::storage_read_csv(
-  container,
-  "newdir/cats.csv",
-  show_col_types = FALSE
-)
-
-# Load file directly from Azure container (by temporarily downloading file
-# and storing it in memory)
-parquet_in_memory <- AzureStor::storage_download(
-  container,
-  src = "newdir/cats.parquet", dest = NULL
-)
-parq_df <- arrow::read_parquet(parquet_in_memory)
-
-# Delete from Azure container (!!!)
-for (blobfile in blob_list$name) {
-  AzureStor::delete_storage_file(container, blobfile)
-}
-```
-
-## Azure Storage in Python
-
-This will use the same environment variables as the R version, just stored in a .env file instead. 
-
-We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
-
-{{< embed azure_python.ipynb echo=true   >}}
-
-
+---
+title: "Storing data safely"
+author: 
+  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
+  - "[Matt Dray](mailto:matt.dray@nhs.net)"
+  - "[Claire Welsh](mailto:claire.welsh8@nhs.net)"
+date: "2024-05-22"
+categories: [learning, R, Python]
+notebook-view:
+  - notebook: azure_python.ipynb
+    title: "Interacting with Azure Storage with Python"
+---
+
+```{r lockfile}
+#| include: FALSE
+renv::use(lockfile = "renv.lock")
+```
+
+::: callout-note
+**UPDATED**: Please see the Addendum to this blog, added 2025-04-03 regarding accessing data from SharePoint
+:::
+
+## Coffee & Coding
+
+In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
+
+In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
+
+## Posit Connect Pins
+
+```{r}
+#| eval: false
+
+
+# A brief intro to using {pins} to store, version, share and protect a dataset
+# on Posit Connect. Documentation: https://pins.rstudio.com/
+
+
+# Setup -------------------------------------------------------------------
+
+
+install.packages(c("pins", "dplyr")) # if not yet installed
+
+suppressPackageStartupMessages({
+  library(pins)
+  library(dplyr) # for wrangling and the 'starwars' demo dataset
+})
+
+board <- board_connect() # will error if you haven't authenticated before
+# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
+# • `server` and `key` not provided for `auth = 'manual'`
+# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
+# • rsconnect package not installed for `auth = 'rsconnect'`
+# Run `rlang::last_trace()` to see where the error occurred.
+
+# To authenticate
+# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
+# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
+# Your browser will open to the Posit Connect web page and you're prompted to
+# for your password. Enter it and you'll be authenticated.
+
+# Once authenticated
+board <- board_connect()
+# Connecting to Posit Connect 2024.03.0 at
+# <https://connect.strategyunitwm.nhs.uk>
+
+board |> pin_list() # see all the pins on that board
+
+
+# Create a pin ------------------------------------------------------------
+
+
+# Write a dataset to the board as a pin
+board |> pin_write(
+  x = starwars,
+  name = "starwars_demo"
+)
+# Guessing `type = 'rds'`
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_exists("starwars_demo")
+# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
+# not "starwars_demo".
+# [1] TRUE
+
+pin_name <- "matt.dray/starwars_demo"
+
+board |> pin_exists(pin_name) # logical, TRUE/FALSE
+board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
+board |> pin_browse(pin_name) # view the pin in the browser
+
+
+# Permissions -------------------------------------------------------------
+
+
+# You can let people see and edit a pin. Log into Posit Connect and select the
+# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
+# the 'sharing' options in the 'Access' tab.
+
+
+# Overwrite and version ---------------------------------------------------
+
+
+starwars_droids <- starwars |>
+  filter(species == "Droid") # beep boop
+
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_versions(pin_name) # see version history
+board |> pin_versions_prune(pin_name, n = 1) # remove history
+board |> pin_versions(pin_name)
+
+# What if you try to overwrite the data but it hasn't changed?
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# ! The hash of pin "matt.dray/starwars_demo" has not changed.
+# • Your pin will not be stored.
+
+
+# Use the pin -------------------------------------------------------------
+
+
+# You can read a pin to your local machine, or access it from a Quarto file
+# or Shiny app hosted on Connect, for example. If the output and the pin are
+# both on Connect, no authentication is required; the board is defaulted to
+# the Posit Connect instance where they're both hosted.
+
+board |>
+  pin_read(pin_name) |> # like you would use e.g. read_csv
+  with(data = _, plot(mass, height)) # wow!
+
+
+# Delete pin --------------------------------------------------------------
+
+
+board |> pin_exists(pin_name) # logical, good function for error handling
+board |> pin_delete(pin_name)
+board |> pin_exists(pin_name)
+
+```
+
+## Azure Storage in R
+
+You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
+
+Below is a sample .Renviron file
+
+```         
+AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
+AZ_STORAGE_CONTAINER=container-name
+AZ_TENANT_ID=long-sequence-of-numbers-and-letters
+AZ_APP_ID=another-long-sequence-of-numbers-and-letters
+```
+
+```{r}
+#| eval: false
+
+install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
+
+# Load all environment variables
+ep_uri <- Sys.getenv("AZ_STORAGE_EP")
+app_id <- Sys.getenv("AZ_APP_ID")
+container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
+tenant <- Sys.getenv("AZ_TENANT_ID")
+
+# Authenticate
+token <- AzureAuth::get_azure_token(
+  "https://storage.azure.com",
+  tenant = tenant,
+  app = app_id,
+  auth_type = "device_code",
+)
+
+# If you have not authenticated before, you will be taken to an external page to
+# authenticate!Use your mlcsu.nhs.uk account.
+
+# Connect to container
+endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
+container <- AzureStor::storage_container(endpoint, container_name)
+
+# List files in container
+blob_list <- AzureStor::list_blobs(container)
+
+# If you get a 403 error when trying to interact with the container, you may
+# have to clear your Azure token and re-authenticate using a different browser.
+# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
+# AzureAuth::get_azure_token() step above.
+
+# Upload specific file to container
+AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
+
+# Upload contents of a local directory to container
+AzureStor::storage_multiupload(container, "data/*", "newdir")
+
+# Check files have uploaded
+blob_list <- AzureStor::list_blobs(container)
+
+# Load file directly from Azure container
+df_from_azure <- AzureStor::storage_read_csv(
+  container,
+  "newdir/cats.csv",
+  show_col_types = FALSE
+)
+
+# Load file directly from Azure container (by temporarily downloading file
+# and storing it in memory)
+parquet_in_memory <- AzureStor::storage_download(
+  container,
+  src = "newdir/cats.parquet", dest = NULL
+)
+parq_df <- arrow::read_parquet(parquet_in_memory)
+
+# Delete from Azure container (!!!)
+for (blobfile in blob_list$name) {
+  AzureStor::delete_storage_file(container, blobfile)
+}
+```
+
+## Azure Storage in Python
+
+This will use the same environment variables as the R version, just stored in a .env file instead.
+
+We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
+
+{{< embed azure_python.ipynb echo=true >}}
+
+## ADDENDUM: Accessing data from SharePoint
+
+SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.
+
+Accessing SharePoint requires user authentication, which also needs to happen when reading data into R from SharePoint. Before you can access SharePoint content from a site, you first need to 'follow' that site.
path: blogs/posts/2024-05-22_storing-data-safely/index.qmd
user: login: matt-dray
body: Subtle tweaks.
```suggestion
Accessing SharePoint requires user authentication, which you'll be prompted for in the browser when you try to access SharePoint from R. Note that you must 'follow' a SharePoint site before you can fetch its content.
```
created_at: 2025-04-04T09:18:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,234 +1,274 @@
----
-title: "Storing data safely"
-author: 
-  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
-  - "[Matt Dray](mailto:matt.dray@nhs.net)"
-date: "2024-05-22"
-categories: [learning, R, Python]
-notebook-view:
-  - notebook: azure_python.ipynb
-    title: "Interacting with Azure Storage with Python"
-
----
-
-```{r lockfile}
-#| include: FALSE
-renv::use(lockfile = "renv.lock")
-```
-
-
-## Coffee & Coding
-
-In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
-
-In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
-
-## Posit Connect Pins
-
-
-```{r}
-#| eval: false
-
-
-# A brief intro to using {pins} to store, version, share and protect a dataset
-# on Posit Connect. Documentation: https://pins.rstudio.com/
-
-
-# Setup -------------------------------------------------------------------
-
-
-install.packages(c("pins", "dplyr")) # if not yet installed
-
-suppressPackageStartupMessages({
-  library(pins)
-  library(dplyr) # for wrangling and the 'starwars' demo dataset
-})
-
-board <- board_connect() # will error if you haven't authenticated before
-# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
-# • `server` and `key` not provided for `auth = 'manual'`
-# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
-# • rsconnect package not installed for `auth = 'rsconnect'`
-# Run `rlang::last_trace()` to see where the error occurred.
-
-# To authenticate
-# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
-# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
-# Your browser will open to the Posit Connect web page and you're prompted to
-# for your password. Enter it and you'll be authenticated.
-
-# Once authenticated
-board <- board_connect()
-# Connecting to Posit Connect 2024.03.0 at
-# <https://connect.strategyunitwm.nhs.uk>
-
-board |> pin_list() # see all the pins on that board
-
-
-# Create a pin ------------------------------------------------------------
-
-
-# Write a dataset to the board as a pin
-board |> pin_write(
-  x = starwars,
-  name = "starwars_demo"
-)
-# Guessing `type = 'rds'`
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_exists("starwars_demo")
-# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
-# not "starwars_demo".
-# [1] TRUE
-
-pin_name <- "matt.dray/starwars_demo"
-
-board |> pin_exists(pin_name) # logical, TRUE/FALSE
-board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
-board |> pin_browse(pin_name) # view the pin in the browser
-
-
-# Permissions -------------------------------------------------------------
-
-
-# You can let people see and edit a pin. Log into Posit Connect and select the
-# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
-# the 'sharing' options in the 'Access' tab.
-
-
-# Overwrite and version ---------------------------------------------------
-
-
-starwars_droids <- starwars |>
-  filter(species == "Droid") # beep boop
-
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_versions(pin_name) # see version history
-board |> pin_versions_prune(pin_name, n = 1) # remove history
-board |> pin_versions(pin_name)
-
-# What if you try to overwrite the data but it hasn't changed?
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# ! The hash of pin "matt.dray/starwars_demo" has not changed.
-# • Your pin will not be stored.
-
-
-# Use the pin -------------------------------------------------------------
-
-
-# You can read a pin to your local machine, or access it from a Quarto file
-# or Shiny app hosted on Connect, for example. If the output and the pin are
-# both on Connect, no authentication is required; the board is defaulted to
-# the Posit Connect instance where they're both hosted.
-
-board |>
-  pin_read(pin_name) |> # like you would use e.g. read_csv
-  with(data = _, plot(mass, height)) # wow!
-
-
-# Delete pin --------------------------------------------------------------
-
-
-board |> pin_exists(pin_name) # logical, good function for error handling
-board |> pin_delete(pin_name)
-board |> pin_exists(pin_name)
-
-```
-
-## Azure Storage in R
-
-You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
-
-Below is a sample .Renviron file
-
-
-```
-AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
-AZ_STORAGE_CONTAINER=container-name
-AZ_TENANT_ID=long-sequence-of-numbers-and-letters
-AZ_APP_ID=another-long-sequence-of-numbers-and-letters
-```
-
-```{r}
-#| eval: false
-
-install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
-
-# Load all environment variables
-ep_uri <- Sys.getenv("AZ_STORAGE_EP")
-app_id <- Sys.getenv("AZ_APP_ID")
-container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
-tenant <- Sys.getenv("AZ_TENANT_ID")
-
-# Authenticate
-token <- AzureAuth::get_azure_token(
-  "https://storage.azure.com",
-  tenant = tenant,
-  app = app_id,
-  auth_type = "device_code",
-)
-
-# If you have not authenticated before, you will be taken to an external page to
-# authenticate!Use your mlcsu.nhs.uk account.
-
-# Connect to container
-endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
-container <- AzureStor::storage_container(endpoint, container_name)
-
-# List files in container
-blob_list <- AzureStor::list_blobs(container)
-
-# If you get a 403 error when trying to interact with the container, you may
-# have to clear your Azure token and re-authenticate using a different browser.
-# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
-# AzureAuth::get_azure_token() step above.
-
-# Upload specific file to container
-AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
-
-# Upload contents of a local directory to container
-AzureStor::storage_multiupload(container, "data/*", "newdir")
-
-# Check files have uploaded
-blob_list <- AzureStor::list_blobs(container)
-
-# Load file directly from Azure container
-df_from_azure <- AzureStor::storage_read_csv(
-  container,
-  "newdir/cats.csv",
-  show_col_types = FALSE
-)
-
-# Load file directly from Azure container (by temporarily downloading file
-# and storing it in memory)
-parquet_in_memory <- AzureStor::storage_download(
-  container,
-  src = "newdir/cats.parquet", dest = NULL
-)
-parq_df <- arrow::read_parquet(parquet_in_memory)
-
-# Delete from Azure container (!!!)
-for (blobfile in blob_list$name) {
-  AzureStor::delete_storage_file(container, blobfile)
-}
-```
-
-## Azure Storage in Python
-
-This will use the same environment variables as the R version, just stored in a .env file instead. 
-
-We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
-
-{{< embed azure_python.ipynb echo=true   >}}
-
-
+---
+title: "Storing data safely"
+author: 
+  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
+  - "[Matt Dray](mailto:matt.dray@nhs.net)"
+  - "[Claire Welsh](mailto:claire.welsh8@nhs.net)"
+date: "2024-05-22"
+categories: [learning, R, Python]
+notebook-view:
+  - notebook: azure_python.ipynb
+    title: "Interacting with Azure Storage with Python"
+---
+
+```{r lockfile}
+#| include: FALSE
+renv::use(lockfile = "renv.lock")
+```
+
+::: callout-note
+**UPDATED**: Please see the Addendum to this blog, added 2025-04-03 regarding accessing data from SharePoint
+:::
+
+## Coffee & Coding
+
+In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
+
+In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
+
+## Posit Connect Pins
+
+```{r}
+#| eval: false
+
+
+# A brief intro to using {pins} to store, version, share and protect a dataset
+# on Posit Connect. Documentation: https://pins.rstudio.com/
+
+
+# Setup -------------------------------------------------------------------
+
+
+install.packages(c("pins", "dplyr")) # if not yet installed
+
+suppressPackageStartupMessages({
+  library(pins)
+  library(dplyr) # for wrangling and the 'starwars' demo dataset
+})
+
+board <- board_connect() # will error if you haven't authenticated before
+# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
+# • `server` and `key` not provided for `auth = 'manual'`
+# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
+# • rsconnect package not installed for `auth = 'rsconnect'`
+# Run `rlang::last_trace()` to see where the error occurred.
+
+# To authenticate
+# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
+# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
+# Your browser will open to the Posit Connect web page and you're prompted to
+# for your password. Enter it and you'll be authenticated.
+
+# Once authenticated
+board <- board_connect()
+# Connecting to Posit Connect 2024.03.0 at
+# <https://connect.strategyunitwm.nhs.uk>
+
+board |> pin_list() # see all the pins on that board
+
+
+# Create a pin ------------------------------------------------------------
+
+
+# Write a dataset to the board as a pin
+board |> pin_write(
+  x = starwars,
+  name = "starwars_demo"
+)
+# Guessing `type = 'rds'`
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_exists("starwars_demo")
+# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
+# not "starwars_demo".
+# [1] TRUE
+
+pin_name <- "matt.dray/starwars_demo"
+
+board |> pin_exists(pin_name) # logical, TRUE/FALSE
+board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
+board |> pin_browse(pin_name) # view the pin in the browser
+
+
+# Permissions -------------------------------------------------------------
+
+
+# You can let people see and edit a pin. Log into Posit Connect and select the
+# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
+# the 'sharing' options in the 'Access' tab.
+
+
+# Overwrite and version ---------------------------------------------------
+
+
+starwars_droids <- starwars |>
+  filter(species == "Droid") # beep boop
+
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_versions(pin_name) # see version history
+board |> pin_versions_prune(pin_name, n = 1) # remove history
+board |> pin_versions(pin_name)
+
+# What if you try to overwrite the data but it hasn't changed?
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# ! The hash of pin "matt.dray/starwars_demo" has not changed.
+# • Your pin will not be stored.
+
+
+# Use the pin -------------------------------------------------------------
+
+
+# You can read a pin to your local machine, or access it from a Quarto file
+# or Shiny app hosted on Connect, for example. If the output and the pin are
+# both on Connect, no authentication is required; the board is defaulted to
+# the Posit Connect instance where they're both hosted.
+
+board |>
+  pin_read(pin_name) |> # like you would use e.g. read_csv
+  with(data = _, plot(mass, height)) # wow!
+
+
+# Delete pin --------------------------------------------------------------
+
+
+board |> pin_exists(pin_name) # logical, good function for error handling
+board |> pin_delete(pin_name)
+board |> pin_exists(pin_name)
+
+```
+
+## Azure Storage in R
+
+You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
+
+Below is a sample .Renviron file
+
+```         
+AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
+AZ_STORAGE_CONTAINER=container-name
+AZ_TENANT_ID=long-sequence-of-numbers-and-letters
+AZ_APP_ID=another-long-sequence-of-numbers-and-letters
+```
+
+```{r}
+#| eval: false
+
+install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
+
+# Load all environment variables
+ep_uri <- Sys.getenv("AZ_STORAGE_EP")
+app_id <- Sys.getenv("AZ_APP_ID")
+container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
+tenant <- Sys.getenv("AZ_TENANT_ID")
+
+# Authenticate
+token <- AzureAuth::get_azure_token(
+  "https://storage.azure.com",
+  tenant = tenant,
+  app = app_id,
+  auth_type = "device_code",
+)
+
+# If you have not authenticated before, you will be taken to an external page to
+# authenticate!Use your mlcsu.nhs.uk account.
+
+# Connect to container
+endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
+container <- AzureStor::storage_container(endpoint, container_name)
+
+# List files in container
+blob_list <- AzureStor::list_blobs(container)
+
+# If you get a 403 error when trying to interact with the container, you may
+# have to clear your Azure token and re-authenticate using a different browser.
+# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
+# AzureAuth::get_azure_token() step above.
+
+# Upload specific file to container
+AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
+
+# Upload contents of a local directory to container
+AzureStor::storage_multiupload(container, "data/*", "newdir")
+
+# Check files have uploaded
+blob_list <- AzureStor::list_blobs(container)
+
+# Load file directly from Azure container
+df_from_azure <- AzureStor::storage_read_csv(
+  container,
+  "newdir/cats.csv",
+  show_col_types = FALSE
+)
+
+# Load file directly from Azure container (by temporarily downloading file
+# and storing it in memory)
+parquet_in_memory <- AzureStor::storage_download(
+  container,
+  src = "newdir/cats.parquet", dest = NULL
+)
+parq_df <- arrow::read_parquet(parquet_in_memory)
+
+# Delete from Azure container (!!!)
+for (blobfile in blob_list$name) {
+  AzureStor::delete_storage_file(container, blobfile)
+}
+```
+
+## Azure Storage in Python
+
+This will use the same environment variables as the R version, just stored in a .env file instead.
+
+We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
+
+{{< embed azure_python.ipynb echo=true >}}
+
+## ADDENDUM: Accessing data from SharePoint
+
+SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.
+
+Accessing SharePoint requires user authentication, which also needs to happen when reading data into R from SharePoint. Before you can access SharePoint content from a site, you first need to 'follow' that site.
+
+To access data on SharePoint,follow these steps:
path: blogs/posts/2024-05-22_storing-data-safely/index.qmd
user: login: matt-dray
body: ```suggestion
To access data on SharePoint, follow these steps:
```
created_at: 2025-04-04T09:20:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,234 +1,274 @@
----
-title: "Storing data safely"
-author: 
-  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
-  - "[Matt Dray](mailto:matt.dray@nhs.net)"
-date: "2024-05-22"
-categories: [learning, R, Python]
-notebook-view:
-  - notebook: azure_python.ipynb
-    title: "Interacting with Azure Storage with Python"
-
----
-
-```{r lockfile}
-#| include: FALSE
-renv::use(lockfile = "renv.lock")
-```
-
-
-## Coffee & Coding
-
-In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
-
-In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
-
-## Posit Connect Pins
-
-
-```{r}
-#| eval: false
-
-
-# A brief intro to using {pins} to store, version, share and protect a dataset
-# on Posit Connect. Documentation: https://pins.rstudio.com/
-
-
-# Setup -------------------------------------------------------------------
-
-
-install.packages(c("pins", "dplyr")) # if not yet installed
-
-suppressPackageStartupMessages({
-  library(pins)
-  library(dplyr) # for wrangling and the 'starwars' demo dataset
-})
-
-board <- board_connect() # will error if you haven't authenticated before
-# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
-# • `server` and `key` not provided for `auth = 'manual'`
-# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
-# • rsconnect package not installed for `auth = 'rsconnect'`
-# Run `rlang::last_trace()` to see where the error occurred.
-
-# To authenticate
-# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
-# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
-# Your browser will open to the Posit Connect web page and you're prompted to
-# for your password. Enter it and you'll be authenticated.
-
-# Once authenticated
-board <- board_connect()
-# Connecting to Posit Connect 2024.03.0 at
-# <https://connect.strategyunitwm.nhs.uk>
-
-board |> pin_list() # see all the pins on that board
-
-
-# Create a pin ------------------------------------------------------------
-
-
-# Write a dataset to the board as a pin
-board |> pin_write(
-  x = starwars,
-  name = "starwars_demo"
-)
-# Guessing `type = 'rds'`
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_exists("starwars_demo")
-# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
-# not "starwars_demo".
-# [1] TRUE
-
-pin_name <- "matt.dray/starwars_demo"
-
-board |> pin_exists(pin_name) # logical, TRUE/FALSE
-board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
-board |> pin_browse(pin_name) # view the pin in the browser
-
-
-# Permissions -------------------------------------------------------------
-
-
-# You can let people see and edit a pin. Log into Posit Connect and select the
-# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
-# the 'sharing' options in the 'Access' tab.
-
-
-# Overwrite and version ---------------------------------------------------
-
-
-starwars_droids <- starwars |>
-  filter(species == "Droid") # beep boop
-
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_versions(pin_name) # see version history
-board |> pin_versions_prune(pin_name, n = 1) # remove history
-board |> pin_versions(pin_name)
-
-# What if you try to overwrite the data but it hasn't changed?
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# ! The hash of pin "matt.dray/starwars_demo" has not changed.
-# • Your pin will not be stored.
-
-
-# Use the pin -------------------------------------------------------------
-
-
-# You can read a pin to your local machine, or access it from a Quarto file
-# or Shiny app hosted on Connect, for example. If the output and the pin are
-# both on Connect, no authentication is required; the board is defaulted to
-# the Posit Connect instance where they're both hosted.
-
-board |>
-  pin_read(pin_name) |> # like you would use e.g. read_csv
-  with(data = _, plot(mass, height)) # wow!
-
-
-# Delete pin --------------------------------------------------------------
-
-
-board |> pin_exists(pin_name) # logical, good function for error handling
-board |> pin_delete(pin_name)
-board |> pin_exists(pin_name)
-
-```
-
-## Azure Storage in R
-
-You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
-
-Below is a sample .Renviron file
-
-
-```
-AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
-AZ_STORAGE_CONTAINER=container-name
-AZ_TENANT_ID=long-sequence-of-numbers-and-letters
-AZ_APP_ID=another-long-sequence-of-numbers-and-letters
-```
-
-```{r}
-#| eval: false
-
-install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
-
-# Load all environment variables
-ep_uri <- Sys.getenv("AZ_STORAGE_EP")
-app_id <- Sys.getenv("AZ_APP_ID")
-container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
-tenant <- Sys.getenv("AZ_TENANT_ID")
-
-# Authenticate
-token <- AzureAuth::get_azure_token(
-  "https://storage.azure.com",
-  tenant = tenant,
-  app = app_id,
-  auth_type = "device_code",
-)
-
-# If you have not authenticated before, you will be taken to an external page to
-# authenticate!Use your mlcsu.nhs.uk account.
-
-# Connect to container
-endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
-container <- AzureStor::storage_container(endpoint, container_name)
-
-# List files in container
-blob_list <- AzureStor::list_blobs(container)
-
-# If you get a 403 error when trying to interact with the container, you may
-# have to clear your Azure token and re-authenticate using a different browser.
-# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
-# AzureAuth::get_azure_token() step above.
-
-# Upload specific file to container
-AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
-
-# Upload contents of a local directory to container
-AzureStor::storage_multiupload(container, "data/*", "newdir")
-
-# Check files have uploaded
-blob_list <- AzureStor::list_blobs(container)
-
-# Load file directly from Azure container
-df_from_azure <- AzureStor::storage_read_csv(
-  container,
-  "newdir/cats.csv",
-  show_col_types = FALSE
-)
-
-# Load file directly from Azure container (by temporarily downloading file
-# and storing it in memory)
-parquet_in_memory <- AzureStor::storage_download(
-  container,
-  src = "newdir/cats.parquet", dest = NULL
-)
-parq_df <- arrow::read_parquet(parquet_in_memory)
-
-# Delete from Azure container (!!!)
-for (blobfile in blob_list$name) {
-  AzureStor::delete_storage_file(container, blobfile)
-}
-```
-
-## Azure Storage in Python
-
-This will use the same environment variables as the R version, just stored in a .env file instead. 
-
-We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
-
-{{< embed azure_python.ipynb echo=true   >}}
-
-
+---
+title: "Storing data safely"
+author: 
+  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
+  - "[Matt Dray](mailto:matt.dray@nhs.net)"
+  - "[Claire Welsh](mailto:claire.welsh8@nhs.net)"
+date: "2024-05-22"
+categories: [learning, R, Python]
+notebook-view:
+  - notebook: azure_python.ipynb
+    title: "Interacting with Azure Storage with Python"
+---
+
+```{r lockfile}
+#| include: FALSE
+renv::use(lockfile = "renv.lock")
+```
+
+::: callout-note
+**UPDATED**: Please see the Addendum to this blog, added 2025-04-03 regarding accessing data from SharePoint
+:::
+
+## Coffee & Coding
+
+In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
+
+In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
+
+## Posit Connect Pins
+
+```{r}
+#| eval: false
+
+
+# A brief intro to using {pins} to store, version, share and protect a dataset
+# on Posit Connect. Documentation: https://pins.rstudio.com/
+
+
+# Setup -------------------------------------------------------------------
+
+
+install.packages(c("pins", "dplyr")) # if not yet installed
+
+suppressPackageStartupMessages({
+  library(pins)
+  library(dplyr) # for wrangling and the 'starwars' demo dataset
+})
+
+board <- board_connect() # will error if you haven't authenticated before
+# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
+# • `server` and `key` not provided for `auth = 'manual'`
+# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
+# • rsconnect package not installed for `auth = 'rsconnect'`
+# Run `rlang::last_trace()` to see where the error occurred.
+
+# To authenticate
+# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
+# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
+# Your browser will open to the Posit Connect web page and you're prompted to
+# for your password. Enter it and you'll be authenticated.
+
+# Once authenticated
+board <- board_connect()
+# Connecting to Posit Connect 2024.03.0 at
+# <https://connect.strategyunitwm.nhs.uk>
+
+board |> pin_list() # see all the pins on that board
+
+
+# Create a pin ------------------------------------------------------------
+
+
+# Write a dataset to the board as a pin
+board |> pin_write(
+  x = starwars,
+  name = "starwars_demo"
+)
+# Guessing `type = 'rds'`
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_exists("starwars_demo")
+# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
+# not "starwars_demo".
+# [1] TRUE
+
+pin_name <- "matt.dray/starwars_demo"
+
+board |> pin_exists(pin_name) # logical, TRUE/FALSE
+board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
+board |> pin_browse(pin_name) # view the pin in the browser
+
+
+# Permissions -------------------------------------------------------------
+
+
+# You can let people see and edit a pin. Log into Posit Connect and select the
+# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
+# the 'sharing' options in the 'Access' tab.
+
+
+# Overwrite and version ---------------------------------------------------
+
+
+starwars_droids <- starwars |>
+  filter(species == "Droid") # beep boop
+
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_versions(pin_name) # see version history
+board |> pin_versions_prune(pin_name, n = 1) # remove history
+board |> pin_versions(pin_name)
+
+# What if you try to overwrite the data but it hasn't changed?
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# ! The hash of pin "matt.dray/starwars_demo" has not changed.
+# • Your pin will not be stored.
+
+
+# Use the pin -------------------------------------------------------------
+
+
+# You can read a pin to your local machine, or access it from a Quarto file
+# or Shiny app hosted on Connect, for example. If the output and the pin are
+# both on Connect, no authentication is required; the board is defaulted to
+# the Posit Connect instance where they're both hosted.
+
+board |>
+  pin_read(pin_name) |> # like you would use e.g. read_csv
+  with(data = _, plot(mass, height)) # wow!
+
+
+# Delete pin --------------------------------------------------------------
+
+
+board |> pin_exists(pin_name) # logical, good function for error handling
+board |> pin_delete(pin_name)
+board |> pin_exists(pin_name)
+
+```
+
+## Azure Storage in R
+
+You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
+
+Below is a sample .Renviron file
+
+```         
+AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
+AZ_STORAGE_CONTAINER=container-name
+AZ_TENANT_ID=long-sequence-of-numbers-and-letters
+AZ_APP_ID=another-long-sequence-of-numbers-and-letters
+```
+
+```{r}
+#| eval: false
+
+install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
+
+# Load all environment variables
+ep_uri <- Sys.getenv("AZ_STORAGE_EP")
+app_id <- Sys.getenv("AZ_APP_ID")
+container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
+tenant <- Sys.getenv("AZ_TENANT_ID")
+
+# Authenticate
+token <- AzureAuth::get_azure_token(
+  "https://storage.azure.com",
+  tenant = tenant,
+  app = app_id,
+  auth_type = "device_code",
+)
+
+# If you have not authenticated before, you will be taken to an external page to
+# authenticate!Use your mlcsu.nhs.uk account.
+
+# Connect to container
+endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
+container <- AzureStor::storage_container(endpoint, container_name)
+
+# List files in container
+blob_list <- AzureStor::list_blobs(container)
+
+# If you get a 403 error when trying to interact with the container, you may
+# have to clear your Azure token and re-authenticate using a different browser.
+# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
+# AzureAuth::get_azure_token() step above.
+
+# Upload specific file to container
+AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
+
+# Upload contents of a local directory to container
+AzureStor::storage_multiupload(container, "data/*", "newdir")
+
+# Check files have uploaded
+blob_list <- AzureStor::list_blobs(container)
+
+# Load file directly from Azure container
+df_from_azure <- AzureStor::storage_read_csv(
+  container,
+  "newdir/cats.csv",
+  show_col_types = FALSE
+)
+
+# Load file directly from Azure container (by temporarily downloading file
+# and storing it in memory)
+parquet_in_memory <- AzureStor::storage_download(
+  container,
+  src = "newdir/cats.parquet", dest = NULL
+)
+parq_df <- arrow::read_parquet(parquet_in_memory)
+
+# Delete from Azure container (!!!)
+for (blobfile in blob_list$name) {
+  AzureStor::delete_storage_file(container, blobfile)
+}
+```
+
+## Azure Storage in Python
+
+This will use the same environment variables as the R version, just stored in a .env file instead.
+
+We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
+
+{{< embed azure_python.ipynb echo=true >}}
+
+## ADDENDUM: Accessing data from SharePoint
+
+SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.
+
+Accessing SharePoint requires user authentication, which also needs to happen when reading data into R from SharePoint. Before you can access SharePoint content from a site, you first need to 'follow' that site.
+
+To access data on SharePoint,follow these steps:
+
+1.  Navigate to the SharePoint page that has the file of interest in it, using your browser.
+
+2.  Click the small star in the top right corner of the window, labelled 'Follow'
path: blogs/posts/2024-05-22_storing-data-safely/index.qmd
user: login: matt-dray
body: Whitespace, period.
```suggestion
2. Click the small star in the top right corner of the window, labelled 'Follow'.
```
created_at: 2025-04-04T09:21:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,234 +1,274 @@
----
-title: "Storing data safely"
-author: 
-  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
-  - "[Matt Dray](mailto:matt.dray@nhs.net)"
-date: "2024-05-22"
-categories: [learning, R, Python]
-notebook-view:
-  - notebook: azure_python.ipynb
-    title: "Interacting with Azure Storage with Python"
-
----
-
-```{r lockfile}
-#| include: FALSE
-renv::use(lockfile = "renv.lock")
-```
-
-
-## Coffee & Coding
-
-In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
-
-In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
-
-## Posit Connect Pins
-
-
-```{r}
-#| eval: false
-
-
-# A brief intro to using {pins} to store, version, share and protect a dataset
-# on Posit Connect. Documentation: https://pins.rstudio.com/
-
-
-# Setup -------------------------------------------------------------------
-
-
-install.packages(c("pins", "dplyr")) # if not yet installed
-
-suppressPackageStartupMessages({
-  library(pins)
-  library(dplyr) # for wrangling and the 'starwars' demo dataset
-})
-
-board <- board_connect() # will error if you haven't authenticated before
-# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
-# • `server` and `key` not provided for `auth = 'manual'`
-# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
-# • rsconnect package not installed for `auth = 'rsconnect'`
-# Run `rlang::last_trace()` to see where the error occurred.
-
-# To authenticate
-# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
-# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
-# Your browser will open to the Posit Connect web page and you're prompted to
-# for your password. Enter it and you'll be authenticated.
-
-# Once authenticated
-board <- board_connect()
-# Connecting to Posit Connect 2024.03.0 at
-# <https://connect.strategyunitwm.nhs.uk>
-
-board |> pin_list() # see all the pins on that board
-
-
-# Create a pin ------------------------------------------------------------
-
-
-# Write a dataset to the board as a pin
-board |> pin_write(
-  x = starwars,
-  name = "starwars_demo"
-)
-# Guessing `type = 'rds'`
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_exists("starwars_demo")
-# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
-# not "starwars_demo".
-# [1] TRUE
-
-pin_name <- "matt.dray/starwars_demo"
-
-board |> pin_exists(pin_name) # logical, TRUE/FALSE
-board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
-board |> pin_browse(pin_name) # view the pin in the browser
-
-
-# Permissions -------------------------------------------------------------
-
-
-# You can let people see and edit a pin. Log into Posit Connect and select the
-# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
-# the 'sharing' options in the 'Access' tab.
-
-
-# Overwrite and version ---------------------------------------------------
-
-
-starwars_droids <- starwars |>
-  filter(species == "Droid") # beep boop
-
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_versions(pin_name) # see version history
-board |> pin_versions_prune(pin_name, n = 1) # remove history
-board |> pin_versions(pin_name)
-
-# What if you try to overwrite the data but it hasn't changed?
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# ! The hash of pin "matt.dray/starwars_demo" has not changed.
-# • Your pin will not be stored.
-
-
-# Use the pin -------------------------------------------------------------
-
-
-# You can read a pin to your local machine, or access it from a Quarto file
-# or Shiny app hosted on Connect, for example. If the output and the pin are
-# both on Connect, no authentication is required; the board is defaulted to
-# the Posit Connect instance where they're both hosted.
-
-board |>
-  pin_read(pin_name) |> # like you would use e.g. read_csv
-  with(data = _, plot(mass, height)) # wow!
-
-
-# Delete pin --------------------------------------------------------------
-
-
-board |> pin_exists(pin_name) # logical, good function for error handling
-board |> pin_delete(pin_name)
-board |> pin_exists(pin_name)
-
-```
-
-## Azure Storage in R
-
-You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
-
-Below is a sample .Renviron file
-
-
-```
-AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
-AZ_STORAGE_CONTAINER=container-name
-AZ_TENANT_ID=long-sequence-of-numbers-and-letters
-AZ_APP_ID=another-long-sequence-of-numbers-and-letters
-```
-
-```{r}
-#| eval: false
-
-install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
-
-# Load all environment variables
-ep_uri <- Sys.getenv("AZ_STORAGE_EP")
-app_id <- Sys.getenv("AZ_APP_ID")
-container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
-tenant <- Sys.getenv("AZ_TENANT_ID")
-
-# Authenticate
-token <- AzureAuth::get_azure_token(
-  "https://storage.azure.com",
-  tenant = tenant,
-  app = app_id,
-  auth_type = "device_code",
-)
-
-# If you have not authenticated before, you will be taken to an external page to
-# authenticate!Use your mlcsu.nhs.uk account.
-
-# Connect to container
-endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
-container <- AzureStor::storage_container(endpoint, container_name)
-
-# List files in container
-blob_list <- AzureStor::list_blobs(container)
-
-# If you get a 403 error when trying to interact with the container, you may
-# have to clear your Azure token and re-authenticate using a different browser.
-# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
-# AzureAuth::get_azure_token() step above.
-
-# Upload specific file to container
-AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
-
-# Upload contents of a local directory to container
-AzureStor::storage_multiupload(container, "data/*", "newdir")
-
-# Check files have uploaded
-blob_list <- AzureStor::list_blobs(container)
-
-# Load file directly from Azure container
-df_from_azure <- AzureStor::storage_read_csv(
-  container,
-  "newdir/cats.csv",
-  show_col_types = FALSE
-)
-
-# Load file directly from Azure container (by temporarily downloading file
-# and storing it in memory)
-parquet_in_memory <- AzureStor::storage_download(
-  container,
-  src = "newdir/cats.parquet", dest = NULL
-)
-parq_df <- arrow::read_parquet(parquet_in_memory)
-
-# Delete from Azure container (!!!)
-for (blobfile in blob_list$name) {
-  AzureStor::delete_storage_file(container, blobfile)
-}
-```
-
-## Azure Storage in Python
-
-This will use the same environment variables as the R version, just stored in a .env file instead. 
-
-We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
-
-{{< embed azure_python.ipynb echo=true   >}}
-
-
+---
+title: "Storing data safely"
+author: 
+  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
+  - "[Matt Dray](mailto:matt.dray@nhs.net)"
+  - "[Claire Welsh](mailto:claire.welsh8@nhs.net)"
+date: "2024-05-22"
+categories: [learning, R, Python]
+notebook-view:
+  - notebook: azure_python.ipynb
+    title: "Interacting with Azure Storage with Python"
+---
+
+```{r lockfile}
+#| include: FALSE
+renv::use(lockfile = "renv.lock")
+```
+
+::: callout-note
+**UPDATED**: Please see the Addendum to this blog, added 2025-04-03 regarding accessing data from SharePoint
+:::
+
+## Coffee & Coding
+
+In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
+
+In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
+
+## Posit Connect Pins
+
+```{r}
+#| eval: false
+
+
+# A brief intro to using {pins} to store, version, share and protect a dataset
+# on Posit Connect. Documentation: https://pins.rstudio.com/
+
+
+# Setup -------------------------------------------------------------------
+
+
+install.packages(c("pins", "dplyr")) # if not yet installed
+
+suppressPackageStartupMessages({
+  library(pins)
+  library(dplyr) # for wrangling and the 'starwars' demo dataset
+})
+
+board <- board_connect() # will error if you haven't authenticated before
+# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
+# • `server` and `key` not provided for `auth = 'manual'`
+# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
+# • rsconnect package not installed for `auth = 'rsconnect'`
+# Run `rlang::last_trace()` to see where the error occurred.
+
+# To authenticate
+# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
+# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
+# Your browser will open to the Posit Connect web page and you're prompted to
+# for your password. Enter it and you'll be authenticated.
+
+# Once authenticated
+board <- board_connect()
+# Connecting to Posit Connect 2024.03.0 at
+# <https://connect.strategyunitwm.nhs.uk>
+
+board |> pin_list() # see all the pins on that board
+
+
+# Create a pin ------------------------------------------------------------
+
+
+# Write a dataset to the board as a pin
+board |> pin_write(
+  x = starwars,
+  name = "starwars_demo"
+)
+# Guessing `type = 'rds'`
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_exists("starwars_demo")
+# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
+# not "starwars_demo".
+# [1] TRUE
+
+pin_name <- "matt.dray/starwars_demo"
+
+board |> pin_exists(pin_name) # logical, TRUE/FALSE
+board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
+board |> pin_browse(pin_name) # view the pin in the browser
+
+
+# Permissions -------------------------------------------------------------
+
+
+# You can let people see and edit a pin. Log into Posit Connect and select the
+# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
+# the 'sharing' options in the 'Access' tab.
+
+
+# Overwrite and version ---------------------------------------------------
+
+
+starwars_droids <- starwars |>
+  filter(species == "Droid") # beep boop
+
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_versions(pin_name) # see version history
+board |> pin_versions_prune(pin_name, n = 1) # remove history
+board |> pin_versions(pin_name)
+
+# What if you try to overwrite the data but it hasn't changed?
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# ! The hash of pin "matt.dray/starwars_demo" has not changed.
+# • Your pin will not be stored.
+
+
+# Use the pin -------------------------------------------------------------
+
+
+# You can read a pin to your local machine, or access it from a Quarto file
+# or Shiny app hosted on Connect, for example. If the output and the pin are
+# both on Connect, no authentication is required; the board is defaulted to
+# the Posit Connect instance where they're both hosted.
+
+board |>
+  pin_read(pin_name) |> # like you would use e.g. read_csv
+  with(data = _, plot(mass, height)) # wow!
+
+
+# Delete pin --------------------------------------------------------------
+
+
+board |> pin_exists(pin_name) # logical, good function for error handling
+board |> pin_delete(pin_name)
+board |> pin_exists(pin_name)
+
+```
+
+## Azure Storage in R
+
+You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
+
+Below is a sample .Renviron file
+
+```         
+AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
+AZ_STORAGE_CONTAINER=container-name
+AZ_TENANT_ID=long-sequence-of-numbers-and-letters
+AZ_APP_ID=another-long-sequence-of-numbers-and-letters
+```
+
+```{r}
+#| eval: false
+
+install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
+
+# Load all environment variables
+ep_uri <- Sys.getenv("AZ_STORAGE_EP")
+app_id <- Sys.getenv("AZ_APP_ID")
+container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
+tenant <- Sys.getenv("AZ_TENANT_ID")
+
+# Authenticate
+token <- AzureAuth::get_azure_token(
+  "https://storage.azure.com",
+  tenant = tenant,
+  app = app_id,
+  auth_type = "device_code",
+)
+
+# If you have not authenticated before, you will be taken to an external page to
+# authenticate!Use your mlcsu.nhs.uk account.
+
+# Connect to container
+endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
+container <- AzureStor::storage_container(endpoint, container_name)
+
+# List files in container
+blob_list <- AzureStor::list_blobs(container)
+
+# If you get a 403 error when trying to interact with the container, you may
+# have to clear your Azure token and re-authenticate using a different browser.
+# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
+# AzureAuth::get_azure_token() step above.
+
+# Upload specific file to container
+AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
+
+# Upload contents of a local directory to container
+AzureStor::storage_multiupload(container, "data/*", "newdir")
+
+# Check files have uploaded
+blob_list <- AzureStor::list_blobs(container)
+
+# Load file directly from Azure container
+df_from_azure <- AzureStor::storage_read_csv(
+  container,
+  "newdir/cats.csv",
+  show_col_types = FALSE
+)
+
+# Load file directly from Azure container (by temporarily downloading file
+# and storing it in memory)
+parquet_in_memory <- AzureStor::storage_download(
+  container,
+  src = "newdir/cats.parquet", dest = NULL
+)
+parq_df <- arrow::read_parquet(parquet_in_memory)
+
+# Delete from Azure container (!!!)
+for (blobfile in blob_list$name) {
+  AzureStor::delete_storage_file(container, blobfile)
+}
+```
+
+## Azure Storage in Python
+
+This will use the same environment variables as the R version, just stored in a .env file instead.
+
+We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
+
+{{< embed azure_python.ipynb echo=true >}}
+
+## ADDENDUM: Accessing data from SharePoint
+
+SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.
+
+Accessing SharePoint requires user authentication, which also needs to happen when reading data into R from SharePoint. Before you can access SharePoint content from a site, you first need to 'follow' that site.
+
+To access data on SharePoint,follow these steps:
+
+1.  Navigate to the SharePoint page that has the file of interest in it, using your browser.
+
+2.  Click the small star in the top right corner of the window, labelled 'Follow'
+
+![](Follow.jpg)
path: blogs/posts/2024-05-22_storing-data-safely/index.qmd
user: login: matt-dray
body: ```suggestion
![](Follow.jpg){fig-alt="A screenshot of the upper-right corner of a SharePoint site in a web browser, highlighting a star-shaped toggle labelled 'follow'."}
```
created_at: 2025-04-04T09:25:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,234 +1,274 @@
----
-title: "Storing data safely"
-author: 
-  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
-  - "[Matt Dray](mailto:matt.dray@nhs.net)"
-date: "2024-05-22"
-categories: [learning, R, Python]
-notebook-view:
-  - notebook: azure_python.ipynb
-    title: "Interacting with Azure Storage with Python"
-
----
-
-```{r lockfile}
-#| include: FALSE
-renv::use(lockfile = "renv.lock")
-```
-
-
-## Coffee & Coding
-
-In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
-
-In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
-
-## Posit Connect Pins
-
-
-```{r}
-#| eval: false
-
-
-# A brief intro to using {pins} to store, version, share and protect a dataset
-# on Posit Connect. Documentation: https://pins.rstudio.com/
-
-
-# Setup -------------------------------------------------------------------
-
-
-install.packages(c("pins", "dplyr")) # if not yet installed
-
-suppressPackageStartupMessages({
-  library(pins)
-  library(dplyr) # for wrangling and the 'starwars' demo dataset
-})
-
-board <- board_connect() # will error if you haven't authenticated before
-# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
-# • `server` and `key` not provided for `auth = 'manual'`
-# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
-# • rsconnect package not installed for `auth = 'rsconnect'`
-# Run `rlang::last_trace()` to see where the error occurred.
-
-# To authenticate
-# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
-# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
-# Your browser will open to the Posit Connect web page and you're prompted to
-# for your password. Enter it and you'll be authenticated.
-
-# Once authenticated
-board <- board_connect()
-# Connecting to Posit Connect 2024.03.0 at
-# <https://connect.strategyunitwm.nhs.uk>
-
-board |> pin_list() # see all the pins on that board
-
-
-# Create a pin ------------------------------------------------------------
-
-
-# Write a dataset to the board as a pin
-board |> pin_write(
-  x = starwars,
-  name = "starwars_demo"
-)
-# Guessing `type = 'rds'`
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_exists("starwars_demo")
-# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
-# not "starwars_demo".
-# [1] TRUE
-
-pin_name <- "matt.dray/starwars_demo"
-
-board |> pin_exists(pin_name) # logical, TRUE/FALSE
-board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
-board |> pin_browse(pin_name) # view the pin in the browser
-
-
-# Permissions -------------------------------------------------------------
-
-
-# You can let people see and edit a pin. Log into Posit Connect and select the
-# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
-# the 'sharing' options in the 'Access' tab.
-
-
-# Overwrite and version ---------------------------------------------------
-
-
-starwars_droids <- starwars |>
-  filter(species == "Droid") # beep boop
-
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_versions(pin_name) # see version history
-board |> pin_versions_prune(pin_name, n = 1) # remove history
-board |> pin_versions(pin_name)
-
-# What if you try to overwrite the data but it hasn't changed?
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# ! The hash of pin "matt.dray/starwars_demo" has not changed.
-# • Your pin will not be stored.
-
-
-# Use the pin -------------------------------------------------------------
-
-
-# You can read a pin to your local machine, or access it from a Quarto file
-# or Shiny app hosted on Connect, for example. If the output and the pin are
-# both on Connect, no authentication is required; the board is defaulted to
-# the Posit Connect instance where they're both hosted.
-
-board |>
-  pin_read(pin_name) |> # like you would use e.g. read_csv
-  with(data = _, plot(mass, height)) # wow!
-
-
-# Delete pin --------------------------------------------------------------
-
-
-board |> pin_exists(pin_name) # logical, good function for error handling
-board |> pin_delete(pin_name)
-board |> pin_exists(pin_name)
-
-```
-
-## Azure Storage in R
-
-You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
-
-Below is a sample .Renviron file
-
-
-```
-AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
-AZ_STORAGE_CONTAINER=container-name
-AZ_TENANT_ID=long-sequence-of-numbers-and-letters
-AZ_APP_ID=another-long-sequence-of-numbers-and-letters
-```
-
-```{r}
-#| eval: false
-
-install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
-
-# Load all environment variables
-ep_uri <- Sys.getenv("AZ_STORAGE_EP")
-app_id <- Sys.getenv("AZ_APP_ID")
-container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
-tenant <- Sys.getenv("AZ_TENANT_ID")
-
-# Authenticate
-token <- AzureAuth::get_azure_token(
-  "https://storage.azure.com",
-  tenant = tenant,
-  app = app_id,
-  auth_type = "device_code",
-)
-
-# If you have not authenticated before, you will be taken to an external page to
-# authenticate!Use your mlcsu.nhs.uk account.
-
-# Connect to container
-endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
-container <- AzureStor::storage_container(endpoint, container_name)
-
-# List files in container
-blob_list <- AzureStor::list_blobs(container)
-
-# If you get a 403 error when trying to interact with the container, you may
-# have to clear your Azure token and re-authenticate using a different browser.
-# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
-# AzureAuth::get_azure_token() step above.
-
-# Upload specific file to container
-AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
-
-# Upload contents of a local directory to container
-AzureStor::storage_multiupload(container, "data/*", "newdir")
-
-# Check files have uploaded
-blob_list <- AzureStor::list_blobs(container)
-
-# Load file directly from Azure container
-df_from_azure <- AzureStor::storage_read_csv(
-  container,
-  "newdir/cats.csv",
-  show_col_types = FALSE
-)
-
-# Load file directly from Azure container (by temporarily downloading file
-# and storing it in memory)
-parquet_in_memory <- AzureStor::storage_download(
-  container,
-  src = "newdir/cats.parquet", dest = NULL
-)
-parq_df <- arrow::read_parquet(parquet_in_memory)
-
-# Delete from Azure container (!!!)
-for (blobfile in blob_list$name) {
-  AzureStor::delete_storage_file(container, blobfile)
-}
-```
-
-## Azure Storage in Python
-
-This will use the same environment variables as the R version, just stored in a .env file instead. 
-
-We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
-
-{{< embed azure_python.ipynb echo=true   >}}
-
-
+---
+title: "Storing data safely"
+author: 
+  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
+  - "[Matt Dray](mailto:matt.dray@nhs.net)"
+  - "[Claire Welsh](mailto:claire.welsh8@nhs.net)"
+date: "2024-05-22"
+categories: [learning, R, Python]
+notebook-view:
+  - notebook: azure_python.ipynb
+    title: "Interacting with Azure Storage with Python"
+---
+
+```{r lockfile}
+#| include: FALSE
+renv::use(lockfile = "renv.lock")
+```
+
+::: callout-note
+**UPDATED**: Please see the Addendum to this blog, added 2025-04-03 regarding accessing data from SharePoint
+:::
+
+## Coffee & Coding
+
+In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
+
+In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
+
+## Posit Connect Pins
+
+```{r}
+#| eval: false
+
+
+# A brief intro to using {pins} to store, version, share and protect a dataset
+# on Posit Connect. Documentation: https://pins.rstudio.com/
+
+
+# Setup -------------------------------------------------------------------
+
+
+install.packages(c("pins", "dplyr")) # if not yet installed
+
+suppressPackageStartupMessages({
+  library(pins)
+  library(dplyr) # for wrangling and the 'starwars' demo dataset
+})
+
+board <- board_connect() # will error if you haven't authenticated before
+# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
+# • `server` and `key` not provided for `auth = 'manual'`
+# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
+# • rsconnect package not installed for `auth = 'rsconnect'`
+# Run `rlang::last_trace()` to see where the error occurred.
+
+# To authenticate
+# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
+# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
+# Your browser will open to the Posit Connect web page and you're prompted to
+# for your password. Enter it and you'll be authenticated.
+
+# Once authenticated
+board <- board_connect()
+# Connecting to Posit Connect 2024.03.0 at
+# <https://connect.strategyunitwm.nhs.uk>
+
+board |> pin_list() # see all the pins on that board
+
+
+# Create a pin ------------------------------------------------------------
+
+
+# Write a dataset to the board as a pin
+board |> pin_write(
+  x = starwars,
+  name = "starwars_demo"
+)
+# Guessing `type = 'rds'`
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_exists("starwars_demo")
+# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
+# not "starwars_demo".
+# [1] TRUE
+
+pin_name <- "matt.dray/starwars_demo"
+
+board |> pin_exists(pin_name) # logical, TRUE/FALSE
+board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
+board |> pin_browse(pin_name) # view the pin in the browser
+
+
+# Permissions -------------------------------------------------------------
+
+
+# You can let people see and edit a pin. Log into Posit Connect and select the
+# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
+# the 'sharing' options in the 'Access' tab.
+
+
+# Overwrite and version ---------------------------------------------------
+
+
+starwars_droids <- starwars |>
+  filter(species == "Droid") # beep boop
+
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_versions(pin_name) # see version history
+board |> pin_versions_prune(pin_name, n = 1) # remove history
+board |> pin_versions(pin_name)
+
+# What if you try to overwrite the data but it hasn't changed?
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# ! The hash of pin "matt.dray/starwars_demo" has not changed.
+# • Your pin will not be stored.
+
+
+# Use the pin -------------------------------------------------------------
+
+
+# You can read a pin to your local machine, or access it from a Quarto file
+# or Shiny app hosted on Connect, for example. If the output and the pin are
+# both on Connect, no authentication is required; the board is defaulted to
+# the Posit Connect instance where they're both hosted.
+
+board |>
+  pin_read(pin_name) |> # like you would use e.g. read_csv
+  with(data = _, plot(mass, height)) # wow!
+
+
+# Delete pin --------------------------------------------------------------
+
+
+board |> pin_exists(pin_name) # logical, good function for error handling
+board |> pin_delete(pin_name)
+board |> pin_exists(pin_name)
+
+```
+
+## Azure Storage in R
+
+You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
+
+Below is a sample .Renviron file
+
+```         
+AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
+AZ_STORAGE_CONTAINER=container-name
+AZ_TENANT_ID=long-sequence-of-numbers-and-letters
+AZ_APP_ID=another-long-sequence-of-numbers-and-letters
+```
+
+```{r}
+#| eval: false
+
+install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
+
+# Load all environment variables
+ep_uri <- Sys.getenv("AZ_STORAGE_EP")
+app_id <- Sys.getenv("AZ_APP_ID")
+container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
+tenant <- Sys.getenv("AZ_TENANT_ID")
+
+# Authenticate
+token <- AzureAuth::get_azure_token(
+  "https://storage.azure.com",
+  tenant = tenant,
+  app = app_id,
+  auth_type = "device_code",
+)
+
+# If you have not authenticated before, you will be taken to an external page to
+# authenticate!Use your mlcsu.nhs.uk account.
+
+# Connect to container
+endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
+container <- AzureStor::storage_container(endpoint, container_name)
+
+# List files in container
+blob_list <- AzureStor::list_blobs(container)
+
+# If you get a 403 error when trying to interact with the container, you may
+# have to clear your Azure token and re-authenticate using a different browser.
+# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
+# AzureAuth::get_azure_token() step above.
+
+# Upload specific file to container
+AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
+
+# Upload contents of a local directory to container
+AzureStor::storage_multiupload(container, "data/*", "newdir")
+
+# Check files have uploaded
+blob_list <- AzureStor::list_blobs(container)
+
+# Load file directly from Azure container
+df_from_azure <- AzureStor::storage_read_csv(
+  container,
+  "newdir/cats.csv",
+  show_col_types = FALSE
+)
+
+# Load file directly from Azure container (by temporarily downloading file
+# and storing it in memory)
+parquet_in_memory <- AzureStor::storage_download(
+  container,
+  src = "newdir/cats.parquet", dest = NULL
+)
+parq_df <- arrow::read_parquet(parquet_in_memory)
+
+# Delete from Azure container (!!!)
+for (blobfile in blob_list$name) {
+  AzureStor::delete_storage_file(container, blobfile)
+}
+```
+
+## Azure Storage in Python
+
+This will use the same environment variables as the R version, just stored in a .env file instead.
+
+We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
+
+{{< embed azure_python.ipynb echo=true >}}
+
+## ADDENDUM: Accessing data from SharePoint
+
+SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.
+
+Accessing SharePoint requires user authentication, which also needs to happen when reading data into R from SharePoint. Before you can access SharePoint content from a site, you first need to 'follow' that site.
+
+To access data on SharePoint,follow these steps:
+
+1.  Navigate to the SharePoint page that has the file of interest in it, using your browser.
+
+2.  Click the small star in the top right corner of the window, labelled 'Follow'
+
+![](Follow.jpg)
+
+3.  Open your environment file, which you can do by running `r file.edit("~/.Renviron")` within the RStudio console area.
+
+4.  Save two new environment variables; one is the name of the site (i.e. the name at the top of the browser screen), and the other is the full file path from below the 'Documents' folder to the file itself, including the file type extension. Save and close the .Renviron file then restart your R session (CTRL \> Shift \> F10 on Windows machines)
path: blogs/posts/2024-05-22_storing-data-safely/index.qmd
user: login: matt-dray
body: I think we should suggest names for the variables and then refer to them in the section below with those names.

```suggestion
4. Save two new environment variables to the `.Renviron` file:
    a. `SP_SITE_NAME`: the name of the site (i.e. the name at the top of the browser screen)
    b. `SP_FILE_PATH`: the full file path from below the `Documents/` folder to the file itself, including the file type extension. 
5. Save and close the `.Renviron` file then either restart your R session (CTRL + Shift + F10 on Windows machines) or run `r readRenviron(".Renviron")` to make the new variables available in your session.
```
created_at: 2025-04-04T09:33:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,234 +1,274 @@
----
-title: "Storing data safely"
-author: 
-  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
-  - "[Matt Dray](mailto:matt.dray@nhs.net)"
-date: "2024-05-22"
-categories: [learning, R, Python]
-notebook-view:
-  - notebook: azure_python.ipynb
-    title: "Interacting with Azure Storage with Python"
-
----
-
-```{r lockfile}
-#| include: FALSE
-renv::use(lockfile = "renv.lock")
-```
-
-
-## Coffee & Coding
-
-In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
-
-In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
-
-## Posit Connect Pins
-
-
-```{r}
-#| eval: false
-
-
-# A brief intro to using {pins} to store, version, share and protect a dataset
-# on Posit Connect. Documentation: https://pins.rstudio.com/
-
-
-# Setup -------------------------------------------------------------------
-
-
-install.packages(c("pins", "dplyr")) # if not yet installed
-
-suppressPackageStartupMessages({
-  library(pins)
-  library(dplyr) # for wrangling and the 'starwars' demo dataset
-})
-
-board <- board_connect() # will error if you haven't authenticated before
-# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
-# • `server` and `key` not provided for `auth = 'manual'`
-# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
-# • rsconnect package not installed for `auth = 'rsconnect'`
-# Run `rlang::last_trace()` to see where the error occurred.
-
-# To authenticate
-# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
-# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
-# Your browser will open to the Posit Connect web page and you're prompted to
-# for your password. Enter it and you'll be authenticated.
-
-# Once authenticated
-board <- board_connect()
-# Connecting to Posit Connect 2024.03.0 at
-# <https://connect.strategyunitwm.nhs.uk>
-
-board |> pin_list() # see all the pins on that board
-
-
-# Create a pin ------------------------------------------------------------
-
-
-# Write a dataset to the board as a pin
-board |> pin_write(
-  x = starwars,
-  name = "starwars_demo"
-)
-# Guessing `type = 'rds'`
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_exists("starwars_demo")
-# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
-# not "starwars_demo".
-# [1] TRUE
-
-pin_name <- "matt.dray/starwars_demo"
-
-board |> pin_exists(pin_name) # logical, TRUE/FALSE
-board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
-board |> pin_browse(pin_name) # view the pin in the browser
-
-
-# Permissions -------------------------------------------------------------
-
-
-# You can let people see and edit a pin. Log into Posit Connect and select the
-# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
-# the 'sharing' options in the 'Access' tab.
-
-
-# Overwrite and version ---------------------------------------------------
-
-
-starwars_droids <- starwars |>
-  filter(species == "Droid") # beep boop
-
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_versions(pin_name) # see version history
-board |> pin_versions_prune(pin_name, n = 1) # remove history
-board |> pin_versions(pin_name)
-
-# What if you try to overwrite the data but it hasn't changed?
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# ! The hash of pin "matt.dray/starwars_demo" has not changed.
-# • Your pin will not be stored.
-
-
-# Use the pin -------------------------------------------------------------
-
-
-# You can read a pin to your local machine, or access it from a Quarto file
-# or Shiny app hosted on Connect, for example. If the output and the pin are
-# both on Connect, no authentication is required; the board is defaulted to
-# the Posit Connect instance where they're both hosted.
-
-board |>
-  pin_read(pin_name) |> # like you would use e.g. read_csv
-  with(data = _, plot(mass, height)) # wow!
-
-
-# Delete pin --------------------------------------------------------------
-
-
-board |> pin_exists(pin_name) # logical, good function for error handling
-board |> pin_delete(pin_name)
-board |> pin_exists(pin_name)
-
-```
-
-## Azure Storage in R
-
-You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
-
-Below is a sample .Renviron file
-
-
-```
-AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
-AZ_STORAGE_CONTAINER=container-name
-AZ_TENANT_ID=long-sequence-of-numbers-and-letters
-AZ_APP_ID=another-long-sequence-of-numbers-and-letters
-```
-
-```{r}
-#| eval: false
-
-install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
-
-# Load all environment variables
-ep_uri <- Sys.getenv("AZ_STORAGE_EP")
-app_id <- Sys.getenv("AZ_APP_ID")
-container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
-tenant <- Sys.getenv("AZ_TENANT_ID")
-
-# Authenticate
-token <- AzureAuth::get_azure_token(
-  "https://storage.azure.com",
-  tenant = tenant,
-  app = app_id,
-  auth_type = "device_code",
-)
-
-# If you have not authenticated before, you will be taken to an external page to
-# authenticate!Use your mlcsu.nhs.uk account.
-
-# Connect to container
-endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
-container <- AzureStor::storage_container(endpoint, container_name)
-
-# List files in container
-blob_list <- AzureStor::list_blobs(container)
-
-# If you get a 403 error when trying to interact with the container, you may
-# have to clear your Azure token and re-authenticate using a different browser.
-# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
-# AzureAuth::get_azure_token() step above.
-
-# Upload specific file to container
-AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
-
-# Upload contents of a local directory to container
-AzureStor::storage_multiupload(container, "data/*", "newdir")
-
-# Check files have uploaded
-blob_list <- AzureStor::list_blobs(container)
-
-# Load file directly from Azure container
-df_from_azure <- AzureStor::storage_read_csv(
-  container,
-  "newdir/cats.csv",
-  show_col_types = FALSE
-)
-
-# Load file directly from Azure container (by temporarily downloading file
-# and storing it in memory)
-parquet_in_memory <- AzureStor::storage_download(
-  container,
-  src = "newdir/cats.parquet", dest = NULL
-)
-parq_df <- arrow::read_parquet(parquet_in_memory)
-
-# Delete from Azure container (!!!)
-for (blobfile in blob_list$name) {
-  AzureStor::delete_storage_file(container, blobfile)
-}
-```
-
-## Azure Storage in Python
-
-This will use the same environment variables as the R version, just stored in a .env file instead. 
-
-We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
-
-{{< embed azure_python.ipynb echo=true   >}}
-
-
+---
+title: "Storing data safely"
+author: 
+  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
+  - "[Matt Dray](mailto:matt.dray@nhs.net)"
+  - "[Claire Welsh](mailto:claire.welsh8@nhs.net)"
+date: "2024-05-22"
+categories: [learning, R, Python]
+notebook-view:
+  - notebook: azure_python.ipynb
+    title: "Interacting with Azure Storage with Python"
+---
+
+```{r lockfile}
+#| include: FALSE
+renv::use(lockfile = "renv.lock")
+```
+
+::: callout-note
+**UPDATED**: Please see the Addendum to this blog, added 2025-04-03 regarding accessing data from SharePoint
+:::
+
+## Coffee & Coding
+
+In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
+
+In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
+
+## Posit Connect Pins
+
+```{r}
+#| eval: false
+
+
+# A brief intro to using {pins} to store, version, share and protect a dataset
+# on Posit Connect. Documentation: https://pins.rstudio.com/
+
+
+# Setup -------------------------------------------------------------------
+
+
+install.packages(c("pins", "dplyr")) # if not yet installed
+
+suppressPackageStartupMessages({
+  library(pins)
+  library(dplyr) # for wrangling and the 'starwars' demo dataset
+})
+
+board <- board_connect() # will error if you haven't authenticated before
+# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
+# • `server` and `key` not provided for `auth = 'manual'`
+# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
+# • rsconnect package not installed for `auth = 'rsconnect'`
+# Run `rlang::last_trace()` to see where the error occurred.
+
+# To authenticate
+# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
+# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
+# Your browser will open to the Posit Connect web page and you're prompted to
+# for your password. Enter it and you'll be authenticated.
+
+# Once authenticated
+board <- board_connect()
+# Connecting to Posit Connect 2024.03.0 at
+# <https://connect.strategyunitwm.nhs.uk>
+
+board |> pin_list() # see all the pins on that board
+
+
+# Create a pin ------------------------------------------------------------
+
+
+# Write a dataset to the board as a pin
+board |> pin_write(
+  x = starwars,
+  name = "starwars_demo"
+)
+# Guessing `type = 'rds'`
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_exists("starwars_demo")
+# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
+# not "starwars_demo".
+# [1] TRUE
+
+pin_name <- "matt.dray/starwars_demo"
+
+board |> pin_exists(pin_name) # logical, TRUE/FALSE
+board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
+board |> pin_browse(pin_name) # view the pin in the browser
+
+
+# Permissions -------------------------------------------------------------
+
+
+# You can let people see and edit a pin. Log into Posit Connect and select the
+# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
+# the 'sharing' options in the 'Access' tab.
+
+
+# Overwrite and version ---------------------------------------------------
+
+
+starwars_droids <- starwars |>
+  filter(species == "Droid") # beep boop
+
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_versions(pin_name) # see version history
+board |> pin_versions_prune(pin_name, n = 1) # remove history
+board |> pin_versions(pin_name)
+
+# What if you try to overwrite the data but it hasn't changed?
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# ! The hash of pin "matt.dray/starwars_demo" has not changed.
+# • Your pin will not be stored.
+
+
+# Use the pin -------------------------------------------------------------
+
+
+# You can read a pin to your local machine, or access it from a Quarto file
+# or Shiny app hosted on Connect, for example. If the output and the pin are
+# both on Connect, no authentication is required; the board is defaulted to
+# the Posit Connect instance where they're both hosted.
+
+board |>
+  pin_read(pin_name) |> # like you would use e.g. read_csv
+  with(data = _, plot(mass, height)) # wow!
+
+
+# Delete pin --------------------------------------------------------------
+
+
+board |> pin_exists(pin_name) # logical, good function for error handling
+board |> pin_delete(pin_name)
+board |> pin_exists(pin_name)
+
+```
+
+## Azure Storage in R
+
+You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
+
+Below is a sample .Renviron file
+
+```         
+AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
+AZ_STORAGE_CONTAINER=container-name
+AZ_TENANT_ID=long-sequence-of-numbers-and-letters
+AZ_APP_ID=another-long-sequence-of-numbers-and-letters
+```
+
+```{r}
+#| eval: false
+
+install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
+
+# Load all environment variables
+ep_uri <- Sys.getenv("AZ_STORAGE_EP")
+app_id <- Sys.getenv("AZ_APP_ID")
+container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
+tenant <- Sys.getenv("AZ_TENANT_ID")
+
+# Authenticate
+token <- AzureAuth::get_azure_token(
+  "https://storage.azure.com",
+  tenant = tenant,
+  app = app_id,
+  auth_type = "device_code",
+)
+
+# If you have not authenticated before, you will be taken to an external page to
+# authenticate!Use your mlcsu.nhs.uk account.
+
+# Connect to container
+endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
+container <- AzureStor::storage_container(endpoint, container_name)
+
+# List files in container
+blob_list <- AzureStor::list_blobs(container)
+
+# If you get a 403 error when trying to interact with the container, you may
+# have to clear your Azure token and re-authenticate using a different browser.
+# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
+# AzureAuth::get_azure_token() step above.
+
+# Upload specific file to container
+AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
+
+# Upload contents of a local directory to container
+AzureStor::storage_multiupload(container, "data/*", "newdir")
+
+# Check files have uploaded
+blob_list <- AzureStor::list_blobs(container)
+
+# Load file directly from Azure container
+df_from_azure <- AzureStor::storage_read_csv(
+  container,
+  "newdir/cats.csv",
+  show_col_types = FALSE
+)
+
+# Load file directly from Azure container (by temporarily downloading file
+# and storing it in memory)
+parquet_in_memory <- AzureStor::storage_download(
+  container,
+  src = "newdir/cats.parquet", dest = NULL
+)
+parq_df <- arrow::read_parquet(parquet_in_memory)
+
+# Delete from Azure container (!!!)
+for (blobfile in blob_list$name) {
+  AzureStor::delete_storage_file(container, blobfile)
+}
+```
+
+## Azure Storage in Python
+
+This will use the same environment variables as the R version, just stored in a .env file instead.
+
+We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
+
+{{< embed azure_python.ipynb echo=true >}}
+
+## ADDENDUM: Accessing data from SharePoint
+
+SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.
+
+Accessing SharePoint requires user authentication, which also needs to happen when reading data into R from SharePoint. Before you can access SharePoint content from a site, you first need to 'follow' that site.
+
+To access data on SharePoint,follow these steps:
+
+1.  Navigate to the SharePoint page that has the file of interest in it, using your browser.
path: blogs/posts/2024-05-22_storing-data-safely/index.qmd
user: login: matt-dray
body: Whitespace.
```suggestion
1. Navigate to the SharePoint page that has the file of interest in it, using your browser.
```
created_at: 2025-04-04T09:34:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,234 +1,274 @@
----
-title: "Storing data safely"
-author: 
-  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
-  - "[Matt Dray](mailto:matt.dray@nhs.net)"
-date: "2024-05-22"
-categories: [learning, R, Python]
-notebook-view:
-  - notebook: azure_python.ipynb
-    title: "Interacting with Azure Storage with Python"
-
----
-
-```{r lockfile}
-#| include: FALSE
-renv::use(lockfile = "renv.lock")
-```
-
-
-## Coffee & Coding
-
-In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
-
-In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
-
-## Posit Connect Pins
-
-
-```{r}
-#| eval: false
-
-
-# A brief intro to using {pins} to store, version, share and protect a dataset
-# on Posit Connect. Documentation: https://pins.rstudio.com/
-
-
-# Setup -------------------------------------------------------------------
-
-
-install.packages(c("pins", "dplyr")) # if not yet installed
-
-suppressPackageStartupMessages({
-  library(pins)
-  library(dplyr) # for wrangling and the 'starwars' demo dataset
-})
-
-board <- board_connect() # will error if you haven't authenticated before
-# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
-# • `server` and `key` not provided for `auth = 'manual'`
-# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
-# • rsconnect package not installed for `auth = 'rsconnect'`
-# Run `rlang::last_trace()` to see where the error occurred.
-
-# To authenticate
-# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
-# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
-# Your browser will open to the Posit Connect web page and you're prompted to
-# for your password. Enter it and you'll be authenticated.
-
-# Once authenticated
-board <- board_connect()
-# Connecting to Posit Connect 2024.03.0 at
-# <https://connect.strategyunitwm.nhs.uk>
-
-board |> pin_list() # see all the pins on that board
-
-
-# Create a pin ------------------------------------------------------------
-
-
-# Write a dataset to the board as a pin
-board |> pin_write(
-  x = starwars,
-  name = "starwars_demo"
-)
-# Guessing `type = 'rds'`
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_exists("starwars_demo")
-# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
-# not "starwars_demo".
-# [1] TRUE
-
-pin_name <- "matt.dray/starwars_demo"
-
-board |> pin_exists(pin_name) # logical, TRUE/FALSE
-board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
-board |> pin_browse(pin_name) # view the pin in the browser
-
-
-# Permissions -------------------------------------------------------------
-
-
-# You can let people see and edit a pin. Log into Posit Connect and select the
-# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
-# the 'sharing' options in the 'Access' tab.
-
-
-# Overwrite and version ---------------------------------------------------
-
-
-starwars_droids <- starwars |>
-  filter(species == "Droid") # beep boop
-
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_versions(pin_name) # see version history
-board |> pin_versions_prune(pin_name, n = 1) # remove history
-board |> pin_versions(pin_name)
-
-# What if you try to overwrite the data but it hasn't changed?
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# ! The hash of pin "matt.dray/starwars_demo" has not changed.
-# • Your pin will not be stored.
-
-
-# Use the pin -------------------------------------------------------------
-
-
-# You can read a pin to your local machine, or access it from a Quarto file
-# or Shiny app hosted on Connect, for example. If the output and the pin are
-# both on Connect, no authentication is required; the board is defaulted to
-# the Posit Connect instance where they're both hosted.
-
-board |>
-  pin_read(pin_name) |> # like you would use e.g. read_csv
-  with(data = _, plot(mass, height)) # wow!
-
-
-# Delete pin --------------------------------------------------------------
-
-
-board |> pin_exists(pin_name) # logical, good function for error handling
-board |> pin_delete(pin_name)
-board |> pin_exists(pin_name)
-
-```
-
-## Azure Storage in R
-
-You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
-
-Below is a sample .Renviron file
-
-
-```
-AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
-AZ_STORAGE_CONTAINER=container-name
-AZ_TENANT_ID=long-sequence-of-numbers-and-letters
-AZ_APP_ID=another-long-sequence-of-numbers-and-letters
-```
-
-```{r}
-#| eval: false
-
-install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
-
-# Load all environment variables
-ep_uri <- Sys.getenv("AZ_STORAGE_EP")
-app_id <- Sys.getenv("AZ_APP_ID")
-container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
-tenant <- Sys.getenv("AZ_TENANT_ID")
-
-# Authenticate
-token <- AzureAuth::get_azure_token(
-  "https://storage.azure.com",
-  tenant = tenant,
-  app = app_id,
-  auth_type = "device_code",
-)
-
-# If you have not authenticated before, you will be taken to an external page to
-# authenticate!Use your mlcsu.nhs.uk account.
-
-# Connect to container
-endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
-container <- AzureStor::storage_container(endpoint, container_name)
-
-# List files in container
-blob_list <- AzureStor::list_blobs(container)
-
-# If you get a 403 error when trying to interact with the container, you may
-# have to clear your Azure token and re-authenticate using a different browser.
-# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
-# AzureAuth::get_azure_token() step above.
-
-# Upload specific file to container
-AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
-
-# Upload contents of a local directory to container
-AzureStor::storage_multiupload(container, "data/*", "newdir")
-
-# Check files have uploaded
-blob_list <- AzureStor::list_blobs(container)
-
-# Load file directly from Azure container
-df_from_azure <- AzureStor::storage_read_csv(
-  container,
-  "newdir/cats.csv",
-  show_col_types = FALSE
-)
-
-# Load file directly from Azure container (by temporarily downloading file
-# and storing it in memory)
-parquet_in_memory <- AzureStor::storage_download(
-  container,
-  src = "newdir/cats.parquet", dest = NULL
-)
-parq_df <- arrow::read_parquet(parquet_in_memory)
-
-# Delete from Azure container (!!!)
-for (blobfile in blob_list$name) {
-  AzureStor::delete_storage_file(container, blobfile)
-}
-```
-
-## Azure Storage in Python
-
-This will use the same environment variables as the R version, just stored in a .env file instead. 
-
-We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
-
-{{< embed azure_python.ipynb echo=true   >}}
-
-
+---
+title: "Storing data safely"
+author: 
+  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
+  - "[Matt Dray](mailto:matt.dray@nhs.net)"
+  - "[Claire Welsh](mailto:claire.welsh8@nhs.net)"
+date: "2024-05-22"
+categories: [learning, R, Python]
+notebook-view:
+  - notebook: azure_python.ipynb
+    title: "Interacting with Azure Storage with Python"
+---
+
+```{r lockfile}
+#| include: FALSE
+renv::use(lockfile = "renv.lock")
+```
+
+::: callout-note
+**UPDATED**: Please see the Addendum to this blog, added 2025-04-03 regarding accessing data from SharePoint
+:::
+
+## Coffee & Coding
+
+In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
+
+In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
+
+## Posit Connect Pins
+
+```{r}
+#| eval: false
+
+
+# A brief intro to using {pins} to store, version, share and protect a dataset
+# on Posit Connect. Documentation: https://pins.rstudio.com/
+
+
+# Setup -------------------------------------------------------------------
+
+
+install.packages(c("pins", "dplyr")) # if not yet installed
+
+suppressPackageStartupMessages({
+  library(pins)
+  library(dplyr) # for wrangling and the 'starwars' demo dataset
+})
+
+board <- board_connect() # will error if you haven't authenticated before
+# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
+# • `server` and `key` not provided for `auth = 'manual'`
+# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
+# • rsconnect package not installed for `auth = 'rsconnect'`
+# Run `rlang::last_trace()` to see where the error occurred.
+
+# To authenticate
+# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
+# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
+# Your browser will open to the Posit Connect web page and you're prompted to
+# for your password. Enter it and you'll be authenticated.
+
+# Once authenticated
+board <- board_connect()
+# Connecting to Posit Connect 2024.03.0 at
+# <https://connect.strategyunitwm.nhs.uk>
+
+board |> pin_list() # see all the pins on that board
+
+
+# Create a pin ------------------------------------------------------------
+
+
+# Write a dataset to the board as a pin
+board |> pin_write(
+  x = starwars,
+  name = "starwars_demo"
+)
+# Guessing `type = 'rds'`
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_exists("starwars_demo")
+# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
+# not "starwars_demo".
+# [1] TRUE
+
+pin_name <- "matt.dray/starwars_demo"
+
+board |> pin_exists(pin_name) # logical, TRUE/FALSE
+board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
+board |> pin_browse(pin_name) # view the pin in the browser
+
+
+# Permissions -------------------------------------------------------------
+
+
+# You can let people see and edit a pin. Log into Posit Connect and select the
+# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
+# the 'sharing' options in the 'Access' tab.
+
+
+# Overwrite and version ---------------------------------------------------
+
+
+starwars_droids <- starwars |>
+  filter(species == "Droid") # beep boop
+
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_versions(pin_name) # see version history
+board |> pin_versions_prune(pin_name, n = 1) # remove history
+board |> pin_versions(pin_name)
+
+# What if you try to overwrite the data but it hasn't changed?
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# ! The hash of pin "matt.dray/starwars_demo" has not changed.
+# • Your pin will not be stored.
+
+
+# Use the pin -------------------------------------------------------------
+
+
+# You can read a pin to your local machine, or access it from a Quarto file
+# or Shiny app hosted on Connect, for example. If the output and the pin are
+# both on Connect, no authentication is required; the board is defaulted to
+# the Posit Connect instance where they're both hosted.
+
+board |>
+  pin_read(pin_name) |> # like you would use e.g. read_csv
+  with(data = _, plot(mass, height)) # wow!
+
+
+# Delete pin --------------------------------------------------------------
+
+
+board |> pin_exists(pin_name) # logical, good function for error handling
+board |> pin_delete(pin_name)
+board |> pin_exists(pin_name)
+
+```
+
+## Azure Storage in R
+
+You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
+
+Below is a sample .Renviron file
+
+```         
+AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
+AZ_STORAGE_CONTAINER=container-name
+AZ_TENANT_ID=long-sequence-of-numbers-and-letters
+AZ_APP_ID=another-long-sequence-of-numbers-and-letters
+```
+
+```{r}
+#| eval: false
+
+install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
+
+# Load all environment variables
+ep_uri <- Sys.getenv("AZ_STORAGE_EP")
+app_id <- Sys.getenv("AZ_APP_ID")
+container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
+tenant <- Sys.getenv("AZ_TENANT_ID")
+
+# Authenticate
+token <- AzureAuth::get_azure_token(
+  "https://storage.azure.com",
+  tenant = tenant,
+  app = app_id,
+  auth_type = "device_code",
+)
+
+# If you have not authenticated before, you will be taken to an external page to
+# authenticate!Use your mlcsu.nhs.uk account.
+
+# Connect to container
+endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
+container <- AzureStor::storage_container(endpoint, container_name)
+
+# List files in container
+blob_list <- AzureStor::list_blobs(container)
+
+# If you get a 403 error when trying to interact with the container, you may
+# have to clear your Azure token and re-authenticate using a different browser.
+# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
+# AzureAuth::get_azure_token() step above.
+
+# Upload specific file to container
+AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
+
+# Upload contents of a local directory to container
+AzureStor::storage_multiupload(container, "data/*", "newdir")
+
+# Check files have uploaded
+blob_list <- AzureStor::list_blobs(container)
+
+# Load file directly from Azure container
+df_from_azure <- AzureStor::storage_read_csv(
+  container,
+  "newdir/cats.csv",
+  show_col_types = FALSE
+)
+
+# Load file directly from Azure container (by temporarily downloading file
+# and storing it in memory)
+parquet_in_memory <- AzureStor::storage_download(
+  container,
+  src = "newdir/cats.parquet", dest = NULL
+)
+parq_df <- arrow::read_parquet(parquet_in_memory)
+
+# Delete from Azure container (!!!)
+for (blobfile in blob_list$name) {
+  AzureStor::delete_storage_file(container, blobfile)
+}
+```
+
+## Azure Storage in Python
+
+This will use the same environment variables as the R version, just stored in a .env file instead.
+
+We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
+
+{{< embed azure_python.ipynb echo=true >}}
+
+## ADDENDUM: Accessing data from SharePoint
+
+SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.
+
+Accessing SharePoint requires user authentication, which also needs to happen when reading data into R from SharePoint. Before you can access SharePoint content from a site, you first need to 'follow' that site.
+
+To access data on SharePoint,follow these steps:
+
+1.  Navigate to the SharePoint page that has the file of interest in it, using your browser.
+
+2.  Click the small star in the top right corner of the window, labelled 'Follow'
+
+![](Follow.jpg)
+
+3.  Open your environment file, which you can do by running `r file.edit("~/.Renviron")` within the RStudio console area.
path: blogs/posts/2024-05-22_storing-data-safely/index.qmd
user: login: matt-dray
body: Bit more on env vars/.Renviron. Maybe {usethis} for opening the file?
```suggestion
3. Open (or create) your project's `.Renviron` file, used for [storing environmental variables](https://docs.posit.co/ide/user/ide/guide/environments/r/managing-r.html#renviron). You can do this by running `r usethis::edit_r_environ()` from the R console.
```
created_at: 2025-04-04T09:34:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,234 +1,274 @@
----
-title: "Storing data safely"
-author: 
-  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
-  - "[Matt Dray](mailto:matt.dray@nhs.net)"
-date: "2024-05-22"
-categories: [learning, R, Python]
-notebook-view:
-  - notebook: azure_python.ipynb
-    title: "Interacting with Azure Storage with Python"
-
----
-
-```{r lockfile}
-#| include: FALSE
-renv::use(lockfile = "renv.lock")
-```
-
-
-## Coffee & Coding
-
-In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
-
-In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
-
-## Posit Connect Pins
-
-
-```{r}
-#| eval: false
-
-
-# A brief intro to using {pins} to store, version, share and protect a dataset
-# on Posit Connect. Documentation: https://pins.rstudio.com/
-
-
-# Setup -------------------------------------------------------------------
-
-
-install.packages(c("pins", "dplyr")) # if not yet installed
-
-suppressPackageStartupMessages({
-  library(pins)
-  library(dplyr) # for wrangling and the 'starwars' demo dataset
-})
-
-board <- board_connect() # will error if you haven't authenticated before
-# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
-# • `server` and `key` not provided for `auth = 'manual'`
-# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
-# • rsconnect package not installed for `auth = 'rsconnect'`
-# Run `rlang::last_trace()` to see where the error occurred.
-
-# To authenticate
-# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
-# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
-# Your browser will open to the Posit Connect web page and you're prompted to
-# for your password. Enter it and you'll be authenticated.
-
-# Once authenticated
-board <- board_connect()
-# Connecting to Posit Connect 2024.03.0 at
-# <https://connect.strategyunitwm.nhs.uk>
-
-board |> pin_list() # see all the pins on that board
-
-
-# Create a pin ------------------------------------------------------------
-
-
-# Write a dataset to the board as a pin
-board |> pin_write(
-  x = starwars,
-  name = "starwars_demo"
-)
-# Guessing `type = 'rds'`
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_exists("starwars_demo")
-# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
-# not "starwars_demo".
-# [1] TRUE
-
-pin_name <- "matt.dray/starwars_demo"
-
-board |> pin_exists(pin_name) # logical, TRUE/FALSE
-board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
-board |> pin_browse(pin_name) # view the pin in the browser
-
-
-# Permissions -------------------------------------------------------------
-
-
-# You can let people see and edit a pin. Log into Posit Connect and select the
-# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
-# the 'sharing' options in the 'Access' tab.
-
-
-# Overwrite and version ---------------------------------------------------
-
-
-starwars_droids <- starwars |>
-  filter(species == "Droid") # beep boop
-
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_versions(pin_name) # see version history
-board |> pin_versions_prune(pin_name, n = 1) # remove history
-board |> pin_versions(pin_name)
-
-# What if you try to overwrite the data but it hasn't changed?
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# ! The hash of pin "matt.dray/starwars_demo" has not changed.
-# • Your pin will not be stored.
-
-
-# Use the pin -------------------------------------------------------------
-
-
-# You can read a pin to your local machine, or access it from a Quarto file
-# or Shiny app hosted on Connect, for example. If the output and the pin are
-# both on Connect, no authentication is required; the board is defaulted to
-# the Posit Connect instance where they're both hosted.
-
-board |>
-  pin_read(pin_name) |> # like you would use e.g. read_csv
-  with(data = _, plot(mass, height)) # wow!
-
-
-# Delete pin --------------------------------------------------------------
-
-
-board |> pin_exists(pin_name) # logical, good function for error handling
-board |> pin_delete(pin_name)
-board |> pin_exists(pin_name)
-
-```
-
-## Azure Storage in R
-
-You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
-
-Below is a sample .Renviron file
-
-
-```
-AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
-AZ_STORAGE_CONTAINER=container-name
-AZ_TENANT_ID=long-sequence-of-numbers-and-letters
-AZ_APP_ID=another-long-sequence-of-numbers-and-letters
-```
-
-```{r}
-#| eval: false
-
-install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
-
-# Load all environment variables
-ep_uri <- Sys.getenv("AZ_STORAGE_EP")
-app_id <- Sys.getenv("AZ_APP_ID")
-container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
-tenant <- Sys.getenv("AZ_TENANT_ID")
-
-# Authenticate
-token <- AzureAuth::get_azure_token(
-  "https://storage.azure.com",
-  tenant = tenant,
-  app = app_id,
-  auth_type = "device_code",
-)
-
-# If you have not authenticated before, you will be taken to an external page to
-# authenticate!Use your mlcsu.nhs.uk account.
-
-# Connect to container
-endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
-container <- AzureStor::storage_container(endpoint, container_name)
-
-# List files in container
-blob_list <- AzureStor::list_blobs(container)
-
-# If you get a 403 error when trying to interact with the container, you may
-# have to clear your Azure token and re-authenticate using a different browser.
-# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
-# AzureAuth::get_azure_token() step above.
-
-# Upload specific file to container
-AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
-
-# Upload contents of a local directory to container
-AzureStor::storage_multiupload(container, "data/*", "newdir")
-
-# Check files have uploaded
-blob_list <- AzureStor::list_blobs(container)
-
-# Load file directly from Azure container
-df_from_azure <- AzureStor::storage_read_csv(
-  container,
-  "newdir/cats.csv",
-  show_col_types = FALSE
-)
-
-# Load file directly from Azure container (by temporarily downloading file
-# and storing it in memory)
-parquet_in_memory <- AzureStor::storage_download(
-  container,
-  src = "newdir/cats.parquet", dest = NULL
-)
-parq_df <- arrow::read_parquet(parquet_in_memory)
-
-# Delete from Azure container (!!!)
-for (blobfile in blob_list$name) {
-  AzureStor::delete_storage_file(container, blobfile)
-}
-```
-
-## Azure Storage in Python
-
-This will use the same environment variables as the R version, just stored in a .env file instead. 
-
-We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
-
-{{< embed azure_python.ipynb echo=true   >}}
-
-
+---
+title: "Storing data safely"
+author: 
+  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
+  - "[Matt Dray](mailto:matt.dray@nhs.net)"
+  - "[Claire Welsh](mailto:claire.welsh8@nhs.net)"
+date: "2024-05-22"
+categories: [learning, R, Python]
+notebook-view:
+  - notebook: azure_python.ipynb
+    title: "Interacting with Azure Storage with Python"
+---
+
+```{r lockfile}
+#| include: FALSE
+renv::use(lockfile = "renv.lock")
+```
+
+::: callout-note
+**UPDATED**: Please see the Addendum to this blog, added 2025-04-03 regarding accessing data from SharePoint
+:::
+
+## Coffee & Coding
+
+In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
+
+In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
+
+## Posit Connect Pins
+
+```{r}
+#| eval: false
+
+
+# A brief intro to using {pins} to store, version, share and protect a dataset
+# on Posit Connect. Documentation: https://pins.rstudio.com/
+
+
+# Setup -------------------------------------------------------------------
+
+
+install.packages(c("pins", "dplyr")) # if not yet installed
+
+suppressPackageStartupMessages({
+  library(pins)
+  library(dplyr) # for wrangling and the 'starwars' demo dataset
+})
+
+board <- board_connect() # will error if you haven't authenticated before
+# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
+# • `server` and `key` not provided for `auth = 'manual'`
+# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
+# • rsconnect package not installed for `auth = 'rsconnect'`
+# Run `rlang::last_trace()` to see where the error occurred.
+
+# To authenticate
+# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
+# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
+# Your browser will open to the Posit Connect web page and you're prompted to
+# for your password. Enter it and you'll be authenticated.
+
+# Once authenticated
+board <- board_connect()
+# Connecting to Posit Connect 2024.03.0 at
+# <https://connect.strategyunitwm.nhs.uk>
+
+board |> pin_list() # see all the pins on that board
+
+
+# Create a pin ------------------------------------------------------------
+
+
+# Write a dataset to the board as a pin
+board |> pin_write(
+  x = starwars,
+  name = "starwars_demo"
+)
+# Guessing `type = 'rds'`
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_exists("starwars_demo")
+# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
+# not "starwars_demo".
+# [1] TRUE
+
+pin_name <- "matt.dray/starwars_demo"
+
+board |> pin_exists(pin_name) # logical, TRUE/FALSE
+board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
+board |> pin_browse(pin_name) # view the pin in the browser
+
+
+# Permissions -------------------------------------------------------------
+
+
+# You can let people see and edit a pin. Log into Posit Connect and select the
+# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
+# the 'sharing' options in the 'Access' tab.
+
+
+# Overwrite and version ---------------------------------------------------
+
+
+starwars_droids <- starwars |>
+  filter(species == "Droid") # beep boop
+
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_versions(pin_name) # see version history
+board |> pin_versions_prune(pin_name, n = 1) # remove history
+board |> pin_versions(pin_name)
+
+# What if you try to overwrite the data but it hasn't changed?
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# ! The hash of pin "matt.dray/starwars_demo" has not changed.
+# • Your pin will not be stored.
+
+
+# Use the pin -------------------------------------------------------------
+
+
+# You can read a pin to your local machine, or access it from a Quarto file
+# or Shiny app hosted on Connect, for example. If the output and the pin are
+# both on Connect, no authentication is required; the board is defaulted to
+# the Posit Connect instance where they're both hosted.
+
+board |>
+  pin_read(pin_name) |> # like you would use e.g. read_csv
+  with(data = _, plot(mass, height)) # wow!
+
+
+# Delete pin --------------------------------------------------------------
+
+
+board |> pin_exists(pin_name) # logical, good function for error handling
+board |> pin_delete(pin_name)
+board |> pin_exists(pin_name)
+
+```
+
+## Azure Storage in R
+
+You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
+
+Below is a sample .Renviron file
+
+```         
+AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
+AZ_STORAGE_CONTAINER=container-name
+AZ_TENANT_ID=long-sequence-of-numbers-and-letters
+AZ_APP_ID=another-long-sequence-of-numbers-and-letters
+```
+
+```{r}
+#| eval: false
+
+install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
+
+# Load all environment variables
+ep_uri <- Sys.getenv("AZ_STORAGE_EP")
+app_id <- Sys.getenv("AZ_APP_ID")
+container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
+tenant <- Sys.getenv("AZ_TENANT_ID")
+
+# Authenticate
+token <- AzureAuth::get_azure_token(
+  "https://storage.azure.com",
+  tenant = tenant,
+  app = app_id,
+  auth_type = "device_code",
+)
+
+# If you have not authenticated before, you will be taken to an external page to
+# authenticate!Use your mlcsu.nhs.uk account.
+
+# Connect to container
+endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
+container <- AzureStor::storage_container(endpoint, container_name)
+
+# List files in container
+blob_list <- AzureStor::list_blobs(container)
+
+# If you get a 403 error when trying to interact with the container, you may
+# have to clear your Azure token and re-authenticate using a different browser.
+# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
+# AzureAuth::get_azure_token() step above.
+
+# Upload specific file to container
+AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
+
+# Upload contents of a local directory to container
+AzureStor::storage_multiupload(container, "data/*", "newdir")
+
+# Check files have uploaded
+blob_list <- AzureStor::list_blobs(container)
+
+# Load file directly from Azure container
+df_from_azure <- AzureStor::storage_read_csv(
+  container,
+  "newdir/cats.csv",
+  show_col_types = FALSE
+)
+
+# Load file directly from Azure container (by temporarily downloading file
+# and storing it in memory)
+parquet_in_memory <- AzureStor::storage_download(
+  container,
+  src = "newdir/cats.parquet", dest = NULL
+)
+parq_df <- arrow::read_parquet(parquet_in_memory)
+
+# Delete from Azure container (!!!)
+for (blobfile in blob_list$name) {
+  AzureStor::delete_storage_file(container, blobfile)
+}
+```
+
+## Azure Storage in Python
+
+This will use the same environment variables as the R version, just stored in a .env file instead.
+
+We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
+
+{{< embed azure_python.ipynb echo=true >}}
+
+## ADDENDUM: Accessing data from SharePoint
+
+SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.
+
+Accessing SharePoint requires user authentication, which also needs to happen when reading data into R from SharePoint. Before you can access SharePoint content from a site, you first need to 'follow' that site.
+
+To access data on SharePoint,follow these steps:
+
+1.  Navigate to the SharePoint page that has the file of interest in it, using your browser.
+
+2.  Click the small star in the top right corner of the window, labelled 'Follow'
+
+![](Follow.jpg)
+
+3.  Open your environment file, which you can do by running `r file.edit("~/.Renviron")` within the RStudio console area.
+
+4.  Save two new environment variables; one is the name of the site (i.e. the name at the top of the browser screen), and the other is the full file path from below the 'Documents' folder to the file itself, including the file type extension. Save and close the .Renviron file then restart your R session (CTRL \> Shift \> F10 on Windows machines)
+
+5.  Use the code below to read in an Excel file (xlsx) into R memory.
path: blogs/posts/2024-05-22_storing-data-safely/index.qmd
user: login: matt-dray
body: ```suggestion
5. Use the code below to read in an Excel file (xlsx) into R memory, or adapt it to read other file types.
```
created_at: 2025-04-04T09:37:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,234 +1,274 @@
----
-title: "Storing data safely"
-author: 
-  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
-  - "[Matt Dray](mailto:matt.dray@nhs.net)"
-date: "2024-05-22"
-categories: [learning, R, Python]
-notebook-view:
-  - notebook: azure_python.ipynb
-    title: "Interacting with Azure Storage with Python"
-
----
-
-```{r lockfile}
-#| include: FALSE
-renv::use(lockfile = "renv.lock")
-```
-
-
-## Coffee & Coding
-
-In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
-
-In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
-
-## Posit Connect Pins
-
-
-```{r}
-#| eval: false
-
-
-# A brief intro to using {pins} to store, version, share and protect a dataset
-# on Posit Connect. Documentation: https://pins.rstudio.com/
-
-
-# Setup -------------------------------------------------------------------
-
-
-install.packages(c("pins", "dplyr")) # if not yet installed
-
-suppressPackageStartupMessages({
-  library(pins)
-  library(dplyr) # for wrangling and the 'starwars' demo dataset
-})
-
-board <- board_connect() # will error if you haven't authenticated before
-# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
-# • `server` and `key` not provided for `auth = 'manual'`
-# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
-# • rsconnect package not installed for `auth = 'rsconnect'`
-# Run `rlang::last_trace()` to see where the error occurred.
-
-# To authenticate
-# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
-# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
-# Your browser will open to the Posit Connect web page and you're prompted to
-# for your password. Enter it and you'll be authenticated.
-
-# Once authenticated
-board <- board_connect()
-# Connecting to Posit Connect 2024.03.0 at
-# <https://connect.strategyunitwm.nhs.uk>
-
-board |> pin_list() # see all the pins on that board
-
-
-# Create a pin ------------------------------------------------------------
-
-
-# Write a dataset to the board as a pin
-board |> pin_write(
-  x = starwars,
-  name = "starwars_demo"
-)
-# Guessing `type = 'rds'`
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_exists("starwars_demo")
-# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
-# not "starwars_demo".
-# [1] TRUE
-
-pin_name <- "matt.dray/starwars_demo"
-
-board |> pin_exists(pin_name) # logical, TRUE/FALSE
-board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
-board |> pin_browse(pin_name) # view the pin in the browser
-
-
-# Permissions -------------------------------------------------------------
-
-
-# You can let people see and edit a pin. Log into Posit Connect and select the
-# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
-# the 'sharing' options in the 'Access' tab.
-
-
-# Overwrite and version ---------------------------------------------------
-
-
-starwars_droids <- starwars |>
-  filter(species == "Droid") # beep boop
-
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# Writing to pin 'matt.dray/starwars_demo'
-
-board |> pin_versions(pin_name) # see version history
-board |> pin_versions_prune(pin_name, n = 1) # remove history
-board |> pin_versions(pin_name)
-
-# What if you try to overwrite the data but it hasn't changed?
-board |> pin_write(
-  starwars_droids,
-  pin_name,
-  type = "rds"
-)
-# ! The hash of pin "matt.dray/starwars_demo" has not changed.
-# • Your pin will not be stored.
-
-
-# Use the pin -------------------------------------------------------------
-
-
-# You can read a pin to your local machine, or access it from a Quarto file
-# or Shiny app hosted on Connect, for example. If the output and the pin are
-# both on Connect, no authentication is required; the board is defaulted to
-# the Posit Connect instance where they're both hosted.
-
-board |>
-  pin_read(pin_name) |> # like you would use e.g. read_csv
-  with(data = _, plot(mass, height)) # wow!
-
-
-# Delete pin --------------------------------------------------------------
-
-
-board |> pin_exists(pin_name) # logical, good function for error handling
-board |> pin_delete(pin_name)
-board |> pin_exists(pin_name)
-
-```
-
-## Azure Storage in R
-
-You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
-
-Below is a sample .Renviron file
-
-
-```
-AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
-AZ_STORAGE_CONTAINER=container-name
-AZ_TENANT_ID=long-sequence-of-numbers-and-letters
-AZ_APP_ID=another-long-sequence-of-numbers-and-letters
-```
-
-```{r}
-#| eval: false
-
-install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
-
-# Load all environment variables
-ep_uri <- Sys.getenv("AZ_STORAGE_EP")
-app_id <- Sys.getenv("AZ_APP_ID")
-container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
-tenant <- Sys.getenv("AZ_TENANT_ID")
-
-# Authenticate
-token <- AzureAuth::get_azure_token(
-  "https://storage.azure.com",
-  tenant = tenant,
-  app = app_id,
-  auth_type = "device_code",
-)
-
-# If you have not authenticated before, you will be taken to an external page to
-# authenticate!Use your mlcsu.nhs.uk account.
-
-# Connect to container
-endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
-container <- AzureStor::storage_container(endpoint, container_name)
-
-# List files in container
-blob_list <- AzureStor::list_blobs(container)
-
-# If you get a 403 error when trying to interact with the container, you may
-# have to clear your Azure token and re-authenticate using a different browser.
-# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
-# AzureAuth::get_azure_token() step above.
-
-# Upload specific file to container
-AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
-
-# Upload contents of a local directory to container
-AzureStor::storage_multiupload(container, "data/*", "newdir")
-
-# Check files have uploaded
-blob_list <- AzureStor::list_blobs(container)
-
-# Load file directly from Azure container
-df_from_azure <- AzureStor::storage_read_csv(
-  container,
-  "newdir/cats.csv",
-  show_col_types = FALSE
-)
-
-# Load file directly from Azure container (by temporarily downloading file
-# and storing it in memory)
-parquet_in_memory <- AzureStor::storage_download(
-  container,
-  src = "newdir/cats.parquet", dest = NULL
-)
-parq_df <- arrow::read_parquet(parquet_in_memory)
-
-# Delete from Azure container (!!!)
-for (blobfile in blob_list$name) {
-  AzureStor::delete_storage_file(container, blobfile)
-}
-```
-
-## Azure Storage in Python
-
-This will use the same environment variables as the R version, just stored in a .env file instead. 
-
-We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
-
-{{< embed azure_python.ipynb echo=true   >}}
-
-
+---
+title: "Storing data safely"
+author: 
+  - "[YiWen Hon](mailto:yiwen.hon1@nhs.net)"
+  - "[Matt Dray](mailto:matt.dray@nhs.net)"
+  - "[Claire Welsh](mailto:claire.welsh8@nhs.net)"
+date: "2024-05-22"
+categories: [learning, R, Python]
+notebook-view:
+  - notebook: azure_python.ipynb
+    title: "Interacting with Azure Storage with Python"
+---
+
+```{r lockfile}
+#| include: FALSE
+renv::use(lockfile = "renv.lock")
+```
+
+::: callout-note
+**UPDATED**: Please see the Addendum to this blog, added 2025-04-03 regarding accessing data from SharePoint
+:::
+
+## Coffee & Coding
+
+In a recent Coffee & Coding session we chatted about storing data safely for use in Reproducible Analytical Pipelines (RAP), and [the slides from the presentation are now available](https://the-strategy-unit.github.io/data_science/presentations/2024-05-16_store-data-safely/). We discussed the use of [Posit Connect Pins](https://docs.posit.co/connect/user/pins/) and [Azure Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/).
+
+In order to avoid duplication, this blog post will not cover the pros and cons of each approach, and will instead focus on documenting the code that was used in our live demonstrations. I would recommend that you look through the slides before using the code in this blogpost and have them alongside, as they provide lots of useful context!
+
+## Posit Connect Pins
+
+```{r}
+#| eval: false
+
+
+# A brief intro to using {pins} to store, version, share and protect a dataset
+# on Posit Connect. Documentation: https://pins.rstudio.com/
+
+
+# Setup -------------------------------------------------------------------
+
+
+install.packages(c("pins", "dplyr")) # if not yet installed
+
+suppressPackageStartupMessages({
+  library(pins)
+  library(dplyr) # for wrangling and the 'starwars' demo dataset
+})
+
+board <- board_connect() # will error if you haven't authenticated before
+# Error in `check_auth()`: ! auth = `auto` has failed to find a way to authenticate:
+# • `server` and `key` not provided for `auth = 'manual'`
+# • Can't find CONNECT_SERVER and CONNECT_API_KEY envvars for `auth = 'envvar'`
+# • rsconnect package not installed for `auth = 'rsconnect'`
+# Run `rlang::last_trace()` to see where the error occurred.
+
+# To authenticate
+# In RStudio: Tools > Global Options > Publishing > Connect... > Posit Connect
+# public URL of the Strategy Unit Posit Connect Server: connect.strategyunitwm.nhs.uk
+# Your browser will open to the Posit Connect web page and you're prompted to
+# for your password. Enter it and you'll be authenticated.
+
+# Once authenticated
+board <- board_connect()
+# Connecting to Posit Connect 2024.03.0 at
+# <https://connect.strategyunitwm.nhs.uk>
+
+board |> pin_list() # see all the pins on that board
+
+
+# Create a pin ------------------------------------------------------------
+
+
+# Write a dataset to the board as a pin
+board |> pin_write(
+  x = starwars,
+  name = "starwars_demo"
+)
+# Guessing `type = 'rds'`
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_exists("starwars_demo")
+# ! Use a fully specified name including user name: "matt.dray/starwars_demo",
+# not "starwars_demo".
+# [1] TRUE
+
+pin_name <- "matt.dray/starwars_demo"
+
+board |> pin_exists(pin_name) # logical, TRUE/FALSE
+board |> pin_meta(pin_name) # metadata, see also 'metadata' arg in pin_write()
+board |> pin_browse(pin_name) # view the pin in the browser
+
+
+# Permissions -------------------------------------------------------------
+
+
+# You can let people see and edit a pin. Log into Posit Connect and select the
+# pin under 'Content'. In the 'Settings' panel on the right-hand side, adjust
+# the 'sharing' options in the 'Access' tab.
+
+
+# Overwrite and version ---------------------------------------------------
+
+
+starwars_droids <- starwars |>
+  filter(species == "Droid") # beep boop
+
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# Writing to pin 'matt.dray/starwars_demo'
+
+board |> pin_versions(pin_name) # see version history
+board |> pin_versions_prune(pin_name, n = 1) # remove history
+board |> pin_versions(pin_name)
+
+# What if you try to overwrite the data but it hasn't changed?
+board |> pin_write(
+  starwars_droids,
+  pin_name,
+  type = "rds"
+)
+# ! The hash of pin "matt.dray/starwars_demo" has not changed.
+# • Your pin will not be stored.
+
+
+# Use the pin -------------------------------------------------------------
+
+
+# You can read a pin to your local machine, or access it from a Quarto file
+# or Shiny app hosted on Connect, for example. If the output and the pin are
+# both on Connect, no authentication is required; the board is defaulted to
+# the Posit Connect instance where they're both hosted.
+
+board |>
+  pin_read(pin_name) |> # like you would use e.g. read_csv
+  with(data = _, plot(mass, height)) # wow!
+
+
+# Delete pin --------------------------------------------------------------
+
+
+board |> pin_exists(pin_name) # logical, good function for error handling
+board |> pin_delete(pin_name)
+board |> pin_exists(pin_name)
+
+```
+
+## Azure Storage in R
+
+You will need an .Renviron file with the four environment variables listed below for the code to work. This .Renviron file should be ignored by git. You can share the contents of .Renviron files with other team members via Teams, email, or Sharepoint.
+
+Below is a sample .Renviron file
+
+```         
+AZ_STORAGE_EP=https://STORAGEACCOUNT.blob.core.windows.net/
+AZ_STORAGE_CONTAINER=container-name
+AZ_TENANT_ID=long-sequence-of-numbers-and-letters
+AZ_APP_ID=another-long-sequence-of-numbers-and-letters
+```
+
+```{r}
+#| eval: false
+
+install.packages(c("AzureAuth", "AzureStor", "arrow")) # if not yet installed
+
+# Load all environment variables
+ep_uri <- Sys.getenv("AZ_STORAGE_EP")
+app_id <- Sys.getenv("AZ_APP_ID")
+container_name <- Sys.getenv("AZ_STORAGE_CONTAINER")
+tenant <- Sys.getenv("AZ_TENANT_ID")
+
+# Authenticate
+token <- AzureAuth::get_azure_token(
+  "https://storage.azure.com",
+  tenant = tenant,
+  app = app_id,
+  auth_type = "device_code",
+)
+
+# If you have not authenticated before, you will be taken to an external page to
+# authenticate!Use your mlcsu.nhs.uk account.
+
+# Connect to container
+endpoint <- AzureStor::blob_endpoint(ep_uri, token = token)
+container <- AzureStor::storage_container(endpoint, container_name)
+
+# List files in container
+blob_list <- AzureStor::list_blobs(container)
+
+# If you get a 403 error when trying to interact with the container, you may
+# have to clear your Azure token and re-authenticate using a different browser.
+# Use AzureAuth::clean_token_directory() to clear your token, then repeat the
+# AzureAuth::get_azure_token() step above.
+
+# Upload specific file to container
+AzureStor::storage_upload(container, "data/ronald.jpeg", "newdir/ronald.jpeg")
+
+# Upload contents of a local directory to container
+AzureStor::storage_multiupload(container, "data/*", "newdir")
+
+# Check files have uploaded
+blob_list <- AzureStor::list_blobs(container)
+
+# Load file directly from Azure container
+df_from_azure <- AzureStor::storage_read_csv(
+  container,
+  "newdir/cats.csv",
+  show_col_types = FALSE
+)
+
+# Load file directly from Azure container (by temporarily downloading file
+# and storing it in memory)
+parquet_in_memory <- AzureStor::storage_download(
+  container,
+  src = "newdir/cats.parquet", dest = NULL
+)
+parq_df <- arrow::read_parquet(parquet_in_memory)
+
+# Delete from Azure container (!!!)
+for (blobfile in blob_list$name) {
+  AzureStor::delete_storage_file(container, blobfile)
+}
+```
+
+## Azure Storage in Python
+
+This will use the same environment variables as the R version, just stored in a .env file instead.
+
+We didn't cover this in the presentation, so it's not in the slides, but the code should be self-explanatory.
+
+{{< embed azure_python.ipynb echo=true >}}
+
+## ADDENDUM: Accessing data from SharePoint
+
+SharePoint is a Microsoft product, which is a content/knowledge management tool. Many teams across the NHS use SharePoint for all sorts of file types that need to be preserved or shared within and between teams, but also need to be kept secure.
+
+Accessing SharePoint requires user authentication, which also needs to happen when reading data into R from SharePoint. Before you can access SharePoint content from a site, you first need to 'follow' that site.
+
+To access data on SharePoint,follow these steps:
+
+1.  Navigate to the SharePoint page that has the file of interest in it, using your browser.
+
+2.  Click the small star in the top right corner of the window, labelled 'Follow'
+
+![](Follow.jpg)
+
+3.  Open your environment file, which you can do by running `r file.edit("~/.Renviron")` within the RStudio console area.
+
+4.  Save two new environment variables; one is the name of the site (i.e. the name at the top of the browser screen), and the other is the full file path from below the 'Documents' folder to the file itself, including the file type extension. Save and close the .Renviron file then restart your R session (CTRL \> Shift \> F10 on Windows machines)
+
+5.  Use the code below to read in an Excel file (xlsx) into R memory.
+
+```{r}
+#| eval: false
+#| echo: true
+
+# Read in the securely saved path and site variables
+sharepoint_site <- Sys.getenv("SITE_NAME")
+
+template_path <- Sys.getenv("PATH_TO_FILE.xlsx")
path: blogs/posts/2024-05-22_storing-data-safely/index.qmd
user: login: matt-dray
body: To match the suggested var names above.
```suggestion
sharepoint_site <- Sys.getenv("SP_SITE_NAME")
template_path <- Sys.getenv("SP_FILE_PATH")
```
created_at: 2025-04-04T09:38:12Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-03T13:11:31Z
body: [PR Preview Action](https://github.com/rossjrw/pr-preview-action) v1.4.8
:---:
Preview removed because the pull request was closed.
2025-04-04 10:10 UTC
<!-- Sticky Pull Request Commentpr-preview -->
author_association: NONE
user: login: ChrisBeeley
created_at: 2025-04-03T16:29:14Z
body: I can get round it I think but I think this PR has the whitespace problem that YiWen addressed in a section of this repo recently 
author_association: MEMBER
user: login: DCEW
created_at: 2025-04-03T16:30:38Z
body: Oh that's odd! I followed all the advice she gave about settings to prevent it, and this PR was created since that! Can't say I really understood it though...
author_association: CONTRIBUTOR
user: login: DCEW
created_at: 2025-04-04T09:20:15Z
body: Incidentally for anyone interested - the problem of PRs showing massive chunks of code highlighted as diffs instead of single lines, can be worked around by adding '?w=1' to the end of the URL! Well done @ChrisBeeley for spotting this! Sadly the git config etc bits didn't stop the whitespace issue from persisting in my most recent PRs. Will look into it at some point.
author_association: CONTRIBUTOR
user: login: matt-dray
created_at: 2025-04-04T09:44:59Z
body: Re whitespace in this PR, won't it also depend on the settings of the original file? So if the original didn't use posix LF, then we're now righting that wrong? 
author_association: CONTRIBUTOR
user: login: ChrisBeeley
created_at: 2025-04-04T09:56:31Z
body: > Re whitespace in this PR, won't it also depend on the settings of the original file? So if the original didn't use posix LF, then we're now righting that wrong?

Maybe you're right Matt, I didn't think of that. Let's just persist with Posix line returns for a bit and hopefully it will go away
author_association: MEMBER
user: login: matt-dray
created_at: 2025-04-04T10:04:27Z
body: > Incidentally for anyone interested - the problem of PRs showing massive chunks of code highlighted as diffs instead of single lines, can be worked around by adding '?w=1' to the end of the URL! Well done @ChrisBeeley for spotting this! Sadly the git config etc bits didn't stop the whitespace issue from persisting in my most recent PRs. Will look into it at some point.

Also see here:

![image](https://github.com/user-attachments/assets/82e61bad-dba7-4244-8630-37f32fbed581)

author_association: CONTRIBUTOR