Repository: Azure/AgentBaker
Pull request #6082: Add function to extract and cache CoreDNS Binary in VHD for localdns
Author: login: SriHarsha001
Created: 2025-03-25T19:51:44Z | Merged: 2025-03-27T22:28:37Z

**What type of PR is this?**
feature

**What this PR does / why we need it**:
This PR adds a function to extract a coredns binary from cached coredns container images in the VHD. It extracts binary of n-1 coredns image version and latest revision version (v.Major.Minor.Patch-Revision) and copies it to /opt/azure/containers/localdns/binary/coredns path. This binary will be used by localdns systemd unit. The function also handles the cleanup of temporary directories and unmounting of images.

**Which issue(s) this PR fixes**:
<!-- 
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes # None

**Requirements**:
Build

- [ ] uses [conventional commit messages](https://www.conventionalcommits.org/)
  <!-- Common commit types:
        build: Build 🏭
        chore: Maintenance 🔧
        ci: Continuous Integration 💜
        docs: Documentation 📘
        feat: Features 🌈
        fix: Bug Fixes 🐞
        perf: Performance Improvements 🚀
        refactor: Code Refactoring 💎
        revert: Revert Change ◀️
        style: Code Style 🎶
        security: Security Fix 🛡️
        test: Testing 💚 -->
- [ ] includes documentation
- [ ] adds unit tests
- [ ] tested upgrade from previous version

**Special notes for your reviewer**:
Not sure if there is a way to write unit tests for this file. 

**Release note**:

```
none
```



Diff:
diff --git a/vhdbuilder/packer/install-dependencies.sh b/vhdbuilder/packer/install-dependencies.sh
index 0b3b539a70b..b571a9be64c 100644
--- a/vhdbuilder/packer/install-dependencies.sh
+++ b/vhdbuilder/packer/install-dependencies.sh
@@ -609,6 +609,113 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDnsBinary() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags (v1.12.0-1 format) and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Function to check version format (vMajor.Minor.Patch).
+  validate_version_format() {
+    local version=$1
+    if [[ ! "$version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+      echo "Error: Invalid coredns version format. Expected vMajor.Minor.Patch, got $version" >> "${VHD_LOGS_FILEPATH}"
+      return 1
+    fi
+    return 0
+  }
+
+  # Determine latest version (eg. v1.12.0-1).
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision. eg - v1.12.0).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    # Extract major.minor.patch (eg - v1.12.0).
+    local vMajorMinorPatch="${tag%-*}"
+    if ! validate_version_format "$vMajorMinorPatch"; then
+      exit 1
+    fi
+
+    if [[ "${vMajorMinorPatch}" != "${latest_vMajorMinorPatch}" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "${previous_coredns_tag}" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
+    local retry_count=0
+    while [[ $retry_count -lt $max_retries ]]; do
+      if ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+        break
+      fi
+      echo "Warning: Failed to mount ${coredns_image_url}, retrying..." >> "${VHD_LOGS_FILEPATH}"
+      sleep 2
+      ((retry_count++))
+    done
+
+    if [[ $retry_count -eq $max_retries ]]; then
+      echo "Error: Failed to mount ${coredns_image_url} after ${max_retries} attempts." >> "${VHD_LOGS_FILEPATH}"
+      exit 1
+    fi
+
+    local coredns_binary="${ctr_temp}/usr/bin/coredns"
+    if [[ -f "${coredns_binary}" ]]; then
+      cp "${coredns_binary}" "${LOCALDNS_BINARY_PATH}/coredns" || {
+        echo "Error: Failed to copy coredns binary of ${previous_coredns_tag}" >> "${VHD_LOGS_FILEPATH}"
+        exit 1
+      }
+      echo "Successfully copied coredns binary of ${previous_coredns_tag}" >> "${VHD_LOGS_FILEPATH}"
+    else
+      echo "Coredns binary not found for ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+    fi
+
+    ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+    rm -rf "${ctr_temp}"
+  done
+
+  # Clear the trap.
+  trap - EXIT ABRT ERR INT PIPE QUIT TERM
+}
+
+extractAndCacheCoreDnsBinary
+
 rm -f ./azcopy # cleanup immediately after usage will return in two downloads
 echo "install-dependencies step completed successfully"
 capture_benchmark "${SCRIPT_NAME}_overall" true
diff --git a/vhdbuilder/packer/test/linux-vhd-content-test.sh b/vhdbuilder/packer/test/linux-vhd-content-test.sh
index 8d202ff5fdb..ec802fb3bac 100644
--- a/vhdbuilder/packer/test/linux-vhd-content-test.sh
+++ b/vhdbuilder/packer/test/linux-vhd-content-test.sh
@@ -1207,6 +1207,85 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDnsBinaryExtractedAndCached"
+  local os_version=$1
+    # Ubuntu 18.04 and 20.04 ship with GLIBC 2.27 and 2.31, respectively.
+  # CoreDNS binary is built with GLIBC 2.32+, which is not compatible with 18.04 and 20.04 OS versions.
+  # Therefore, we skip the test for these OS versions here.
+  # Validation in AKS RP will be done to ensure localdns is not enabled for these OS versions.
+  if [[ ${os_version} == "18.04" || ${os_version} == "20.04" ]]; then
+    # For Ubuntu 18.04 and 20.04, the coredns binary is located in /opt/azure/containers/localdns/binary/coredns
+    echo "$test: CoreDNS is not supported on OS version: ${os_version}"
+    return 0
+  fi
+
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+
+  echo "$test: Checking for existence of coredns binary at ${binaryPath}"
+
+  if [[ ! -f "${binaryPath}" ]]; then
+    err "$test: coredns binary does not exist at ${binaryPath}"
+    return 1
+  fi
+
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    err "$test: No CoreDNS images found in the local container images"
+    return 1
+  fi
+
+  # Extract available coredns image tags (v1.12.0-1 format) and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version (eg. v1.12.0-1).
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision. eg - v1.12.0).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    # Extract major.minor.patch (eg - v1.12.0).
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "${vMajorMinorPatch}" != "${latest_vMajorMinorPatch}" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after the next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "$test: Warning: Previous version not found, using the latest version: ${latest_coredns_tag}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  local expectedVersion="$previous_coredns_tag"
+  local expectedVersionWithoutV="${expectedVersion#v}"
+  echo "$test: Expected CoreDNS version (n-1 latest revision): ${expectedVersionWithoutV}"
+
+  # Get the actual version from the extracted CoreDNS binary
+  local actualVersion
+  actualVersion=$("$binaryPath" --version | awk -F'-' '{print $2}')
+
+  local actualVersionWithoutV="${actualVersion#v}"
+  if [[ -z "${actualVersionWithoutV}" ]]; then
+    err "$test: Failed to retrieve CoreDNS version from $binaryPath"
+    return 1
+  fi
+
+  echo "$test: Verify extracted CoreDNS version: ${actualVersionWithoutV}"
+
+  if [[ "${actualVersion%-*}" != "${expectedVersionWithoutV%-*}" ]]; then
+    echo "$test: Extracted CoreDNS version: ${actualVersion} does not match expected version: ${expectedVersionWithoutV}"
+    return 1
+  fi
+
+  echo "$test: Expected version: ${expectedVersionWithoutV} of coredns binary is extracted and cached at ${binaryPath}"
+  return 0
+}
+
 # As we call these tests, we need to bear in mind how the test results are processed by the
 # the caller in run-tests.sh. That code uses az vm run-command invoke to run this script
 # on a VM. It then looks at stderr to see if any errors were reported. Notably it doesn't
@@ -1248,4 +1327,5 @@ testUmaskSettings
 testContainerImagePrefetchScript
 testAKSNodeControllerBinary
 testAKSNodeControllerService
-testLtsKernel $OS_VERSION $OS_SKU $ENABLE_FIPS
\ No newline at end of file
+testLtsKernel $OS_VERSION $OS_SKU $ENABLE_FIPS
+testCoreDnsBinaryExtractedAndCached $OS_VERSION
\ No newline at end of file


Reviews:
user: login: yewmsft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T23:04:32Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T23:09:57Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T23:12:51Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T23:37:47Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T23:43:14Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T23:44:16Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T23:44:23Z
user: login: cameronmeissner
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T23:48:40Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T00:01:48Z
user: login: yewmsft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T02:50:16Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T06:00:20Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T06:27:55Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T06:31:43Z
user: login: Devinwong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T17:29:58Z
user: login: Devinwong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T17:30:31Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T19:14:21Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T19:17:48Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T19:19:57Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T19:21:30Z
user: login: Devinwong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T19:35:40Z
user: login: Devinwong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T19:37:06Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T20:11:47Z
user: login: yewmsft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T20:41:27Z
user: login: yewmsft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T20:53:45Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T20:56:58Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T20:58:20Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T21:22:38Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T22:29:11Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T22:29:43Z
user: login: yewmsft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-27T01:18:46Z
user: login: yewmsft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-27T01:20:11Z
user: login: yewmsft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-27T01:26:24Z
user: login: Devinwong
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-27T16:51:33Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T19:44:47Z
user: login: yewmsft
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-27T20:57:14Z
user: login: SriHarsha001
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T21:00:39Z
user: login: yewmsft
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-27T21:08:08Z

Review comments:
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: do we have UT for this?
created_at: 2025-03-25T22:08:49Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: what if ${LOCALDNS_BINARY_PATH} already exists? Maybe clean this up as well?
created_at: 2025-03-25T22:09:34Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: handle anomalies here? what if the version is not in major.minor.patch-rev format?
created_at: 2025-03-25T22:12:22Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "n-1 major-minor tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: "n-1 minor" ?
created_at: 2025-03-25T22:50:21Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: grab the url as well, so you don't need to loop again down below?
created_at: 2025-03-25T22:59:01Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: keep track of the url as well maybe?
created_at: 2025-03-25T22:59:24Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: nevermind, it is a different property
created_at: 2025-03-25T23:00:36Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: nevermind, it is in a different property
created_at: 2025-03-25T23:01:19Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "n-1 major-minor tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    if ! ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+      echo "Failed to mount ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      continue
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: break? 
created_at: 2025-03-25T23:03:35Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: Not sure where to write unit tests for this file. Can somebody please point me to a file where I can write unit tests for my function ? 
created_at: 2025-03-25T23:09:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: https://www.man7.org/linux/man-pages/man1/mkdir.1.html 
-p, --parents
              no error if existing, make parent directories as needed

If ${LOCALDNS_BINARY_PATH} already exists, mkdir -p will not do anything and no error is thrown. 
created_at: 2025-03-25T23:12:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: that is my point. If the directory already exists and have old binaries inside that directory, your mkdir -p will just do nothing, and then you will continue to unpack into that directory, which can be mixing different versions of binaries in there.
created_at: 2025-03-25T23:30:41Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: As long as the image tags are valid CoreDNS versions in the format vX.Y.Z, the extraction of binary from coredns version (previous version to the latest version) will work. Reference - https://github.com/Azure/dalec-build-defs/blob/main/specs/coredns/coredns-1.12.0.yml

For the below - binary of v1.9.4 coredns version will be extracted. 

    {
      "downloadURL": "mcr.microsoft.com/oss/v2/kubernetes/coredns:*",
      "amd64OnlyVersions": [],
      "multiArchVersionsV2": [
        {
          "renovateTag": "registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/coredns",
          "latestVersion": "v1.11.3"
        },
        {
          "renovateTag": "registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/coredns",
          "latestVersion": "v1.9.4"
        },
		{
          "renovateTag": "registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/coredns",
          "latestVersion": "v1.11.3"
        },
        {
          "renovateTag": "registry=https://mcr.microsoft.com, name=oss/v2/kubernetes/coredns",
          "latestVersion": "v1.9.4"
        }
      ],
      "windowsVersions": []
    },
created_at: 2025-03-25T23:37:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "n-1 major-minor tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: Updated the comments - "Previous version (ie n-1, n being the latest version) tag not found, using the latest version:"
created_at: 2025-03-25T23:43:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: yeah
created_at: 2025-03-25T23:44:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: yeah
created_at: 2025-03-25T23:44:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
path: vhdbuilder/packer/install-dependencies.sh
user: login: cameronmeissner
body: @SriHarsha001 for now you can try adding something here to test: https://github.com/Azure/AgentBaker/blob/master/vhdbuilder/packer/test/linux-vhd-content-test.sh, in the meantime we'll work on onboarding these scripts to shellspec which will make them unit testable
created_at: 2025-03-25T23:48:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "n-1 major-minor tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    if ! ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+      echo "Failed to mount ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      continue
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: I did not choose to break here, so that in the event of mount failure, it logs the failure, removes the temporary directory, and continues with the next image in the loop. This will give another chance to extract the binary from another revision version of the same tag. 
created_at: 2025-03-26T00:01:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: we can add content validations in https://github.com/Azure/AgentBaker/blob/master/vhdbuilder/packer/test/linux-vhd-content-test.sh#L1221C1-L1251C47
and verify with pipeline: https://msazure.visualstudio.com/CloudNativeCompute/_build?definitionId=172842&_a=summary
created_at: 2025-03-26T00:03:13Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: linux-vhd-content-test.sh was executed at the end of the vhd building and verifies certain contents are present on the vhd. We can just add a new validation function in there, say the coreDns binary should be extracted at the desired location after vhd building. 
created_at: 2025-03-26T00:04:48Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: as long as the test could fail when someone messed up the version format, we should be good. 
created_at: 2025-03-26T01:14:02Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "n-1 major-minor tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    if ! ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+      echo "Failed to mount ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      continue
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: but will the next image match? 
if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]];
created_at: 2025-03-26T01:15:09Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "n-1 major-minor tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    if ! ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+      echo "Failed to mount ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      continue
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: Will there be multiple images from coredns_image_list[] that matches previous_coredns_tag? If so, we will be extracting the binaries multiple times.
created_at: 2025-03-26T01:17:39Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: Got it. I thought this func will get called only once at the build time of VHD. 
created_at: 2025-03-26T06:00:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: Made the code changes. 
created_at: 2025-03-26T06:27:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "n-1 major-minor tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    if ! ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+      echo "Failed to mount ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      continue
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: previous_coredns_tag which will be in v.M.m.p-r format will only match one coredns image url. You are right, the next image will not match. So will not be extracting binaries multiple times. I have added a retry logic to handle transient image mount failure and exit if retries did not succeed. 
created_at: 2025-03-26T06:31:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,96 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version (ie n-1, n being the latest version) tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
+    local retry_count=0
+    while [[ $retry_count -lt $max_retries ]]; do
+      if ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+        break
+      fi
+      echo "Warning: Failed to mount ${coredns_image_url}, retrying..." >> "${VHD_LOGS_FILEPATH}"
+      sleep 2
+      ((retry_count++))
+    done
+
+    if [[ $retry_count -eq $max_retries ]]; then
+      echo "Error: Failed to mount ${coredns_image_url} after $max_retries attempts." >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: Devinwong
body: curious: do we still need this if there is already a trap?
created_at: 2025-03-26T17:29:57Z
author_association: COLLABORATOR
diff_hunk: @@ -609,6 +609,96 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version (ie n-1, n being the latest version) tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
+    local retry_count=0
+    while [[ $retry_count -lt $max_retries ]]; do
+      if ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+        break
+      fi
+      echo "Warning: Failed to mount ${coredns_image_url}, retrying..." >> "${VHD_LOGS_FILEPATH}"
+      sleep 2
+      ((retry_count++))
+    done
+
+    if [[ $retry_count -eq $max_retries ]]; then
+      echo "Error: Failed to mount ${coredns_image_url} after $max_retries attempts." >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      exit 1
+    fi
+
+    local coredns_binary="${ctr_temp}/usr/bin/coredns"
+    if [[ -f "${coredns_binary}" ]]; then
+      cp "${coredns_binary}" "${LOCALDNS_BINARY_PATH}/coredns"
+      echo "Copied coredns binary of ${previous_coredns_tag} to ${LOCALDNS_BINARY_PATH}/coredns" >> "${VHD_LOGS_FILEPATH}"
+    else
+      echo "Coredns binary not found for ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+    fi
+
+    ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+    rm -rf "${ctr_temp}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: Devinwong
body: curious: do we still need this if there is already a trap?
created_at: 2025-03-26T17:30:31Z
author_association: COLLABORATOR
diff_hunk: @@ -609,6 +609,96 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version (ie n-1, n being the latest version) tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
+    local retry_count=0
+    while [[ $retry_count -lt $max_retries ]]; do
+      if ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+        break
+      fi
+      echo "Warning: Failed to mount ${coredns_image_url}, retrying..." >> "${VHD_LOGS_FILEPATH}"
+      sleep 2
+      ((retry_count++))
+    done
+
+    if [[ $retry_count -eq $max_retries ]]; then
+      echo "Error: Failed to mount ${coredns_image_url} after $max_retries attempts." >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      exit 1
+    fi
+
+    local coredns_binary="${ctr_temp}/usr/bin/coredns"
+    if [[ -f "${coredns_binary}" ]]; then
+      cp "${coredns_binary}" "${LOCALDNS_BINARY_PATH}/coredns"
+      echo "Copied coredns binary of ${previous_coredns_tag} to ${LOCALDNS_BINARY_PATH}/coredns" >> "${VHD_LOGS_FILEPATH}"
+    else
+      echo "Coredns binary not found for ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+    fi
+
+    ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+    rm -rf "${ctr_temp}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: The trap function will only be triggered after the function exits or if any of the specified signals are encountered during the execution. The cleanup command lines here are still necessary for immediate cleanup within the function when the function completes. 
created_at: 2025-03-26T19:14:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,96 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version (ie n-1, n being the latest version) tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
+    local retry_count=0
+    while [[ $retry_count -lt $max_retries ]]; do
+      if ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+        break
+      fi
+      echo "Warning: Failed to mount ${coredns_image_url}, retrying..." >> "${VHD_LOGS_FILEPATH}"
+      sleep 2
+      ((retry_count++))
+    done
+
+    if [[ $retry_count -eq $max_retries ]]; then
+      echo "Error: Failed to mount ${coredns_image_url} after $max_retries attempts." >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      exit 1
+    fi
+
+    local coredns_binary="${ctr_temp}/usr/bin/coredns"
+    if [[ -f "${coredns_binary}" ]]; then
+      cp "${coredns_binary}" "${LOCALDNS_BINARY_PATH}/coredns"
+      echo "Copied coredns binary of ${previous_coredns_tag} to ${LOCALDNS_BINARY_PATH}/coredns" >> "${VHD_LOGS_FILEPATH}"
+    else
+      echo "Coredns binary not found for ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+    fi
+
+    ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+    rm -rf "${ctr_temp}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: Please let me know if you think otherwise. 
created_at: 2025-03-26T19:17:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,96 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version (ie n-1, n being the latest version) tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
+    local retry_count=0
+    while [[ $retry_count -lt $max_retries ]]; do
+      if ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+        break
+      fi
+      echo "Warning: Failed to mount ${coredns_image_url}, retrying..." >> "${VHD_LOGS_FILEPATH}"
+      sleep 2
+      ((retry_count++))
+    done
+
+    if [[ $retry_count -eq $max_retries ]]; then
+      echo "Error: Failed to mount ${coredns_image_url} after $max_retries attempts." >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: Sure, I have removed the cleanup command. Relying on trap to cleanup on exit. 
created_at: 2025-03-26T19:19:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: Added a func to check version format in the extraction code. 
created_at: 2025-03-26T19:21:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,96 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version (ie n-1, n being the latest version) tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
+    local retry_count=0
+    while [[ $retry_count -lt $max_retries ]]; do
+      if ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+        break
+      fi
+      echo "Warning: Failed to mount ${coredns_image_url}, retrying..." >> "${VHD_LOGS_FILEPATH}"
+      sleep 2
+      ((retry_count++))
+    done
+
+    if [[ $retry_count -eq $max_retries ]]; then
+      echo "Error: Failed to mount ${coredns_image_url} after $max_retries attempts." >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      exit 1
+    fi
+
+    local coredns_binary="${ctr_temp}/usr/bin/coredns"
+    if [[ -f "${coredns_binary}" ]]; then
+      cp "${coredns_binary}" "${LOCALDNS_BINARY_PATH}/coredns"
+      echo "Copied coredns binary of ${previous_coredns_tag} to ${LOCALDNS_BINARY_PATH}/coredns" >> "${VHD_LOGS_FILEPATH}"
+    else
+      echo "Coredns binary not found for ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+    fi
+
+    ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+    rm -rf "${ctr_temp}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: Devinwong
body: I am not familiar with the trap command either. I guess it is like `defer` in go. So I checked with copilot.
![image](https://github.com/user-attachments/assets/8443accd-d795-4e4d-bdfd-e78e8710b83d)
Just an FYI
created_at: 2025-03-26T19:35:40Z
author_association: COLLABORATOR
diff_hunk: @@ -609,6 +609,96 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version (ie n-1, n being the latest version) tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
+    local retry_count=0
+    while [[ $retry_count -lt $max_retries ]]; do
+      if ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+        break
+      fi
+      echo "Warning: Failed to mount ${coredns_image_url}, retrying..." >> "${VHD_LOGS_FILEPATH}"
+      sleep 2
+      ((retry_count++))
+    done
+
+    if [[ $retry_count -eq $max_retries ]]; then
+      echo "Error: Failed to mount ${coredns_image_url} after $max_retries attempts." >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      exit 1
+    fi
+
+    local coredns_binary="${ctr_temp}/usr/bin/coredns"
+    if [[ -f "${coredns_binary}" ]]; then
+      cp "${coredns_binary}" "${LOCALDNS_BINARY_PATH}/coredns"
+      echo "Copied coredns binary of ${previous_coredns_tag} to ${LOCALDNS_BINARY_PATH}/coredns" >> "${VHD_LOGS_FILEPATH}"
+    else
+      echo "Coredns binary not found for ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+    fi
+
+    ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+    rm -rf "${ctr_temp}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: Devinwong
body: Unresolving this comment so that you can see my reply. Feel free to resolve again if you think it's still necessary to keep it.
created_at: 2025-03-26T19:37:05Z
author_association: COLLABORATOR
diff_hunk: @@ -609,6 +609,96 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version and latest from previous major-minor version.
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version (ie n-1, n being the latest version) tag not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
+    local retry_count=0
+    while [[ $retry_count -lt $max_retries ]]; do
+      if ctr -n k8s.io images mount "${coredns_image_url}" "${ctr_temp}" >/dev/null; then
+        break
+      fi
+      echo "Warning: Failed to mount ${coredns_image_url}, retrying..." >> "${VHD_LOGS_FILEPATH}"
+      sleep 2
+      ((retry_count++))
+    done
+
+    if [[ $retry_count -eq $max_retries ]]; then
+      echo "Error: Failed to mount ${coredns_image_url} after $max_retries attempts." >> "${VHD_LOGS_FILEPATH}"
+      rm -rf "${ctr_temp}"
+      exit 1
+    fi
+
+    local coredns_binary="${ctr_temp}/usr/bin/coredns"
+    if [[ -f "${coredns_binary}" ]]; then
+      cp "${coredns_binary}" "${LOCALDNS_BINARY_PATH}/coredns"
+      echo "Copied coredns binary of ${previous_coredns_tag} to ${LOCALDNS_BINARY_PATH}/coredns" >> "${VHD_LOGS_FILEPATH}"
+    else
+      echo "Coredns binary not found for ${coredns_image_url}" >> "${VHD_LOGS_FILEPATH}"
+    fi
+
+    ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+    rm -rf "${ctr_temp}"
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: Thank you for sharing and taking a look. I think it is better and safe to keep the cleanup command at the end of the function execution. 
created_at: 2025-03-26T20:11:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,113 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDnsBinary() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags (v1.12.0-1 format) and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Function to check version format (vMajor.Minor.Patch).
+  validate_version_format() {
+    local version=$1
+    if [[ ! "$version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+      echo "Error: Invalid coredns version format. Expected vMajor.Minor.Patch, got $version" >> "${VHD_LOGS_FILEPATH}"
+      return 1
+    fi
+    return 0
+  }
+
+  # Determine latest version (eg. v1.12.0-1).
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision. eg - v1.12.0).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    # Extract major.minor.patch (eg - v1.12.0).
+    local vMajorMinorPatch="${tag%-*}"
+    if ! validate_version_format "$vMajorMinorPatch"; then
+      exit 1
+    fi
+
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
path: vhdbuilder/packer/install-dependencies.sh
user: login: yewmsft
body: I am not familiar with vhd building retries, so not sure if 3 times is enough for mounting errors. But we can keep this for now. If the failure rate is high during build, we can tune it up.
created_at: 2025-03-26T20:41:27Z
author_association: MEMBER
diff_hunk: @@ -1207,6 +1207,27 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDNSBinaryExtractedAndCached"
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+
+  echo "$test: Checking for existence of coredns binary at $binaryPath"
+  
+  if [[ ! -f "$binaryPath" ]]; then
+    err "$test: coredns binary does not exist at $binaryPath"
+    return 1
+  fi
+  
+  if [[ ! -x "$binaryPath" ]]; then
+    err "$test: coredns binary exists but is not executable at $binaryPath"
+    return 1
+  fi
+  
+  echo "$test: coredns binary is properly extracted and cached at $binaryPath"
path: vhdbuilder/packer/test/linux-vhd-content-test.sh
user: login: yewmsft
body: do we want to verify the version as well?
created_at: 2025-03-26T20:53:38Z
author_association: MEMBER
diff_hunk: @@ -609,6 +609,113 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDnsBinary() {
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    echo "Error: No coredns images found."
+    exit 1
+  fi
+
+  rm -rf "${LOCALDNS_BINARY_PATH}" || exit 1
+  mkdir -p "${LOCALDNS_BINARY_PATH}" || exit 1
+
+  cleanup_coredns_imports() {
+    set +e
+    if [[ -n "${ctr_temp}" ]]; then
+      ctr -n k8s.io images unmount "${ctr_temp}" >/dev/null
+      rm -rf "${ctr_temp}"
+    fi
+  }
+  trap cleanup_coredns_imports EXIT ABRT ERR INT PIPE QUIT TERM
+
+  # Extract available coredns image tags (v1.12.0-1 format) and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Function to check version format (vMajor.Minor.Patch).
+  validate_version_format() {
+    local version=$1
+    if [[ ! "$version" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+      echo "Error: Invalid coredns version format. Expected vMajor.Minor.Patch, got $version" >> "${VHD_LOGS_FILEPATH}"
+      return 1
+    fi
+    return 0
+  }
+
+  # Determine latest version (eg. v1.12.0-1).
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision. eg - v1.12.0).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    # Extract major.minor.patch (eg - v1.12.0).
+    local vMajorMinorPatch="${tag%-*}"
+    if ! validate_version_format "$vMajorMinorPatch"; then
+      exit 1
+    fi
+
+    if [[ "$vMajorMinorPatch" != "$latest_vMajorMinorPatch" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "Warning: Previous version not found, using the latest version: $latest_coredns_tag" >> "${VHD_LOGS_FILEPATH}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  # Extract the CoreDNS binary for the selected version.
+  for coredns_image_url in "${coredns_image_list[@]}"; do
+    if [[ "${coredns_image_url##*:}" != "$previous_coredns_tag" ]]; then
+      continue
+    fi
+
+    ctr_temp="$(mktemp -d)"
+    local max_retries=3
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: oh ok sure.
created_at: 2025-03-26T20:56:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1207,6 +1207,27 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDNSBinaryExtractedAndCached"
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+
+  echo "$test: Checking for existence of coredns binary at $binaryPath"
+  
+  if [[ ! -f "$binaryPath" ]]; then
+    err "$test: coredns binary does not exist at $binaryPath"
+    return 1
+  fi
+  
+  if [[ ! -x "$binaryPath" ]]; then
+    err "$test: coredns binary exists but is not executable at $binaryPath"
+    return 1
+  fi
+  
+  echo "$test: coredns binary is properly extracted and cached at $binaryPath"
path: vhdbuilder/packer/test/linux-vhd-content-test.sh
user: login: SriHarsha001
body: yes, I was thinking about that as well. But thinking about a way to determine the version in the script. 
created_at: 2025-03-26T20:58:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1207,6 +1207,27 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDNSBinaryExtractedAndCached"
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+
+  echo "$test: Checking for existence of coredns binary at $binaryPath"
+  
+  if [[ ! -f "$binaryPath" ]]; then
+    err "$test: coredns binary does not exist at $binaryPath"
+    return 1
+  fi
+  
+  if [[ ! -x "$binaryPath" ]]; then
+    err "$test: coredns binary exists but is not executable at $binaryPath"
+    return 1
+  fi
+  
+  echo "$test: coredns binary is properly extracted and cached at $binaryPath"
path: vhdbuilder/packer/test/linux-vhd-content-test.sh
user: login: SriHarsha001
body: Any ideas ? 
created_at: 2025-03-26T21:22:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1207,6 +1207,27 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDNSBinaryExtractedAndCached"
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+
+  echo "$test: Checking for existence of coredns binary at $binaryPath"
+  
+  if [[ ! -f "$binaryPath" ]]; then
+    err "$test: coredns binary does not exist at $binaryPath"
+    return 1
+  fi
+  
+  if [[ ! -x "$binaryPath" ]]; then
+    err "$test: coredns binary exists but is not executable at $binaryPath"
+    return 1
+  fi
+  
+  echo "$test: coredns binary is properly extracted and cached at $binaryPath"
path: vhdbuilder/packer/test/linux-vhd-content-test.sh
user: login: SriHarsha001
body: Never mind, I figured out.
created_at: 2025-03-26T22:29:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -609,6 +609,84 @@ if [[ -n ${PRIVATE_PACKAGES_URL} ]]; then
   done
 fi
 
+LOCALDNS_BINARY_PATH="/opt/azure/containers/localdns/binary"
+# This function extracts CoreDNS binary from cached coredns images (n-1 image version and latest revision version)
+# and copies it to - /opt/azure/containers/localdns/binary/coredns.
+# The binary is later used by localdns systemd unit.
+# The function also handles the cleanup of temporary directories and unmounting of images.
+extractAndCacheCoreDNSBinaries() {
path: vhdbuilder/packer/install-dependencies.sh
user: login: SriHarsha001
body: I have added the test script. 
created_at: 2025-03-26T22:29:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1207,6 +1207,74 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDnsBinaryExtractedAndCached"
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+
+  echo "$test: Checking for existence of coredns binary at ${binaryPath}"
+
+  if [[ ! -f "${binaryPath}" ]]; then
+    err "$test: coredns binary does not exist at ${binaryPath}"
+    return 1
+  fi
+
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    err "$test: No CoreDNS images found in the local container images"
+    return 1
+  fi
+
+  # Extract available coredns image tags (v1.12.0-1 format) and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version (eg. v1.12.0-1).
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision. eg - v1.12.0).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    # Extract major.minor.patch (eg - v1.12.0).
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "${vMajorMinorPatch}" != "${latest_vMajorMinorPatch}" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after the next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "$test: Warning: Previous version not found, using the latest version: $latest_coredns_tag"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  local expectedVersion="$previous_coredns_tag"
+  local expectedVersionWithoutV="${expectedVersion#v}"
+  echo "$test: Expected CoreDNS version (n-1 latest revision): ${expectedVersionWithoutV}"
+
+  # Get the actual version from the extracted CoreDNS binary
+  local actualVersion
+  actualVersion=$("$binaryPath" --version | awk -F'-' '{print $2}')
path: vhdbuilder/packer/test/linux-vhd-content-test.sh
user: login: yewmsft
body: this line fails at 2004fips VHDs due to missing dependency lib: GLIBC_2.34
/opt/azure/containers/localdns/binary/coredns: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /opt/azure/containers/localdns/binary/coredns)
created_at: 2025-03-27T01:18:46Z
author_association: MEMBER
diff_hunk: @@ -1207,6 +1207,74 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDnsBinaryExtractedAndCached"
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+
+  echo "$test: Checking for existence of coredns binary at ${binaryPath}"
+
+  if [[ ! -f "${binaryPath}" ]]; then
+    err "$test: coredns binary does not exist at ${binaryPath}"
+    return 1
+  fi
+
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    err "$test: No CoreDNS images found in the local container images"
+    return 1
+  fi
+
+  # Extract available coredns image tags (v1.12.0-1 format) and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version (eg. v1.12.0-1).
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision. eg - v1.12.0).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    # Extract major.minor.patch (eg - v1.12.0).
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "${vMajorMinorPatch}" != "${latest_vMajorMinorPatch}" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after the next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "$test: Warning: Previous version not found, using the latest version: $latest_coredns_tag"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  local expectedVersion="$previous_coredns_tag"
+  local expectedVersionWithoutV="${expectedVersion#v}"
+  echo "$test: Expected CoreDNS version (n-1 latest revision): ${expectedVersionWithoutV}"
+
+  # Get the actual version from the extracted CoreDNS binary
+  local actualVersion
+  actualVersion=$("$binaryPath" --version | awk -F'-' '{print $2}')
path: vhdbuilder/packer/test/linux-vhd-content-test.sh
user: login: yewmsft
body: we need to do 2 things:
1. exclude running this tests on 2004fips vhds
2. do not support localdns on 2004fips vhds, which will not be a problem, since it is eol soon.
created_at: 2025-03-27T01:20:11Z
author_association: MEMBER
diff_hunk: @@ -1207,6 +1207,74 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDnsBinaryExtractedAndCached"
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+
+  echo "$test: Checking for existence of coredns binary at ${binaryPath}"
+
+  if [[ ! -f "${binaryPath}" ]]; then
+    err "$test: coredns binary does not exist at ${binaryPath}"
+    return 1
+  fi
+
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    err "$test: No CoreDNS images found in the local container images"
+    return 1
+  fi
+
+  # Extract available coredns image tags (v1.12.0-1 format) and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version (eg. v1.12.0-1).
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision. eg - v1.12.0).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    # Extract major.minor.patch (eg - v1.12.0).
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "${vMajorMinorPatch}" != "${latest_vMajorMinorPatch}" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after the next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "$test: Warning: Previous version not found, using the latest version: $latest_coredns_tag"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  local expectedVersion="$previous_coredns_tag"
+  local expectedVersionWithoutV="${expectedVersion#v}"
+  echo "$test: Expected CoreDNS version (n-1 latest revision): ${expectedVersionWithoutV}"
+
+  # Get the actual version from the extracted CoreDNS binary
+  local actualVersion
+  actualVersion=$("$binaryPath" --version | awk -F'-' '{print $2}')
path: vhdbuilder/packer/test/linux-vhd-content-test.sh
user: login: yewmsft
body: you can just do something like [this](https://github.com/Azure/AgentBaker/blob/4c73681f022385e275a3779e62de344271b6169a/vhdbuilder/packer/test/linux-vhd-content-test.sh#L443)
if [[ ${os_version} == "18.04" || ${os_version} == "20.04" ]]; then 
exit 0
created_at: 2025-03-27T01:26:24Z
author_association: MEMBER
diff_hunk: @@ -1207,6 +1207,74 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDnsBinaryExtractedAndCached"
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+
+  echo "$test: Checking for existence of coredns binary at ${binaryPath}"
+
+  if [[ ! -f "${binaryPath}" ]]; then
+    err "$test: coredns binary does not exist at ${binaryPath}"
+    return 1
+  fi
+
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    err "$test: No CoreDNS images found in the local container images"
+    return 1
+  fi
+
+  # Extract available coredns image tags (v1.12.0-1 format) and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version (eg. v1.12.0-1).
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision. eg - v1.12.0).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    # Extract major.minor.patch (eg - v1.12.0).
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "${vMajorMinorPatch}" != "${latest_vMajorMinorPatch}" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after the next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "$test: Warning: Previous version not found, using the latest version: $latest_coredns_tag"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  local expectedVersion="$previous_coredns_tag"
+  local expectedVersionWithoutV="${expectedVersion#v}"
+  echo "$test: Expected CoreDNS version (n-1 latest revision): ${expectedVersionWithoutV}"
+
+  # Get the actual version from the extracted CoreDNS binary
+  local actualVersion
+  actualVersion=$("$binaryPath" --version | awk -F'-' '{print $2}')
path: vhdbuilder/packer/test/linux-vhd-content-test.sh
user: login: SriHarsha001
body: I have added the check in test file. Please take another look. 
created_at: 2025-03-27T19:44:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1207,6 +1207,84 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDnsBinaryExtractedAndCached"
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+
+  echo "$test: Checking for existence of coredns binary at ${binaryPath}"
+
+  if [[ ! -f "${binaryPath}" ]]; then
+    err "$test: coredns binary does not exist at ${binaryPath}"
+    return 1
+  fi
+
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    err "$test: No CoreDNS images found in the local container images"
+    return 1
+  fi
+
+  # Extract available coredns image tags (v1.12.0-1 format) and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version (eg. v1.12.0-1).
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision. eg - v1.12.0).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    # Extract major.minor.patch (eg - v1.12.0).
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "${vMajorMinorPatch}" != "${latest_vMajorMinorPatch}" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after the next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "$test: Warning: Previous version not found, using the latest version: ${latest_coredns_tag}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  local expectedVersion="$previous_coredns_tag"
+  local expectedVersionWithoutV="${expectedVersion#v}"
+  echo "$test: Expected CoreDNS version (n-1 latest revision): ${expectedVersionWithoutV}"
+
+  # Ubuntu 18.04 and 20.04 ship with GLIBC 2.27 and 2.31, respectively.
+  # CoreDNS binary is built with GLIBC 2.32+, which is not compatible with 18.04 and 20.04 OS versions.
+  # Therefore, we skip the test for these OS versions here.
+  # Validation in AKS RP will be done to ensure localdns is not enabled for these OS versions.
+  if [[ ${os_version} == "18.04" || ${os_version} == "20.04" ]]; then
path: vhdbuilder/packer/test/linux-vhd-content-test.sh
user: login: yewmsft
body: should move this to top 
created_at: 2025-03-27T20:57:14Z
author_association: MEMBER
diff_hunk: @@ -1207,6 +1207,84 @@ checkPerformanceData() {
   return 0
 }
 
+testCoreDnsBinaryExtractedAndCached() {
+  local test="CoreDnsBinaryExtractedAndCached"
+  local localdnsBinaryDir="/opt/azure/containers/localdns/binary"
+  local binaryPath="$localdnsBinaryDir/coredns"
+  local coredns_image_list=($(ctr -n k8s.io images list -q | grep coredns))
+
+  echo "$test: Checking for existence of coredns binary at ${binaryPath}"
+
+  if [[ ! -f "${binaryPath}" ]]; then
+    err "$test: coredns binary does not exist at ${binaryPath}"
+    return 1
+  fi
+
+  if [[ ${#coredns_image_list[@]} -eq 0 ]]; then
+    err "$test: No CoreDNS images found in the local container images"
+    return 1
+  fi
+
+  # Extract available coredns image tags (v1.12.0-1 format) and sort them in descending order.
+  local sorted_coredns_tags=($(for image in "${coredns_image_list[@]}"; do echo "${image##*:}"; done | sort -V -r))
+
+  # Determine latest version (eg. v1.12.0-1).
+  local latest_coredns_tag="${sorted_coredns_tags[0]}"
+  # Extract major.minor.patch (removes -revision. eg - v1.12.0).
+  local latest_vMajorMinorPatch="${latest_coredns_tag%-*}"
+
+  local previous_coredns_tag=""
+  # Iterate through the sorted list to find the next highest major-minor version.
+  for tag in "${sorted_coredns_tags[@]}"; do
+    # Extract major.minor.patch (eg - v1.12.0).
+    local vMajorMinorPatch="${tag%-*}"
+    if [[ "${vMajorMinorPatch}" != "${latest_vMajorMinorPatch}" ]]; then
+      previous_coredns_tag="$tag"
+      # Break the loop after the next highest major-minor version is found.
+      break
+    fi
+  done
+
+  if [[ -z "${previous_coredns_tag}" ]]; then
+    echo "$test: Warning: Previous version not found, using the latest version: ${latest_coredns_tag}"
+    previous_coredns_tag="$latest_coredns_tag"
+  fi
+
+  local expectedVersion="$previous_coredns_tag"
+  local expectedVersionWithoutV="${expectedVersion#v}"
+  echo "$test: Expected CoreDNS version (n-1 latest revision): ${expectedVersionWithoutV}"
+
+  # Ubuntu 18.04 and 20.04 ship with GLIBC 2.27 and 2.31, respectively.
+  # CoreDNS binary is built with GLIBC 2.32+, which is not compatible with 18.04 and 20.04 OS versions.
+  # Therefore, we skip the test for these OS versions here.
+  # Validation in AKS RP will be done to ensure localdns is not enabled for these OS versions.
+  if [[ ${os_version} == "18.04" || ${os_version} == "20.04" ]]; then
path: vhdbuilder/packer/test/linux-vhd-content-test.sh
user: login: SriHarsha001
body: ok moved it. 
created_at: 2025-03-27T21:00:39Z
author_association: CONTRIBUTOR

PR comments:
user: login: SriHarsha001
created_at: 2025-03-26T22:31:05Z
body: Manually tested both the functions on an AKS agent node.

```
./testCoreDnsBinaryExtractedAndCached.sh
CoreDnsBinaryExtractedAndCached: Checking for existence of coredns binary at /opt/azure/containers/localdns/binary/coredns
CoreDnsBinaryExtractedAndCached: Expected CoreDNS version (n-1 latest revision): 1.9.4-5
CoreDnsBinaryExtractedAndCached: Verify extracted CoreDNS version: 1.9.4
CoreDnsBinaryExtractedAndCached: Expected version: 1.9.4-5 of coredns binary is extracted and cached at /opt/azure/containers/localdns/binary/coredns
```
and 
```
./extractAndCacheCoreDnsBinary.sh

cat /opt/azure/vhd-install.complete | grep coredns
  - mcr.microsoft.com/oss/v2/kubernetes/coredns:v1.11.3-6
  - mcr.microsoft.com/oss/v2/kubernetes/coredns:v1.9.4-5
Copied coredns binary of v1.9.4-5 to /opt/azure/containers/localdns/binary/coredns
Successfully copied coredns binary of v1.9.4-5
```
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-27T21:11:37Z
body: No changes to cached containers or packages on Windows VHDs

<!-- thollander/actions-comment-pull-request "WINDOWS_VHD_CONTENT_DIFF" -->
author_association: CONTRIBUTOR