Repository: microsoft/navcontainerhelper
Pull request #3901: Improve performance of NuGet Download
Author: login: OleWunschmann
Created: 2025-04-15T11:29:41Z | Merged: 2025-04-18T05:29:17Z

The following example results in a long runtime caused by unnecessary app downloads and high number of web requests:
```powershell
Import-Module BcContainerHelper -Force -DisableNameChecking
$bcContainerHelperConfig.TrustedNuGetFeeds = @()
$bcContainerHelperConfig.TrustedNuGetFeeds += @([PSCustomObject]@{ 'Url' = 'https://dynamicssmb2.pkgs.visualstudio.com/DynamicsBCPublicFeeds/_packaging/MSApps/nuget/v3/index.json'; 'Token' = ''; 'Patterns' = @('*'); 'Fingerprints' = @() })
$bcContainerHelperConfig.TrustedNuGetFeeds += @([PSCustomObject]@{ 'Url' = 'https://dynamicssmb2.pkgs.visualstudio.com/DynamicsBCPublicFeeds/_packaging/MSSymbols/nuget/v3/index.json'; 'Token' = ''; 'Patterns' = @('*'); 'Fingerprints' = @() })
$bcContainerHelperConfig.TrustedNuGetFeeds += @([PSCustomObject]@{ 'Url' = 'https://dynamicssmb2.pkgs.visualstudio.com/DynamicsBCPublicFeeds/_packaging/AppSourceSymbols/nuget/v3/index.json'; 'Token' = ''; 'Patterns' = @('*'); 'Fingerprints' = @() })
Measure-Command {
        Download-BcNuGetPackageToFolder -packageName Microsoft.Application -folder C:\Temp\Test -downloadDependencies all -select 'LatestMatching' -installedPlatform '25.0'
} | Format-Table
```

This PR introduces the following improvements:
- Reduce the number of downloads by:
  - Only retrieving the nuspec to check the package and its dependencies.
  - Checking all direct dependencies before downloading (both the package and its dependencies).
- Reduce the number of web requests by caching the search results for a package name per feed (for 10 minutes).



Diff:
diff --git a/BC.HelperFunctions.ps1 b/BC.HelperFunctions.ps1
index 37ae10c10..d6c335095 100644
--- a/BC.HelperFunctions.ps1
+++ b/BC.HelperFunctions.ps1
@@ -121,6 +121,7 @@ function Get-ContainerHelperConfig {
             "IsGitLab" = ($env:GITLAB_CI -eq "true")
             "useApproximateVersion" = $false
             "useSqlServerModule" = $false
+            "NuGetSearchResultsCacheRetentionPeriod" = 600 # 10 minutes
         }
 
         if ($isInsider) {
diff --git a/NuGet/Download-BcNuGetPackageToFolder.ps1 b/NuGet/Download-BcNuGetPackageToFolder.ps1
index e8183ddfe..0e4172e91 100644
--- a/NuGet/Download-BcNuGetPackageToFolder.ps1
+++ b/NuGet/Download-BcNuGetPackageToFolder.ps1
@@ -144,11 +144,19 @@ try {
         }
         else {
             Write-Host "Best match for package name $($packageName) Version $($version): $packageId Version $packageVersion from $($feed.Url)"
-            $package = $feed.DownloadPackage($packageId, $packageVersion)
-            $nuspec = Get-Content (Join-Path $package '*.nuspec' -Resolve) -Encoding UTF8
-            Write-Verbose "NUSPEC:"
-            $nuspec | ForEach-Object { Write-Verbose $_ }
-            $manifest = [xml]$nuspec
+            $package = ''
+            $manifest = $feed.DownloadNuSpec($packageId, $packageVersion)
+            
+            if ($VerbosePreference -ne 'SilentlyContinue') {
+                Write-Verbose "NUSPEC:"
+                $manifestStringWriter = New-Object System.IO.StringWriter
+                $manifestXmlWriter = [System.Xml.XmlWriter]::Create($manifestStringWriter, (New-Object System.Xml.XmlWriterSettings -Property @{ Indent = $true}))
+                $manifest.Save($manifestXmlWriter)
+                $manifestXmlWriter.Dispose()
+                $manifestStringWriter.ToString() -split [System.Environment]::NewLine | Write-Verbose
+                $manifestStringWriter.Dispose()
+            }
+
             $appId = ''
             if ($manifest.package.metadata.PSObject.Properties.Name -eq 'title') {
                 $appName = $manifest.package.metadata.title
@@ -180,6 +188,7 @@ try {
             else {
                 $dependencies = @()
             }
+            $dependenciesToDownload = @()
             foreach($dependency in $dependencies) {
                 if (-not $installedPlatform) {
                     $installedPlatform = $installedApps+$returnValue | Where-Object { $_ -and $_.Name -eq 'Platform' } | Select-Object -ExpandProperty Version
@@ -267,19 +276,29 @@ try {
                     }
                 }
                 if ($downloadIt) {
-                    if ($dependencyVersion.StartsWith('[') -and $select -eq 'Exact') {
-                        # Downloading Microsoft packages for a specific version
-                        $dependencyVersion = $version
-                    }
-                    $returnValue += Download-BcNuGetPackageToFolder -nuGetServerUrl $nuGetServerUrl -nuGetToken $nuGetToken -packageName $dependencyId -version $dependencyVersion -folder $package -copyInstalledAppsToFolder $copyInstalledAppsToFolder -installedPlatform $installedPlatform -installedCountry $installedCountry -installedApps @($installedApps + $returnValue) -downloadDependencies $downloadDependencies -verbose:($VerbosePreference -eq 'Continue') -select $select -allowPrerelease:$allowPrerelease -checkLocalVersion
+                    $dependenciesToDownload += @( $dependency )
                 }
             }
             if ($dependenciesErr) {
                 # If we are looking for the earliest/latest matching version, then we can try to find another version
                 $excludeVersions += $packageVersion
-                Remove-Item -Path $package -Recurse -Force
                 continue
             }
+            
+            # Download the package
+            $package = $feed.DownloadPackage($packageId, $packageVersion)
+
+            foreach ($dependency in $dependenciesToDownload) {
+                # Download the dependencies of the package
+                $dependencyVersion = $dependency.Version
+                $dependencyId = $dependency.Id
+                if ($dependencyVersion.StartsWith('[') -and $select -eq 'Exact') {
+                    # Downloading Microsoft packages for a specific version
+                    $dependencyVersion = $version
+                }
+                $returnValue += Download-BcNuGetPackageToFolder -nuGetServerUrl $nuGetServerUrl -nuGetToken $nuGetToken -packageName $dependencyId -version $dependencyVersion -folder $package -copyInstalledAppsToFolder $copyInstalledAppsToFolder -installedPlatform $installedPlatform -installedCountry $installedCountry -installedApps @($installedApps + $returnValue) -downloadDependencies $downloadDependencies -verbose:($VerbosePreference -eq 'Continue') -select $select -allowPrerelease:$allowPrerelease -checkLocalVersion
+            }
+
             if ($installedCountry -and (Test-Path (Join-Path $package $installedCountry) -PathType Container)) {
                 # NuGet packages of Runtime packages might exist in different versions for different countries
                 # The runtime package might contain C# invoke calls with different methodis for different countries
diff --git a/NuGet/NuGetFeedClass.ps1 b/NuGet/NuGetFeedClass.ps1
index 0ec6d6e52..0beaf1586 100644
--- a/NuGet/NuGetFeedClass.ps1
+++ b/NuGet/NuGetFeedClass.ps1
@@ -16,6 +16,9 @@ class NuGetFeed {
 
     [hashtable] $orgType = @{}
 
+    [hashtable] $searchResultsCache = @{}
+    [int]       $searchResultsCacheRetentionPeriod = $bcContainerHelperConfig.NuGetSearchResultsCacheRetentionPeriod
+
     NuGetFeed([string] $nuGetServerUrl, [string] $nuGetToken, [string[]] $patterns, [string[]] $fingerprints) {
         $this.url = $nuGetServerUrl
         $this.token = $nuGetToken
@@ -83,7 +86,27 @@ class NuGetFeed {
     }
 
     [hashtable[]] Search([string] $packageName) {
-        if ($this.searchQueryServiceUrl -match '^https://nuget.pkg.github.com/(.*)/query$') {
+        $useCache = $this.searchResultsCacheRetentionPeriod -gt 0
+        $hasCache = $this.searchResultsCache.ContainsKey($packageName)
+        if ($hasCache) {
+            if ($useCache) {
+                # Clear cache older than the retention period
+                $clearCache = $this.searchResultsCache[$packageName].timestamp.AddSeconds($this.searchResultsCacheRetentionPeriod) -lt (Get-Date)
+            } else {
+                # Clear cache if we are not using it
+                $clearCache = $true
+            }
+            if ($clearCache) {
+                $this.searchResultsCache.Remove($packageName)
+                $hasCache = $false
+            }
+        }
+
+        if ($useCache -and $hasCache) { 
+            Write-Host "Search package using cache"
+            $matching = $this.searchResultsCache[$packageName].matching
+        } 
+        elseif ($this.searchQueryServiceUrl -match '^https://nuget.pkg.github.com/(.*)/query$') {
             # GitHub support for SearchQueryService is unstable and is not usable
             # use GitHub API instead
             # GitHub API unfortunately doesn't support filtering, so we need to filter ourselves
@@ -141,6 +164,15 @@ class NuGetFeed {
         else {
             Write-Host "$($matching.count) matching packages found"
         }
+
+        if ($useCache -and !$hasCache) {
+            # Cache the search results
+            $this.searchResultsCache[$packageName] = @{
+                matching = $matching
+                timestamp = (Get-Date)
+            }
+        }
+
         return $matching | ForEach-Object { Write-Host "- $($_.id)"; $_ }
     }
 
@@ -371,6 +403,11 @@ class NuGetFeed {
         try {
             Invoke-RestMethod -UseBasicParsing -Uri $this.packagePublishUrl -ContentType "multipart/form-data; boundary=$boundary" -Method Put -Headers $headers -inFile $tmpFile | Out-Host
             Write-Host -ForegroundColor Green "NuGet package successfully submitted"
+
+            # Clear matching search results caches
+            @( $this.searchResultsCache.Keys ) | 
+                Where-Object { $package -like "*$($_)*" } | 
+                ForEach-Object { $this.searchResultsCache.Remove($_) }
         }
         catch [System.Net.WebException] {
             if ($_.Exception.Status -eq "ProtocolError" -and $_.Exception.Response -is [System.Net.HttpWebResponse]) {
diff --git a/ReleaseNotes.txt b/ReleaseNotes.txt
index 0fdb1d57e..c22da674c 100644
--- a/ReleaseNotes.txt
+++ b/ReleaseNotes.txt
@@ -6,6 +6,9 @@ During Run-AlPipeline, apps which are only published are reported as being insta
 Use standard app filename for app instead of the app name inside the nuget package
 Allow vsixFile to be a local file for New-BCCompilerFolder (3831)
 Fixes issues determining test symbols for W1 and OnPrem artifacts not containing versions in their name (3839)
+Add cache for search results of a NuGet feed
+Add new setting NuGetSearchResultsCacheRetentionPeriod (default: 600) to set the retention period (in seconds) for the search result cache of NuGet feeds (0 to disable the cache)
+Download-BcNuGetPackageToFolder checks all direct dependency versions of a package based on the NuSpec before downloading the whole package and its dependencies
 
 6.1.3
 Add setting ExcludeBuilds, which is an array of NuGet range specifications to exclude from Get-BcArtifactUrl. See more about range specification here: https://learn.microsoft.com/en-us/nuget/concepts/package-versioning?tabs=semver20sort#version-ranges


Reviews:
user: login: freddydk
body: This looks really good - a few minor comments.
Also, please add a few lines in the release notes specifying what is done and the new setting (when that is done) and the default value of that.

state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T21:03:32Z
user: login: OleWunschmann
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T09:58:01Z
user: login: freddydk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T10:02:23Z
user: login: OleWunschmann
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T10:03:06Z
user: login: freddydk
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T04:57:35Z
user: login: OleWunschmann
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T13:54:08Z
user: login: freddydk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T14:51:04Z
user: login: OleWunschmann
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T21:32:10Z
user: login: freddydk
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T05:28:09Z

Review comments:
diff_hunk: @@ -144,11 +144,16 @@ try {
         }
         else {
             Write-Host "Best match for package name $($packageName) Version $($version): $packageId Version $packageVersion from $($feed.Url)"
-            $package = $feed.DownloadPackage($packageId, $packageVersion)
-            $nuspec = Get-Content (Join-Path $package '*.nuspec' -Resolve) -Encoding UTF8
+            $package = ''
+            $manifest = $feed.DownloadNuSpec($packageId, $packageVersion)
             Write-Verbose "NUSPEC:"
-            $nuspec | ForEach-Object { Write-Verbose $_ }
-            $manifest = [xml]$nuspec
+            $manifestStringWriter = New-Object System.IO.StringWriter
path: NuGet/Download-BcNuGetPackageToFolder.ps1
user: login: freddydk
body: I guess we could have the entire section from 149 to 155 behind an `if ($verbosepreference -ne 'SilentlyContinue')`

created_at: 2025-04-15T20:57:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -83,7 +85,20 @@ class NuGetFeed {
     }
 
     [hashtable[]] Search([string] $packageName) {
-        if ($this.searchQueryServiceUrl -match '^https://nuget.pkg.github.com/(.*)/query$') {
+        $useCache = $this.searchResultsCache.ContainsKey($packageName)
path: NuGet/NuGetFeedClass.ps1
user: login: freddydk
body: Would it make sense to remove items from the cache if a package is pushed?
created_at: 2025-04-15T20:59:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -83,7 +85,20 @@ class NuGetFeed {
     }
 
     [hashtable[]] Search([string] $packageName) {
-        if ($this.searchQueryServiceUrl -match '^https://nuget.pkg.github.com/(.*)/query$') {
+        $useCache = $this.searchResultsCache.ContainsKey($packageName)
+        if ($useCache) {
+            # Clear cache older than 10 minutes
+            if ($this.searchResultsCache[$packageName].timestamp -lt (Get-Date).AddMinutes(-10)) {
path: NuGet/NuGetFeedClass.ps1
user: login: freddydk
body: Please add the cache retention period to the bcContainerHelper settings in bc.helperfunctions.ps1 - this allows people to control this (or remove it) if needed - and make sure that 0 means do not use cache.
created_at: 2025-04-15T21:01:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -83,7 +85,20 @@ class NuGetFeed {
     }
 
     [hashtable[]] Search([string] $packageName) {
-        if ($this.searchQueryServiceUrl -match '^https://nuget.pkg.github.com/(.*)/query$') {
+        $useCache = $this.searchResultsCache.ContainsKey($packageName)
path: NuGet/NuGetFeedClass.ps1
user: login: OleWunschmann
body: Good point. 
My suggestion would be to remove the whole cache to keep it simple.
Another solution would be to only remove items with keys that match the pushed package name (`$packageName -like "*$Key*"`).
Which option would you prefer?
created_at: 2025-04-16T09:58:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -83,7 +85,20 @@ class NuGetFeed {
     }
 
     [hashtable[]] Search([string] $packageName) {
-        if ($this.searchQueryServiceUrl -match '^https://nuget.pkg.github.com/(.*)/query$') {
+        $useCache = $this.searchResultsCache.ContainsKey($packageName)
path: NuGet/NuGetFeedClass.ps1
user: login: freddydk
body: The latter - running with incremental builds, we will publish packages if they don't already exist - it would be good that the cache isn't cleared all the time (even though it might not matter at all)

created_at: 2025-04-16T10:02:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -144,11 +144,16 @@ try {
         }
         else {
             Write-Host "Best match for package name $($packageName) Version $($version): $packageId Version $packageVersion from $($feed.Url)"
-            $package = $feed.DownloadPackage($packageId, $packageVersion)
-            $nuspec = Get-Content (Join-Path $package '*.nuspec' -Resolve) -Encoding UTF8
+            $package = ''
+            $manifest = $feed.DownloadNuSpec($packageId, $packageVersion)
             Write-Verbose "NUSPEC:"
-            $nuspec | ForEach-Object { Write-Verbose $_ }
-            $manifest = [xml]$nuspec
+            $manifestStringWriter = New-Object System.IO.StringWriter
path: NuGet/Download-BcNuGetPackageToFolder.ps1
user: login: OleWunschmann
body: I also thougth about it and will add the condition
created_at: 2025-04-16T10:03:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,16 +85,31 @@ class NuGetFeed {
     }
 
     [hashtable[]] Search([string] $packageName) {
-        $useCache = $this.searchResultsCache.ContainsKey($packageName)
-        if ($useCache) {
-            # Clear cache older than 10 minutes
-            if ($this.searchResultsCache[$packageName].timestamp -lt (Get-Date).AddMinutes(-10)) {
+        $cacheRetentionPeriod = 600 # 10 minutes
path: NuGet/NuGetFeedClass.ps1
user: login: freddydk
body: I would prefer to have a class variable like:

[int] $cacheRetentionPeriod = $bcContainerHelperConfig.NuGetSearchResultsCacheRetentionPeriod

and then use $this.cacheRetentionPeriod in the search function (avoiding the two signatures of search)
created_at: 2025-04-17T04:55:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,16 +85,31 @@ class NuGetFeed {
     }
 
     [hashtable[]] Search([string] $packageName) {
-        $useCache = $this.searchResultsCache.ContainsKey($packageName)
-        if ($useCache) {
-            # Clear cache older than 10 minutes
-            if ($this.searchResultsCache[$packageName].timestamp -lt (Get-Date).AddMinutes(-10)) {
+        $cacheRetentionPeriod = 600 # 10 minutes
path: NuGet/NuGetFeedClass.ps1
user: login: OleWunschmann
body: I will remove the second signiture.

I also thought about a class variable but the current solution allows the change of the retention period between searches.
But I can change it if you still prefer the class variable.
created_at: 2025-04-17T13:54:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,16 +85,31 @@ class NuGetFeed {
     }
 
     [hashtable[]] Search([string] $packageName) {
-        $useCache = $this.searchResultsCache.ContainsKey($packageName)
-        if ($useCache) {
-            # Clear cache older than 10 minutes
-            if ($this.searchResultsCache[$packageName].timestamp -lt (Get-Date).AddMinutes(-10)) {
+        $cacheRetentionPeriod = 600 # 10 minutes
path: NuGet/NuGetFeedClass.ps1
user: login: freddydk
body: My problem was really that I had to test what $script: would do in this case - I find it harder to read.
Maybe that's just me:-)
In the end - the setting is only for allowing people to add a setting in AL-Go to change this / disable this if things stops working for some reason.
created_at: 2025-04-17T14:51:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -85,16 +85,31 @@ class NuGetFeed {
     }
 
     [hashtable[]] Search([string] $packageName) {
-        $useCache = $this.searchResultsCache.ContainsKey($packageName)
-        if ($useCache) {
-            # Clear cache older than 10 minutes
-            if ($this.searchResultsCache[$packageName].timestamp -lt (Get-Date).AddMinutes(-10)) {
+        $cacheRetentionPeriod = 600 # 10 minutes
path: NuGet/NuGetFeedClass.ps1
user: login: OleWunschmann
body: I thought about it and made your suggested change.

My original implementation had no config and always cached for 10 minutes.
So nothing is lost with your suggestion and it keeps it simpler and - as you mentioned - easier to read.
created_at: 2025-04-17T21:32:10Z
author_association: CONTRIBUTOR

PR comments:
user: login: OleWunschmann
created_at: 2025-04-15T11:48:53Z
body: @microsoft-github-policy-service agree company="Cosmo Consult"
author_association: CONTRIBUTOR