Repository: Kotlin/dokka (kotlin)

Issue:
V2 Gradle Plugin: Failed to download package-list
**Describe the bug**
Using the Dokka v2 Gradle plugin, I see this error when generating API docs:

> \> Task :couchbase-lite-ee-paging:dokkaGeneratePublicationHtml
Failed to download package-list from https://kotlinlang.org/api/kotlinx-datetime/kotlinx-datetime/package-list, this might suggest that remote resource is not available, module is empty or dokka output got corrupted

Even though the package-list is available at that URL: https://kotlinlang.org/api/kotlinx-datetime/kotlinx-datetime/package-list

**Expected behaviour**
kotlinx-datetime is [known](https://github.com/Kotlin/kotlinx-datetime/issues/268) to have its package-list in a non-standard location, which is why I [specify the correct URL](https://github.com/jeffdgr8/kotbase/blob/acea5e8f3d68e056c80c0817870f709b75f6f0a2/buildSrc/src/main/kotlin/dokka-convention.gradle.kts#L29) in my library. This worked correctly with the Dokka v1 Gradle plugin.

It's not clear if this is a false-positive error or if it's somehow not using the correct URL, even though it's logging the URL provided.

**To Reproduce**
Run the `dokkaGenerate` task on my [Kotbase library](https://github.com/jeffdgr8/kotbase) and observe the logs.

**Dokka configuration**
See [dokka-convention.kts](https://github.com/jeffdgr8/kotbase/blob/acea5e8f3d68e056c80c0817870f709b75f6f0a2/buildSrc/src/main/kotlin/dokka-convention.gradle.kts) and [build.gradle.kts](https://github.com/jeffdgr8/kotbase/blob/acea5e8f3d68e056c80c0817870f709b75f6f0a2/build.gradle.kts#L26)

**Installation**
- Operating system: macOS
- Build tool: Gradle v8.11.1
- Dokka version: 2.0.0

**Additional context**
The [generated API docs](https://kotbase.dev/api/) appear to correctly link to the kotlinx-datetime docs, e.g. [`Instant` from `getDate`](https://kotbase.dev/api/couchbase-lite/kotbase/-document/get-date.html), despite the error.

Pull request:
Fixes #4018

While writing documentation inside [cryptography-kotlin](https://github.com/whyoleg/cryptography-kotlin) and checking Dokka's output, I was a bit surprised that running the `dokkaGenerateModuleHtml` task for some module with **just four files took 55 seconds**.

The module in question: [cryptography-serialization/pem](https://github.com/whyoleg/cryptography-kotlin/tree/daedb820faa364eddb7c3ac4920df6b2ae17ad33/cryptography-serialization/pem/src/commonMain/kotlin)

After a small profiling session, I found out that there is **no CPU work or memory allocations** happening most of the time after the documentable model is created. Total memory used ~300 MB.
That's how I found out that the threads are mostly **just waiting** somewhere in the `PackageList` code.
That was strange, as there are just two external documentation links added: stdlib and kotlinx-io.
So looks like we do a lot of requests there?
I think so, sometimes, I think that I could even hit rate-limiting on kotlinlang.org, or something like this, because all package lists were resolved with `Failed to download package-list from ...` and the task finished super fast :)

Using local package lists solves the issue, and the same task is now executed in **9 seconds**.

So I decided, why not cache them inside Dokka during execution? Results:

* pem module:
  * before: 55 seconds
  * after: **7.4 seconds** (~7 times faster)
* total Dokka tasks execution time across the whole project (12 modules, 5 of them have just 1 class)
  * before: 354 seconds (longest per module: 53)
  * after: **94 seconds (longest per module: 15)** (~4 times faster)

I've additionally checked kotlinx-io (only `okio` has additional to the stdlib package list):

* bytestring module: 36.5 vs 10.8 seconds (~3.5 times faster)
* core module: 37.7 vs 8.2 (~4.5 times faster)
* okio module: 33.9 vs 4.5 (~7.5 times faster)

So, on average, the speed up is `4 * package lists count` file :)
And, as by default, Dokka configures external links to the kotlin stdlib, does this mean that Dokka will now be **4 times faster**?

> Note: those tests, of course, are not fully representative, because they should be run in better isolation and with more iterations, but still, it's not just several percent % change, it's **hundreds % change!**

Regarding the PR itself, feel free to suggest another way to clean up that cache. As we use `PackageList.load` also during multi-module generation (with files), we still go through the cache.

Patch:
diff --git a/dokka-subprojects/plugin-all-modules-page/src/main/kotlin/org/jetbrains/dokka/allModulesPage/AllModulesPageGeneration.kt b/dokka-subprojects/plugin-all-modules-page/src/main/kotlin/org/jetbrains/dokka/allModulesPage/AllModulesPageGeneration.kt
index ae2a55c865..c8e6ec9a17 100644
--- a/dokka-subprojects/plugin-all-modules-page/src/main/kotlin/org/jetbrains/dokka/allModulesPage/AllModulesPageGeneration.kt
+++ b/dokka-subprojects/plugin-all-modules-page/src/main/kotlin/org/jetbrains/dokka/allModulesPage/AllModulesPageGeneration.kt
@@ -6,6 +6,7 @@ package org.jetbrains.dokka.allModulesPage
 
 import org.jetbrains.dokka.CoreExtensions
 import org.jetbrains.dokka.Timer
+import org.jetbrains.dokka.base.resolvers.shared.PackageList
 import org.jetbrains.dokka.generation.Generation
 import org.jetbrains.dokka.pages.RootPageNode
 import org.jetbrains.dokka.plugability.DokkaContext
@@ -21,7 +22,7 @@ public class AllModulesPageGeneration(private val context: DokkaContext) : Gener
     private val allModulesPagePlugin by lazy { context.plugin<AllModulesPagePlugin>() }
     private val templatingPlugin by lazy { context.plugin<TemplatingPlugin>() }
 
-    override fun Timer.generate() {
+    override fun Timer.generate(): Unit = try {
         report("Processing submodules")
         val generationContext = processSubmodules()
 
@@ -42,6 +43,8 @@ public class AllModulesPageGeneration(private val context: DokkaContext) : Gener
 
         report("Running post-actions")
         runPostActions()
+    } finally {
+        PackageList.clearCache()
     }
 
     override val generationName: String = "index page for project"
diff --git a/dokka-subprojects/plugin-base/api/plugin-base.api b/dokka-subprojects/plugin-base/api/plugin-base.api
index 1128c70d40..6a72efb4f5 100644
--- a/dokka-subprojects/plugin-base/api/plugin-base.api
+++ b/dokka-subprojects/plugin-base/api/plugin-base.api
@@ -853,6 +853,7 @@ public final class org/jetbrains/dokka/base/resolvers/shared/PackageList {
 }
 
 public final class org/jetbrains/dokka/base/resolvers/shared/PackageList$Companion {
+	public final fun clearCache ()V
 	public final fun load (Ljava/net/URL;IZ)Lorg/jetbrains/dokka/base/resolvers/shared/PackageList;
 	public static synthetic fun load$default (Lorg/jetbrains/dokka/base/resolvers/shared/PackageList$Companion;Ljava/net/URL;IZILjava/lang/Object;)Lorg/jetbrains/dokka/base/resolvers/shared/PackageList;
 }
diff --git a/dokka-subprojects/plugin-base/src/main/kotlin/org/jetbrains/dokka/base/generation/SingleModuleGeneration.kt b/dokka-subprojects/plugin-base/src/main/kotlin/org/jetbrains/dokka/base/generation/SingleModuleGeneration.kt
index 8193cc7b0c..250f0850d5 100644
--- a/dokka-subprojects/plugin-base/src/main/kotlin/org/jetbrains/dokka/base/generation/SingleModuleGeneration.kt
+++ b/dokka-subprojects/plugin-base/src/main/kotlin/org/jetbrains/dokka/base/generation/SingleModuleGeneration.kt
@@ -10,6 +10,7 @@ import org.jetbrains.dokka.DokkaConfiguration
 import org.jetbrains.dokka.DokkaException
 import org.jetbrains.dokka.Timer
 import org.jetbrains.dokka.base.DokkaBase
+import org.jetbrains.dokka.base.resolvers.shared.PackageList
 import org.jetbrains.dokka.generation.Generation
 import org.jetbrains.dokka.generation.exitGenerationGracefully
 import org.jetbrains.dokka.model.DModule
@@ -23,7 +24,7 @@ import org.jetbrains.dokka.utilities.report
 
 public class SingleModuleGeneration(private val context: DokkaContext) : Generation {
 
-    override fun Timer.generate() {
+    override fun Timer.generate(): Unit = try {
         report("Validity check")
         validityCheck(context)
 
@@ -56,6 +57,8 @@ public class SingleModuleGeneration(private val context: DokkaContext) : Generat
         runPostActions()
 
         reportAfterRendering()
+    } finally {
+        PackageList.clearCache()
     }
 
     override val generationName: String = "documentation for ${context.configuration.moduleName}"
diff --git a/dokka-subprojects/plugin-base/src/main/kotlin/org/jetbrains/dokka/base/resolvers/shared/PackageList.kt b/dokka-subprojects/plugin-base/src/main/kotlin/org/jetbrains/dokka/base/resolvers/shared/PackageList.kt
index 9dc7540f5a..d4e3ece314 100644
--- a/dokka-subprojects/plugin-base/src/main/kotlin/org/jetbrains/dokka/base/resolvers/shared/PackageList.kt
+++ b/dokka-subprojects/plugin-base/src/main/kotlin/org/jetbrains/dokka/base/resolvers/shared/PackageList.kt
@@ -5,6 +5,7 @@
 package org.jetbrains.dokka.base.resolvers.shared
 
 import java.net.URL
+import java.util.concurrent.ConcurrentHashMap
 
 public typealias Module = String
 
@@ -35,10 +36,42 @@ public data class PackageList(
         public const val DOKKA_PARAM_PREFIX: String = "\$dokka"
         public const val SINGLE_MODULE_NAME: String = ""
 
+        private val cache: ConcurrentHashMap<Pair<URL, Int>, PackageList> = ConcurrentHashMap()
+
+        public fun clearCache() {
+            cache.clear()
+        }
+
+        /**
+         * During the Dokka execution pipeline, usually, there exists only several instances of `PackageList`.
+         * Those are created to resolve links to external documentation, or in the case of multi-module documentation
+         * to resolve links to other modules.
+         * But because of the way Dokka works, it calls `PackageList.load` multiple times with the same URL.
+         * This causes downloading the same file again and again.
+         *
+         * So to avoid unnecessary computations and save a lot of CPU resources, we cache it.
+         *
+         * [clearCache] is used to clear the cache.
+         * It is called via [org.jetbrains.dokka.renderers.PostAction] in both single and multi-module [org.jetbrains.dokka.generation.Generation] pipelines
+         */
         public fun load(url: URL, jdkVersion: Int, offlineMode: Boolean = false): PackageList? {
             if (offlineMode && url.protocol.toLowerCase() != "file")
                 return null
 
+            return cache.getOrPut(url to jdkVersion) {
+                // we cache nothing on failure (when `download` returns null)
+                download(url, jdkVersion) ?: return null
+            }
+        }
+
+        internal fun loadWithoutCache(url: URL, jdkVersion: Int, offlineMode: Boolean = false): PackageList? {
+            if (offlineMode && url.protocol.toLowerCase() != "file")
+                return null
+
+            return download(url, jdkVersion)
+        }
+
+        private fun download(url: URL, jdkVersion: Int): PackageList? {
             val packageListStream = runCatching { url.readContent() }.onFailure {
                 println("Failed to download package-list from $url, this might suggest that remote resource is not available," +
                         " module is empty or dokka output got corrupted")


Test patch:
diff --git a/dokka-subprojects/plugin-base-test-utils/src/main/kotlin/org/jetbrains/dokka/base/testApi/testRunner/baseTestApi.kt b/dokka-subprojects/plugin-base-test-utils/src/main/kotlin/org/jetbrains/dokka/base/testApi/testRunner/baseTestApi.kt
index c776c06fe0..cb4338bad9 100644
--- a/dokka-subprojects/plugin-base-test-utils/src/main/kotlin/org/jetbrains/dokka/base/testApi/testRunner/baseTestApi.kt
+++ b/dokka-subprojects/plugin-base-test-utils/src/main/kotlin/org/jetbrains/dokka/base/testApi/testRunner/baseTestApi.kt
@@ -8,6 +8,7 @@ import org.jetbrains.dokka.CoreExtensions
 import org.jetbrains.dokka.DokkaConfiguration
 import org.jetbrains.dokka.DokkaGenerator
 import org.jetbrains.dokka.base.generation.SingleModuleGeneration
+import org.jetbrains.dokka.base.resolvers.shared.PackageList
 import org.jetbrains.dokka.model.DModule
 import org.jetbrains.dokka.pages.RootPageNode
 import org.jetbrains.dokka.plugability.DokkaContext
@@ -28,7 +29,7 @@ public class BaseDokkaTestGenerator(
     additionalPlugins: List<DokkaPlugin> = emptyList()
 ) : DokkaTestGenerator<BaseTestMethods>(configuration, logger, testMethods, additionalPlugins) {
 
-    override fun generate() {
+    override fun generate(): Unit = try {
         with(testMethods) {
             val dokkaGenerator = DokkaGenerator(configuration, logger)
 
@@ -65,6 +66,8 @@ public class BaseDokkaTestGenerator(
 
             singleModuleGeneration.reportAfterRendering()
         }
+    } finally {
+        PackageList.clearCache()
     }
 }
 
diff --git a/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/DefaultExternalLocationProviderTest.kt b/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/DefaultExternalLocationProviderTest.kt
index 6535031e7c..38476ccde7 100644
--- a/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/DefaultExternalLocationProviderTest.kt
+++ b/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/DefaultExternalLocationProviderTest.kt
@@ -32,7 +32,7 @@ class DefaultExternalLocationProviderTest : BaseAbstractTest() {
 
     private fun getTestLocationProvider(context: DokkaContext? = null): DefaultExternalLocationProvider {
         val dokkaContext = context ?: DokkaContext.create(configuration, logger, emptyList())
-        val packageList = PackageList.load(packageListURL, 8, true)!!
+        val packageList = PackageList.loadWithoutCache(packageListURL, 8, true)!!
         val externalDocumentation =
             ExternalDocumentation(URL(kotlinLang), packageList)
         return DefaultExternalLocationProvider(externalDocumentation, ".html", dokkaContext)
diff --git a/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/Dokka010ExternalLocationProviderTest.kt b/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/Dokka010ExternalLocationProviderTest.kt
index 0d1881ff03..65b858d12c 100644
--- a/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/Dokka010ExternalLocationProviderTest.kt
+++ b/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/Dokka010ExternalLocationProviderTest.kt
@@ -32,7 +32,7 @@ class Dokka010ExternalLocationProviderTest : BaseAbstractTest() {
 
     private fun getTestLocationProvider(context: DokkaContext? = null): Dokka010ExternalLocationProvider {
         val dokkaContext = context ?: DokkaContext.create(configuration, logger, emptyList())
-        val packageList = PackageList.load(packageListURL, 8, true)!!
+        val packageList = PackageList.loadWithoutCache(packageListURL, 8, true)!!
         val externalDocumentation =
             ExternalDocumentation(URL(kotlinLang), packageList)
         return Dokka010ExternalLocationProvider(externalDocumentation, ".html", dokkaContext)
diff --git a/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/JavadocExternalLocationProviderTest.kt b/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/JavadocExternalLocationProviderTest.kt
index fee76ddb19..799afe68ae 100644
--- a/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/JavadocExternalLocationProviderTest.kt
+++ b/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/JavadocExternalLocationProviderTest.kt
@@ -33,7 +33,7 @@ class JavadocExternalLocationProviderTest : BaseAbstractTest() {
 
     private fun getTestLocationProvider(context: DokkaContext? = null): DefaultExternalLocationProvider {
         val dokkaContext = context ?: DokkaContext.create(configuration, logger, emptyList())
-        val packageList = PackageList.load(jdkPackageListURL, 8, true)!!
+        val packageList = PackageList.loadWithoutCache(jdkPackageListURL, 8, true)!!
         val externalDocumentation =
             ExternalDocumentation(URL(jdk), packageList)
         return JavadocExternalLocationProvider(externalDocumentation, "--", "-", dokkaContext)
diff --git a/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/MultiModuleLinkingTest.kt b/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/MultiModuleLinkingTest.kt
index ff2be29afc..568a58bb93 100644
--- a/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/MultiModuleLinkingTest.kt
+++ b/dokka-subprojects/plugin-base/src/test/kotlin/locationProvider/MultiModuleLinkingTest.kt
@@ -32,7 +32,7 @@ class MultiModuleLinkingTest : BaseAbstractTest() {
 
     private fun getTestLocationProvider(context: DokkaContext? = null): DefaultExternalLocationProvider {
         val dokkaContext = context ?: DokkaContext.create(configuration, logger, emptyList())
-        val packageList = PackageList.load(packageListURL, 8, true)!!
+        val packageList = PackageList.loadWithoutCache(packageListURL, 8, true)!!
         val externalDocumentation =
             ExternalDocumentation(URL(exampleDomain), packageList)
         return DefaultExternalLocationProvider(externalDocumentation, ".html", dokkaContext)
@@ -40,7 +40,7 @@ class MultiModuleLinkingTest : BaseAbstractTest() {
 
     private fun getStdlibTestLocationProvider(context: DokkaContext? = null): DefaultExternalLocationProvider {
         val dokkaContext = context ?: DokkaContext.create(configuration, logger, emptyList())
-        val packageList = PackageList.load(stdlibPackageListURL, 8, true)!!
+        val packageList = PackageList.loadWithoutCache(stdlibPackageListURL, 8, true)!!
         val externalDocumentation =
                 ExternalDocumentation(URL(kotlinLang), packageList)
         return DefaultExternalLocationProvider(externalDocumentation, ".html", dokkaContext)
diff --git a/dokka-subprojects/plugin-templating/src/test/kotlin/org/jetbrains/dokka/templates/TestTemplatingGeneration.kt b/dokka-subprojects/plugin-templating/src/test/kotlin/org/jetbrains/dokka/templates/TestTemplatingGeneration.kt
index 02f5baaaae..6edea389a1 100644
--- a/dokka-subprojects/plugin-templating/src/test/kotlin/org/jetbrains/dokka/templates/TestTemplatingGeneration.kt
+++ b/dokka-subprojects/plugin-templating/src/test/kotlin/org/jetbrains/dokka/templates/TestTemplatingGeneration.kt
@@ -5,6 +5,7 @@
 package org.jetbrains.dokka.templates
 
 import org.jetbrains.dokka.Timer
+import org.jetbrains.dokka.base.resolvers.shared.PackageList
 import org.jetbrains.dokka.generation.Generation
 import org.jetbrains.dokka.plugability.DokkaContext
 import org.jetbrains.dokka.plugability.plugin
@@ -15,12 +16,14 @@ class TestTemplatingGeneration(private val context: DokkaContext) : Generation {
 
     val templatingPlugin by lazy { context.plugin<TemplatingPlugin>() }
 
-    override fun Timer.generate() {
+    override fun Timer.generate(): Unit = try {
         report("Processing submodules")
         processSubmodules()
 
         report("Finishing processing")
         finishProcessing()
+    } finally {
+        PackageList.clearCache()
     }
 
     fun processSubmodules() =