Repository: dhis2/dhis2-core
Pull request #20348: fix: OpenAPI generation flakiness where sometimes a referenced schema is not included 
Author: login: jbee
Created: 2025-03-20T16:06:11Z | Merged: 2025-03-24T15:08:30Z

### Summary
The main change is to use collections that maintain a defined order to avoid having the order of operations vary between runs and between JDKs. This mainly means to use `TreeSet` and `TreeMap` where possible, use `toCollection` instead of `toSet` when collecting streams to a set, and avoid `Set.copyOf`. This is either to eliminate the issue or to make it reproducible.

The second change is how schema usages are tracked before schemas considered unused are removed (see `OpenApiGenerator.addPathSchemaRefs`). I am not sure the old code did miss to follow (expand) a type in some cases but in any case the new code is more clear on that.

Overall I am not sure this really fixed the flakiness. I could reproduce the issue about 1/10 runs locally at the start. After doing the main work I failed to get it again. At the very least it seems much more unlikely to happen. 

**Findings (for future debug if needed)**

I found that the schema complained about by validation as missing is indeed missing and that it is the only one out of all that is gone. At the time of debugging there were 921 schemas for a full API but sometimes it would just be 920. My main suspect is that due to reordering in the processing and an issue with the identification of included schemas the missing schema would sometimes be removed as "unused". But I failed to confirm that. 

**Other Fixes**

* fixed test scenarios use of  `scope` parameter value syntax (worked because the entire API also included the part under test)
* fixed entity type of some objects I noticed were not accurate
* fixed class types extending `Collection` to also map to an array schema
* fixed map with fixed key type (1 generic) also map to object schema
* fixed `JsonMixed` to map to _any_ schema
* fixed `StreamingResponseBody` to map to _any_ schema


Diff:
diff --git a/dhis-2/dhis-test-web-api/src/test/java/org/hisp/dhis/webapi/controller/OpenApiControllerTest.java b/dhis-2/dhis-test-web-api/src/test/java/org/hisp/dhis/webapi/controller/OpenApiControllerTest.java
index c621d3a81d19..8f819faa5b38 100644
--- a/dhis-2/dhis-test-web-api/src/test/java/org/hisp/dhis/webapi/controller/OpenApiControllerTest.java
+++ b/dhis-2/dhis-test-web-api/src/test/java/org/hisp/dhis/webapi/controller/OpenApiControllerTest.java
@@ -59,7 +59,6 @@
 import org.hisp.dhis.webapi.openapi.OpenApiObject.ParameterObject;
 import org.hisp.dhis.webapi.openapi.OpenApiObject.ResponseObject;
 import org.hisp.dhis.webapi.openapi.OpenApiObject.SchemaObject;
-import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
 import org.openapitools.codegen.DefaultGenerator;
 import org.openapitools.codegen.config.CodegenConfigurator;
@@ -90,7 +89,6 @@ void testGetOpenApiDocumentJson() {
   }
 
   @Test
-  @Disabled("JB: temporary until we find the cause of flakiness")
   void testGetOpenApiDocumentJson_NoValidationErrors() {
     JsonObject doc =
         GET("/openapi/openapi.json?failOnNameClash=true&failOnInconsistency=true").content();
@@ -132,7 +130,7 @@ void testGetOpenApiDocument_ScopeFilter() {
   @Test
   void testGetOpenApiDocumentHtml_DomainFilter() {
     String html =
-        GET("/openapi/openapi.html?scope=entity.DataElement", Accept(TEXT_HTML_VALUE))
+        GET("/openapi/openapi.html?scope=entity:DataElement", Accept(TEXT_HTML_VALUE))
             .content(TEXT_HTML_VALUE);
     assertContains("#DataElement", html);
   }
@@ -140,7 +138,7 @@ void testGetOpenApiDocumentHtml_DomainFilter() {
   @Test
   void testGetOpenApiDocument_DefaultValue() {
     // defaults in parameter objects (from Property analysis)
-    JsonObject users = GET("/openapi/openapi.json?scope=path./api/users").content();
+    JsonObject users = GET("/openapi/openapi.json?scope=path:/api/users").content();
     JsonObject sharedParams = users.getObject("components.parameters");
     assertEquals(50, sharedParams.getNumber("{GistParams.pageSize}.schema.default").integer());
     assertEquals(
@@ -148,7 +146,7 @@ void testGetOpenApiDocument_DefaultValue() {
     assertTrue(sharedParams.getBoolean("{GistParams.translate}.schema.default").booleanValue());
 
     // defaults in individual parameters (from endpoint method parameter analysis)
-    JsonObject fileResources = GET("/openapi/openapi.json?scope=path./api/fileResources").content();
+    JsonObject fileResources = GET("/openapi/openapi.json?scope=path:/api/fileResources").content();
     JsonObject domain =
         fileResources
             .get("paths./api/fileResources/.post.parameters")
@@ -159,7 +157,7 @@ void testGetOpenApiDocument_DefaultValue() {
             .orElse(JsonMixed.of("{}"));
     assertEquals("DATA_VALUE", domain.getString("schema.default").string());
 
-    JsonObject audits = GET("/openapi/openapi.json?scope=path./api/audits").content();
+    JsonObject audits = GET("/openapi/openapi.json?scope=path:/api/audits").content();
     JsonObject pageSize =
         audits
             .getArray("paths./api/audits/trackedEntity.get.parameters")
@@ -222,7 +220,7 @@ void testGetOpenApiDocument_GetObjectListResponse() {
 
   @Test
   void testGetOpenApiDocument_ReadOnly() {
-    JsonObject doc = GET("/openapi/openapi.json?scope=entity.JobConfiguration").content();
+    JsonObject doc = GET("/openapi/openapi.json?scope=entity:JobConfiguration").content();
     JsonObject jobConfiguration = doc.getObject("components.schemas.JobConfiguration");
     JsonObject jobConfigurationParams = doc.getObject("components.schemas.JobConfigurationParams");
     assertTrue(jobConfiguration.isObject());
@@ -243,7 +241,6 @@ void testGetOpenApiDocument_ReadOnly() {
   }
 
   @Test
-  @Disabled("JB: temporary until we find the cause of flakiness")
   void testGetOpenApiDocument_CodeGeneration() throws IOException {
     JsonObject doc = GET("/openapi/openapi.json?failOnNameClash=true").content();
 
diff --git a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/controller/CompleteDataSetRegistrationController.java b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/controller/CompleteDataSetRegistrationController.java
index 8a9a57ff7025..673b40040e1a 100644
--- a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/controller/CompleteDataSetRegistrationController.java
+++ b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/controller/CompleteDataSetRegistrationController.java
@@ -56,7 +56,6 @@
 import org.hisp.dhis.dataset.CompleteDataSetRegistrationService;
 import org.hisp.dhis.dataset.DataSet;
 import org.hisp.dhis.dataset.DataSetService;
-import org.hisp.dhis.datavalue.DataValue;
 import org.hisp.dhis.dxf2.common.ImportOptions;
 import org.hisp.dhis.dxf2.dataset.CompleteDataSetRegistrationExchangeService;
 import org.hisp.dhis.dxf2.dataset.ExportParams;
@@ -90,8 +89,9 @@
  * @author Morten Olav Hansen <mortenoh@gmail.com>
  * @author Halvdan Hoem Grelland <halvdan@dhis2.org>
  */
+@OpenApi.EntityType(CompleteDataSetRegistration.class)
 @OpenApi.Document(
-    entity = DataValue.class,
+    entity = CompleteDataSetRegistration.class,
     classifiers = {"team:platform", "purpose:data"})
 @Controller
 @ApiVersion({DhisApiVersion.DEFAULT, DhisApiVersion.ALL})
diff --git a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/controller/notification/ProgramNotificationInstanceController.java b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/controller/notification/ProgramNotificationInstanceController.java
index e818533290d5..9b8346862195 100644
--- a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/controller/notification/ProgramNotificationInstanceController.java
+++ b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/controller/notification/ProgramNotificationInstanceController.java
@@ -64,6 +64,7 @@
 /**
  * @author Zubair Asghar
  */
+@OpenApi.EntityType(ProgramNotificationInstance.class)
 @OpenApi.Document(
     entity = ProgramNotificationInstance.class,
     classifiers = {"team:tracker", "purpose:data"})
diff --git a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/Api.java b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/Api.java
index 868f297acf22..db9f6f4c8661 100644
--- a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/Api.java
+++ b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/Api.java
@@ -30,22 +30,20 @@
 package org.hisp.dhis.webapi.openapi;
 
 import static java.util.Comparator.comparing;
-import static java.util.stream.Collectors.toSet;
+import static java.util.stream.Collectors.toCollection;
 
 import java.lang.reflect.AnnotatedElement;
 import java.lang.reflect.Method;
 import java.lang.reflect.Type;
+import java.lang.reflect.WildcardType;
 import java.util.ArrayList;
 import java.util.EnumMap;
 import java.util.EnumSet;
-import java.util.LinkedHashMap;
-import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.TreeMap;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.TreeSet;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.function.Function;
 import java.util.function.Supplier;
@@ -131,19 +129,17 @@ record Scope(
   Components components = new Components();
 
   /**
-   * Note that this needs to use the {@link ConcurrentSkipListMap} as most other maps do not allow
-   * to be modified from within a callback that itself is adding an entry like {@link
-   * Map#computeIfAbsent(Object, Function)}. Here, while one {@link Schema} is resolved more {@link
-   * Schema} might be added.
+   * Note that this needs tu use a {@link Map} that allows to be modified while running a call
+   * {@link Map#computeIfAbsent(Object, Function)}. Here, while one {@link Schema} is resolved more
+   * {@link Schema} might be added.
    */
-  Map<Class<?>, Schema> schemas = new ConcurrentSkipListMap<>(comparing(Class::getName));
+  Map<Class<?>, Schema> schemas = newClassMap();
 
   /**
    * First level key is the {@link SchemaGenerator} type, second level is the {@link
    * Schema#getRawType()} of the generated {@link Schema}.
    */
-  Map<Class<?>, Map<Class<?>, Schema>> generatorSchemas =
-      new ConcurrentSkipListMap<>(comparing(Class::getName));
+  Map<Class<?>, Map<Class<?>, Schema>> generatorSchemas = newClassMap();
 
   @Data
   @NoArgsConstructor
@@ -181,13 +177,13 @@ public Stream<T> stream() {
   @Value
   static class Components {
     /** Only the shared schemas of the API by their unique name */
-    Map<String, Schema> schemas = new LinkedHashMap<>();
+    Map<String, Schema> schemas = new TreeMap<>();
 
     /**
      * Shared parameters originating from parameter object classes. These are reused purely for sake
      * of removing duplication from the resulting OpenAPI document.
      */
-    Map<Class<?>, List<Parameter>> parameters = new ConcurrentHashMap<>();
+    Map<Class<?>, List<Parameter>> parameters = newClassMap();
   }
 
   @Value
@@ -233,7 +229,7 @@ public static class Endpoint {
 
     @EqualsAndHashCode.Include Set<RequestMethod> methods = EnumSet.noneOf(RequestMethod.class);
 
-    @EqualsAndHashCode.Include Set<String> paths = new LinkedHashSet<>();
+    @EqualsAndHashCode.Include Set<String> paths = new TreeSet<>();
 
     Maybe<RequestBody> requestBody = new Maybe<>();
 
@@ -362,12 +358,17 @@ Response add(Set<Header> headers) {
 
   @Value
   @EqualsAndHashCode(onlyExplicitlyIncluded = true)
-  static class Header {
+  static class Header implements Comparable<Header> {
     @EqualsAndHashCode.Include String name;
 
     String description;
 
     Schema type;
+
+    @Override
+    public int compareTo(@Nonnull Header other) {
+      return name.compareTo(other.name);
+    }
   }
 
   @Value
@@ -438,7 +439,11 @@ boolean isOutput() {
   public static class Schema {
 
     public static Schema ofAny(java.lang.reflect.Type source) {
-      if (source != Object.class) log.warn("OpenAPI failed to analyse the type: " + source);
+      if (source == OpenApi.EntityType.class) {
+        log.warn("OpenAPI substitute is undefined for: " + source);
+      } else if (source != Object.class
+          && !(source instanceof WildcardType wt && "?".equals(wt.toString())))
+        log.warn("OpenAPI failed to analyse the type: " + source);
       return new Schema(Type.ANY, source, Object.class, null).asSingleton();
     }
 
@@ -584,7 +589,7 @@ Set<String> getRequiredProperties() {
       return getProperties().stream()
           .filter(property -> Boolean.TRUE.equals(property.getRequired()))
           .map(Property::getName)
-          .collect(toSet());
+          .collect(toCollection(TreeSet::new));
     }
 
     Schema getElementType() {
@@ -634,4 +639,9 @@ Api.Schema sealed() {
       return properties.stream().allMatch(p -> p.type.isSingleton()) ? asSingleton() : this;
     }
   }
+
+  @Nonnull
+  private static <T> TreeMap<Class<?>, T> newClassMap() {
+    return new TreeMap<>(comparing(Class::getName));
+  }
 }
diff --git a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiClassifications.java b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiClassifications.java
index 11bb0f1c9551..9aa9b186df8f 100644
--- a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiClassifications.java
+++ b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiClassifications.java
@@ -30,9 +30,10 @@
 package org.hisp.dhis.webapi.openapi;
 
 import static java.util.Collections.unmodifiableMap;
+import static java.util.Comparator.comparing;
 import static java.util.Comparator.comparingInt;
+import static java.util.stream.Collectors.toCollection;
 import static java.util.stream.Collectors.toMap;
-import static java.util.stream.Collectors.toUnmodifiableSet;
 
 import java.util.LinkedHashMap;
 import java.util.List;
@@ -113,7 +114,9 @@ static int matchesCount(
 
   static Set<Class<?>> matches(
       @Nonnull Set<Class<?>> controllers, @Nonnull Map<String, Set<String>> filters) {
-    return controllers.stream().filter(c -> matches(filters, c)).collect(toUnmodifiableSet());
+    return controllers.stream()
+        .filter(c -> matches(filters, c))
+        .collect(toCollection((() -> new TreeSet<>(comparing(Class::getName)))));
   }
 
   private static boolean matches(@Nonnull Map<String, Set<String>> filters, Class<?> controller) {
diff --git a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiDescriptions.java b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiDescriptions.java
index 2fcb7875cd29..d82130b704b7 100644
--- a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiDescriptions.java
+++ b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiDescriptions.java
@@ -34,10 +34,10 @@
 import static java.util.stream.Collectors.joining;
 
 import java.io.InputStream;
-import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Scanner;
+import java.util.TreeMap;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ConcurrentSkipListMap;
 import java.util.function.UnaryOperator;
@@ -90,7 +90,7 @@ final class ApiDescriptions {
 
   private final Class<?> target;
 
-  private final Map<String, String> entries = new HashMap<>();
+  private final Map<String, String> entries = new TreeMap<>();
 
   static ApiDescriptions of(Class<?> target) {
     return CACHE.computeIfAbsent(target, ApiDescriptions::ofUncached);
diff --git a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiExtractor.java b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiExtractor.java
index d2a3bb43d438..fae9f2e591f3 100644
--- a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiExtractor.java
+++ b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiExtractor.java
@@ -31,6 +31,8 @@
 
 import static java.util.Arrays.copyOfRange;
 import static java.util.Arrays.stream;
+import static java.util.Comparator.comparing;
+import static java.util.stream.Collectors.toCollection;
 import static java.util.stream.Collectors.toList;
 import static java.util.stream.Collectors.toSet;
 import static java.util.stream.Collectors.toUnmodifiableSet;
@@ -64,7 +66,8 @@
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
-import java.util.concurrent.ConcurrentHashMap;
+import java.util.TreeMap;
+import java.util.TreeSet;
 import java.util.function.Function;
 import java.util.function.UnaryOperator;
 import java.util.stream.Stream;
@@ -111,7 +114,7 @@ final class ApiExtractor {
 
   record Configuration(boolean ignoreTypeAs) {}
 
-  private static final Map<Class<?>, Api.SchemaGenerator> GENERATORS = new ConcurrentHashMap<>();
+  private static final Map<Class<?>, Api.SchemaGenerator> GENERATORS = newClassMap();
 
   public static void register(Class<?> type, Api.SchemaGenerator generator) {
     GENERATORS.put(type, generator);
@@ -200,7 +203,9 @@ private Api.Endpoint extractEndpoint(Api.Controller controller, EndpointMapping
 
     // request media types
     Set<MediaType> consumes =
-        mapping.consumes().stream().map(MediaType::parseMediaType).collect(toSet());
+        mapping.consumes().stream()
+            .map(MediaType::parseMediaType)
+            .collect(toCollection(TreeSet::new));
     if (consumes.isEmpty()) {
       // assume JSON if nothing is set explicitly
       consumes.add(MediaType.APPLICATION_JSON);
@@ -286,7 +291,9 @@ private Map<HttpStatus, Api.Response> extractResponses(
       Api.Endpoint endpoint, EndpointMapping mapping, Set<MediaType> consumes) {
     Method source = mapping.source();
     Set<MediaType> produces =
-        mapping.produces().stream().map(MediaType::parseMediaType).collect(toSet());
+        mapping.produces().stream()
+            .map(MediaType::parseMediaType)
+            .collect(toCollection(TreeSet::new));
     if (produces.isEmpty()) {
       // either make symmetric or assume JSON as standard
       if (consumes.contains(MediaType.APPLICATION_JSON)
@@ -412,7 +419,7 @@ private Map<HttpStatus, Api.Response> extractResponses(
                         header.name(),
                         header.description(),
                         extractSchema(endpoint, null, header.type())))
-            .collect(toSet());
+            .collect(toCollection(TreeSet::new));
     Set<MediaType> produces =
         response.mediaTypes().length == 0
             ? defaultProduces
@@ -664,7 +671,7 @@ private static Api.Schema extractGeneratorSchema(
             .getIn()
             .getIn()
             .getGeneratorSchemas()
-            .computeIfAbsent(genType, key -> new ConcurrentHashMap<>());
+            .computeIfAbsent(genType, key -> newClassMap());
     schema.getSharedName().setValue(getGeneratorTypeSharedName(genType, schema));
     if (schema.isShared()) {
       Api.Schema shared = genTypes.putIfAbsent(ofType, schema);
@@ -673,6 +680,11 @@ private static Api.Schema extractGeneratorSchema(
     return type == genType ? schema : Api.Schema.ofArray(type, type).withElements(schema).sealed();
   }
 
+  @Nonnull
+  private static <T> TreeMap<Class<?>, T> newClassMap() {
+    return new TreeMap<>(comparing(Class::getName));
+  }
+
   private static String getGeneratorTypeSharedName(Class<?> genType, Api.Schema schema) {
     Class<?> of = schema.getRawType();
     Api.Schema.Type type = schema.getType();
@@ -711,6 +723,9 @@ private Api.Schema extractClassSchema(Api.Endpoint endpoint, Class<?> type) {
     if (s != null) {
       return s;
     }
+    if (type.isAnnotation()) {
+      return Api.Schema.ofAny(type);
+    }
     UnaryOperator<Api.Schema> addShared =
         schema -> {
           schema
@@ -815,8 +830,9 @@ private Api.Schema extractTypeSchema(Api.Endpoint endpoint, Type source) {
       if (rawType == Class.class) {
         return Api.Schema.ofSimple(rawType);
       }
-      Type typeArg0 = pt.getActualTypeArguments()[0];
-      if (Collection.class.isAssignableFrom(rawType) && rawType.isInterface()
+      Type[] actualTypes = pt.getActualTypeArguments();
+      Type typeArg0 = actualTypes[0];
+      if (Collection.class.isAssignableFrom(rawType) && actualTypes.length == 1
           || rawType == Iterable.class
           || rawType == Stream.class
           || rawType == JsonList.class) {
@@ -826,13 +842,22 @@ private Api.Schema extractTypeSchema(Api.Endpoint endpoint, Type source) {
             .withElements(extractTypeSchema(endpoint, typeArg0))
             .sealed();
       }
-      if (Map.class.isAssignableFrom(rawType) && rawType.isInterface()) {
+      if (Map.class.isAssignableFrom(rawType) && actualTypes.length == 2) {
         return Api.Schema.ofObject(source, rawType)
             .withEntries(
-                extractTypeSchema(endpoint, typeArg0),
-                extractTypeSchema(endpoint, pt.getActualTypeArguments()[1]))
+                extractTypeSchema(endpoint, typeArg0), extractTypeSchema(endpoint, actualTypes[1]))
             .sealed();
       }
+      if (Map.class.isAssignableFrom(rawType) && actualTypes.length == 1) {
+        Type extended = rawType.getGenericSuperclass();
+        if (extended instanceof ParameterizedType pst) {
+          return Api.Schema.ofObject(source, rawType)
+              .withEntries(
+                  extractTypeSchema(endpoint, pst.getActualTypeArguments()[0]),
+                  extractTypeSchema(endpoint, typeArg0))
+              .sealed();
+        }
+      }
       if (JsonMap.class.isAssignableFrom(rawType)) {
         return Api.Schema.ofObject(source, rawType)
             .withEntries(
diff --git a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiIntegrator.java b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiIntegrator.java
index 2ba9fd4c7131..3260bc6c903c 100644
--- a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiIntegrator.java
+++ b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/ApiIntegrator.java
@@ -32,20 +32,16 @@
 import static java.lang.String.format;
 import static java.util.Comparator.comparing;
 import static java.util.stream.Collectors.joining;
-import static java.util.stream.Collectors.toSet;
 
 import java.lang.reflect.Member;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.EnumMap;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.TreeMap;
-import java.util.concurrent.ConcurrentHashMap;
+import java.util.TreeSet;
 import java.util.function.Consumer;
 import java.util.function.UnaryOperator;
 import java.util.regex.Pattern;
@@ -174,7 +170,7 @@ private void validateParameter(Parameter p) {
    * <p>All shared types are transferred to {@link Api.Components#getSchemas()}.
    */
   private void nameSharedSchemas() {
-    Map<String, List<Api.Schema>> sharedSchemasByName = new LinkedHashMap<>();
+    Map<String, List<Api.Schema>> sharedSchemasByName = new TreeMap<>();
     Consumer<Api.Schema> addSchema =
         schema ->
             sharedSchemasByName
@@ -229,7 +225,7 @@ private void nameSharedSchemas() {
    */
   private void nameSharedParameters() {
     Map<String, List<Map.Entry<Class<?>, List<Api.Parameter>>>> sharedParametersByName =
-        new HashMap<>();
+        new TreeMap<>();
     api.getComponents()
         .getParameters()
         .entrySet()
@@ -251,7 +247,7 @@ private void nameSharedParameters() {
             params.get(0).getValue().forEach(p -> p.getSharedName().setValue(name));
           } else {
             // there might be a clash if a parameter name occurs in more than one
-            Set<String> usedNames = new HashSet<>();
+            Set<String> usedNames = new TreeSet<>();
             params.stream()
                 .flatMap(e -> e.getValue().stream())
                 .forEach(
@@ -287,9 +283,9 @@ private void checkNoParameterNameClash(
           if (params.size() > 1) {
             List<Api.Parameter> paramsInNamespace =
                 params.stream().flatMap(e -> e.getValue().stream()).toList();
-            Set<String> names =
-                paramsInNamespace.stream().map(Api.Parameter::getName).collect(toSet());
-            if (paramsInNamespace.size() > names.size()) {
+            int names =
+                (int) paramsInNamespace.stream().map(Api.Parameter::getName).distinct().count();
+            if (paramsInNamespace.size() > names) {
               throw new IllegalStateException(createParamNameClashMessage(name, params));
             }
           }
@@ -346,16 +342,16 @@ private Api.Schema generateInputSchema(Api.Schema output) {
     }
     Api.Schema input = Api.Schema.ofObject(output.getSource(), schemaType);
     output.getInput().setValue(input);
+    input.getDirection().setValue(Api.Schema.Direction.IN);
     if (output.isShared()) {
       String name = output.getSharedName().getValue() + "Params";
       Map<String, Api.Schema> schemas = api.getComponents().getSchemas();
       if (schemas.containsKey(name)) name = output.getSharedName().getValue() + "_Params";
       input.getSharedName().setValue(name);
       api.getGeneratorSchemas()
-          .computeIfAbsent(Api.Schema.Direction.class, k -> new ConcurrentHashMap<>())
+          .computeIfAbsent(Api.Schema.Direction.class, k -> newClassMap())
           .put(schemaType, input);
     }
-    input.getDirection().setValue(Api.Schema.Direction.IN);
     output.getProperties().stream()
         .filter(Api.Property::isInput)
         .forEach(p -> input.getProperties().add(p.withType(generateInputReferenceSchema(p))));
@@ -374,7 +370,7 @@ private Api.Schema generateIdObject(Api.Schema of) {
     Class<?> schemaType = of.getIdentifyAs();
     Api.Schema object = Api.Schema.ofObject(of.getSource(), schemaType);
     Map<Class<?>, Api.Schema> idSchemas =
-        api.getGeneratorSchemas().computeIfAbsent(UID.class, key -> new ConcurrentHashMap<>());
+        api.getGeneratorSchemas().computeIfAbsent(UID.class, key -> newClassMap());
 
     Api.Schema idType =
         idSchemas.computeIfAbsent(
@@ -388,6 +384,11 @@ private Api.Schema generateIdObject(Api.Schema of) {
     return object;
   }
 
+  @Nonnull
+  private static <T> TreeMap<Class<?>, T> newClassMap() {
+    return new TreeMap<>(comparing(Class::getName));
+  }
+
   /*
    * 3. Add description texts from markdown files to the Api model
    */
@@ -552,7 +553,7 @@ private Map<String, Api.Endpoint> groupEndpointsByFragment(
     if (endpoints.size() == 1) return Map.of("", endpoints.get(0));
 
     Api.Endpoint defaultEndpoint = getDefaultEndpoint(endpoints);
-    Map<String, Api.Endpoint> endpointsByFragment = new HashMap<>();
+    Map<String, Api.Endpoint> endpointsByFragment = new TreeMap<>();
     for (Api.Endpoint endpoint : endpoints) {
       if (!endpoint.equals(defaultEndpoint)) {
         if (isMergeable(endpoint, defaultEndpoint)) {
diff --git a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/DirectType.java b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/DirectType.java
index 35ea551e21f5..cc2746b1aad3 100644
--- a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/DirectType.java
+++ b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/DirectType.java
@@ -61,6 +61,7 @@
 import org.hisp.dhis.jsontree.JsonArray;
 import org.hisp.dhis.jsontree.JsonBoolean;
 import org.hisp.dhis.jsontree.JsonDate;
+import org.hisp.dhis.jsontree.JsonMixed;
 import org.hisp.dhis.jsontree.JsonNumber;
 import org.hisp.dhis.jsontree.JsonObject;
 import org.hisp.dhis.jsontree.JsonString;
@@ -78,6 +79,7 @@
 import org.locationtech.jts.geom.Geometry;
 import org.springframework.core.io.InputStreamResource;
 import org.springframework.web.multipart.MultipartFile;
+import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody;
 
 /**
  * A type mapping between a Java type and its OpenAPI form that is specified directly in terms of an
@@ -247,6 +249,7 @@ private static void share(Class<?> source) {
     oneOf(JsonPointer.class, schema -> schema.type("string"));
 
     oneOf(JsonValue.class, schema -> schema.type("any"));
+    oneOf(JsonMixed.class, schema -> schema.type("any"));
     oneOf(JsonObject.class, schema -> schema.type("object"));
     oneOf(JsonArray.class, schema -> schema.type("array"));
     oneOf(JsonString.class, schema -> schema.type("string"));
@@ -256,6 +259,8 @@ private static void share(Class<?> source) {
 
     oneOf(Geometry.class, schema -> schema.type("object"));
     oneOf(JobParameters.class, schema -> schema.type("object"));
+
+    oneOf(StreamingResponseBody.class, schema -> schema.type("any"));
   }
 
   private static <T extends Enum<T>> List<String> enums(Class<T> type, Function<T, String> name) {
diff --git a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/OpenApiGenerator.java b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/OpenApiGenerator.java
index 47037e5b48aa..cf977c81ca3a 100644
--- a/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/OpenApiGenerator.java
+++ b/dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/OpenApiGenerator.java
@@ -37,12 +37,12 @@
 
 import java.util.ArrayList;
 import java.util.Collection;
-import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.TreeMap;
+import java.util.TreeSet;
 import lombok.Builder;
 import lombok.Value;
 import lombok.extern.slf4j.Slf4j;
@@ -122,13 +122,13 @@ public static String generate(
   private final Api api;
   private final Info info;
   private final OpenApiGenerationParams params;
-  private final Map<String, List<Api.Endpoint>> endpointsByBaseOperationId = new HashMap<>();
+  private final Map<String, List<Api.Endpoint>> endpointsByBaseOperationId = new TreeMap<>();
 
   /**
    * Accumulates the schema references used during generation of the OpenAPI {@code paths} part.
    * This is later used to ensure no schema is added to the output that isn't actually referenced.
    */
-  private final Set<String> pathSchemaRefs = new HashSet<>();
+  private final Set<String> pathSchemaRefs = new TreeSet<>();
 
   private OpenApiGenerator(
       Api api,
@@ -592,7 +592,7 @@ private void retainUsedSchemasOnly() {
     // in that set
     Set<String> expanded = new HashSet<>();
     // needs a copy as we modify the iterated collection in the forEach
-    Set.copyOf(pathSchemaRefs).forEach(name -> addPathSchemaRefs(name, expanded));
+    new TreeSet<>(pathSchemaRefs).forEach(name -> addPathSchemaRefs(name, expanded));
     // now remove those schemas that are not referenced
     api.getComponents().getSchemas().keySet().removeIf(name -> !pathSchemaRefs.contains(name));
   }
@@ -608,9 +608,13 @@ private void addPathSchemaRefs(String name, Set<String> expanded) {
   private void addPathSchemaRefs(Api.Schema schema, Set<String> expanded) {
     if (schema.isShared()) {
       String name = schema.getSharedName().getValue();
-      addPathSchemaRefs(name, expanded);
-      if (expanded.contains(name)) return;
+      if (!expanded.contains(name)) {
+        pathSchemaRefs.add(name);
+        expanded.add(name);
+        schema.getProperties().forEach(p -> addPathSchemaRefs(p.getType(), expanded));
+      }
+    } else {
+      schema.getProperties().forEach(p -> addPathSchemaRefs(p.getType(), expanded));
     }
-    schema.getProperties().forEach(p -> addPathSchemaRefs(p.getType(), expanded));
   }
 }


Reviews:
user: login: github-advanced-security[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T16:21:04Z
user: login: david-mackessy
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T10:24:56Z
user: login: netroms
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T15:08:27Z

Review comments:
diff_hunk: @@ -131,19 +129,17 @@
   Components components = new Components();
 
   /**
-   * Note that this needs to use the {@link ConcurrentSkipListMap} as most other maps do not allow
-   * to be modified from within a callback that itself is adding an entry like {@link
-   * Map#computeIfAbsent(Object, Function)}. Here, while one {@link Schema} is resolved more {@link
-   * Schema} might be added.
+   * Note that this needs tu use a {@link Map} that allows to be modified while running a call
+   * {@link Map#computeIfAbsent(Object, Function)}. Here, while one {@link Schema} is resolved more
+   * {@link Schema} might be added.
    */
-  Map<Class<?>, Schema> schemas = new ConcurrentSkipListMap<>(comparing(Class::getName));
+  Map<Class<?>, Schema> schemas = newClassMap();
path: dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/Api.java
user: login: github-advanced-security[bot]
body: ## Exposing internal representation

getSchemas exposes the internal representation stored in field schemas. The value may be modified [after this call to getSchemas](1).

[Show more details](https://github.com/dhis2/dhis2-core/security/code-scanning/9602)
created_at: 2025-03-20T16:21:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -181,13 +177,13 @@
   @Value
   static class Components {
     /** Only the shared schemas of the API by their unique name */
-    Map<String, Schema> schemas = new LinkedHashMap<>();
+    Map<String, Schema> schemas = new TreeMap<>();
path: dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/Api.java
user: login: github-advanced-security[bot]
body: ## Exposing internal representation

getSchemas exposes the internal representation stored in field schemas. The value may be modified [after this call to getSchemas](1).

[Show more details](https://github.com/dhis2/dhis2-core/security/code-scanning/9600)
created_at: 2025-03-20T16:21:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -233,7 +229,7 @@
 
     @EqualsAndHashCode.Include Set<RequestMethod> methods = EnumSet.noneOf(RequestMethod.class);
 
-    @EqualsAndHashCode.Include Set<String> paths = new LinkedHashSet<>();
+    @EqualsAndHashCode.Include Set<String> paths = new TreeSet<>();
path: dhis-2/dhis-web-api/src/main/java/org/hisp/dhis/webapi/openapi/Api.java
user: login: github-advanced-security[bot]
body: ## Exposing internal representation

getPaths exposes the internal representation stored in field paths. The value may be modified [after this call to getPaths](1).

[Show more details](https://github.com/dhis2/dhis2-core/security/code-scanning/9601)
created_at: 2025-03-20T16:21:04Z
author_association: CONTRIBUTOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-03-24T09:45:13Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=dhis2_dhis2-core&pullRequest=20348) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=dhis2_dhis2-core&pullRequest=20348&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=dhis2_dhis2-core&pullRequest=20348&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=dhis2_dhis2-core&pullRequest=20348&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=dhis2_dhis2-core&pullRequest=20348&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=dhis2_dhis2-core&pullRequest=20348&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=dhis2_dhis2-core&pullRequest=20348)


author_association: NONE