Repository: apache/druid
Pull request #17860: Web console: add turboLoadingNodes
Author: login: vogievetsky
Labels: name: Area - Documentation
name: Area - Web Console
Created: 2025-04-01T21:30:45Z | Merged: 2025-04-02T22:56:34Z

The web console follow up to https://github.com/apache/druid/pull/17775

<img width="534" alt="image" src="https://github.com/user-attachments/assets/aa9c0dfc-c024-41d2-a046-3a22a419ca0c" />



Diff:
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index 5723d8926e4d..91168d9471a1 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -953,7 +953,7 @@ The following table shows the dynamic configuration properties for the Coordinat
 |`decommissioningNodes`|List of Historical servers to decommission. Coordinator will not assign new segments to decommissioning servers, and segments will be moved away from them to be placed on non-decommissioning servers at the maximum rate specified by `maxSegmentsToMove`.|none|
 |`pauseCoordination`|Boolean flag for whether or not the Coordinator should execute its various duties of coordinating the cluster. Setting this to true essentially pauses all coordination work while allowing the API to remain up. Duties that are paused include all classes that implement the `CoordinatorDuty` interface. Such duties include: segment balancing, segment compaction, submitting kill tasks for unused segments (if enabled), logging of used segments in the cluster, marking of newly unused or overshadowed segments, matching and execution of load/drop rules for used segments, unloading segments that are no longer marked as used from Historical servers. An example of when an admin may want to pause coordination would be if they are doing deep storage maintenance on HDFS name nodes with downtime and don't want the Coordinator to be directing Historical nodes to hit the name node with API requests until maintenance is done and the deep store is declared healthy for use again.|false|
 |`replicateAfterLoadTimeout`|Boolean flag for whether or not additional replication is needed for segments that have failed to load due to the expiry of `druid.coordinator.load.timeout`. If this is set to true, the Coordinator will attempt to replicate the failed segment on a different historical server. This helps improve the segment availability if there are a few slow Historicals in the cluster. However, the slow Historical may still load the segment later and the Coordinator may issue drop requests if the segment is over-replicated.|false|
-|`turboLoadingNodes`| List of Historical servers to place in turbo loading mode. These servers use a larger thread-pool to load segments faster but at the cost of query performance. For servers specified in `turboLoadingNodes`, `druid.coordinator.loadqueuepeon.http.batchSize` is ignored and the coordinator uses the value of the respective `numLoadingThreads` instead. |none|
+|`turboLoadingNodes`| Experimental. List of Historical servers to place in turbo loading mode. These servers use a larger thread-pool to load segments faster but at the cost of query performance. For servers specified in `turboLoadingNodes`, `druid.coordinator.loadqueuepeon.http.batchSize` is ignored and the coordinator uses the value of the respective `numLoadingThreads` instead.<br/>Please use this config with caution. All servers should eventually be removed from this list once the segment loading on the respective historicals is finished. |none|
 
 ##### Smart segment loading
 
diff --git a/web-console/src/components/auto-form/auto-form.tsx b/web-console/src/components/auto-form/auto-form.tsx
index 7026ca5d5dc6..746e7dc4ce84 100644
--- a/web-console/src/components/auto-form/auto-form.tsx
+++ b/web-console/src/components/auto-form/auto-form.tsx
@@ -28,7 +28,7 @@ import { IconNames } from '@blueprintjs/icons';
 import type { JSX } from 'react';
 import React from 'react';
 
-import { deepDelete, deepGet, deepSet, durationSanitizer } from '../../utils';
+import { deepDelete, deepGet, deepSet, durationSanitizer, EXPERIMENTAL_ICON } from '../../utils';
 import { ArrayInput } from '../array-input/array-input';
 import { FancyNumericInput } from '../fancy-numeric-input/fancy-numeric-input';
 import { FormGroupWithInfo } from '../form-group-with-info/form-group-with-info';
@@ -65,6 +65,7 @@ export interface Field<M> {
   max?: number;
   zeroMeansUndefined?: boolean;
   height?: string;
+  experimental?: Functor<M, boolean>;
   disabled?: Functor<M, boolean>;
   defined?: Functor<M, boolean | undefined>;
   required?: Functor<M, boolean>;
@@ -509,10 +510,20 @@ export class AutoForm<T extends Record<string, any>> extends React.PureComponent
     if (!model) return;
 
     const label = field.label || AutoForm.makeLabelName(field.name);
+    const experimental = AutoForm.evaluateFunctor(field.experimental, model, false);
     return (
       <FormGroupWithInfo
         key={field.name}
-        label={label}
+        label={
+          experimental ? (
+            <>
+              {`${label} `}
+              {EXPERIMENTAL_ICON}
+            </>
+          ) : (
+            label
+          )
+        }
         info={field.info ? <PopoverText>{field.info}</PopoverText> : undefined}
       >
         {this.renderFieldInput(field)}
diff --git a/web-console/src/components/header-bar/__snapshots__/header-bar.spec.tsx.snap b/web-console/src/components/header-bar/__snapshots__/header-bar.spec.tsx.snap
index d8e883e50e4c..d713f2fa86f1 100644
--- a/web-console/src/components/header-bar/__snapshots__/header-bar.spec.tsx.snap
+++ b/web-console/src/components/header-bar/__snapshots__/header-bar.spec.tsx.snap
@@ -168,7 +168,15 @@ exports[`HeaderBar matches snapshot 1`] = `
             disabled={false}
             href="#explore"
             icon="map"
-            label="(experimental)"
+            labelElement={
+              <Blueprint5.Icon
+                autoLoad={true}
+                data-tooltip="Experimental"
+                icon="lab-test"
+                intent="warning"
+                tagName="span"
+              />
+            }
             multiline={false}
             popoverProps={{}}
             selected={false}
diff --git a/web-console/src/components/header-bar/header-bar.tsx b/web-console/src/components/header-bar/header-bar.tsx
index 46c1e9340fde..140edc051c1b 100644
--- a/web-console/src/components/header-bar/header-bar.tsx
+++ b/web-console/src/components/header-bar/header-bar.tsx
@@ -44,6 +44,7 @@ import {
 import { Capabilities } from '../../helpers';
 import { getLink } from '../../links';
 import {
+  EXPERIMENTAL_ICON,
   localStorageGetJson,
   LocalStorageKeys,
   localStorageRemove,
@@ -153,7 +154,7 @@ export const HeaderBar = React.memo(function HeaderBar(props: HeaderBarProps) {
       <MenuItem
         icon={IconNames.MAP}
         text="Explore"
-        label="(experimental)"
+        labelElement={EXPERIMENTAL_ICON}
         href="#explore"
         disabled={!capabilities.hasSql()}
         selected={active === 'explore'}
diff --git a/web-console/src/druid-models/coordinator-dynamic-config/coordinator-dynamic-config.tsx b/web-console/src/druid-models/coordinator-dynamic-config/coordinator-dynamic-config.tsx
index bdd47a317225..4a7224dbee4d 100644
--- a/web-console/src/druid-models/coordinator-dynamic-config/coordinator-dynamic-config.tsx
+++ b/web-console/src/druid-models/coordinator-dynamic-config/coordinator-dynamic-config.tsx
@@ -38,6 +38,7 @@ export interface CoordinatorDynamicConfig {
   replicateAfterLoadTimeout?: boolean;
   useRoundRobinSegmentAssignment?: boolean;
   smartSegmentLoading?: boolean;
+  turboLoadingNodes?: string[];
 
   // Undocumented
   debugDimensions?: any;
@@ -156,6 +157,26 @@ export const COORDINATOR_DYNAMIC_CONFIG_FIELDS: Field<CoordinatorDynamicConfig>[
       </>
     ),
   },
+  {
+    name: 'turboLoadingNodes',
+    type: 'string-array',
+    experimental: true,
+    info: (
+      <>
+        <p>
+          List of Historical servers to place in turbo loading mode. These servers use a larger
+          thread-pool to load segments faster but at the cost of query performance. For servers
+          specified in <Code>turboLoadingNodes</Code>,{' '}
+          <Code>druid.coordinator.loadqueuepeon.http.batchSize</Code> is ignored and the coordinator
+          uses the value of the respective <Code>numLoadingThreads</Code> instead.
+        </p>
+        <p>
+          Please use this config with caution. All servers should eventually be removed from this
+          list once the segment loading on the respective historicals is finished.
+        </p>
+      </>
+    ),
+  },
   {
     name: 'killDataSourceWhitelist',
     label: 'Kill datasource whitelist',
diff --git a/web-console/src/utils/general.tsx b/web-console/src/utils/general.tsx
index d204c7a89e07..1cb6765b86f8 100644
--- a/web-console/src/utils/general.tsx
+++ b/web-console/src/utils/general.tsx
@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 
-import { Classes, Intent } from '@blueprintjs/core';
+import { Classes, Icon, Intent } from '@blueprintjs/core';
 import type { IconName } from '@blueprintjs/icons';
 import { IconNames } from '@blueprintjs/icons';
 import copy from 'copy-to-clipboard';
@@ -708,3 +708,7 @@ export function toggle<T>(xs: readonly T[], x: T, eq?: (a: T, b: T) => boolean):
   const e = eq || ((a, b) => a === b);
   return xs.find(_ => e(_, x)) ? xs.filter(d => !e(d, x)) : xs.concat([x]);
 }
+
+export const EXPERIMENTAL_ICON = (
+  <Icon icon={IconNames.LAB_TEST} intent={Intent.WARNING} data-tooltip="Experimental" />
+);
diff --git a/web-console/src/views/sql-data-loader-view/schema-step/schema-step.tsx b/web-console/src/views/sql-data-loader-view/schema-step/schema-step.tsx
index 232c6635361f..f3ee9fa2adc4 100644
--- a/web-console/src/views/sql-data-loader-view/schema-step/schema-step.tsx
+++ b/web-console/src/views/sql-data-loader-view/schema-step/schema-step.tsx
@@ -22,7 +22,6 @@ import {
   ButtonGroup,
   Callout,
   FormGroup,
-  Icon,
   Intent,
   Menu,
   MenuDivider,
@@ -83,6 +82,7 @@ import {
   dataTypeToIcon,
   deepSet,
   DruidError,
+  EXPERIMENTAL_ICON,
   filterMap,
   oneOf,
   queryDruidSql,
@@ -107,8 +107,6 @@ import { RollupAnalysisPane } from './rollup-analysis-pane/rollup-analysis-pane'
 
 import './schema-step.scss';
 
-const EXPERIMENTAL_ICON = <Icon icon={IconNames.WARNING_SIGN} title="Experimental" />;
-
 const queryRunner = new QueryRunner();
 
 function digestQueryString(queryString: string): {


Reviews:
user: login: capistrant
body: I think the CI is broken cuz of docusaurus upgrade which is being looked into separately... upon further review maybe merging latest master into your branch would give better idea on the CI fail. I think your branch is behind master a bit and missing the docusaurus change all together.  
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T22:35:42Z
user: login: cryptoe
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T05:32:08Z
user: login: vogievetsky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T14:58:54Z
user: login: vogievetsky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T14:59:55Z
user: login: kgyrtkirk
body: +1
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-02T22:26:31Z

Review comments:
diff_hunk: @@ -156,6 +157,19 @@ export const COORDINATOR_DYNAMIC_CONFIG_FIELDS: Field<CoordinatorDynamicConfig>[
       </>
     ),
   },
+  {
+    name: 'turboLoadingNodes',
+    type: 'string-array',
+    info: (
+      <>
+        List of Historical servers to place in turbo loading mode. These servers use a larger
+        thread-pool to load segments faster but at the cost of query performance. For servers
+        specified in <Code>turboLoadingNodes</Code>,{' '}
+        <Code>druid.coordinator.loadqueuepeon.http.batchSize</Code> is ignored and the coordinator
+        uses the value of the respective <Code>numLoadingThreads</Code> instead.
path: web-console/src/druid-models/coordinator-dynamic-config/coordinator-dynamic-config.tsx
user: login: cryptoe
body: ```suggestion
        uses the value of the respective <Code>numLoadingThreads</Code> instead. Please use this config with caution. All servers should eventually be removed from this list once the segment loading on the respective historicals is finished.
```
created_at: 2025-04-02T05:31:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -156,6 +157,19 @@ export const COORDINATOR_DYNAMIC_CONFIG_FIELDS: Field<CoordinatorDynamicConfig>[
       </>
     ),
   },
+  {
+    name: 'turboLoadingNodes',
+    type: 'string-array',
+    info: (
+      <>
+        List of Historical servers to place in turbo loading mode. These servers use a larger
path: web-console/src/druid-models/coordinator-dynamic-config/coordinator-dynamic-config.tsx
user: login: cryptoe
body: ```suggestion
        [Experimental] List of Historical servers to place in turbo loading mode. These servers use a larger
```
created_at: 2025-04-02T05:32:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -156,6 +157,19 @@ export const COORDINATOR_DYNAMIC_CONFIG_FIELDS: Field<CoordinatorDynamicConfig>[
       </>
     ),
   },
+  {
+    name: 'turboLoadingNodes',
+    type: 'string-array',
+    info: (
+      <>
+        List of Historical servers to place in turbo loading mode. These servers use a larger
+        thread-pool to load segments faster but at the cost of query performance. For servers
+        specified in <Code>turboLoadingNodes</Code>,{' '}
+        <Code>druid.coordinator.loadqueuepeon.http.batchSize</Code> is ignored and the coordinator
+        uses the value of the respective <Code>numLoadingThreads</Code> instead.
path: web-console/src/druid-models/coordinator-dynamic-config/coordinator-dynamic-config.tsx
user: login: vogievetsky
body: this text is just copied from the docs so the same change should probably be done there. I can do it.

note: the philosophy I have is that the console is the docs that came to life. The docs should always be the source of truth.
created_at: 2025-04-02T14:58:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -156,6 +157,19 @@ export const COORDINATOR_DYNAMIC_CONFIG_FIELDS: Field<CoordinatorDynamicConfig>[
       </>
     ),
   },
+  {
+    name: 'turboLoadingNodes',
+    type: 'string-array',
+    info: (
+      <>
+        List of Historical servers to place in turbo loading mode. These servers use a larger
path: web-console/src/druid-models/coordinator-dynamic-config/coordinator-dynamic-config.tsx
user: login: vogievetsky
body: Experimental features get ` (experimental)` at the end of their label (not just the description). Will add. Let's add it to the docs also
created_at: 2025-04-02T14:59:55Z
author_association: CONTRIBUTOR

PR comments:
user: login: capistrant
created_at: 2025-04-02T13:40:25Z
body: @cryptoe if we are wanting to call this out as experimental for druid 33, do we also need to update https://github.com/apache/druid/blob/master/docs/configuration/index.md to say the same in docs?
author_association: CONTRIBUTOR
user: login: cryptoe
created_at: 2025-04-03T03:13:40Z
body: > @cryptoe if we are wanting to call this out as experimental for druid 33, do we also need to update https://github.com/apache/druid/blob/master/docs/configuration/index.md to say the same in docs?


Yup I think we should call it experimental in the docs as well. 

author_association: CONTRIBUTOR