Repository: jupyterlab/jupyterlab
Pull request #17363: If subshells are supported by the kernel, send comm messages to subshells
Author: login: martinRenou
Labels: name: enhancement
name: documentation
name: pkg:services
name: pkg:apputils
name: status:resolved-locked
description: Closed inactive issues are locked after a while. Please open a new issue for related discussion.
Created: 2025-03-04T13:01:43Z | Merged: 2025-03-18T20:41:47Z

## References

This builds on top of the work done in both ipykernel and JupyterLab for subshells support:
- https://github.com/ipython/ipykernel/pull/1249
- https://github.com/jupyterlab/jupyterlab/pull/16963

## Code changes

The code changes are only on `Comm` message handling. We're now sending all comm messages on a subshell (one subshell per-comm). There is no public API change.

Subshells are only used if the kernel supports it.

## User-facing changes

This improves a lot the ipywidgets usage, allowing to execute callbacks even if a blocking execute request is being processed.

### BEFORE THIS PR:

https://github.com/user-attachments/assets/55d1bdb2-36e9-4ebe-bb83-e13194a1da98

### AFTER THIS PR:

https://github.com/user-attachments/assets/393850e6-8fd8-48c7-9448-e4ed23728cc2

## Backwards-incompatible changes

None


Diff:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f1097e211846..fa19314bda13 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -76,6 +76,13 @@ A new console panel will open allowing access to variables from the main session
 
 <img alt="The subshell console" src="https://raw.githubusercontent.com/jupyterlab/jupyterlab/main/docs/source/getting_started/changelog_assets/4.4-subshells.png" class="jp-screenshot">
 
+JupyterLab can automatically create subshells for comm messages if kernels support this feature.
+This enables concurrent execution of `ipywidgets` callbacks without requiring a user action.
+This behavior can be changed from the settings, with options to:
+- disable that feature to not use subshells for comms
+- (default) automatically create one subshell per comm-target (e.g. one subshell for all `ipywidgets`)
+- automatically create one subshell per comm (this can lead to issues if creating many comms)
+
 ### Context menu opt-out
 
 The context menu, available under right mouse click in browsers, can now be disabled globally
diff --git a/packages/apputils-extension/schema/running-sessions-status.json b/packages/apputils-extension/schema/running-sessions-status.json
deleted file mode 100644
index e21a16270875..000000000000
--- a/packages/apputils-extension/schema/running-sessions-status.json
+++ /dev/null
@@ -1,14 +0,0 @@
-{
-  "title": "Running Terminals and Kernels",
-  "description": "Running Terminals and Kernels settings",
-  "additionalProperties": false,
-  "properties": {
-    "showStatusBarItem": {
-      "type": "boolean",
-      "title": "Show the status bar item",
-      "description": "Whether to show the running terminals and kernels item in the status bar",
-      "default": true
-    }
-  },
-  "type": "object"
-}
diff --git a/packages/apputils-extension/schema/sessions-settings.json b/packages/apputils-extension/schema/sessions-settings.json
new file mode 100644
index 000000000000..a529c33749a5
--- /dev/null
+++ b/packages/apputils-extension/schema/sessions-settings.json
@@ -0,0 +1,25 @@
+{
+  "title": "Terminals and Kernels sessions settings",
+  "description": "Terminals and Kernels sessions settings",
+  "additionalProperties": false,
+  "properties": {
+    "showStatusBarItem": {
+      "type": "boolean",
+      "title": "Show the status bar item",
+      "description": "Whether to show the running terminals and kernels item in the status bar",
+      "default": true
+    },
+    "commsOverSubshells": {
+      "type": "string",
+      "title": "Kernel Comms over subshells",
+      "description": "Whether comm messages should be sent to kernel subshells, if the kernel supports it.",
+      "default": "perCommTarget",
+      "oneOf": [
+        { "const": "disabled", "title": "Disabled" },
+        { "const": "perComm", "title": "One subshell per comm" },
+        { "const": "perCommTarget", "title": "One subshell per comm-target" }
+      ]
+    }
+  },
+  "type": "object"
+}
diff --git a/packages/apputils-extension/src/index.ts b/packages/apputils-extension/src/index.ts
index 986c8b55132b..a95c581860c5 100644
--- a/packages/apputils-extension/src/index.ts
+++ b/packages/apputils-extension/src/index.ts
@@ -40,6 +40,7 @@ import { notificationPlugin } from './notificationplugin';
 import { Palette } from './palette';
 import { settingsConnector, settingsPlugin } from './settingsplugin';
 import { kernelStatus, runningSessionsStatus } from './statusbarplugin';
+import { subshellsSettings } from './subshell-settings';
 import { themesPaletteMenuPlugin, themesPlugin } from './themesplugins';
 import { toolbarRegistry } from './toolbarregistryplugin';
 import { workspacesPlugin } from './workspacesplugin';
@@ -741,6 +742,7 @@ const plugins: JupyterFrontEndPlugin<any>[] = [
   print,
   resolver,
   runningSessionsStatus,
+  subshellsSettings,
   sanitizer,
   settingsConnector,
   settingsPlugin,
diff --git a/packages/apputils-extension/src/statusbarplugin.ts b/packages/apputils-extension/src/statusbarplugin.ts
index f336b7070c23..4daa6996132f 100644
--- a/packages/apputils-extension/src/statusbarplugin.ts
+++ b/packages/apputils-extension/src/statusbarplugin.ts
@@ -199,10 +199,12 @@ export const runningSessionsStatus: JupyterFrontEndPlugin<void> = {
         }
       };
 
-      void settingRegistry.load(runningSessionsStatus.id).then(settings => {
-        onSettingsUpdated(settings);
-        settings.changed.connect(onSettingsUpdated);
-      });
+      void settingRegistry
+        .load('@jupyterlab/apputils-extension:sessions-settings')
+        .then(settings => {
+          onSettingsUpdated(settings);
+          settings.changed.connect(onSettingsUpdated);
+        });
     } else {
       registerItem();
     }
diff --git a/packages/apputils-extension/src/subshell-settings.ts b/packages/apputils-extension/src/subshell-settings.ts
new file mode 100644
index 000000000000..a88e2735a200
--- /dev/null
+++ b/packages/apputils-extension/src/subshell-settings.ts
@@ -0,0 +1,51 @@
+/* -----------------------------------------------------------------------------
+| Copyright (c) Jupyter Development Team.
+| Distributed under the terms of the Modified BSD License.
+|----------------------------------------------------------------------------*/
+
+import {
+  JupyterFrontEnd,
+  JupyterFrontEndPlugin
+} from '@jupyterlab/application';
+import { CommsOverSubshells } from '@jupyterlab/services';
+import { ISettingRegistry } from '@jupyterlab/settingregistry';
+
+/*
+ * A plugin loading kernel subshell settings.
+ */
+export const subshellsSettings: JupyterFrontEndPlugin<void> = {
+  id: '@jupyterlab/apputils-extension:subshell-settings',
+  description: 'Kernel subshell settings.',
+  autoStart: true,
+  requires: [],
+  optional: [ISettingRegistry],
+  activate: (
+    app: JupyterFrontEnd,
+    settingRegistry: ISettingRegistry | null
+  ) => {
+    if (settingRegistry) {
+      app.started
+        .then(async () => {
+          const subshellsSettings = await settingRegistry.load(
+            '@jupyterlab/apputils-extension:sessions-settings'
+          );
+
+          const commsOverSubshells = subshellsSettings.get('commsOverSubshells')
+            .composite as CommsOverSubshells;
+
+          app.serviceManager.kernels.commsOverSubshells = commsOverSubshells;
+
+          subshellsSettings.changed.connect(() => {
+            const commsOverSubshells = subshellsSettings.get(
+              'commsOverSubshells'
+            ).composite as CommsOverSubshells;
+            app.serviceManager.kernels.commsOverSubshells = commsOverSubshells;
+          });
+        })
+        .catch(reason => {
+          console.error('Fail to load settings for the subshells.');
+          console.error(reason);
+        });
+    }
+  }
+};
diff --git a/packages/services/src/kernel/comm.ts b/packages/services/src/kernel/comm.ts
index 4f0c0f2cc1cb..936a7cfcdd6e 100644
--- a/packages/services/src/kernel/comm.ts
+++ b/packages/services/src/kernel/comm.ts
@@ -1,7 +1,7 @@
 // Copyright (c) Jupyter Development Team.
 // Distributed under the terms of the Modified BSD License.
 
-import { JSONObject } from '@lumino/coreutils';
+import { JSONObject, PromiseDelegate } from '@lumino/coreutils';
 
 import { DisposableDelegate } from '@lumino/disposable';
 
@@ -9,6 +9,15 @@ import * as Kernel from './kernel';
 
 import * as KernelMessage from './messages';
 
+/**
+ * The Comm Over Subshell Enum
+ */
+export enum CommsOverSubshells {
+  Disabled = 'disabled',
+  PerComm = 'perComm',
+  PerCommTarget = 'perCommTarget'
+}
+
 /**
  * Comm channel handler.
  */
@@ -20,12 +29,23 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     target: string,
     id: string,
     kernel: Kernel.IKernelConnection,
-    disposeCb: () => void
+    disposeCb: () => void,
+    commsOverSubshells?: CommsOverSubshells
   ) {
     super(disposeCb);
     this._id = id;
     this._target = target;
     this._kernel = kernel;
+
+    // Clean subshell ids upon restart
+    this._kernel.statusChanged.connect(() => {
+      if (this._kernel.status === 'restarting') {
+        this._cleanSubshells();
+      }
+    });
+
+    this.commsOverSubshells =
+      commsOverSubshells ?? CommsOverSubshells.PerCommTarget;
   }
 
   /**
@@ -42,6 +62,39 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return this._target;
   }
 
+  /**
+   * The current subshell id.
+   */
+  get subshellId(): string | null {
+    return this._subshellId;
+  }
+
+  /**
+   * Promise that resolves when the subshell started, if any
+   */
+  get subshellStarted(): Promise<void> {
+    return this._subshellStarted.promise;
+  }
+
+  /**
+   * Whether comms are running on a subshell, or not
+   */
+  get commsOverSubshells(): CommsOverSubshells {
+    return this._commsOverSubshells;
+  }
+
+  /**
+   * Set whether comms are running on subshells, or not
+   */
+  set commsOverSubshells(value: CommsOverSubshells) {
+    this._commsOverSubshells = value;
+    if (this._commsOverSubshells === CommsOverSubshells.Disabled) {
+      this._maybeCloseSubshell();
+    } else {
+      void this._maybeStartSubshell();
+    }
+  }
+
   /**
    * Get the callback for a comm close event.
    *
@@ -112,7 +165,7 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
       channel: 'shell',
       username: this._kernel.username,
       session: this._kernel.clientId,
-      subshellId: this._kernel.subshellId,
+      subshellId: this._subshellId || this._kernel.subshellId,
       content: {
         comm_id: this._id,
         target_name: this._target,
@@ -146,7 +199,7 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
       channel: 'shell',
       username: this._kernel.username,
       session: this._kernel.clientId,
-      subshellId: this._kernel.subshellId,
+      subshellId: this._subshellId || this._kernel.subshellId,
       content: {
         comm_id: this._id,
         data: data
@@ -181,7 +234,7 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
       channel: 'shell',
       username: this._kernel.username,
       session: this._kernel.clientId,
-      subshellId: this._kernel.subshellId,
+      subshellId: this._subshellId || this._kernel.subshellId,
       content: {
         comm_id: this._id,
         data: data ?? {}
@@ -197,6 +250,7 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
         channel: 'iopub',
         username: this._kernel.username,
         session: this._kernel.clientId,
+        subshellId: this._subshellId || this._kernel.subshellId,
         content: {
           comm_id: this._id,
           data: data ?? {}
@@ -212,6 +266,71 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return future;
   }
 
+  dispose(): void {
+    this._maybeCloseSubshell();
+
+    super.dispose();
+  }
+
+  private _cleanSubshells() {
+    CommHandler._commTargetSubShellsId = {};
+  }
+
+  private async _maybeStartSubshell() {
+    await this._kernel.info;
+    if (!this._kernel.supportsSubshells) {
+      return;
+    }
+
+    if (this._commsOverSubshells === CommsOverSubshells.PerComm) {
+      // Create subshell
+      const replyMsg = await this._kernel.requestCreateSubshell({}).done;
+      this._subshellId = replyMsg.content.subshell_id;
+      this._subshellStarted.resolve();
+      return;
+    }
+
+    // One shell per comm-target
+    if (CommHandler._commTargetSubShellsId[this._target]) {
+      this._subshellId = await CommHandler._commTargetSubShellsId[this._target];
+      this._subshellStarted.resolve();
+    } else {
+      // Create subshell
+      CommHandler._commTargetSubShellsId[this._target] = this._kernel
+        .requestCreateSubshell({})
+        .done.then(replyMsg => {
+          this._subshellId = replyMsg.content.subshell_id;
+          return this._subshellId;
+        });
+      await CommHandler._commTargetSubShellsId[this._target];
+      this._subshellStarted.resolve();
+    }
+  }
+
+  private _maybeCloseSubshell() {
+    // Only close subshell if we have one subshell per comm
+    if (this._commsOverSubshells !== CommsOverSubshells.PerComm) {
+      this._subshellId = null;
+      return;
+    }
+
+    if (this._subshellId && this._kernel.status !== 'dead') {
+      this._kernel.requestDeleteSubshell(
+        { subshell_id: this._subshellId },
+        true
+      );
+    }
+    this._subshellId = null;
+  }
+
+  private _subshellStarted = new PromiseDelegate<void>();
+  private static _commTargetSubShellsId: {
+    [targetName: string]: Promise<string> | null;
+  } = {};
+
+  private _commsOverSubshells: CommsOverSubshells;
+  private _subshellId: string | null = null;
+
   private _target = '';
   private _id = '';
   private _kernel: Kernel.IKernelConnection;
diff --git a/packages/services/src/kernel/default.ts b/packages/services/src/kernel/default.ts
index fcefa9031d89..1889342e3895 100644
--- a/packages/services/src/kernel/default.ts
+++ b/packages/services/src/kernel/default.ts
@@ -7,7 +7,7 @@ import { JSONExt, JSONObject, PromiseDelegate, UUID } from '@lumino/coreutils';
 
 import { ISignal, Signal } from '@lumino/signaling';
 
-import { ServerConnection } from '..';
+import { CommsOverSubshells, ServerConnection } from '..';
 
 import { CommHandler } from './comm';
 
@@ -60,6 +60,8 @@ export class KernelConnection implements Kernel.IKernelConnection {
     this._clientId = options.clientId ?? UUID.uuid4();
     this._username = options.username ?? '';
     this.handleComms = options.handleComms ?? true;
+    this._commsOverSubshells =
+      options.commsOverSubshells ?? CommsOverSubshells.PerCommTarget;
     this._subshellId = options.subshellId ?? null;
 
     this._createSocket();
@@ -86,6 +88,31 @@ export class KernelConnection implements Kernel.IKernelConnection {
    */
   readonly handleComms: boolean;
 
+  /**
+   * Whether comm messages should be sent to kernel subshells, if the
+   * kernel supports it.
+   *
+   * #### Notes
+   * Sending comm messages over subshells allows processing comms whilst
+   * processing execute-request on the "main shell". This prevents blocking
+   * comm processing.
+   * Options are:
+   * - disabled: not using subshells
+   * - one subshell per comm-target (default)
+   * - one subshell per comm (can lead to issues if creating many comms)
+   */
+  get commsOverSubshells(): CommsOverSubshells {
+    return this._commsOverSubshells;
+  }
+
+  set commsOverSubshells(value: CommsOverSubshells) {
+    this._commsOverSubshells = value;
+
+    for (const [_, comm] of this._comms) {
+      comm.commsOverSubshells = value;
+    }
+  }
+
   /**
    * A signal emitted when the kernel status changes.
    */
@@ -268,6 +295,7 @@ export class KernelConnection implements Kernel.IKernelConnection {
       // handleComms defaults to false since that is safer
       handleComms: false,
       kernelAPIClient: this._kernelAPIClient,
+      commsOverSubshells: CommsOverSubshells.Disabled,
       ...options
     });
   }
@@ -1012,9 +1040,15 @@ export class KernelConnection implements Kernel.IKernelConnection {
       throw new Error('Comm is already created');
     }
 
-    const comm = new CommHandler(targetName, commId, this, () => {
-      this._unregisterComm(commId);
-    });
+    const comm = new CommHandler(
+      targetName,
+      commId,
+      this,
+      () => {
+        this._unregisterComm(commId);
+      },
+      this._commsOverSubshells
+    );
     this._comms.set(commId, comm);
     return comm;
   }
@@ -1282,7 +1316,7 @@ export class KernelConnection implements Kernel.IKernelConnection {
         KernelMessage.IShellControlMessage
       >
     >();
-    this._comms = new Map<string, Kernel.IComm>();
+    this._comms = new Map<string, CommHandler>();
     this._displayIdToParentIds.clear();
     this._msgIdToDisplayIds.clear();
   }
@@ -1321,7 +1355,8 @@ export class KernelConnection implements Kernel.IKernelConnection {
       this,
       () => {
         this._unregisterComm(content.comm_id);
-      }
+      },
+      this.commsOverSubshells
     );
     this._comms.set(content.comm_id, comm);
 
@@ -1802,6 +1837,9 @@ export class KernelConnection implements Kernel.IKernelConnection {
   );
   private _selectedProtocol: string = '';
 
+  private _commsOverSubshells: CommsOverSubshells =
+    CommsOverSubshells.PerCommTarget;
+
   private _futures = new Map<
     string,
     KernelFutureHandler<
@@ -1809,7 +1847,7 @@ export class KernelConnection implements Kernel.IKernelConnection {
       KernelMessage.IShellControlMessage
     >
   >();
-  private _comms = new Map<string, Kernel.IComm>();
+  private _comms = new Map<string, CommHandler>();
   private _targetRegistry: {
     [key: string]: (
       comm: Kernel.IComm,
diff --git a/packages/services/src/kernel/index.ts b/packages/services/src/kernel/index.ts
index bce73e9076d4..06ee22e2ac46 100644
--- a/packages/services/src/kernel/index.ts
+++ b/packages/services/src/kernel/index.ts
@@ -2,10 +2,17 @@
 // Distributed under the terms of the Modified BSD License.
 
 // Namespace some of our modules for convenience and backwards compatibility.
+import { CommsOverSubshells } from './comm';
 import * as Kernel from './kernel';
 import * as KernelMessage from './messages';
 import * as KernelAPI from './restapi';
 import { KernelConnection } from './default';
 
 export * from './manager';
-export { Kernel, KernelMessage, KernelAPI, KernelConnection };
+export {
+  Kernel,
+  KernelMessage,
+  KernelAPI,
+  KernelConnection,
+  CommsOverSubshells
+};
diff --git a/packages/services/src/kernel/kernel.ts b/packages/services/src/kernel/kernel.ts
index 9a86ffac069b..8dea7ce072c5 100644
--- a/packages/services/src/kernel/kernel.ts
+++ b/packages/services/src/kernel/kernel.ts
@@ -7,7 +7,7 @@ import { IDisposable, IObservableDisposable } from '@lumino/disposable';
 
 import { ISignal } from '@lumino/signaling';
 
-import { ServerConnection } from '..';
+import { CommsOverSubshells, ServerConnection } from '..';
 
 import * as KernelMessage from './messages';
 
@@ -111,6 +111,21 @@ export interface IKernelConnection extends IObservableDisposable {
    */
   handleComms: boolean;
 
+  /**
+   * Whether comm messages should be sent to kernel subshells, if the
+   * kernel supports it.
+   *
+   * #### Notes
+   * Sending comm messages over subshells allows processing comms whilst
+   * processing execute-request on the "main shell". This prevents blocking
+   * comm processing.
+   * Options are:
+   * - disabled: not using subshells
+   * - one subshell per comm-target (default)
+   * - one subshell per comm (can lead to issues if creating many comms)
+   */
+  commsOverSubshells?: CommsOverSubshells;
+
   /**
    * Whether the kernel connection has pending input.
    *
@@ -654,6 +669,21 @@ export namespace IKernelConnection {
      */
     handleComms?: boolean;
 
+    /**
+     * Whether comm messages should be sent to kernel subshells, if the
+     * kernel supports it.
+     *
+     * #### Notes
+     * Sending comm messages over subshells allows processing comms whilst
+     * processing execute-request on the "main shell". This prevents blocking
+     * comm processing.
+     * Options are:
+     * - disabled: not using subshells
+     * - one subshell per comm-target (default)
+     * - one subshell per comm (can lead to issues if creating many comms)
+     */
+    commsOverSubshells?: CommsOverSubshells;
+
     /**
      * The unique identifier for the kernel client.
      */
@@ -707,6 +737,21 @@ export interface IManager extends IBaseManager {
    */
   readonly runningCount: number;
 
+  /**
+   * Whether comm messages should be sent to kernel subshells, if the
+   * kernel supports it.
+   *
+   * #### Notes
+   * Sending comm messages over subshells allows processing comms whilst
+   * processing execute-request on the "main shell". This prevents blocking
+   * comm processing.
+   * Options are:
+   * - disabled: not using subshells
+   * - one subshell per comm-target (default)
+   * - one subshell per comm (can lead to issues if creating many comms)
+   */
+  commsOverSubshells?: CommsOverSubshells;
+
   /**
    * Force a refresh of the running kernels.
    *
@@ -896,6 +941,11 @@ export interface IComm extends IDisposable {
    */
   readonly targetName: string;
 
+  /**
+   * Whether comms are running on subshell or not.
+   */
+  commsOverSubshells?: CommsOverSubshells;
+
   /**
    * Callback for a comm close event.
    *
diff --git a/packages/services/src/kernel/manager.ts b/packages/services/src/kernel/manager.ts
index a71a71757ff9..591b1d14e44a 100644
--- a/packages/services/src/kernel/manager.ts
+++ b/packages/services/src/kernel/manager.ts
@@ -3,7 +3,7 @@
 
 import { Poll } from '@lumino/polling';
 import { ISignal, Signal } from '@lumino/signaling';
-import { KernelSpec, ServerConnection } from '..';
+import { CommsOverSubshells, KernelSpec, ServerConnection } from '..';
 import * as Kernel from './kernel';
 import { BaseManager } from '../basemanager';
 import { IKernelOptions, KernelAPIClient } from './restapi';
@@ -123,6 +123,9 @@ export class KernelManager extends BaseManager implements Kernel.IManager {
         }
       }
     }
+
+    options.commsOverSubshells = this._commsOverSubshells;
+
     const kernelConnection = new KernelConnection({
       handleComms,
       ...options,
@@ -157,6 +160,21 @@ export class KernelManager extends BaseManager implements Kernel.IManager {
     return this._models.size;
   }
 
+  /**
+   * Whether comms are running on subshell or not.
+   */
+  get commsOverSubshells(): CommsOverSubshells {
+    return this._commsOverSubshells;
+  }
+
+  set commsOverSubshells(value: CommsOverSubshells) {
+    this._commsOverSubshells = value;
+
+    for (const connection of this._kernelConnections) {
+      connection.commsOverSubshells = value;
+    }
+  }
+
   /**
    * Force a refresh of the running kernels.
    *
@@ -337,6 +355,8 @@ export class KernelManager extends BaseManager implements Kernel.IManager {
     }
   }
 
+  private _commsOverSubshells: CommsOverSubshells =
+    CommsOverSubshells.PerCommTarget;
   private _isReady = false;
   private _ready: Promise<void>;
   private _kernelConnections = new Set<KernelConnection>();
diff --git a/packages/services/test/kernel/comm.spec.ts b/packages/services/test/kernel/comm.spec.ts
index 424f81f2af7b..1c6bcd7ec92e 100644
--- a/packages/services/test/kernel/comm.spec.ts
+++ b/packages/services/test/kernel/comm.spec.ts
@@ -3,7 +3,13 @@
 
 import { isFulfilled, JupyterServer } from '@jupyterlab/testing';
 import { PromiseDelegate } from '@lumino/coreutils';
-import { Kernel, KernelManager, KernelMessage } from '../../src';
+import {
+  CommsOverSubshells,
+  Kernel,
+  KernelManager,
+  KernelMessage
+} from '../../src';
+import { CommHandler } from '../../src/kernel/comm';
 
 const BLIP = `
 from ipykernel.comm import Comm
@@ -42,6 +48,7 @@ describe('jupyter.services - Comm', () => {
   let server: JupyterServer;
   let kernelManager: KernelManager;
   let kernel: Kernel.IKernelConnection;
+  let echoKernel: Kernel.IKernelConnection;
 
   jest.retryTimes(3);
 
@@ -50,6 +57,7 @@ describe('jupyter.services - Comm', () => {
     await server.start();
     kernelManager = new KernelManager();
     kernel = await kernelManager.startNew({ name: 'ipython' });
+    echoKernel = await kernelManager.startNew();
   }, 30000);
 
   afterEach(() => {
@@ -72,6 +80,83 @@ describe('jupyter.services - Comm', () => {
         expect(typeof comm.commId).toBe('string');
       });
 
+      it('should not spawn a subshell by default', () => {
+        const comm = echoKernel.createComm('test') as CommHandler;
+        expect(comm.subshellId).toBeNull();
+      });
+
+      it('should spawn a subshell per-comm', async () => {
+        await echoKernel.info;
+
+        expect(echoKernel.supportsSubshells).toBeTruthy();
+
+        echoKernel.commsOverSubshells = CommsOverSubshells.PerComm;
+
+        const comm = echoKernel.createComm('testTarget') as CommHandler;
+        const comm2 = echoKernel.createComm('testTarget') as CommHandler;
+
+        expect(echoKernel.commsOverSubshells).toEqual(
+          CommsOverSubshells.PerComm
+        );
+        expect(comm.commsOverSubshells).toEqual(CommsOverSubshells.PerComm);
+
+        await comm.subshellStarted;
+        await comm2.subshellStarted;
+
+        expect(comm.subshellId).not.toBeNull();
+        expect(comm2.subshellId).not.toBeNull();
+
+        expect(comm.subshellId).not.toEqual(comm2.subshellId);
+      });
+
+      it('should dispose the subshell per-comm', async () => {
+        await echoKernel.info;
+
+        expect(echoKernel.supportsSubshells).toBeTruthy();
+
+        echoKernel.commsOverSubshells = CommsOverSubshells.PerComm;
+
+        const comm = echoKernel.createComm('testTarget') as CommHandler;
+        await comm.subshellStarted;
+        expect(comm.subshellId).not.toBeNull();
+
+        const replyMsg = await echoKernel.requestListSubshell({}).done;
+        const before = replyMsg.content.subshell_id.length;
+
+        comm.dispose();
+
+        const replyMsg2 = await echoKernel.requestListSubshell({}).done;
+        const after = replyMsg2.content.subshell_id.length;
+
+        // We should have less subshells after the comm disposal than before
+        expect(after).toBeLessThan(before);
+      });
+
+      it('should spawn a subshell per-comm-target', async () => {
+        await echoKernel.info;
+
+        expect(echoKernel.supportsSubshells).toBeTruthy();
+
+        echoKernel.commsOverSubshells = CommsOverSubshells.PerCommTarget;
+
+        const comm = echoKernel.createComm('testTarget') as CommHandler;
+        await comm.subshellStarted;
+        expect(comm.subshellId).not.toBeNull();
+
+        const replyMsg = await echoKernel.requestListSubshell({}).done;
+        const before = replyMsg.content.subshell_id.length;
+
+        const comm2 = echoKernel.createComm('testTarget') as CommHandler;
+        await comm2.subshellStarted;
+        expect(comm2.subshellId).not.toBeNull();
+
+        const replyMsg2 = await echoKernel.requestListSubshell({}).done;
+        const after = replyMsg2.content.subshell_id.length;
+
+        // We shouldn't have more subshells before and after the second comm creation
+        expect(before).toEqual(after);
+      });
+
       it('should use the given id', () => {
         const comm = kernel.createComm('test', '1234');
         expect(comm.targetName).toBe('test');


Reviews:
user: login: ianthomas23
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T13:11:49Z
user: login: github-advanced-security[bot]
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-05T10:46:54Z
user: login: krassowski
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T17:08:34Z
user: login: krassowski
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T17:11:42Z
user: login: martinRenou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T08:09:24Z
user: login: martinRenou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T08:25:59Z
user: login: martinRenou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T08:43:56Z
user: login: krassowski
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T10:39:30Z
user: login: krassowski
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T17:59:37Z
user: login: ianthomas23
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T13:43:49Z
user: login: ianthomas23
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T13:47:02Z
user: login: fleming79
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T22:05:38Z
user: login: ianthomas23
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T07:55:18Z
user: login: martinRenou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T15:51:22Z
user: login: martinRenou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T15:53:33Z
user: login: fleming79
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T00:34:06Z
user: login: martinRenou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-14T08:05:10Z
user: login: ianthomas23
body: The use of subshells looks good to me. Approving with just a nit about some comments.

Thanks @martinRenou!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T09:47:56Z
user: login: krassowski
body: Can you add a paragraph about user implications in user-facing changelog under existing subshells section?
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-14T10:03:23Z
user: login: krassowski
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T15:18:01Z
user: login: krassowski
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T09:19:24Z
user: login: krassowski
body: I wonder if we could make it less disrupting for any downstreams which re-implement interfaces by making the additions optional.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T10:46:53Z
user: login: krassowski
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T11:39:23Z
user: login: martinRenou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T11:47:26Z
user: login: krassowski
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T12:30:19Z
user: login: martinRenou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T12:35:00Z
user: login: krassowski
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T12:55:21Z
user: login: martinRenou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T13:52:54Z
user: login: martinRenou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T13:55:28Z
user: login: krassowski
body: Thank you @martinRenou!

Also thank you @ianthomas23 and @fleming79 for reviews!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-18T15:52:38Z
user: login: jtpio
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T17:23:26Z

Review comments:
diff_hunk: @@ -212,6 +214,31 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return future;
   }
 
+  dispose(): void {
+    super.dispose();
+
+    this._maybeCloseSubshell();
+  }
+
+  private async _maybeStartSubshell() {
+    if (this._kernel.supportsSubshells) {
path: packages/services/src/kernel/comm.ts
user: login: ianthomas23
body: There should be an `await this._kernel.info` before this, as we can only tell if a kernel supports subshells after we've received the kernel info which contains the appropriate information.
created_at: 2025-03-04T13:11:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,6 +80,29 @@
    */
   readonly handleComms: boolean;
 
+  /**
+   * Whether comm messages should be sent to kernel subshells, if the
+   * kernel supports it.
+   *
+   * #### Notes
+   * Sending comm messages over subshells allows processing comms whilst
+   * processing execute-request on the "main shell". This prevents blocking
+   * comm processing.
+   * If enabled, we'll create one subshell per-comm, this may lead to issue
+   * if many comms are open, so it's disabled by default.
+   */
+  get commsOverSubshells(): boolean {
+    return this._commsOverSubshells;
+  };
+
+  set commsOverSubshells(value: boolean) {
+    this._commsOverSubshells = value;
+
+    for (const [_, comm] of this._comms) {
+      const handler = comm as CommHandler;
path: packages/services/src/kernel/default.ts
user: login: github-advanced-security[bot]
body: ## Unused variable, import, function or class

Unused variable handler.

[Show more details](https://github.com/jupyterlab/jupyterlab/security/code-scanning/1075)
created_at: 2025-03-05T10:46:54Z
author_association: NONE
diff_hunk: @@ -0,0 +1,13 @@
+{
+  "title": "Kernel Subshells",
path: packages/application-extension/schema/kernel-subshells.json
user: login: krassowski
body: I think we need to consolidate kernel settings in a single place. There is also "Kernel dialogs" which only has one setting.
created_at: 2025-03-05T17:08:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,13 @@
+{
+  "title": "Kernel Subshells",
path: packages/application-extension/schema/kernel-subshells.json
user: login: krassowski
body: Also "Running Terminals and Kernels" with just one setting.
created_at: 2025-03-05T17:11:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,13 @@
+{
+  "title": "Kernel Subshells",
path: packages/application-extension/schema/kernel-subshells.json
user: login: martinRenou
body: Should I merge them all in some "Terminals and Kernels" settings?
created_at: 2025-03-06T08:09:24Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,13 @@
+{
+  "title": "Kernel Subshells",
path: packages/application-extension/schema/kernel-subshells.json
user: login: martinRenou
body: That would be a breaking change, wouldn't it be? Since user settings would stop working.
created_at: 2025-03-06T08:25:59Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,13 @@
+{
+  "title": "Kernel Subshells",
path: packages/application-extension/schema/kernel-subshells.json
user: login: martinRenou
body: I've moved the subshells settings closer to the existing kernel settings
created_at: 2025-03-06T08:43:56Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,13 @@
+{
+  "title": "Kernel Subshells",
path: packages/application-extension/schema/kernel-subshells.json
user: login: krassowski
body: > That would be a breaking change, wouldn't it be? Since user settings would stop working.

No, adding settings to existing sections is fine. Also, that particular one was not even released. I think this should block release but not necessarily this PR.
created_at: 2025-03-06T10:39:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,117 @@
+/* -----------------------------------------------------------------------------
+| Copyright (c) Jupyter Development Team.
+| Distributed under the terms of the Modified BSD License.
+|----------------------------------------------------------------------------*/
+
+import {
+  JupyterFrontEnd,
+  JupyterFrontEndPlugin
+} from '@jupyterlab/application';
+import { RunningSessions } from '@jupyterlab/apputils';
+import { CommsOverSubshells } from '@jupyterlab/services';
+import { ISettingRegistry } from '@jupyterlab/settingregistry';
+import { IStatusBar } from '@jupyterlab/statusbar';
+import { ITranslator } from '@jupyterlab/translation';
+import { IDisposable } from '@lumino/disposable';
+import { KeyboardEvent } from 'react';
+
+/*
+ * A plugin providing running terminals and sessions settings
+ */
+export const sessionsSettings: JupyterFrontEndPlugin<void> = {
+  id: '@jupyterlab/apputils-extension:sessions-settings',
+  description: 'Sessions and terminals settings.',
+  autoStart: true,
+  requires: [IStatusBar, ITranslator],
+  optional: [ISettingRegistry],
+  activate: (
+    app: JupyterFrontEnd,
+    statusBar: IStatusBar,
+    translator: ITranslator,
+    settingRegistry: ISettingRegistry | null
+  ) => {
+    const createStatusItem = () => {
+      const item = new RunningSessions({
+        onClick: () => app.shell.activateById('jp-running-sessions'),
path: packages/apputils-extension/src/sessions-settings.ts
user: login: krassowski
body: You do not need to move the code for the `@jupyterlab/apputils-extension:running-sessions-status` plugin; it can be left where it was; the only part that needs to be moved is what settings ID it is reading from.
created_at: 2025-03-11T17:59:37Z
author_association: MEMBER
diff_hunk: @@ -212,6 +257,61 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return future;
   }
 
+  dispose(): void {
+    this._maybeCloseSubshell();
+
+    super.dispose();
+  }
+
+  private async _maybeStartSubshell() {
+    await this._kernel.info;
+    if (!this._kernel.supportsSubshells) {
+      return;
+    }
+
+    if (this._commsOverSubshells === CommsOverSubshells.PerComm) {
+      // Create subshell
+      const replyMsg = await this._kernel.requestCreateSubshell({}).done;
+      this._subshellId = replyMsg.content.subshell_id;
+      this._subshellStarted.resolve();
+      return;
+    }
+
+    // One shell per comm-target
path: packages/services/src/kernel/comm.ts
user: login: ianthomas23
body: I'm carefully testing this locally, and if you create multiple widgets in the same notebook cell using `PerCommTarget` you get `PerComm` behaviour. This is, I believe, because we get multiple concurrent `_maybeStartSubshell` calls and we can't update the `_commTargetSubShellsId` until we have received the first subshell ID, by which time we have requested all the others.
created_at: 2025-03-12T13:43:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -212,6 +257,61 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return future;
   }
 
+  dispose(): void {
+    this._maybeCloseSubshell();
+
+    super.dispose();
+  }
+
+  private async _maybeStartSubshell() {
+    await this._kernel.info;
+    if (!this._kernel.supportsSubshells) {
+      return;
+    }
+
+    if (this._commsOverSubshells === CommsOverSubshells.PerComm) {
+      // Create subshell
+      const replyMsg = await this._kernel.requestCreateSubshell({}).done;
+      this._subshellId = replyMsg.content.subshell_id;
+      this._subshellStarted.resolve();
+      return;
+    }
+
+    // One shell per comm-target
path: packages/services/src/kernel/comm.ts
user: login: ianthomas23
body: I suppose we could store some sort of `pendingRequests` information and not repeat a request if it is already pending but instead wait for that request to be resolved.
created_at: 2025-03-12T13:47:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -212,6 +257,61 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return future;
   }
 
+  dispose(): void {
+    this._maybeCloseSubshell();
+
+    super.dispose();
+  }
+
+  private async _maybeStartSubshell() {
+    await this._kernel.info;
+    if (!this._kernel.supportsSubshells) {
+      return;
+    }
+
+    if (this._commsOverSubshells === CommsOverSubshells.PerComm) {
+      // Create subshell
+      const replyMsg = await this._kernel.requestCreateSubshell({}).done;
+      this._subshellId = replyMsg.content.subshell_id;
+      this._subshellStarted.resolve();
+      return;
+    }
+
+    // One shell per comm-target
path: packages/services/src/kernel/comm.ts
user: login: fleming79
body: I'm very excited to see this new feature emerging!

I gained some experience related to this topic by writing a draft [PR](https://github.com/jupyter-widgets/ipywidgets/pull/3922) for ipywidgets. The motivation was around enabling the usage of Ipywidgets outside of Notebooks and Consoles for [a modified version of Ipylab ](https://github.com/fleming79/ipylab). 

One issue I noticed in writing that PR was related to how comm messages are managed by `KernelConnection`:

*Extract from KernelConnection in Jupyterlab*
```
  /**
   * Handle comm messages
   *
   * #### Notes
   * The comm message protocol currently has implicit assumptions that only
   * one kernel connection is handling comm messages. This option allows a
   * kernel connection to opt out of handling comms.
   *
   * See https://github.com/jupyter/jupyter_client/issues/263
   */
  readonly handleComms: boolean;
```

Although it says it is an 'opt out' option, what it really means is that the life of the comms is tied to the `KernelConnection` of whichever connection is made first, generally a notebook or console session.  In the PR mentioned above, comms is re-establishing when the original `SessionConnection` is closed.

I think you can keep this new feature simple but powerful,  I offer the following suggestions:
1. Modify KernelConnction so comms is de-coupled from `KernelConnection` (but still respect `handleComms===false`). The life of comms is tied to the life of the `Kernel` rather than the `KernelConnection`.
2.  Provide a separate subshell per-comm-target per-kernel (rather than per `KernelConnection`). Both the comm-target and kernel ID can be known in advance to ensure the correct number of subshells are created.
3. Drop: *one subshell per comm*.


created_at: 2025-03-12T22:05:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -212,6 +257,61 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return future;
   }
 
+  dispose(): void {
+    this._maybeCloseSubshell();
+
+    super.dispose();
+  }
+
+  private async _maybeStartSubshell() {
+    await this._kernel.info;
+    if (!this._kernel.supportsSubshells) {
+      return;
+    }
+
+    if (this._commsOverSubshells === CommsOverSubshells.PerComm) {
+      // Create subshell
+      const replyMsg = await this._kernel.requestCreateSubshell({}).done;
+      this._subshellId = replyMsg.content.subshell_id;
+      this._subshellStarted.resolve();
+      return;
+    }
+
+    // One shell per comm-target
path: packages/services/src/kernel/comm.ts
user: login: ianthomas23
body: > I suppose we could store some sort of `pendingRequests` information and not repeat a request if it is already pending but instead wait for that request to be resolved.

On second thoughts we could just change `_commTargetSubShellsId` to store `Promise`s of subshell IDs, and create a `Promise` and add it to the collection before the first request subshell call, resolving on its reply. Then other uses of the same target could just await the Promise.
created_at: 2025-03-13T07:55:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -212,6 +257,61 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return future;
   }
 
+  dispose(): void {
+    this._maybeCloseSubshell();
+
+    super.dispose();
+  }
+
+  private async _maybeStartSubshell() {
+    await this._kernel.info;
+    if (!this._kernel.supportsSubshells) {
+      return;
+    }
+
+    if (this._commsOverSubshells === CommsOverSubshells.PerComm) {
+      // Create subshell
+      const replyMsg = await this._kernel.requestCreateSubshell({}).done;
+      this._subshellId = replyMsg.content.subshell_id;
+      this._subshellStarted.resolve();
+      return;
+    }
+
+    // One shell per comm-target
path: packages/services/src/kernel/comm.ts
user: login: martinRenou
body: Thanks for your comment @fleming79 !

> Drop: one subshell per comm

Other opinions on this? I could see the benefit of allowing one subshell per-comm, even though it may get out of hand quickly.

> Provide a separate subshell per-comm-target per-kernel (rather than per KernelConnection). Both the comm-target and kernel ID can be known in advance to ensure the correct number of subshells are created.

Since the subshell ID per-target is kept as a static property of `CommHandler` now, that shouldnt be a problem I guess?
created_at: 2025-03-13T15:51:21Z
author_association: MEMBER
diff_hunk: @@ -212,6 +257,61 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return future;
   }
 
+  dispose(): void {
+    this._maybeCloseSubshell();
+
+    super.dispose();
+  }
+
+  private async _maybeStartSubshell() {
+    await this._kernel.info;
+    if (!this._kernel.supportsSubshells) {
+      return;
+    }
+
+    if (this._commsOverSubshells === CommsOverSubshells.PerComm) {
+      // Create subshell
+      const replyMsg = await this._kernel.requestCreateSubshell({}).done;
+      this._subshellId = replyMsg.content.subshell_id;
+      this._subshellStarted.resolve();
+      return;
+    }
+
+    // One shell per comm-target
path: packages/services/src/kernel/comm.ts
user: login: martinRenou
body: > create a Promise and add it to the collection before the first request subshell call

Good catch @ianthomas23  I've updated the PR as you suggested
created_at: 2025-03-13T15:53:33Z
author_association: MEMBER
diff_hunk: @@ -212,6 +257,61 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return future;
   }
 
+  dispose(): void {
+    this._maybeCloseSubshell();
+
+    super.dispose();
+  }
+
+  private async _maybeStartSubshell() {
+    await this._kernel.info;
+    if (!this._kernel.supportsSubshells) {
+      return;
+    }
+
+    if (this._commsOverSubshells === CommsOverSubshells.PerComm) {
+      // Create subshell
+      const replyMsg = await this._kernel.requestCreateSubshell({}).done;
+      this._subshellId = replyMsg.content.subshell_id;
+      this._subshellStarted.resolve();
+      return;
+    }
+
+    // One shell per comm-target
path: packages/services/src/kernel/comm.ts
user: login: fleming79
body: In ipywidgets both `layout` and `style` are also widgets. Meaning possibly 3 subshells for a single widget.
created_at: 2025-03-14T00:34:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -212,6 +257,61 @@ export class CommHandler extends DisposableDelegate implements Kernel.IComm {
     return future;
   }
 
+  dispose(): void {
+    this._maybeCloseSubshell();
+
+    super.dispose();
+  }
+
+  private async _maybeStartSubshell() {
+    await this._kernel.info;
+    if (!this._kernel.supportsSubshells) {
+      return;
+    }
+
+    if (this._commsOverSubshells === CommsOverSubshells.PerComm) {
+      // Create subshell
+      const replyMsg = await this._kernel.requestCreateSubshell({}).done;
+      this._subshellId = replyMsg.content.subshell_id;
+      this._subshellStarted.resolve();
+      return;
+    }
+
+    // One shell per comm-target
path: packages/services/src/kernel/comm.ts
user: login: martinRenou
body: Yep 👍🏽 Meaning one should only use one subshell per-comm when using small number of widgets. From our testing it seems less than a hundred of widgets works well.
created_at: 2025-03-14T08:05:10Z
author_association: MEMBER
diff_hunk: @@ -82,6 +84,29 @@ export class KernelConnection implements Kernel.IKernelConnection {
    */
   readonly handleComms: boolean;
 
+  /**
+   * Whether comm messages should be sent to kernel subshells, if the
+   * kernel supports it.
+   *
+   * #### Notes
+   * Sending comm messages over subshells allows processing comms whilst
+   * processing execute-request on the "main shell". This prevents blocking
+   * comm processing.
+   * If enabled, we'll create one subshell per-comm, this may lead to issue
path: packages/services/src/kernel/default.ts
user: login: ianthomas23
body: This text can be updated to also mention one subshell per comm target, and that that is the default.
created_at: 2025-03-14T09:45:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,19 @@ export interface IKernelConnection extends IObservableDisposable {
    */
   handleComms: boolean;
 
+  /**
+   * Whether comm messages should be sent to kernel subshells, if the
+   * kernel supports it.
+   *
+   * #### Notes
+   * Sending comm messages over subshells allows processing comms whilst
+   * processing execute-request on the "main shell". This prevents blocking
+   * comm processing.
+   * If enabled, we'll create one subshell per-comm, this may lead to issue
path: packages/services/src/kernel/kernel.ts
user: login: ianthomas23
body: Same here.
created_at: 2025-03-14T09:45:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -646,6 +659,19 @@ export namespace IKernelConnection {
      */
     handleComms?: boolean;
 
+    /**
+     * Whether comm messages should be sent to kernel subshells, if the
+     * kernel supports it.
+     *
+     * #### Notes
+     * Sending comm messages over subshells allows processing comms whilst
+     * processing execute-request on the "main shell". This prevents blocking
+     * comm processing.
+     * If enabled, we'll create one subshell per-comm, this may lead to issue
path: packages/services/src/kernel/kernel.ts
user: login: ianthomas23
body: Same here.
created_at: 2025-03-14T09:45:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -699,6 +725,19 @@ export interface IManager extends IBaseManager {
    */
   readonly runningCount: number;
 
+  /**
+   * Whether comm messages should be sent to kernel subshells, if the
+   * kernel supports it.
+   *
+   * #### Notes
+   * Sending comm messages over subshells allows processing comms whilst
+   * processing execute-request on the "main shell". This prevents blocking
+   * comm processing.
+   * If enabled, we'll create one subshell per-comm, this may lead to issue
path: packages/services/src/kernel/kernel.ts
user: login: ianthomas23
body: Same here.
created_at: 2025-03-14T09:46:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,25 @@
+{
+  "title": "Terminals and Kernels sessions settings",
+  "description": "Terminals and Kernels sessions settings",
+  "additionalProperties": false,
+  "properties": {
+    "showStatusBarItem": {
+      "type": "boolean",
+      "title": "Show the status bar item",
+      "description": "Whether to show the running terminals and kernels item in the status bar",
+      "default": true
+    },
path: packages/apputils-extension/schema/sessions-settings.json
user: login: krassowski
body: Maybe this should be just "Kernel settings" and later we can also fold this orphan in there:

![image](https://github.com/user-attachments/assets/5f7ae75d-6639-45e3-899e-d2763d1b7d61)


My reasoning is:
- the terminals are not shown in status bar unless any is open
- we can rework the behaviour of `showStatusBarItem` to only affect kernels
- we could add another `showStatusBarItem` for terminal specifically, as we already have "Terminal" settings section:

![image](https://github.com/user-attachments/assets/8392c99b-2faf-4193-be0e-adc299c3ae40)

Basically I would like to decouple these so that we do not have "terminals and kernels" section in settings just because the "running sessions" encompasses both in code.

CC @afshin @jtpio?
created_at: 2025-03-14T10:02:29Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,25 @@
+{
+  "title": "Terminals and Kernels sessions settings",
+  "description": "Terminals and Kernels sessions settings",
+  "additionalProperties": false,
+  "properties": {
+    "showStatusBarItem": {
+      "type": "boolean",
+      "title": "Show the status bar item",
+      "description": "Whether to show the running terminals and kernels item in the status bar",
+      "default": true
+    },
path: packages/apputils-extension/schema/sessions-settings.json
user: login: krassowski
body: > I would like to decouple these so that we do not have "terminals and kernels" section in settings just because the "running sessions" encompasses both in code.

To make it clear, I do not suggest making this change in this PR. I am only asking if this is the direction we should take (so in this PR we would just rename this settings section to "Kernels", and possibly the file to `kernels-settings.json` or similar).
created_at: 2025-03-17T15:18:01Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,25 @@
+{
+  "title": "Terminals and Kernels sessions settings",
+  "description": "Terminals and Kernels sessions settings",
+  "additionalProperties": false,
+  "properties": {
+    "showStatusBarItem": {
+      "type": "boolean",
+      "title": "Show the status bar item",
+      "description": "Whether to show the running terminals and kernels item in the status bar",
+      "default": true
+    },
path: packages/apputils-extension/schema/sessions-settings.json
user: login: krassowski
body: I opened https://github.com/jupyterlab/jupyterlab/issues/17406 to track this.
created_at: 2025-03-18T09:19:24Z
author_association: MEMBER
diff_hunk: @@ -76,6 +76,13 @@ A new console panel will open allowing access to variables from the main session
 
 <img alt="The subshell console" src="https://raw.githubusercontent.com/jupyterlab/jupyterlab/main/docs/source/getting_started/changelog_assets/4.4-subshells.png" class="jp-screenshot">
 
+Alternatively, JupyterLab allows automatically creating subshells for comm messages if kernels support this feature.
+This has the by-product of allowing to execute `ipywidgets` callbacks concurrently without requiring a user action.
+This behavior can be changed from the settings, the user can either:
path: CHANGELOG.md
user: login: krassowski
body: ```suggestion
JupyterLab can automatically create subshells for comm messages if kernels support this feature.
This enables concurrent execution of `ipywidgets` callbacks without requiring a user action.
This behavior can be changed from the settings, with options to:
```
created_at: 2025-03-18T10:37:35Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,51 @@
+/* -----------------------------------------------------------------------------
+| Copyright (c) Jupyter Development Team.
+| Distributed under the terms of the Modified BSD License.
+|----------------------------------------------------------------------------*/
+
+import {
+  JupyterFrontEnd,
+  JupyterFrontEndPlugin
+} from '@jupyterlab/application';
+import { CommsOverSubshells } from '@jupyterlab/services';
+import { ISettingRegistry } from '@jupyterlab/settingregistry';
+
+/*
+ * A plugin providing running terminals and sessions settings
+ */
+export const subshellsSettings: JupyterFrontEndPlugin<void> = {
+  id: '@jupyterlab/apputils-extension:sessions-settings',
+  description: 'Sessions and terminals settings.',
path: packages/apputils-extension/src/subshell-settings.ts
user: login: krassowski
body: ```suggestion
 * A plugin loading kernel subshell settings.
 */
export const subshellsSettings: JupyterFrontEndPlugin<void> = {
  id: '@jupyterlab/apputils-extension:subshell-settings',
  description: 'Kernel subshell settings.',
```
created_at: 2025-03-18T10:39:07Z
author_association: MEMBER
diff_hunk: @@ -707,6 +737,21 @@ export interface IManager extends IBaseManager {
    */
   readonly runningCount: number;
 
+  /**
+   * Whether comm messages should be sent to kernel subshells, if the
+   * kernel supports it.
+   *
+   * #### Notes
+   * Sending comm messages over subshells allows processing comms whilst
+   * processing execute-request on the "main shell". This prevents blocking
+   * comm processing.
+   * Options are:
+   * - disabled: not using subshells
+   * - one subshell per comm-target (default)
+   * - one subshell per comm (can lead to issues if creating many comms)
+   */
+  commsOverSubshells: CommsOverSubshells;
path: packages/services/src/kernel/kernel.ts
user: login: krassowski
body: Can this be optional?


```suggestion
  commsOverSubshells?: CommsOverSubshells;
```
created_at: 2025-03-18T10:42:40Z
author_association: MEMBER
diff_hunk: @@ -896,6 +941,21 @@ export interface IComm extends IDisposable {
    */
   readonly targetName: string;
 
+  /**
+   * The subshell ID, null if not using a subshell.
+   */
+  subshellId: string | null;
+
+  /**
+   * Promise that resolves when the subshell started, if any
+   */
+  get subshellStarted(): Promise<void>;
+
+  /**
+   * Whether comms are running on subshell or not.
+   */
+  commsOverSubshells: CommsOverSubshells;
+
path: packages/services/src/kernel/kernel.ts
user: login: krassowski
body: This is a non-optional expansion of the interface; should these be optional?
created_at: 2025-03-18T10:45:19Z
author_association: MEMBER
diff_hunk: @@ -157,6 +160,18 @@ export class KernelManager extends BaseManager implements Kernel.IManager {
     return this._models.size;
   }
 
+  get commsOverSubshells(): CommsOverSubshells {
path: packages/services/src/kernel/manager.ts
user: login: krassowski
body: ```suggestion
  /**
   * Whether comms are running on subshell or not.
   */
  get commsOverSubshells(): CommsOverSubshells {
```
created_at: 2025-03-18T10:45:59Z
author_association: MEMBER
diff_hunk: @@ -896,6 +941,21 @@ export interface IComm extends IDisposable {
    */
   readonly targetName: string;
 
+  /**
+   * The subshell ID, null if not using a subshell.
+   */
+  subshellId: string | null;
+
+  /**
+   * Promise that resolves when the subshell started, if any
+   */
+  get subshellStarted(): Promise<void>;
path: packages/services/src/kernel/kernel.ts
user: login: krassowski
body: @martinRenou to highlight, `subshellStarted` and `subshellId` are also non-optional additions (and it appears mostly used in tests?)
created_at: 2025-03-18T11:39:23Z
author_association: MEMBER
diff_hunk: @@ -896,6 +941,21 @@ export interface IComm extends IDisposable {
    */
   readonly targetName: string;
 
+  /**
+   * The subshell ID, null if not using a subshell.
+   */
+  subshellId: string | null;
+
+  /**
+   * Promise that resolves when the subshell started, if any
+   */
+  get subshellStarted(): Promise<void>;
path: packages/services/src/kernel/kernel.ts
user: login: martinRenou
body: yeah I'm annoyed to have added this. But this is needed for testing

created_at: 2025-03-18T11:47:26Z
author_association: MEMBER
diff_hunk: @@ -896,6 +941,21 @@ export interface IComm extends IDisposable {
    */
   readonly targetName: string;
 
+  /**
+   * The subshell ID, null if not using a subshell.
+   */
+  subshellId: string | null;
+
+  /**
+   * Promise that resolves when the subshell started, if any
+   */
+  get subshellStarted(): Promise<void>;
path: packages/services/src/kernel/kernel.ts
user: login: krassowski
body: Could casting in tests to `as CommHandler` make the tests pass while narrowing the required API?
created_at: 2025-03-18T12:30:19Z
author_association: MEMBER
diff_hunk: @@ -896,6 +941,21 @@ export interface IComm extends IDisposable {
    */
   readonly targetName: string;
 
+  /**
+   * The subshell ID, null if not using a subshell.
+   */
+  subshellId: string | null;
+
+  /**
+   * Promise that resolves when the subshell started, if any
+   */
+  get subshellStarted(): Promise<void>;
path: packages/services/src/kernel/kernel.ts
user: login: martinRenou
body: What do you mean by narrowing the required API?
created_at: 2025-03-18T12:35:00Z
author_association: MEMBER
diff_hunk: @@ -896,6 +941,21 @@ export interface IComm extends IDisposable {
    */
   readonly targetName: string;
 
+  /**
+   * The subshell ID, null if not using a subshell.
+   */
+  subshellId: string | null;
+
+  /**
+   * Promise that resolves when the subshell started, if any
+   */
+  get subshellStarted(): Promise<void>;
path: packages/services/src/kernel/kernel.ts
user: login: krassowski
body: I mean `IComm` would have fewer required attributes.

Edit so I mean removing `subshellStarted` and `subshellId` from here maybe (or making them optional)? Not sure what is the best, I do not want to add yo your workload, I just worry that releasing it as-is will break downstreams.
created_at: 2025-03-18T12:55:21Z
author_association: MEMBER
diff_hunk: @@ -896,6 +941,21 @@ export interface IComm extends IDisposable {
    */
   readonly targetName: string;
 
+  /**
+   * The subshell ID, null if not using a subshell.
+   */
+  subshellId: string | null;
+
+  /**
+   * Promise that resolves when the subshell started, if any
+   */
+  get subshellStarted(): Promise<void>;
path: packages/services/src/kernel/kernel.ts
user: login: martinRenou
body: Gotcha, I misunderstood the casting to `as CommHandler` in tests. Let me try to do it.
created_at: 2025-03-18T13:52:54Z
author_association: MEMBER
diff_hunk: @@ -896,6 +941,21 @@ export interface IComm extends IDisposable {
    */
   readonly targetName: string;
 
+  /**
+   * The subshell ID, null if not using a subshell.
+   */
+  subshellId: string | null;
+
+  /**
+   * Promise that resolves when the subshell started, if any
+   */
+  get subshellStarted(): Promise<void>;
path: packages/services/src/kernel/kernel.ts
user: login: martinRenou
body: Done!
created_at: 2025-03-18T13:55:28Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,51 @@
+/* -----------------------------------------------------------------------------
+| Copyright (c) Jupyter Development Team.
+| Distributed under the terms of the Modified BSD License.
+|----------------------------------------------------------------------------*/
+
+import {
+  JupyterFrontEnd,
+  JupyterFrontEndPlugin
+} from '@jupyterlab/application';
+import { CommsOverSubshells } from '@jupyterlab/services';
+import { ISettingRegistry } from '@jupyterlab/settingregistry';
+
+/*
+ * A plugin loading kernel subshell settings.
+ */
+export const subshellsSettings: JupyterFrontEndPlugin<void> = {
+  id: '@jupyterlab/apputils-extension:subshell-settings',
path: packages/apputils-extension/src/subshell-settings.ts
user: login: jtpio
body: Noting it would be nice to update the generated list of plugins to include this new one.

But it would be better if CI does it automatically, or let us know: https://github.com/jupyterlab/jupyterlab/issues/17393
created_at: 2025-03-19T17:23:26Z
author_association: MEMBER

PR comments:
user: login: jupyterlab-probot[bot]
created_at: 2025-03-04T13:01:46Z
body: Thanks for making a pull request to jupyterlab!
To try out this branch on [binder](https://mybinder.org), follow this link: [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/martinRenou/jupyterlab/comm_subshells?urlpath=lab)
author_association: NONE
user: login: ianthomas23
created_at: 2025-03-04T13:14:14Z
body: I wonder if we should add the ability to check if a comm is using a subshell, and what the subshell ID is. I don't have a particular use case in mind, but I imagine it could be useful for debugging.
author_association: CONTRIBUTOR
user: login: krassowski
created_at: 2025-03-04T15:05:51Z
body: Exciting! I am not familiar with comms very much, is using subshells by default (if ipykernel 7 or another kernel supporting them is used) compatible with all comms consumers, or will there be any changes required for these to work?
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-04T15:15:39Z
body: > Exciting! I am not familiar with comms very much, is using subshells by default (if ipykernel 7 or another kernel supporting them is used) compatible with all comms consumers, or will there be any changes required for these to work?

It will continue working as before and use the "main shell" if sub-shells are not supported, since I'm checking `if (this._kernel.supportsSubshells)`. So for now only ipykernel 7.0.0a0 would see a change in behavior.
author_association: MEMBER
user: login: krassowski
created_at: 2025-03-04T15:17:44Z
body: Yes, I understand that. I do not understand if the other side of the comm, like ipywidgets required any modifications, or if it "just" works with subshells.
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-04T15:33:23Z
body: > Yes, I understand that. I do not understand if the other side of the comm, like ipywidgets required any modifications, or if it "just" works with subshells.

Gotcha, yes, it just works without any change in ipywidgets! The subshell is created from the front-end, and the kernel will dispatch comm messages to the right subshells automatically. ipywidgets auto-magically executes the comm message callbacks in the proper shell.
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-04T17:06:04Z
body: Putting this on draft for now. We should battle test this. Creating one subshell per-comm may not be the smartest move since I've seen users create thousands of widgets.
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-04T17:11:54Z
body: Wondering if we should create a subshell per comm-target basis instead. Meaning, one subshell for all ipywidgets.
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-05T14:23:27Z
body: I make that functionality disabled by default, and allowing to enable it through settings:

https://github.com/user-attachments/assets/91f22423-f2a0-49a2-9fe7-5b9c61a9d25e

This mitigates my concerns since it won't be breaking any current workflow, we can see it as a feature preview.


author_association: MEMBER
user: login: ianthomas23
created_at: 2025-03-06T10:35:25Z
body: For a test for this I think it might be sufficient to test that a subshell is created and deleted as expected without having to actually run any code in the subshell. So both with and without a subshell:

1. Create a comm.
2. See how many subshells the kernel has, either via a `requestListSubshell` message on the control channel https://github.com/jupyterlab/jupyterlab/blob/d470c501f50ad7075413cd89967a1a8a332b9a2f/packages/services/src/kernel/kernel.ts#L375 or by executing the `%subshell` magic command and parsing the return.
3. Delete the comm.
4. Recheck how many subshells there are.
author_association: CONTRIBUTOR
user: login: ianthomas23
created_at: 2025-03-06T12:37:16Z
body: I've tried stress-testing this locally on an M3 mac using `ipykernel 7.0.0a0` with 150 sliders all linked together:

https://github.com/user-attachments/assets/a15e5c7e-d008-4688-9cf4-a341cb0fbdb9

This uses 453 subshells and 920 threads in the python kernel process (each subshell is actually 2 threads as an extra one is used to signal stopping the real subshell thread). This is obviously a silly extreme example but it works reliably. I can increase this to 166 sliders (501 subshells, 1016 threads) but not 167 sliders (504 subshells, 1022 threads) which fails on trying to start a python thread. This success/failure boundary for me is repeatable, so it may well be a resource limit that I am reaching rather than some unreliability in the subshell implementation.

It looks to me that the approach in this PR is good enough for us to continue with and refine later. Eventually we will probably not want to have a separate subshell for each comm, and we'll also need some checking of the eventual failure in `ipykernel` but I don't intend to do that until the `anyio` changes are finalised as that changes the implementation.
author_association: CONTRIBUTOR
user: login: trungleduc
created_at: 2025-03-07T13:42:27Z
body: > Wondering if we should create a subshell per comm-target basis instead. Meaning, one subshell for all ipywidgets.

👍 this sounds good! is there any downside of this approach?
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-07T16:53:49Z
body: > > Wondering if we should create a subshell per comm-target basis instead. Meaning, one subshell for all ipywidgets.
> 
> 👍 this sounds good! is there any downside of this approach?

Widgets would be blocking each other. It would be a better situation than today but still not perfect.
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-11T09:55:02Z
body: @krassowski I merged the settings together. Please tell me what you think!

@ianthomas23 @trungleduc I added the option to use one subshell per-comm or per-comm-target

![Screenshot From 2025-03-11 10-54-41](https://github.com/user-attachments/assets/5ddfc03c-510a-4f9c-b0b7-1758df4f4494)

author_association: MEMBER
user: login: ianthomas23
created_at: 2025-03-11T10:05:57Z
body: I think this needs at least a minimal test like the one I outlined above to check that a subshell is created and deleted correctly depending on the settings.  
author_association: CONTRIBUTOR
user: login: trungleduc
created_at: 2025-03-11T10:15:50Z
body: > I outlined above to check that a subshell is created and deleted correctly depending on the settings.

👍 is the subshell closed when I clear the cell output containing the widgets?
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-11T10:24:07Z
body: Will look into adding a test 👍🏽 

> is the subshell closed when I clear the cell output containing the widgets

It should be closed when disposing the comm in case it's one subshell per-comm
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-11T11:54:32Z
body: Added some tests for each case! 
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-11T13:03:04Z
body: Some tests are flaky right now. Looks unrelated to the PR.
author_association: MEMBER
user: login: krassowski
created_at: 2025-03-11T18:05:12Z
body: > I added the option to use one subshell per-comm or per-comm-target

Would you like to make one of these the default over 'disabled'? Or is there a reason not to do that?
If there is a reason not to, would it be desirable to change the default just for the duration of the beta period?

> Some tests are flaky right now. Looks unrelated to the PR.

Yes, it looks like there was some issue on GH Actions earlier today, see https://github.com/jupyterlab/jupyterlab/issues/17381
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-11T19:22:16Z
body: I guess defaulting to one subshell per comm target makes sense. It should improve the current state, without putting too much load on the subshell implementation which is still early.

Will do!
author_association: MEMBER
user: login: krassowski
created_at: 2025-03-13T18:36:33Z
body: Picked up a conflict in `packages/services/src/kernel/default.ts`
author_association: MEMBER
user: login: fleming79
created_at: 2025-03-13T23:34:58Z
body: Are there any thread safe issues that need consideration (kernel side)?

Would one subshell per kernel be adequate for most use cases?
author_association: CONTRIBUTOR
user: login: fleming79
created_at: 2025-03-14T05:13:32Z
body: From an `asyncio` perspective I think each `subshell` will have its own event loop. 

For example.

``` python
# in notebook

import asyncio
loop = asyncio.get_running_loop()
id(loop)
```

``` python
# in a subshell console for the notebook kernel

print(id(asyncio.get_running_loop()))

async def test():
    print("hello")

await loop.create_task(test())
# Raises RuntimeError: attached to a different loop
```

author_association: CONTRIBUTOR
user: login: martinRenou
created_at: 2025-03-14T08:10:47Z
body: > Picked up a conflict in packages/services/src/kernel/default.ts

Rebased and fixed! Thanks

> Would one subshell per kernel be adequate for most use cases?

We're close to this when using one subshell per comm target (there aren't many libraries out there using comms). And I made that the default in the PR.
author_association: MEMBER
user: login: ianthomas23
created_at: 2025-03-14T09:34:59Z
body: Using one subshell per comm target (2 subshells overall) here are 400 linked sliders:

https://github.com/user-attachments/assets/2897a728-e42f-4e55-8b24-d632df2565c9

500 linked sliders doesn't work for me with or without subshells so that is some unrelated limit.

Although one subshell per comm may not be a good solution long term, I would like to keep it available now as it is a good way to stress the subshell implementation to find problems.
author_association: CONTRIBUTOR
user: login: trungleduc
created_at: 2025-03-14T09:54:27Z
body: I think it's useful to have a short explanation about each option of the comm subshell setting. So that users know the implications of their choice.
author_association: MEMBER
user: login: krassowski
created_at: 2025-03-17T15:22:56Z
body: It picked up a conflict in `kernel/manager.ts` from https://github.com/jupyterlab/jupyterlab/pull/17395.

If we want to include this PR in 4.4.0 we need to finalize it this week and ideally release another beta before Wednesday. Do you all think this is feasible?
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-17T15:35:05Z
body: Since the latest review comments are about docstrings, it should be feasible. I will put some time on this tomorrow morning.
author_association: MEMBER
user: login: martinRenou
created_at: 2025-03-18T08:22:25Z
body: Done!
author_association: MEMBER
user: login: fleming79
created_at: 2025-03-20T00:28:10Z
body: I tried Version 4.4.0b2 with my custom version of [ipylab](https://github.com/fleming79/ipylab) and found using subshells for comms breaks things. I haven't tracked down the exact reason but would like to warn that the default setting could also break things for other users.

When switching to 'Disabled' it works correctly.

author_association: CONTRIBUTOR
user: login: fleming79
created_at: 2025-03-20T06:38:58Z
body: I had a look inside Ipykernel and wonder if maybe it might be worth considering running execute requests in a task with queue.
This enables widget comms to pass whilst awaiting in a cell.

![image](https://github.com/user-attachments/assets/6705ba68-ceae-4ac0-af67-1ba1f789b8d7)

### Ipykernel patch

As a proof of concept. Just overwrite the method in ipykernel.

``` python 
# ipykernel.ipykernel.py  ~ line 356

    async def execute_request(self, stream, ident, parent):
        """Override for cell output - cell reconciliation."""
        parent_header = extract_header(parent)
        self._associate_new_top_level_threads_with(parent_header)
        import asyncio
        if not hasattr(self, '_exec_queue'):
            self._exec_queue = asyncio.Queue()
            self._executor_task = None
        await self._exec_queue.put((super().execute_request, (stream, ident, parent)))
        if not self._executor_task or self._executor_task.done():
            async def executor_task():
                while True:
                    execute_request, args  = await self._exec_queue.get()
                    await execute_request(*args)
            self._executor_task = asyncio.create_task(executor_task())
```

### Notebook

``` python

import asyncio
from ipywidgets import IntSlider, HTML, __version__
import time

slider = IntSlider()
value = HTML()
print('ipywidgets', __version__)
display(slider)
display(value)

async def run_till_100():
    while True:
        time.sleep(0.1) # not a good idea as it blocks the main thread
        await asyncio.sleep(0.1)
        value.value = str(slider.value)
        if slider.value == 100:
            return 'Done'
print(await run_till_100())

```

author_association: CONTRIBUTOR
user: login: ianthomas23
created_at: 2025-03-20T09:43:23Z
body: @fleming79 Maybe I am misunderstanding this, but if I try your example locally it works if subshells are enabled and fails if subshells are disabled:

https://github.com/user-attachments/assets/b7821dbc-a2a9-48c8-bfcc-f4c1273dcd1f

This is using ipykernel 7.0.0a1, ipywidgets 8.1.5 and jupyterlab 4.4.0b2 on macOS. The only problem I see is that the execute count is not updated, which is a known problem.


author_association: CONTRIBUTOR
user: login: fleming79
created_at: 2025-03-20T21:49:57Z
body: Apologies @ianthomas23, the example I'd provided was flawed. I've re-written the example [above](https://github.com/jupyterlab/jupyterlab/pull/17363#issuecomment-2739355243). The kernel patch could be used instead of "subshells for comms" to allow widget comms to pass.

author_association: CONTRIBUTOR
user: login: fleming79
created_at: 2025-03-21T21:35:46Z
body: Because each subshell provides a different thread, it also has a different asyncio event loop.  

Here is an example illustrating the problem of using subshells for comms when considering asyncio.

I'd recommend the default setting for 'Kernel Comms over subshells' is changed to 'Disabled' to avoid disturbing current users who use asyncio.


``` python
#  Cell 1

from ipywidgets import Button, HBox, HTML
import time
import asyncio

button_set = Button(description = "Set")
button_start = Button(description = "Start")
result = HTML()
event = asyncio.Event()

button_set.on_click(lambda _: event.set())
button_start.on_click(lambda _: asyncio.create_task(my_function()))

async def my_function():
    event.clear()
    result.value = "Waiting for event. Click 'set'"
    try:
        await event.wait()
        result.value = f"{time.ctime()} okay"
    except Exception as e:
        result.value = f"{time.ctime()} {e!r}"
display(result)
display(HBox([button_start, button_set]))

```


``` python 
# Cell 2
await my_function()
```


### With subshell

![image](https://github.com/user-attachments/assets/a4066f06-0adf-4efc-8344-a60780116426)

![image](https://github.com/user-attachments/assets/c2d1984e-4776-4230-9c87-81800460244f)


### Without subshell
(using [ipykernel patch](https://github.com/jupyterlab/jupyterlab/pull/17363#issuecomment-2739355243))


![image](https://github.com/user-attachments/assets/426fb37b-4b64-470c-8738-458495a17b46)

![image](https://github.com/user-attachments/assets/d9f92e42-29cd-48df-9ee5-d8bb1432d960)



author_association: CONTRIBUTOR
user: login: ianthomas23
created_at: 2025-03-27T11:10:30Z
body: OK, so your latest example doesn't work with `ipykernel` 7.0.0a1 either with or without use of subshells. I assume it used to work in the past. Ipykernel has changes in two areas, using `anyio` instead of `tornado`/`asyncio`, and subshells. It looks like your example is affected by both changes, and the former are greater changes that the latter. If you think that `ipykernel` needs something like your patch then you should probably report it there to get the attention of the `anyio` experts.

As you say, your example doesn't work with subshells because `asyncio` doesn't work across threads. It can be changed to work with subshells by using `threading.Event` rather than `asyncio.Event` and ensuring that `my_function` is run in the main thread not a subshell:
```python
# Cell 1
from ipywidgets import Button, HBox, HTML
import time
import asyncio
from threading import Event

button_set = Button(description = "Set")
button_start = Button(description = "Start")
result = HTML()

start_event = Event()
set_event = Event()

button_set.on_click(lambda _: set_event.set())
button_start.on_click(lambda _: start_event.set())

async def my_function():
    start_event.wait()
    start_event.clear()
    set_event.clear()
    result.value = "Waiting for event. Click 'set'"
    try:
        set_event.wait()
        result.value = f"{time.ctime()} okay"
    except Exception as e:
        result.value = f"{time.ctime()} {e!r}"

display(result)
display(HBox([button_start, button_set]))
```
and cell 2 unchanged. This works for me using `ipykernel 7.0.0a1` without your patch.


author_association: CONTRIBUTOR
user: login: fleming79
created_at: 2025-03-27T22:44:05Z
body: @ianthomas23 - Thank you for taking the time to trial the suggestion. I'm sorry to hear the suggested patch didn't work for you straight up. It does work, but you need to pause for a few seconds after restarting the kernel. 

Here is a screenshot of the patched kernel (7.0.0.a1) with Jupyterlab after running the example.

![image](https://github.com/user-attachments/assets/727d3585-f32a-453b-ae99-71cbeea933b5)

Note: the patch was only a proof of concept, not a working fix (for example stdio output incorrectly always ends up at the same cell).

The second example I posted above was to illustrate problems existing users could encounter. In the example it was due to unexpectedly having two separate event loops where previously there was only one. The point being that enabling subshell for comms by default could break things for a lot of users.

Another issue I noticed was that debug breakpoints weren't working where previously they were (using a launch request in vscode).


To summarize my observations/suggestions:
1. Kernel subshells for widget comms  could break things for user code involving widget comms that is not designed to be thread safe. For asyncio based code it introduces one or more event loops depending on where the call originated and the subshell setting in Jupyterlab. 
2. An alternate to using subshells for widget comms could be to modify the kernel to run execute requests in a queue (modifying ipykernel). The benefit of not using threads for widget comms being that existing non thread safe code will continue to work without alteration.


author_association: CONTRIBUTOR
user: login: ianthomas23
created_at: 2025-03-28T08:09:38Z
body: @fleming79 To clarify, I didn't use your patch. My sentence "OK, so your latest example doesn't work with ipykernel 7.0.0a1 either with or without use of subshells" should be taken literally.

Ipykernel 7 will be a major release, it is not backward compatible for everything which is why it is a major release. I personally expect all sync code to work unmodified, but probably any semi-complex async code may be broken and need to be changed. I expect other projects will pin to ipykernel<7 initially until they have made the necessary changes in their code.

For your specific suggestions about ipykernel, please please please report them to ipykernel.

Personally I think the default use of subshells in this merged PR should remain as it is, i.e. one subshell per comm target. The inability of ipywidgets to support updates in the presence of a blocking task has been a bug for years, and is fixed by this PR. Yes there will be some use cases using some async workarounds that will be broken by this, but in many cases those workarounds will not be required any more. Users and projects affected by this but not pinning ipykernel<7 can change the setting to disable use of subshells for comms in the usual manner.
author_association: CONTRIBUTOR