pluginbridge / FAILED-APPROACHES.md

Sitar118

docs: update all context files for v0.5.2 session (2026-05-17)

dcfc9c9 2 months ago

11 kB

PluginBridge — Failed Approaches & Architecture Decisions Log

Everything we tried that didn't work, why it failed, and what we did instead. Use this to avoid repeating mistakes or to revisit with better solutions later.

1. Out-of-Process GUI via Floating Window (Sprint 2, v0.2)

What we tried: Host the plugin in a separate Helper process. Show the plugin's GUI as a floating window from the Helper process, overlaying Ableton.

What worked:

Audio routing via shared memory ✅
Crash safety (Helper crashes, Ableton survives) ✅
GUI showed correctly when Ableton was in WINDOWED mode ✅

What broke: When Ableton was in macOS FULLSCREEN, the plugin GUI opened in a separate Space (different virtual desktop), not over Ableton.

Fixes attempted (all failed in fullscreen):

Attempt	Code	Result
`NSApplicationActivationPolicyAccessory`	Set in `main.cpp initialise()`	No Dock icon ✅ but broke fullscreen behavior ❌
`setLevel: 3` (NSFloatingWindowLevel)	In `showAsFloatingPanel()`	Not high enough — fullscreen apps are above level 3 ❌
`setLevel: 25` (NSStatusWindowLevel)	In `showAsFloatingPanel()`	Still opens in separate Space ❌
`setCollectionBehavior: canJoinAllSpaces + fullScreenAuxiliary` (257)	Before `setVisible(true)`	Still separate Space ❌
`setCollectionBehavior` + `stationary` + `ignoresCycle`	Full flags (1<<0 \| 1<<4 \| 1<<6 \| 1<<8)	Still separate Space ❌
`orderFrontRegardless`	After setting level	Window appears but wrong Space ❌
`setHidesOnDeactivate: 0`	Combined with above	No effect on Space problem ❌
`setAlwaysOnTop(true)` (JUCE method)	Reverted to original approach	Was original working code but didn't fix fullscreen either ❌

Root cause: macOS fullscreen creates a dedicated compositor Space. Apple prevents other processes from injecting windows into a fullscreen app's Space. No combination of window level, collection behavior, or activation policy reliably works for cross-process window display in fullscreen. This is a macOS platform limitation, not a code bug.

What we did instead: Switched to in-process hosting (v0.3). Plugin GUI is embedded inside PluginBridge's own editor. Ableton manages the window. No separate process window needed.

2. X Button (Close) Not Working on Helper Window (Sprint 2, v0.2)

What happened: The Helper's floating window had minimize and close (X) buttons visible, but clicking X did nothing.

Root cause: When the Helper ran as NSApplicationActivationPolicyAccessory, macOS doesn't deliver close-button clicks to accessory-policy processes the same way. The native title bar's close button was visible but non-functional.

Fix by Claude Code (commit 0632681): Removed makeAccessoryProcess(), added showAsFloatingPanel() with objc_msgSend level tricks. This fixed the X button but broke the fullscreen behavior (see #1 above).

What we did instead: In v0.3 (in-process hosting), there's no separate window at all. The plugin editor is a child component inside PluginBridge's editor. No close button needed — the DAW manages the window lifecycle.

3. Plugin Crashes Crashing Ableton (v0.3 initial)

What happened: After moving to in-process hosting, loading Kickstart 2 crashed Ableton (because the plugin crash happens inside Ableton's process).

Root cause: loadPlugin() called createPluginInstance() directly without first testing if the plugin was safe. Some plugins (Kickstart 2, Ozone 12, Gullfoss) call abort() or show NSAlert during loading, which kills the host process.

Fix attempted that was missing: The design called for Helper-based test loading BEFORE in-process load, but it was left as a // TODO stub.

What we did: Implemented testPluginSafety() — spawns PluginBridgeHelper --test "/path" before loading. If Helper survives (exit 0) → safe to load in-process. If Helper crashes → add to blocklist → show "Not compatible" → Ableton stays alive.

4. In-Process Hosting = No Crash Safety for Audio Processing

Current limitation (v0.3): If a plugin passes the test-load but crashes during processBlock() later (rare but possible), Ableton will still crash. The Helper test only validates loading, not runtime stability.

Potential future solutions:

Wrap processBlock() in a signal handler (fragile, not recommended)
Use setjmp/longjmp around processBlock (undefined behavior with C++)
Move back to out-of-process audio routing BUT keep GUI in-process (hybrid approach)
Accept the risk — plugins that crash during audio processing are very rare

Status: Accepted risk. Most crashes happen during loading, not during audio processing.

5. Plugin Scanning via PluginDirectoryScanner (Early research, never shipped)

What was planned: Use JUCE's PluginDirectoryScanner to scan all installed plugins on first run.

Why it was rejected: Some plugins execute code during scanning (show NSAlert dialogs, call abort()). This would crash the scanner — or worse, crash Ableton if scanning happened in-process.

What we did instead: Read .vst3 and .component filenames from disk (no code execution). Only actually load a plugin when the user selects it (after Helper test).

6. `addDefaultFormats()` — Deleted in JUCE 8

What happened: Early code used formatManager.addDefaultFormats() which doesn't exist in JUCE 8.

Fix: Explicitly register formats:

formatManager.addFormat(new juce::VST3PluginFormat());
formatManager.addFormat(new juce::AudioUnitPluginFormat());

7. `MSG_NOSIGNAL` — Doesn't Exist on macOS

What happened: Socket code used MSG_NOSIGNAL flag which is Linux-only.

Fix:

signal(SIGPIPE, SIG_IGN);  // Global
int nosigpipe = 1;
setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &nosigpipe, sizeof(nosigpipe));  // Per-socket

8. `juce::Thread::setCurrentThreadPriority()` — Removed in JUCE 8

What happened: Audio thread priority setting via JUCE API no longer exists.

Fix: Use native pthread:

struct sched_param param;
param.sched_priority = sched_get_priority_max(SCHED_FIFO);
pthread_setschedparam(pthread_self(), SCHED_FIFO, &param);

9. `create_audio_track()` Serialization Error (Ableton LOM, not PluginBridge)

Context: From the broader AI Music Production system, not PluginBridge-specific.

What happened: song.create_audio_track() in Ableton's LOM returns a "not JSON serializable" error.

Reality: The track IS created. The error is in the return value serialization, not the action.

Workaround: Always check len(song.tracks) to confirm track was created, ignore the error message.

Architecture Evolution Summary

v0.1 (Sprint 1): Plugin shell + MCP server only (no hosting)
v0.2 (Sprint 2): Out-of-process hosting via shared memory
                  ✅ Crash-safe audio
                  ❌ GUI in separate window → fullscreen broken
v0.3 (Sprint 3): In-process hosting (like SnappySnap)
                  ✅ GUI embedded in Ableton's window
                  ✅ Helper tests plugin safety before loading
                  ❌ No crash safety during audio processBlock (accepted risk)

10. CAContext + CALayerHost — Cross-Process GUI Embedding (v0.4, 2026-05)

What we tried: After the floating-window fullscreen failure (see #1), we tried the macOS private API for sharing a CALayer tree across processes:

Helper (server): CAContext contextWithCGSConnection:options: — wraps the plugin's GUI layer and exposes it as a uint32_t contextId
Plugin (client): CALayerHost.contextId = contextId — a CALayer subclass that renders the remote context inline in JUCE's NSView

The contextId was sent over the existing IPC socket. No Mach port bootstrapping needed.

Also fixed a secondary bug: the Helper's NSWindow was never ordered front. Fix was to position it off-screen at (-32000, -32000) and call orderFront:.

What happened:

Build succeeded, API calls compiled and ran without errors
CALayerHost was created and added as a sublayer to JUCE's NSView
The plugin GUI never appeared inside the PluginBridge editor window
Debug showed: contextId was valid (non-zero), CALayerHost was attached, but nothing rendered

Why it failed: Ableton Live runs inside the macOS app sandbox / process isolation model. CAContext cross-process layer sharing requires both processes to be in the same WindowServer session with compatible entitlements. Ableton's process environment appears to block this — the compositor receives the layer tree but doesn't render it into a foreign process's view hierarchy.

This is the same fundamental limitation as the floating-window approach: Apple does not provide a reliable, supported API for one process to embed rendering from another process's view hierarchy.

Attempts made:

Attempt	Result
`CAContext` + off-screen NSWindow + `orderFront:` at (-32000,-32000)	API worked, nothing rendered ❌
`CALayerHost` added as sublayer to JUCE NSView root layer	Layer hierarchy correct, blank ❌
Explicit `setNeedsDisplay` / `setNeedsLayout` on CALayerHost	No effect ❌
Sending contextId immediately after `createEditor()`	Same result ❌

Files changed (then reverted):

Source/Shared/PrivateCA.h — declared CAContext, CALayerHost, CGSMainConnectionID
Source/Helper/HelperPluginHost.mm — showGui() using CAContext
Source/Plugin/PluginBridgeEditor.mm — connectRemoteLayer() using CALayerHost

What we did instead: Moved to the hybrid safety-scan architecture (v0.5):

Plugins that pass a safety scan are loaded in-process — their createEditor() runs inside PluginBridge's process, producing a standard JUCE AudioProcessorEditor* that embeds natively
Plugins that fail the scan are loaded only in Helper for audio-only mode (no GUI)
No cross-process GUI required at all

This trades crash isolation for reliability: safe plugins (99% of the library) get full GUI; only crash-prone plugins (iZotope Ozone, Neutron, a few others) are audio-only.

Revisit Candidates (Future)

Problem	Possible Future Solution	Difficulty
Fullscreen floating window from separate process	Wait for Apple to provide a supported API, or use private `CGSOrderWindow` API	Hard, fragile
Runtime crash safety (processBlock)	Hybrid: audio in Helper, GUI in-process. Two instances synced via IPC.	Very hard
Blocklist is manual (per-machine)	Share blocklist via cloud / community database	Medium
Test-load adds 2-3 second delay first time	Cache results aggressively, pre-test in background on startup	Medium
Some plugins crash in createEditor() not createInstance()	Test GUI creation in Helper too (spawn with --test-gui flag)	Medium