npm package: @malv/base-app
Description: Shared utilities and scripts for malv apps
Author: Ilias Nikolaidis Olsson
Latest version: 1.1.7

# @malv/base-app

**Framework Component Type:** Build Tools & Development Infrastructure

## Purpose

Build tools, development scripts, and local infrastructure server for MALV applications. This package provides everything needed to develop, build, and deploy MALV apps with automatic local infrastructure setup.

## Architecture

```
@malv/base-app
├── Development Scripts
│   ├── malv-dev.js              # Multi-app batch coordinator
│   ├── malv-dev-cloudflare.js   # Single-app Cloudflare Workers dev
│   └── malv-dev-cloudflare-multi.js  # Multi-app orchestrator
│
├── Code Generators (15+)
│   ├── generate-all.js          # Run all generators
│   ├── generate-token-types.js  # TokenPayloads.ts from tokens.json
│   ├── generate-event-handler-types.js  # Event handler types
│   ├── generate-types.js        # Type definitions
│   └── ...
│
├── Infrastructure Server (Port 59459)
│   ├── /apps      # App metadata & renderers
│   ├── /search    # Semantic search with embeddings
│   ├── /token     # Token signing & verification
│   ├── /storage   # File-based storage operations
│   └── /queue     # Tool dispatch queue with batching
│
└── Build & Deploy
    ├── malv-build.js            # Production build
    ├── malv-deploy-cloudflare.js  # Deploy to Cloudflare
    └── setup-production.js      # Prepare production bundle
```

---

## Development Workflow

### Quick Start

```bash
# Start a single app in development mode
cd packages/malv-apps/my-app
yarn run dev

# Or use malv-dev directly
malv-dev --port 4557

# Start multiple apps (batched automatically)
# Terminal 1:
cd packages/malv-apps/auth && malv-dev
# Terminal 2 (within 4 seconds):
cd packages/malv-apps/conversation && malv-dev
# Terminal 3 (within 4 seconds):
cd packages/malv-apps/orchestrator && malv-dev
```

### How Multi-App Batching Works

The `malv-dev` command implements a **batch coordinator pattern** that automatically groups multiple apps started within a 4-second window:

```
┌─────────────────────────────────────────────────────────────────┐
│                    Multi-App Batch Coordinator                  │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Terminal 1: malv-dev     ──┐                                   │
│                             │                                   │
│  Terminal 2: malv-dev     ──┼──► Batch Coordinator              │
│                             │    (4-second window)              │
│  Terminal 3: malv-dev     ──┘           │                       │
│                                         ▼                       │
│                              malv-dev-cloudflare-multi.js       │
│                                         │                       │
│                    ┌────────────────────┼────────────────────┐  │
│                    ▼                    ▼                    ▼  │
│              Wrangler #1          Wrangler #2          Wrangler #3
│              (App 1)              (App 2)              (App 3)  │
│                                                                 │
│                    Shared Infrastructure Server                 │
│                         (Port 59459)                            │
└─────────────────────────────────────────────────────────────────┘
```

**Coordination Flow:**

1. **Leader Election**: First app to register becomes the "leader"
2. **Batch Collection**: Leader waits 4 seconds for more apps to join
3. **Multi-App Spawn**: Leader launches `malv-dev-cloudflare-multi.js` with all apps
4. **Non-Leaders Exit**: Other processes exit after multi-app process starts
5. **Fallback**: If multi-app process already running, new apps start in single-app mode

**State File**: `~/.malv/dev-coordinator.json`

```json
{
  "apps": [
    { "path": "/path/to/app1", "port": 4557, "envPath": null, "pid": 12345 },
    { "path": "/path/to/app2", "port": 4558, "envPath": null, "pid": 12346 }
  ],
  "leaderPid": 12345,
  "processRunning": false,
  "lastUpdate": 1234567890
}
```

---

## Infrastructure Server

During local development, all infrastructure services are provided by a **unified Node.js HTTP server** on port `59459`. This server is automatically started when you run any app with `malv-dev` or `malv-dev-cloudflare`.

### Singleton Pattern

Only one infrastructure server runs at a time, regardless of how many apps are started:

```
App 1 starts → Acquires lock → Starts server (owner)
App 2 starts → Lock exists → Connects to existing server
App 3 starts → Lock exists → Connects to existing server
```

**Lock File**: `~/.malv/infrastructure-server.lock`

### Available Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/health` | GET | Server health check with running app count |
| `/apps` | GET | List all running apps |
| `/apps/{appName}/tools.json` | GET | Tool definitions for an app |
| `/apps/{appName}/tools-with-output-token/{tokenName}.json` | GET | Tools that output a specific token |
| `/apps/{appName}/objects.json` | GET | All object definitions for an app |
| `/apps/{appName}/objects/{objectType}.json` | GET | Single object definition by type |
| `/apps/{appName}/events.json` | GET | Event definitions |
| `/apps/{appName}/tokens.json` | GET | Token definitions |
| `/apps/{appName}/web-renderers/{name}.js` | GET | Bundled web renderer |
| `/apps/{appName}/objects/{type}/icons/{name}.js` | GET | Bundled icon |
| `/search?q=<query>` | GET | Semantic search across tools/objects |
| `/perception` | GET | List all perception files from all running apps |
| `/perception/{appName}` | GET | Get perception files for a specific app |
| `/token/sign-token` | POST | Sign a JWT token |
| `/token/get-public-keys` | POST | Get public keys for verification |
| `/storage` | POST | Storage operations (read/write/list/delete) |
| `/storage/upload` | PUT | Direct file upload |
| `/queue` | POST | Queue tool for batched background execution |
| `/queue/status` | GET | Queue status (pending items per app) |

### Route Details

#### Apps Routes (`/apps/*`)

Serves app metadata and bundled assets from running apps:

```bash
# List running apps
curl http://localhost:59459/apps

# Get tools for an app
curl http://localhost:59459/apps/@malv/auth/tools.json

# Get tools that output a specific token (for prerequisite discovery)
curl http://localhost:59459/apps/@malv/gmail/tools-with-output-token/gmail_access.json

# Get a bundled web renderer
curl http://localhost:59459/apps/@malv/tables/web-renderers/table.js
```

**Prerequisite Tool Discovery**: The `/tools-with-output-token/{tokenName}.json` endpoint returns tools that have the specified token in their `output_tokens`. This is used by the orchestrator to find prerequisite tools - for example, when `list_emails` requires `gmail_access` token that the user doesn't have, the orchestrator queries this endpoint to discover `login_gmail` which can provide that token.

**Bundling**: Renderers are served from `dist/standalone/` if pre-built, otherwise bundled on-demand from TypeScript source using Rollup.

#### Search Routes (`/search`)

Semantic search using HuggingFace embeddings (`Xenova/bge-base-en-v1.5`):

```bash
# Search for tools related to "email"
curl -H "X-App-ID: @malv/orchestrator" \
     -H "X-App-Secret: <secret>" \
     "http://localhost:59459/search?q=email&types=tools&limit=10"

# Search storage data with security key filtering
curl -H "X-App-ID: @malv/orchestrator" \
     -H "X-App-Secret: <secret>" \
     "http://localhost:59459/search?q=project%20goals&types=storage&securityKeys=[\"account:user123\"]"
```

**Response:**
```json
{
  "query": "email",
  "types": ["tools"],
  "count": 3,
  "results": [
    {
      "type": "tools",
      "appName": "@malv/gmail",
      "appUrl": "http://localhost:4560",
      "name": "list_emails",
      "description": "Lists emails from Gmail inbox",
      "similarity": 0.92
    }
  ]
}
```

**How it works:**
1. Apps generate embeddings at build time (stored in `embeddings/embeddings.json`)
2. Search endpoint loads embeddings from all running apps
3. Creates embedding for query and calculates cosine similarity
4. Returns results sorted by similarity

**Storage Search**: In addition to tools and objects, the search endpoint supports semantic search over app storage data. Embeddings are automatically generated when apps write to storage, enabling AI-powered cross-app data discovery with permission-aware filtering via security keys. See [SEMANTIC_STORAGE.md](./SEMANTIC_STORAGE.md) for complete documentation.

#### Perception Routes (`/perception`)

Fetch perception files that define how the AI should perceive users based on their tokens:

```bash
# Get all perception files from all running apps
curl http://localhost:59459/perception

# Get perception files for a specific app
curl http://localhost:59459/perception/@malv/research
# Or using short name:
curl http://localhost:59459/perception/research
```

**Response (all apps):**
```json
{
  "@malv/research": [
    {
      "name": "create_project",
      "tokens": { "absent": { "@malv/research": "project" } },
      "perception": "Wants to create their first research project",
      "tasks": ["Guide user into creating a project"]
    },
    {
      "name": "guide_user",
      "tokens": { "exists": { "@malv/research": "project" } },
      "perception": "Working on a specific research project",
      "tasks": ["Edit goal, objective, questions of the research project"]
    }
  ]
}
```

**Response (single app):**
```json
{
  "app": "@malv/research",
  "perceptions": [
    {
      "name": "create_project",
      "tokens": { "absent": { "@malv/research": "project" } },
      "perception": "Wants to create their first research project",
      "tasks": ["Guide user into creating a project"]
    }
  ]
}
```

Apps define perception files in `perception/*.json`. The orchestrator evaluates conditions against available tokens and storage data to determine which perceptions apply.

**Supported conditions:**
- `tokens.exists` / `tokens.absent` - Check token presence
- `storage` - Query actual storage values (e.g., check if a field is null or empty)

**Storage condition example:**
```json
{
  "tokens": { "exists": { "@malv/research": "project" } },
  "storage": {
    "@malv/research": {
      "/teams/<project.teamId>/research/projects/<project.projectId>/config.json": {
        "objective": { "operator": "equals", "value": null }
      }
    }
  },
  "perception": "Wants us to set objective for research project",
  "tasks": ["Edit objective"]
}
```

Storage paths support `<tokenType.field>` templates resolved from token payloads. Operators include: `equals`, `notEquals`, `exists`, `notExists`, `isEmpty`, `isNotEmpty`.

#### Token Routes (`/token/*`)

Sign and verify JWT tokens with Ed25519 signatures:

```bash
# Sign a token
curl -X POST http://localhost:59459/token/sign-token \
  -H "Content-Type: application/json" \
  -d '{
    "appId": "@malv/auth",
    "appSecret": "<secret>",
    "type": "account",
    "payload": { "userId": "user-123", "team": "team-456" },
    "expiresIn": 3600
  }'
```

**Token Structure:**
```json
{
  "appId": "@malv/auth",
  "type": "account",
  "iss": "malv-local-dev",
  "sub": "user-123",
  "iat": 1234567890,
  "exp": 1234571490,
  "aud": "storage",
  "storage_permissions": [
    { "type": "read", "prefix": "/teams/team-456/" },
    { "type": "write", "prefix": "/teams/team-456/" }
  ]
}
```

**Storage Permissions**: Permissions are extracted from the app's `storage.json` file at token signing time. See [STORAGE_LOGIC.md](./STORAGE_LOGIC.md) for the complete permission embedding and validation flow.

#### Storage Routes (`/storage`)

File-based storage with token authentication:

```bash
# Write a file
curl -X POST http://localhost:59459/storage \
  -H "Content-Type: application/json" \
  -H "X-Provided-Tokens: [\"<signed-token>\"]" \
  -d '{
    "operation": "write",
    "path": "/teams/team-456/data.json",
    "data": {"key": "value"}
  }'

# Read a file
curl -X POST http://localhost:59459/storage \
  -H "X-Provided-Tokens: [\"<signed-token>\"]" \
  -d '{"operation": "read", "path": "/teams/team-456/data.json"}'
```

**Permission System:**
1. **App Secret Auth**: Apps with valid secret have full access to their own storage
2. **Token Permissions**: Check permissions embedded in signed tokens (from `storage.json`)
3. **Cross-App Permissions**: Validate against target app's `storage.json` definitions

For detailed information on how permissions are embedded in tokens and validated during storage access, see [STORAGE_LOGIC.md](./STORAGE_LOGIC.md).

---

## App Registration & Health Monitoring

Running apps are tracked in a registry for discovery and health monitoring.

**Registry File**: `~/.malv/running-apps.json`

```json
{
  "@malv/auth": {
    "path": "/path/to/auth",
    "url": "http://localhost:4557",
    "healthy": 1234567890,
    "pid": 12345
  },
  "@malv/orchestrator": {
    "path": "/path/to/orchestrator",
    "url": "http://localhost:4558",
    "healthy": 1234567892,
    "pid": 12346
  }
}
```

**Health Monitoring:**
- Each app sends health checks every 30 seconds
- Apps with no health update for 60 seconds are considered stale
- Stale apps are automatically removed from registry
- PID verification prevents cross-process unregistration

---

## Code Generators

The package includes 15+ code generators that create TypeScript types from declarative JSON schemas.

### Available Generators

| Script | Input | Output | Description |
|--------|-------|--------|-------------|
| `generate-all.js` | - | - | Runs all generators |
| `generate-token-types.js` | `tokens.json` | `TokenPayloads.ts` | Token payload types |
| `generate-event-handler-types.js` | `events.json` | `EventHandlerTypes.ts` | Event handler types |
| `generate-types.js` | `tools.json` | Type definitions | Tool input/output types |
| `generate-environment.js` | `environment.json` | `Environment.ts` | Environment config types |
| `generate-exports.js` | `package.json` | Export mappings | Package export configuration |
| `generate-icons.js` | `objects.json` | Icon scaffolding | Object icon templates |
| `generate-object-renders.js` | `objects.json` | Renderer scaffolding | Web renderer templates |
| `generate-object-types.js` | `objects.json` | Object type definitions | Object metadata types |

### Example: Token Type Generation

**Input (`tokens.json`):**
```json
{
  "account": {
    "schema": {
      "type": "object",
      "properties": {
        "userId": { "type": "string" },
        "team": { "type": "string" },
        "role": { "type": "string", "enum": ["owner", "admin", "member"] }
      },
      "required": ["userId", "team", "role"]
    }
  }
}
```

**Output (`TokenPayloads.ts`):**
```typescript
export type TokenPayloads = {
  account: {
    userId: string;
    team: string;
    role: "owner" | "admin" | "member";
  };
};
```


### Running Generators

```bash
# Generate all types
yarn run generate

# Generate specific types
yarn run generate:tokens
yarn run generate:events
yarn run generate:objects
```

---

## Build & Deploy Scripts

### Development Scripts

| Script | Description |
|--------|-------------|
| `malv-dev.js` | Multi-app batch coordinator |
| `malv-dev-cloudflare.js` | Single-app Cloudflare Workers development |
| `malv-dev-cloudflare-multi.js` | Multi-app orchestrator |
| `malv-dev-cloudflare-setup.js` | Setup Cloudflare environment |
| `malv-dev-server.js` | Local development server |

### Build Scripts

| Script | Description |
|--------|-------------|
| `malv-build.js` | Production build with Rollup |
| `malv-build-standalone.js` | Standalone bundle build |
| `setup-production.js` | Prepare production artifacts |
| `setup-cloudflare.js` | Setup Cloudflare deployment |

### Deploy Scripts

| Script | Description |
|--------|-------------|
| `malv-deploy-cloudflare.js` | Deploy to Cloudflare Workers |
| `publish-github.js` | Publish to GitHub Packages |
| `publish-npmjs.js` | Publish to npm registry |

---

## CLI Commands

Apps typically expose these commands via `package.json`:

```json
{
  "scripts": {
    "dev": "malv-dev",
    "dev:cloudflare": "malv-dev-cloudflare",
    "generate": "generate-all",
    "generate:tokens": "generate-token-types",
    "generate:events": "generate-event-handler-types",
    "build": "malv-build",
    "setup-production": "setup-production",
    "deploy": "malv-deploy-cloudflare"
  }
}
```

### Command Options

**`malv-dev`**
```bash
malv-dev [options]

Options:
  --port <port>     Specify port (default: auto-allocate from 4557)
  --env, -e <path>  Custom .env file path
```

**`malv-dev-cloudflare-multi`**
```bash
malv-dev-cloudflare-multi <app-args...>

App argument format: path[:port][:env-path]

Examples:
  malv-dev-cloudflare-multi ./app1 ./app2
  malv-dev-cloudflare-multi ./app1:4557 ./app2:4558
  malv-dev-cloudflare-multi ./app1:4557:/path/.env ./app2::/other/.env
```

---

## File Locations

### System Files

| Path | Description |
|------|-------------|
| `~/.malv/` | MALV configuration directory |
| `~/.malv/running-apps.json` | Running apps registry |
| `~/.malv/dev-coordinator.json` | Multi-app batch coordinator state |
| `~/.malv/infrastructure-server.lock` | Infrastructure server lock file |
| `~/.malv/secrets/{appName}` | App secrets storage |
| `~/.malv/cloudflare-state/` | Cloudflare persistent state |

### Per-App Files

| Path | Description |
|------|-------------|
| `tmp/cloudflare/` | Generated Cloudflare configuration |
| `tmp/cloudflare/wrangler.toml` | Wrangler configuration |
| `.dev.vars` | Environment variables for wrangler |
| `dist/standalone/` | Pre-built standalone bundles |
| `embeddings/embeddings.json` | Semantic search embeddings |

---

## App Initialization Flow

When starting an app with `malv-dev-cloudflare`, the following happens:

```
1. Validate Environment
   └─ Check required env vars, custom env path

2. Run Setup Script
   └─ malv-dev-cloudflare-setup.js
      └─ Generate wrangler.toml, worker entry point

3. Run Code Generators
   └─ generate-all.js
      ├─ TokenPayloads.ts
      ├─ EventHandlerTypes.ts
      └─ Environment.ts

4. Validate TypeScript
   └─ Check for syntax errors (exit on error)

5. Allocate Port
   └─ Use specified port or find available (starting 4557)

6. Build Standalone Bundles
   └─ Rollup build for renderers and icons

7. Generate App Secret
   └─ Create or retrieve from ~/.malv/secrets/{appName}

8. Write Environment
   └─ .dev.vars with SECRET, APPS_CDN_URL, SEARCH_URL

9. Start Wrangler
   └─ wrangler dev --config tmp/cloudflare/wrangler.toml

10. Register App
    └─ Add to ~/.malv/running-apps.json

11. Start Infrastructure Server
    └─ If not already running, start on port 59459

12. Start Health Monitoring
    └─ 30-second interval health checks

13. Setup File Watchers
    └─ Watch tools.json, objects.json, .env for changes
```

---

## Wrangler Integration

Apps run as Cloudflare Workers using Wrangler in development mode.

### Generated Configuration

**`tmp/cloudflare/wrangler.toml`:**
```toml
name = "malv-auth-dev"
main = "worker.ts"
compatibility_date = "2024-01-01"

[vars]
ENVIRONMENT = "development"

[[d1_databases]]
binding = "DB"
database_name = "malv-dev"
database_id = "local"
```

### Worker Entry Point

**`tmp/cloudflare/worker.ts`:**
```typescript
import { createWorker } from '@malv/base-app/cloudflare-worker';
import tools from '../../dist/tools.js';

export default createWorker({ tools });
```

---

## Cloudflare Worker Template

The package includes a Cloudflare Worker template that:
- Handles tool execution requests
- Manages capabilities (storage, token, event)
- Streams responses via SSE
- Captures output tokens

**Location**: `bin/cloudflare-worker/worker.ts`

### Request-Scoped Logging

The worker uses `AsyncLocalStorage` to provide request-scoped `console.log` capture. This prevents cross-request I/O errors when multiple requests run concurrently in the same Workers isolate.

**Key files:**
- `toolExecutionContext.ts` - AsyncLocalStorage instance for execution context
- `setupConsoleCapture.ts` - Global console.log override (imported first in worker.ts)

Each tool execution runs inside `toolExecutionContext.run()`, ensuring logs are routed to the correct request's SSE stream.

---

## Dependencies

### Runtime Dependencies
- `miniflare` - Local Cloudflare Workers runtime
- `wrangler` - Cloudflare Workers CLI
- `rollup` - Module bundler
- `@rollup/plugin-typescript` - TypeScript support
- `@rollup/plugin-commonjs` - CommonJS support
- `chokidar` - File watching

### Peer Dependencies
- `@malv/types` - Type definitions
- `@malv/shared` - Shared utilities
- `@malv/base` - Core framework

---

## Best Practices

1. **Use Multi-App Mode**: Start apps within 4 seconds to batch them together
2. **Let Infrastructure Start Automatically**: Don't manually start the infrastructure server
3. **Run Generators After Schema Changes**: Always run `yarn generate` after modifying `tokens.json`, `events.json`, or `objects.json`
4. **Pre-Build Standalone Bundles**: Run `malv-build-standalone` for faster renderer serving
5. **Check Registry on Issues**: If apps aren't discovered, check `~/.malv/running-apps.json`
6. **Clean State on Errors**: Delete `~/.malv/` directory to reset all state

---

## Troubleshooting

### Infrastructure Server Won't Start

```bash
# Check if server is running
curl http://localhost:59459/health

# Remove stale lock file
rm ~/.malv/infrastructure-server.lock

# Restart apps
```

### Apps Not Discovered

```bash
# Check running apps registry
cat ~/.malv/running-apps.json

# Check app health
curl http://localhost:4557/health

# Clean stale entries
# Restart all apps
```

### Port Already in Use

```bash
# Find process using port
lsof -i :4557

# Kill process
kill <pid>

# Or specify different port
malv-dev --port 4560
```

### Generator Errors

```bash
# Validate JSON schemas
cat tokens.json | jq .

# Run specific generator with debug
node bin/generate-token-types.js --debug
```

---

## Related Documentation

- [SEMANTIC_STORAGE.md](./SEMANTIC_STORAGE.md) - Semantic storage search: automatic embedding generation, security keys, and cross-app data discovery
- [SEMANTIC_STORAGE_LOCATION_FILTERING.md](./SEMANTIC_STORAGE_LOCATION_FILTERING.md) - Location-based filtering for precise semantic storage searches
- [STORAGE_LOGIC.md](./STORAGE_LOGIC.md) - Storage permission embedding and validation flow

## Related Packages

- **@malv/types** - TypeScript type definitions
- **@malv/shared** - Shared utilities
- **@malv/base** - Core framework implementation
- **@malv/dev** - Development CLI tools
- **@malv/infrastructure** - Production infrastructure services