Skill: oblio-falootin/sql-dreamer (version 0.1.0)

# openclaw-sql-dreamer — SKILL.md > OpenClaw skill: SQL-backed memory corpus feeder and dream output archiver. ## What This Skill Does OpenClaw's native dreamer reads `memory/YYYY-MM-DD.md` files — raw session logs containing noise, errors, and real insights all mixed together. This skill: 1. **Pre-dream feed** — queries SQL for high-importance memories, writes a clean daily file 2. **Post-dream archiver** — stores dream outputs (light/REM/deep) to SQL tables durably 3. **File cleanup** — prunes dream output files older than N days The OpenClaw dreamer runs unchanged. This skill wraps around it. ## Installation ```bash git clone https://github.com/High-Falootin/openclaw-SQL-dreamer.git cd openclaw-SQL-dreamer pip install -r requirements.txt cp config/example.yml config/config.yml # Edit config/config.yml python sql/migrate.py ``` ## Configuration Reference All settings live in `config/config.yml` (never committed — see `.gitignore`). | Field | Type | Default | Description | |---|---|---|---| | `sql.server` | string | — | SQL Server hostname | | `sql.database` | string | — | Database name | | `sql.username` | string | — | SQL username | | `sql.password` | string | — | Via env `SQL_PASSWORD` | | `corpus.importance_threshold` | int | 7 | Min importance (1-10) for dream corpus | | `corpus.lookback_days` | int | 2 | Days back to pull memories | | `dreaming.workspace_dir` | string | — | OpenClaw workspace root | | `dreaming.archive_after_days` | int | 7 | Days before dream files are pruned | | `confluence.enabled` | bool | false | Enable Confluence publishing | ## Scripts | Script | When | What | |---|---|---| | `scripts/pre_dream_sql_feed.py` | 3:00 AM (before dream) | Queries SQL, writes clean memory file | | `scripts/post_dream_archiver.py` | 4:00 AM (after dream) | Archives dream outputs to SQL, prunes old files | | `scripts/confluence_dream_publisher.py` | 4:30 AM (optional) | Pushes wiki syntheses to Confluence | ## SQL Tables Created Run `python sql/migrate.py` to create: - `dreams.DreamCorpus` — memories queued for each cycle - `dreams.DreamLight` — light sleep candidates - `dreams.DreamREM` — REM themes and reflections - `dreams.DreamDeep` — deep sleep promotions ## Crontab Example ```cron # Pre-dream SQL feed (30 min before your OpenClaw dream cron) 0 7 * * * python /path/to/scripts/pre_dream_sql_feed.py # Post-dream archiver (1 hour after dream cycle) 0 8 * * * python /path/to/scripts/post_dream_archiver.py # Confluence publisher (optional) 30 8 * * * python /path/to/scripts/confluence_dream_publisher.py ``` ## Security This is a public repo. **Never commit:** - `.env` files - `config/config.yml` - Passwords, tokens, API keys Use environment variables for all secrets: ```bash export SQL_PASSWORD="..." export CONFLUENCE_API_TOKEN="..." ``` ## Updated Crontab (after HFTC-28) ```cron # Pre-dream SQL feed (30 min before dream cycle) 0 7 * * * python /path/to/scripts/pre_dream_sql_feed.py # Post-dream archiver + phase signal reconciler (1 hr after dream) 0 8 * * * python /path/to/scripts/post_dream_archiver.py 5 8 * * * python /path/to/scripts/phase_signal_reconciler.py # Confluence publisher (optional, 90 min after dream) 30 8 * * * python /path/to/scripts/confluence_dream_publisher.py ```

path: GETTING_STARTED.md
content: # GETTING STARTED: SQL Connector + SQL Memory + SQL Dreamer

**For:** Users who want to use all three skills in a standalone OpenClaw installation  
**Time Required:** 15 minutes setup + 5 minutes configuration  
**Prerequisites:** OpenClaw installed, SQL Server (cloud or local), basic CLI knowledge

---

## Overview: How These 3 Skills Work Together

```
┌─────────────────────────────────────────────────────────────────┐
│                                                                 │
│  Your OpenClaw agents                                           │
│  ↓                                                              │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ SQL Memory (clawbot-sql-memory)                          │   │
│  │ ├─ Stores: facts, decisions, incidents, lessons learned │   │
│  │ ├─ Table: memory.Memories                               │   │
│  │ └─ Uses: SQL Connector to read/write                     │   │
│  └──────────────────────────────────────────────────────────┘   │
│  ↓ (daily at 3:00 AM)                                           │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ SQL Dreamer (openclaw-sql-dreamer)                       │   │
│  │ ├─ Reads: high-importance memories from SQL Memory      │   │
│  │ ├─ Runs: OpenClaw's native dreaming pipeline            │   │
│  │ ├─ Stores: dream results back to SQL                    │   │
│  │ └─ Uses: SQL Connector for all I/O                       │   │
│  └──────────────────────────────────────────────────────────┘   │
│  ↓ (optional: publish to Confluence)                            │
│  Confluence (Dream reports)                                     │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Foundation: SQL Connector (clawbot-sql-connector)
├─ Handles: All database communication
├─ Supports: Cloud (Azure, site4now) or local SQL Server
└─ Used by: SQL Memory and SQL Dreamer
```

**Key insight:** SQL Connector is the foundation. SQL Memory uses it. SQL Dreamer depends on both.

---

## Step 1: Install SQL Connector

The foundation — handles all database connections for the other two skills.

```bash
# Install via ClawHub
clawhub install sql-connector

# Or manually
pip install clawbot-sql-connector pymssql python-dotenv
```

### Configure SQL Connector

In your OpenClaw workspace root, create a `.env` file:

```bash
cd ~/.openclaw/workspace
# Or wherever your OpenClaw workspace is
cp .env.example .env
```

Edit `.env` and add SQL Server credentials:

**If using Cloud SQL Server (Azure, site4now, etc.):**
```env
SQL_CLOUD_SERVER=your-server.database.windows.net
SQL_CLOUD_DATABASE=your_database_name
SQL_CLOUD_USER=your_username
SQL_CLOUD_PASSWORD=your_password
SQL_DEFAULT_BACKEND=cloud
```

**If using Local SQL Server (on-prem):**
```env
SQL_LOCAL_SERVER=10.0.0.110
SQL_LOCAL_DATABASE=your_database_name
SQL_LOCAL_USER=your_username
SQL_LOCAL_PASSWORD=your_password
SQL_DEFAULT_BACKEND=local
```

### Verify Connection

```bash
python3 << 'PYTHON'
from sql_connector import get_connector

db = get_connector()
if db.ping():
    print("✅ Connected to SQL Server")
else:
    print("❌ Connection failed")
PYTHON
```

---

## Step 2: Install SQL Memory

Persistent memory for your agents. Stores everything they learn.

```bash
clawhub install sql-memory
```

### Create Database Schema

SQL Memory needs tables in your SQL Server. Create them once:

**Option A (Automatic — Recommended):**
```bash
# Find where sql-memory installed
pip show clawbot-sql-memory | grep Location
# Output example: Location: /home/username/.venv/lib/python3.10/site-packages/clawbot_sql_memory

# Go there and run setup
cd /path/from/above/clawbot_sql_memory
python3 setup_schema.py
```

**Option B (Manual — If Option A fails):**

Open your SQL Server (SSMS, Azure Data Studio, or sqlcmd) and run:

```sql
CREATE SCHEMA memory;
GO

CREATE TABLE memory.Memories (
    id          UNIQUEIDENTIFIER PRIMARY KEY DEFAULT newid(),
    category    NVARCHAR(100)  NOT NULL,
    key         NVARCHAR(255)  NOT NULL,
    content     NVARCHAR(MAX)  NOT NULL,
    importance  INT            DEFAULT 3,
    tags        NVARCHAR(500)  DEFAULT '',
    status      NVARCHAR(50)   DEFAULT 'active',
    created_at  DATETIME2      DEFAULT GETUTCDATE(),
    updated_at  DATETIME2      DEFAULT GETUTCDATE()
);

CREATE TABLE memory.TaskQueue (
    id          UNIQUEIDENTIFIER PRIMARY KEY DEFAULT newid(),
    agent       NVARCHAR(100)  NOT NULL,
    task_type   NVARCHAR(100)  NOT NULL,
    payload     NVARCHAR(MAX)  DEFAULT '',
    priority    INT            DEFAULT 5,
    status      NVARCHAR(50)   DEFAULT 'pending',
    retries     INT            DEFAULT 0,
    model_hint  NVARCHAR(100)  DEFAULT '',
    created_at  DATETIME2      DEFAULT GETUTCDATE(),
    updated_at  DATETIME2      DEFAULT GETUTCDATE(),
    claimed_at  DATETIME2      NULL,
    completed_at DATETIME2     NULL,
    error       NVARCHAR(MAX)  DEFAULT ''
);

CREATE TABLE memory.ActivityLog (
    id          UNIQUEIDENTIFIER PRIMARY KEY DEFAULT newid(),
    event_type  NVARCHAR(100)  NOT NULL,
    agent       NVARCHAR(100)  DEFAULT '',
    description NVARCHAR(MAX)  DEFAULT '',
    metadata    NVARCHAR(MAX)  DEFAULT '',
    created_at  DATETIME2      DEFAULT GETUTCDATE()
);
```

### Verify Setup

```bash
python3 << 'PYTHON'
from sql_memory import SQLMemory

mem = SQLMemory('cloud')  # or 'local' if using local server
mem.remember(
    category='facts',
    key='test_memory',
    content='This is a test memory',
    importance=5
)

result = mem.recall(category='facts', key='test_memory')
if result:
    print("✅ SQL Memory working")
else:
    print("❌ Could not retrieve memory")
PYTHON
```

---

## Step 3: Install SQL Dreamer

Automated dreaming powered by SQL-backed memories.

```bash
clawhub install sql-dreamer
```

### Create Dream Schema

SQL Dreamer needs its own tables. Create them once:

```bash
# Find where sql-dreamer installed
pip show openclaw-sql-dreamer | grep Location
# Output example: Location: /home/username/.venv/lib/python3.10/site-packages/openclaw_sql_dreamer

cd /path/from/above/openclaw_sql_dreamer
python3 sql/migrate.py
```

### Configure SQL Dreamer

Create configuration file:

```bash
cd ~/.openclaw/workspace
mkdir -p config
cp /path/to/sql_dreamer/config/example.yml config/sql_dreamer.yml
```

Edit `config/sql_dreamer.yml`:

```yaml
sql:
  # Use same credentials as .env file
  backend: "cloud"  # or "local"
  # sql_connector will read from .env automatically

corpus:
  importance_threshold: 7    # Only remember items score ≥ 7 (1-10 scale)
  lookback_days: 2            # Look back 2 days for memories

dreaming:
  workspace_dir: "/home/username/.openclaw/workspace"  # YOUR WORKSPACE PATH
  phases:
    light:
      enabled: true
    rem:
      enabled: true
    deep:
      enabled: true
  archive_after_days: 7       # Delete dream files after 7 days

confluence:
  enabled: false              # Set to true to publish dreams to Confluence
  # (See section below for Confluence setup)
```

### Find Your Workspace Path

```bash
# Find your OpenClaw workspace
which openclaw
# Output: /usr/local/bin/openclaw (or similar)

openclaw status
# Output will show: repo=/home/username/.openclaw/workspace
# Use that path in the config above
```

### Verify Schema Creation

```bash
python3 << 'PYTHON'
from sql_connector import get_connector

db = get_connector()
tables = db.query("SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='dreams'")
if tables:
    print("✅ Dream tables created")
    for row in tables:
        print(f"  - {row['TABLE_NAME']}")
else:
    print("❌ Dream tables not found")
PYTHON
```

---

## Step 4: Schedule Dream Cycles (Crontab)

Add these lines to your crontab to run dreams automatically:

```bash
crontab -e
```

Paste these lines (adjust times to your preference):

```cron
# Pre-dream: Curate memories from SQL (3:00 AM)
0 3 * * * /usr/bin/python3 ~/.openclaw/workspace/scripts/pre_dream_sql_feed.py >> ~/.openclaw/logs/pre_dream.log 2>&1

# Post-dream: Archive dream outputs (4:00 AM, 1 hour after native dreamer)
0 4 * * * /usr/bin/python3 ~/.openclaw/workspace/scripts/post_dream_archiver.py >> ~/.openclaw/logs/post_dream.log 2>&1

# (Optional) Confluence publisher: Publish dreams to Confluence (4:30 AM)
# 30 4 * * * /usr/bin/python3 ~/.openclaw/workspace/scripts/confluence_dream_publisher.py >> ~/.openclaw/logs/confluence.log 2>&1
```

### Find Your Python Path

```bash
which python3
# Output: /usr/bin/python3 (use this in crontab)
```

### Find Script Paths

After installing via ClawHub, scripts are installed to:

```bash
pip show openclaw-sql-dreamer | grep Location
# Location: /home/username/.venv/lib/python3.10/site-packages/openclaw_sql_dreamer

# Scripts are in: /path/from/above/scripts/
```

---

## Step 5: Verify Everything Works

### Test 1: Can SQL Connector connect?

```bash
python3 << 'PYTHON'
from sql_connector import get_connector
db = get_connector()
print("✅ SQL Connector OK" if db.ping() else "❌ SQL Connector FAILED")
PYTHON
```

### Test 2: Can SQL Memory store/retrieve?

```bash
python3 << 'PYTHON'
from sql_memory import SQLMemory
mem = SQLMemory('cloud')
mem.remember(category='test', key='verify', content='Works!', importance=7)
result = mem.recall(category='test', key='verify')
print("✅ SQL Memory OK" if result else "❌ SQL Memory FAILED")
PYTHON
```

### Test 3: Can SQL Dreamer see the tables?

```bash
python3 << 'PYTHON'
from sql_connector import get_connector
db = get_connector()
tables = db.query("SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA IN ('memory', 'dreams')")
print(f"✅ Found {len(tables)} tables:" if tables else "❌ No tables found")
for row in tables:
    print(f"  - {row['TABLE_NAME']}")
PYTHON
```

---

## What Happens on First Run?

After setup, here's what to expect:

**Day 1:** 
- You have empty SQL Memory (no memories yet)
- First dream cycle runs but finds nothing to dream about
- Dream output files are created but mostly empty
- ✅ This is normal

**Days 2-7:**
- Your agents store memories in SQL Memory
- Each night, dreams include those memories
- Dream quality improves as memory grows
- ✅ This is expected

**After 1 week:**
- Dreams should show patterns (themes, lasting truths)
- First synthetic memories promoted to long-term storage
- System is working as designed
- ✅ This is success

---

## Troubleshooting

### "Connection refused" or "Cannot connect to SQL Server"

**Check:**
1. SQL Server is running and accessible
2. `.env` credentials are correct
3. Firewall allows connection on port 1433 (SQL Server default)
4. Network can reach the server

```bash
# Test connection manually
python3 -c "from sql_connector import get_connector; get_connector().ping()" && echo "OK" || echo "FAILED"
```

### "Table not found" or "Schema not found"

**Check:**
1. Did you run `setup_schema.py` for SQL Memory?
2. Did you run `python3 sql/migrate.py` for SQL Dreamer?
3. Do the tables exist in your SQL Server?

```bash
# List all tables
python3 << 'PYTHON'
from sql_connector import get_connector
db = get_connector()
tables = db.query("SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLES ORDER BY TABLE_SCHEMA, TABLE_NAME")
for row in tables:
    print(f"{row['TABLE_SCHEMA']}.{row['TABLE_NAME']}")
PYTHON
```

### "Script not found" in crontab

**Check:**
1. Find the actual script path: `pip show openclaw-sql-dreamer | grep Location`
2. Update crontab with full path: `/full/path/to/scripts/pre_dream_sql_feed.py`

### Dreams are always empty

**Expected if:**
- First run (no memories in SQL Memory yet)
- Memory importance is too high (raise threshold in config)

**Check:**
1. Do you have memories in SQL Memory? Query: `SELECT COUNT(*) FROM memory.Memories`
2. Are their importance scores ≥ 7? Check `importance` column
3. Lower threshold in `sql_dreamer.yml`: `importance_threshold: 5`

### Logs show errors

Check log files:

```bash
tail -50 ~/.openclaw/logs/pre_dream.log
tail -50 ~/.openclaw/logs/post_dream.log
```

---

## Next Steps

1. **Read individual skill READMEs** for advanced configuration
   - clawbot-sql-connector: Advanced retry logic, multi-tenant setup
   - clawbot-sql-memory: Semantic search, rollups, activity logging
   - openclaw-sql-dreamer: Confluence publishing, custom synthesizers

2. **Integrate with your agents** — use SQL Memory to store agent findings
   ```python
   from sql_memory import SQLMemory
   mem = SQLMemory('cloud')
   mem.remember(category='findings', key='research_123', content='...', importance=8)
   ```

3. **Set up Confluence publishing** (optional) — share dreams with team
   - See openclaw-sql-dreamer README → Confluence Integration

---

## Support

- **SQL Connector Issues:** https://github.com/VeXHarbinger/clawbot-sql-connector/issues
- **SQL Memory Issues:** https://github.com/VeXHarbinger/clawbot-sql-memory/issues
- **SQL Dreamer Issues:** https://github.com/High-Falootin/openclaw-sql-dreamer/issues

---

**Last updated:** 2026-04-28  
**Version:** 1.0 (stable for production use)


path: README.md
content: # openclaw-sql-dreamer

> An OpenClaw skill that replaces file-based memory ingestion with SQL-backed storage, while preserving full dreaming pipeline compatibility.

---

## Why This Exists

OpenClaw's native `memory-core` dreaming system reads from `memory/YYYY-MM-DD.md` files — flat text logs that contain everything: debug noise, error traces, real decisions, and meaningful insights all mixed together. The dreamer can't distinguish signal from noise without a priority filter.

This skill solves that by:

1. **Feeding the dreamer from SQL** — queries your database for high-importance memories (configurable threshold) and writes a clean daily memory file before the dream cycle runs.
2. **Archiving dream outputs back to SQL** — after each dream cycle, structured dream data (light/REM/deep phases) is stored durably in SQL instead of living only as files.
3. **Keeping the filesystem lean** — dream output files older than N days are pruned automatically.
4. **Providing a stable foundation** for future wiki-to-Confluence publishing.

The OpenClaw dreamer still runs exactly as designed. This skill is a wrapper around it, not a replacement.

---

## Architecture

```
                ┌─────────────────────────────────┐
                │         SQL Database              │
                │   memory.Memories (importance)    │
                │   dreams.DreamCorpus              │
                │   dreams.DreamLight               │
                │   dreams.DreamREM                 │
                │   dreams.DreamDeep                │
                └────────┬─────────────┬────────────┘
                         │             │
                [3:00 AM]│             │[4:00 AM]
                pre_dream│             │post_dream
                sql_feed │             │archiver
                         ▼             │
                memory/YYYY-MM-DD.md   │
                (clean, curated)        │
                         │             │
                [3:30 AM]│             │
                ┌────────▼─────────┐   │
                │  OpenClaw native │   │
                │  dream cycle     │   │
                │  (unchanged)     │   │
                └────────┬─────────┘   │
                         │             │
                memory/dreaming/        │
                ├── light/YYYY-MM-DD.md │
                ├── rem/YYYY-MM-DD.md   │
                └── deep/YYYY-MM-DD.md  │
                         └─────────────┘
                                       │
                              [4:30 AM]│ (optional)
                              confluence_publisher
                                       │
                                       ▼
                              Confluence Memory Palace
```

---

## Prerequisites

- **OpenClaw** installed and configured
- **SQL Connector skill** (v2.0.0+, **required**) — the secure SQL Server bridge this skill depends on
  - Install via: `clawhub install sql-connector` (recommended) or `pip install clawbot-sql-connector`
- **SQL Memory skill** (recommended) — provides the `memory.Memories` table structure this skill reads from
  - Install via: `clawhub install sql-memory`
  - Without it, you'll need to create `memory.Memories` manually or populate via another method
- **SQL Server** (on-prem, cloud Azure SQL, site4now, etc.) — accessible from your network
- **Python** 3.10+
- `.env` file with database credentials (see Configuration)

**Note:** `memory-core` is OpenClaw's native dreaming system (built-in to OpenClaw). This skill wraps it — no separate install needed.

---

## Quick Start

### Option 1: Install via ClawHub (recommended)

```bash
# Install this skill and its dependencies
clawhub install sql-dreamer
clawhub install sql-connector   # required — SQL Server bridge
clawhub install sql-memory       # recommended — SQL-backed memory tables
```

### Option 2: Clone from GitHub

```bash
# 1. Clone the repo
git clone https://github.com/High-Falootin/openclaw-SQL-dreamer.git
cd openclaw-SQL-dreamer

# 2. Install dependencies (skip if running inside OpenClaw — already installed)
pip install -r requirements.txt

# 3. Copy and fill in config
cp config/example.yml config/config.yml
# Edit config/config.yml with your SQL credentials and settings

# 4. Run schema migration
python sql/migrate.py

# 5. Test the pre-dream feed
python scripts/pre_dream_sql_feed.py --dry-run

# 6. Add to crontab (adjust for your timezone)
# See: Crontab Integration section below
```

---

## Configuration

Copy `config/example.yml` and fill in your values. **Never commit `config/config.yml`** — it's in `.gitignore`.

```yaml
# config/example.yml — copy to config/config.yml and fill in values

sql:
  server: "your-sql-server-hostname"   # e.g. 10.0.0.110 or sql.example.com
  database: "Oblio_Memories"            # your database name
  username: "your_sql_username"         # SQL auth username
  password: ""                          # leave blank, set via env SQL_PASSWORD
  # Alternatively, use a full connection string:
  # connection_string: "DRIVER={ODBC Driver 17 for SQL Server};SERVER=...;"

corpus:
  importance_threshold: 7     # Min importance score to include in dream corpus (1-10)
  lookback_days: 2            # How many days back to pull memories

dreaming:
  workspace_dir: "/path/to/.openclaw/workspace"  # OpenClaw workspace root
  phases:
    light:
      enabled: true
    rem:
      enabled: true
    deep:
      enabled: true
  archive_after_days: 7       # Delete dream .md files older than this

confluence:
  enabled: false              # Set true to enable Confluence publishing
  domain: ""                  # e.g. yourorg.atlassian.net
  email: ""                   # Atlassian account email
  api_token: ""               # leave blank, set via env CONFLUENCE_API_TOKEN
  space_key: ""               # Confluence space key
  parent_page_id: ""          # Parent page ID for Memory Palace
```

**Environment variables (override config, never commit values):**

```bash
SQL_PASSWORD=your_sql_password
CONFLUENCE_API_TOKEN=your_token
```

---

## Crontab Integration

Add these entries to your crontab (`crontab -e`). Adjust timezone as needed (shown in EDT = UTC-4):

```cron
# Pre-dream SQL feed: populate clean memory file from SQL (30 min before dream cycle)
0 7 * * * /path/to/python /path/to/openclaw-SQL-dreamer/scripts/pre_dream_sql_feed.py >> /var/log/oblio/pre_dream.log 2>&1

# OpenClaw native dream cycle: 3:30 AM EDT (07:30 UTC) — configured in openclaw.json
# This runs automatically via OpenClaw's memory-core cron. No crontab entry needed.

# Post-dream archiver: archive outputs to SQL + cleanup (1 hr after dream cycle)
0 8 * * * /path/to/python /path/to/openclaw-SQL-dreamer/scripts/post_dream_archiver.py >> /var/log/oblio/post_dream.log 2>&1

# Confluence publisher (optional, runs after archiver)
30 8 * * * /path/to/python /path/to/openclaw-SQL-dreamer/scripts/confluence_dream_publisher.py >> /var/log/oblio/confluence.log 2>&1
```

---

## SQL Schema Overview

All tables live in the `dreams` schema. Run `sql/migrate.py` to create them.

| Table | Purpose |
|---|---|
| `dreams.DreamCorpus` | Curated memories queued for each dream cycle (filtered from `memory.Memories`) |
| `dreams.DreamLight` | Light sleep candidates and phase signal equivalents |
| `dreams.DreamREM` | REM sleep themes and pattern reflections |
| `dreams.DreamDeep` | Deep sleep promotions — durable recall entries |

---

## How the Three Dream Phases Work

OpenClaw's dreaming system runs three sequential phases each night:

### Light Sleep
Ingests recent daily memory files and session transcripts. Ranks entries by recency + recall frequency. Writes top N candidates as "Imported Insights" — things that surfaced recently and deserve attention.

**SQL mapping:** `dreams.DreamLight` stores each candidate with its confidence score, evidence source, recall count, and status (staged → promoted).

### REM Sleep
Processes short-term recall store. Finds patterns and themes across multiple sessions. Reflects on recurring concepts (what kept surfacing?). Synthesizes "Possible Lasting Truths."

**SQL mapping:** `dreams.DreamREM` stores theme entries with frequency counts, supporting evidence, and confidence levels.

### Deep Sleep
Promotes high-scoring short-term memories to durable (MEMORY.md / long-term). Uses weighted scoring: recall frequency × recency × query diversity. Only entries meeting minimum score + recall thresholds are promoted.

**SQL mapping:** `dreams.DreamDeep` stores promotions with scoring breakdown and promotion timestamp.

---

## Testing

### Unit Tests (71 tests)

Test individual components in isolation:

```bash
pytest tests/test_sql_connector.py
pytest tests/test_pre_dream_sql_feed.py
pytest tests/test_post_dream_archiver.py
pytest tests/test_phase_signal_reconciler.py
pytest tests/test_light_sleep_synthesizer.py
```

**Or run all unit tests:**

```bash
pytest tests/ -v
```

### Integration Tests (10 tests) — NEW in HFTC-35

Test the complete pipeline from empty state:

```bash
pytest tests/test_integration_pipeline.py -v
```

What these tests verify:

1. **Empty-state initialization** — skill can start from zero (no files, no DB state)
2. **Memory file creation** — pre_dream_sql_feed.py correctly creates memory/YYYY-MM-DD.md
3. **Full pipeline** — complete workflow: pre-feed → native dreamer → post-archiver
4. **Dream output validation** — dream files have correct structure (light/REM/deep phases)
5. **Noise filtering** — low-importance items excluded, high-importance preserved
6. **Confluence compatibility** — outputs ready for wiki publishing

These tests use a **mock dreamer** (tests/mock_dreamer.py) that simulates OpenClaw's native dreaming without requiring the actual dreamer to run. This allows us to test the full pipeline deterministically.

### Running All Tests

```bash
pytest tests/ -v  # All 81 tests (71 unit + 10 integration)
```

**Expected output:** 81 passed, 8 skipped (skipped tests require live DB connection)

### Test Coverage

- **Phase 1 (Scripts):** SQL connector, pre-dream feed, post-dream archiver, phase signal reconciler
- **Phase 2 (Synthesis):** Light sleep scoring, theme extraction, deep sleep promotion
- **Phase 3 (Integration):** Full pipeline from empty state, empty-state initialization, noise filtering, wiki compatibility

---

## Contributing

1. Fork the repo
2. Create a feature branch: `git checkout -b feature/your-feature`
3. Make changes with tests
4. Run `pytest tests/` — all tests must pass
5. Open a PR against `development` branch

**Never commit secrets, credentials, or personal data.** This is a public repository.



---

## Publishing & Versions

**Published to:** [clawhub.ai](https://clawhub.ai/skills/sql-dreamer) as `sql-dreamer`

**Install:** `clawhub install sql-dreamer`

**Current version:** 0.1.0

**Version policy:** Stable releases only. We run this in production nightly and publish after thorough validation.

**Compatibility:**
- Python 3.10+
- OpenClaw (memory-core dreaming enabled)
- SQL Server 2019+ (including Azure SQL)
- Requires: `sql-connector` skill v2.0.0+
- Recommends: `sql-memory` skill for `memory.Memories` table

---

## License

MIT — see [LICENSE](LICENSE)

---

## Related

- [OpenClaw](https://openclaw.ai) — the AI assistant platform this skill extends
- [ClawHub](https://clawhub.ai) — the OpenClaw skill registry
- [sql-connector](https://clawhub.ai/skills/sql-connector) — **required** — sealed SQL Server bridge (`clawhub install sql-connector`)
- [sql-memory](https://clawhub.ai/skills/sql-memory) — **recommended** — SQL-backed memory tables (`clawhub install sql-memory`)


path: SKILL_REFERENCE.md
content: # SQL Skills Reference Architecture

**For:** Developers and users who want to understand how SQL Connector, SQL Memory, and SQL Dreamer work together.

---

## Quick Reference

| Skill | Purpose | Depends On | Key Files |
|-------|---------|-----------|-----------|
| **sql-connector** | Foundation: connects to SQL Server | Nothing | `.env` (credentials) |
| **sql-memory** | Persistent memory store | sql-connector | `memory.Memories` table |
| **sql-dreamer** | Automated dream synthesis | sql-connector + sql-memory | `dreams.*` tables, `config/sql_dreamer.yml` |

---

## Data Flow Diagram

```
┌────────────────────────────────────────────────────────────────┐
│                                                                │
│                    Your OpenClaw Agents                        │
│                    (agents, codebase)                          │
│                                                                │
│                    Store facts/decisions in:                   │
│                                                                │
└──────────────────────┬─────────────────────────────────────────┘
                       │
                       ▼
    ┌─────────────────────────────────────┐
    │     SQL Memory API                  │
    │  from sql_memory import SQLMemory   │
    │  mem.remember(...)                  │
    │  mem.recall(...)                    │
    └──────────────┬──────────────────────┘
                   │
                   ▼
    ┌──────────────────────────────────────────┐
    │      SQL Connector                       │
    │  (handles credentials, retries, etc)     │
    └──────────────┬───────────────────────────┘
                   │
                   ▼
    ┌──────────────────────────────────────────┐
    │      SQL Server (Cloud or Local)         │
    │                                          │
    │      memory.Memories                     │
    │      memory.TaskQueue                    │
    │      memory.ActivityLog                  │
    │                                          │
    │      dreams.DreamCorpus                  │
    │      dreams.DreamLight                   │
    │      dreams.DreamREM                     │
    │      dreams.DreamDeep                    │
    │                                          │
    └──────────────────────────────────────────┘
                   ▲
                   │
    ┌──────────────┴──────────────┐
    │                             │
    │  [3:00 AM] SQL Dreamer      │
    │  pre_dream_sql_feed.py      │
    │  (queries Memories)         │
    │                             │
    │  [3:30 AM] Native Dreamer   │
    │  (runs via OpenClaw)        │
    │                             │
    │  [4:00 AM] SQL Dreamer      │
    │  post_dream_archiver.py     │
    │  (stores results)           │
    │                             │
    └─────────────────────────────┘
                   │
                   ▼ (optional)
            Confluence
         (Dream Portal)
```

---

## Dependency Chain

### Level 1: SQL Connector (Foundation)

```python
from sql_connector import get_connector

db = get_connector()  # Uses credentials from .env
# That's it. You can now query SQL Server.
```

**Requirements:**
- `.env` file with SQL Server credentials
- SQL Server running and accessible

**Provides:**
- `query()` — SELECT statements
- `execute()` — INSERT/UPDATE/DELETE
- `scalar()` — single value queries
- `ping()` — health check

---

### Level 2: SQL Memory (Depends on Level 1)

```python
from sql_memory import SQLMemory

mem = SQLMemory('cloud')  # Uses SQL Connector internally
mem.remember(
    category='facts',
    key='my_memory',
    content='Learned something',
    importance=7  # 1-10 scale
)

fact = mem.recall(category='facts', key='my_memory')
```

**Requirements:**
- SQL Connector installed and configured
- `memory.*` tables created (run setup_schema.py)

**Provides:**
- `remember()` — store a memory
- `recall()` — retrieve a memory
- `search_memories()` — full-text search
- `log_event()` — activity logging
- Task queueing for agents

**Data stored in:**
- `memory.Memories` — your facts/decisions/incidents
- `memory.TaskQueue` — agent work items
- `memory.ActivityLog` — audit trail

---

### Level 3: SQL Dreamer (Depends on Level 1 + Level 2)

```yaml
# config/sql_dreamer.yml

sql:
  backend: "cloud"  # Uses SQL Connector

corpus:
  importance_threshold: 7  # Read from memory.Memories

dreaming:
  workspace_dir: "/path/to/.openclaw/workspace"
  phases:
    light: true
    rem: true
    deep: true
```

**Requirements:**
- SQL Connector installed and configured
- SQL Memory installed with `memory.Memories` table populated
- `dreams.*` tables created (run sql/migrate.py)
- OpenClaw native dreamer installed and running

**Reads from:**
- `memory.Memories` (importance ≥ threshold)

**Writes to:**
- `dreams.DreamCorpus` (queued memories)
- `dreams.DreamLight` (light sleep candidates)
- `dreams.DreamREM` (REM sleep themes)
- `dreams.DreamDeep` (deep sleep promotions)

**Outputs to:**
- `memory/dreaming/light/YYYY-MM-DD.md`
- `memory/dreaming/rem/YYYY-MM-DD.md`
- `memory/dreaming/deep/YYYY-MM-DD.md`

---

## Configuration Map

### `.env` (SQL Connector)

Located: `~/.openclaw/workspace/.env`

```env
# For cloud SQL Server
SQL_CLOUD_SERVER=your-server.database.windows.net
SQL_CLOUD_DATABASE=your_database
SQL_CLOUD_USER=your_user@company
SQL_CLOUD_PASSWORD=xxxxx
SQL_DEFAULT_BACKEND=cloud

# Or for local SQL Server
SQL_LOCAL_SERVER=10.0.0.110
SQL_LOCAL_DATABASE=your_database
SQL_LOCAL_USER=oblio
SQL_LOCAL_PASSWORD=xxxxx
SQL_DEFAULT_BACKEND=local
```

### `config/sql_dreamer.yml` (SQL Dreamer)

Located: `~/.openclaw/workspace/config/sql_dreamer.yml`

```yaml
sql:
  backend: "cloud"  # Inherits from .env

corpus:
  importance_threshold: 7      # 1-10, higher = fewer but better memories
  lookback_days: 2             # How many days back to query

dreaming:
  workspace_dir: "/home/user/.openclaw/workspace"
  phases:
    light:
      enabled: true
      max_candidates: 20
    rem:
      enabled: true
    deep:
      enabled: true
  archive_after_days: 7        # Delete dream files after N days

confluence:
  enabled: false               # Set true to publish
  domain: "yourorg.atlassian.net"
  email: "[REDACTED_SECRET]"
  api_token: "xxxxx"           # From Atlassian account settings
  space_key: "YOUR_SPACE"      # Confluence space ID
  parent_page_id: "12345"      # Parent page for Memory Palace
```

---

## Database Schema Map

### SQL Memory Tables

```sql
-- Stores your memories
CREATE TABLE memory.Memories (
    id          UNIQUEIDENTIFIER,  -- Unique ID
    category    NVARCHAR(100),     -- 'facts', 'decisions', 'incidents', 'lessons_learned'
    key         NVARCHAR(255),     -- Unique key per category
    content     NVARCHAR(MAX),     -- The actual memory text
    importance  INT,               -- 1-10 scale
    tags        NVARCHAR(500),     -- Optional tags
    status      NVARCHAR(50),      -- 'active', 'archived'
    created_at  DATETIME2,         -- When stored
    updated_at  DATETIME2          -- When last modified
);

-- Work items for agents
CREATE TABLE memory.TaskQueue (
    id          UNIQUEIDENTIFIER,
    agent       NVARCHAR(100),     -- Agent name
    task_type   NVARCHAR(100),     -- Type of work
    payload     NVARCHAR(MAX),     -- JSON data
    priority    INT,               -- 1-10, lower = sooner
    status      NVARCHAR(50),      -- 'pending', 'claimed', 'done', 'failed'
    retries     INT,               -- How many times retried
    created_at  DATETIME2,
    claimed_at  DATETIME2,
    completed_at DATETIME2,
    error       NVARCHAR(MAX)      -- Error message if failed
);

-- Activity log (audit trail)
CREATE TABLE memory.ActivityLog (
    id          UNIQUEIDENTIFIER,
    event_type  NVARCHAR(100),     -- 'memory_stored', 'task_queued', 'error'
    agent       NVARCHAR(100),
    description NVARCHAR(MAX),
    metadata    NVARCHAR(MAX),     -- JSON
    created_at  DATETIME2
);
```

### SQL Dreamer Tables

```sql
-- Memories queued for this dream cycle
CREATE TABLE dreams.DreamCorpus (
    id          UNIQUEIDENTIFIER,
    dream_date  DATE,              -- Which night
    memory_id   UNIQUEIDENTIFIER,  -- From memory.Memories
    importance  INT,               -- Inherited from memory
    source      NVARCHAR(100),     -- 'memory' or 'session'
    created_at  DATETIME2
);

-- Light sleep phase results
CREATE TABLE dreams.DreamLight (
    id          UNIQUEIDENTIFIER,
    dream_date  DATE,
    snippet     NVARCHAR(MAX),     -- The insight text
    confidence  FLOAT,             -- 0.0-1.0
    recall_count INT,              -- How many times this surfaced
    status      NVARCHAR(50),      -- 'staged', 'promoted'
    created_at  DATETIME2
);

-- REM sleep phase results
CREATE TABLE dreams.DreamREM (
    id          UNIQUEIDENTIFIER,
    dream_date  DATE,
    theme_text  NVARCHAR(MAX),     -- The theme
    frequency   INT,               -- How often this appeared
    confidence  FLOAT,
    created_at  DATETIME2
);

-- Deep sleep phase results
CREATE TABLE dreams.DreamDeep (
    id          UNIQUEIDENTIFIER,
    dream_date  DATE,
    snippet     NVARCHAR(MAX),     -- The promoted memory
    ranked      INT,               -- Rank by score
    promoted    BIT,               -- Was this promoted?
    created_at  DATETIME2
);
```

---

## Environment Variable Map

| Variable | Used By | Default | Required |
|----------|---------|---------|----------|
| `SQL_CLOUD_SERVER` | sql-connector | — | Only if using cloud |
| `SQL_CLOUD_DATABASE` | sql-connector | — | Only if using cloud |
| `SQL_CLOUD_USER` | sql-connector | — | Only if using cloud |
| `SQL_CLOUD_PASSWORD` | sql-connector | — | Only if using cloud |
| `SQL_LOCAL_SERVER` | sql-connector | — | Only if using local |
| `SQL_LOCAL_DATABASE` | sql-connector | — | Only if using local |
| `SQL_LOCAL_USER` | sql-connector | — | Only if using local |
| `SQL_LOCAL_PASSWORD` | sql-connector | — | Only if using local |
| `SQL_DEFAULT_BACKEND` | sql-connector | `cloud` | No (optional) |
| `CONFLUENCE_API_TOKEN` | sql-dreamer | — | Only if using Confluence |

---

## Common Patterns

### Pattern 1: Store an Agent Finding

```python
from sql_memory import SQLMemory

mem = SQLMemory('cloud')

# Your agent discovers something important
mem.remember(
    category='facts',
    key='discovered_pattern_001',
    content='Customer X always prefers Y over Z',
    importance=8,  # High importance = included in dreams
    tags='customer_behavior,shopping'
)
```

### Pattern 2: Query Memories for Your Own Use

```python
from sql_memory import SQLMemory

mem = SQLMemory('cloud')

# Get a specific memory
fact = mem.recall(category='facts', key='discovered_pattern_001')
if fact:
    print(fact['content'])

# Search for memories matching a pattern
results = mem.search_memories('customer behavior')
for memory in results:
    print(f"{memory['key']}: {memory['content'][:100]}")
```

### Pattern 3: Manual Dream Cycle

If you don't want to use crontab:

```bash
# Run pre-dream feed
python3 ~/.openclaw/workspace/scripts/pre_dream_sql_feed.py

# Native OpenClaw dreamer runs (in memory-core)
# This is automatic

# Run post-dream archiver
python3 ~/.openclaw/workspace/scripts/post_dream_archiver.py
```

### Pattern 4: Check Dream Results

```python
from sql_connector import get_connector

db = get_connector()

# Get last night's light sleep
light = db.query("""
    SELECT snippet, confidence, recall_count
    FROM dreams.DreamLight
    WHERE dream_date = CAST(GETDATE() AS DATE)
    ORDER BY confidence DESC
""")

for candidate in light:
    print(f"{candidate['snippet']} (confidence: {candidate['confidence']})")
```

---

## Troubleshooting by Layer

### Layer 1: SQL Connector Issues

**"Cannot connect to SQL Server"**
- Check `.env` credentials
- Verify SQL Server is running: `ping your-server.database.windows.net`
- Check firewall: SQL Server uses port 1433

**"Connection timeout"**
- Server might be down or unreachable
- Check network connectivity
- Try manually connecting with SQL Server Management Studio

### Layer 2: SQL Memory Issues

**"Tables not found"**
- Did you run `setup_schema.py`?
- Verify tables exist: `SELECT * FROM memory.Memories`

**"Cannot store/retrieve memories"**
- Check Layer 1: Is SQL Connector working?
- Check permissions: Does your SQL user have CREATE/SELECT/INSERT rights?

### Layer 3: SQL Dreamer Issues

**"Dreams are always empty"**
- Check `memory.Memories` table has data
- Check `importance_threshold` in config (might be too high)
- Run `pre_dream_sql_feed.py` manually and check output

**"Cron jobs not running"**
- Check crontab: `crontab -l`
- Check logs: `tail ~/.openclaw/logs/pre_dream.log`
- Run scripts manually to see errors

---

## Version Compatibility

| Component | Version | Status |
|-----------|---------|--------|
| sql-connector | 2.0+ | Stable |
| sql-memory | 0.5+ | Alpha (stable for production, API may evolve) |
| sql-dreamer | 1.0+ | Stable |
| Python | 3.9+ | Required |
| SQL Server | 2019+ | Required (Azure SQL supported) |

---

## References

- **SQL Connector API:** See `clawbot-sql-connector` README
- **SQL Memory API:** See `clawbot-sql-memory` README
- **SQL Dreamer Config:** See `openclaw-sql-dreamer` README
- **Setup Guide:** See `GETTING_STARTED.md`

---

**Last updated:** 2026-04-28


path: config/example.yml
content: # sql configuration - EXAMPLE ONLY
# Copy this to config/config.yml and fill in your values
# NEVER commit config/config.yml (it is gitignored)

sql:
  server: "your-sql-server-hostname"    # e.g. 10.0.0.110 or sql.example.com
  database: "your_database_name"         # e.g. Oblio_Memories
  username: "your_sql_username"
  # password: leave blank, set env SQL_PASSWORD instead
  # connection_string: "DRIVER={ODBC Driver 17 for SQL Server};SERVER=...;"

corpus:
  importance_threshold: 7     # Min importance (1-10) to enter dream corpus
  lookback_days: 2             # How many days back to pull memories

dreaming:
  workspace_dir: "/absolute/path/to/.openclaw/workspace"
  phases:
    light:
      enabled: true
    rem:
      enabled: true
    deep:
      enabled: true
  archive_after_days: 7        # Prune dream .md files older than this

confluence:
  enabled: false
  domain: ""                   # e.g. yourorg.atlassian.net
  email: ""
  # api_token: leave blank, set env CONFLUENCE_API_TOKEN instead
  space_key: ""
  parent_page_id: ""           # Confluence page ID for Memory Palace parent


path: openclaw.skill.json
content: {
  "id": "openclaw-sql-dreamer",
  "name": "OpenClaw SQL Dreamer",
  "version": "0.1.0",
  "description": "SQL-backed memory corpus feeder and dream output archiver for OpenClaw. Replaces file-based memory ingestion with curated SQL queries, archives dream phase outputs (light/REM/deep) to SQL, and keeps the filesystem lean.",
  "author": "Oblio Falootin <[REDACTED_SECRET]>",
  "license": "MIT",
  "keywords": ["openclaw", "memory", "sql", "dreaming", "sql-server", "archive", "corpus"],
  "homepage": "https://github.com/High-Falootin/openclaw-SQL-dreamer",
  "repository": {
    "type": "git",
    "url": "https://github.com/High-Falootin/openclaw-SQL-dreamer.git"
  },
  "skills": ["./skills"],
  "configSchema": {
    "type": "object",
    "additionalProperties": false,
    "required": ["sql", "corpus", "dreaming"],
    "properties": {
      "sql": {
        "type": "object",
        "description": "SQL Server connection settings",
        "additionalProperties": false,
        "properties": {
          "server": { "type": "string", "description": "SQL Server hostname or IP" },
          "database": { "type": "string", "description": "Database name" },
          "username": { "type": "string", "description": "SQL auth username" },
          "password": { "type": "string", "description": "SQL auth password — prefer env SQL_PASSWORD" },
          "connection_string": { "type": "string", "description": "Full ODBC connection string (overrides server/database/username/password)" }
        }
      },
      "corpus": {
        "type": "object",
        "description": "Controls which memories are pulled into the dream corpus",
        "additionalProperties": false,
        "properties": {
          "importance_threshold": {
            "type": "integer",
            "minimum": 1,
            "maximum": 10,
            "default": 7,
            "description": "Minimum importance score (1-10) for a memory to enter the dream corpus"
          },
          "lookback_days": {
            "type": "integer",
            "minimum": 1,
            "default": 2,
            "description": "How many days back to pull memories for the nightly corpus"
          }
        }
      },
      "dreaming": {
        "type": "object",
        "description": "Dream cycle and archival settings",
        "additionalProperties": false,
        "properties": {
          "workspace_dir": {
            "type": "string",
            "description": "Absolute path to OpenClaw workspace root (where memory/ lives)"
          },
          "phases": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
              "light": { "type": "object", "properties": { "enabled": { "type": "boolean", "default": true } } },
              "rem":   { "type": "object", "properties": { "enabled": { "type": "boolean", "default": true } } },
              "deep":  { "type": "object", "properties": { "enabled": { "type": "boolean", "default": true } } }
            }
          },
          "archive_after_days": {
            "type": "integer",
            "minimum": 1,
            "default": 7,
            "description": "Delete dream output .md files older than this many days"
          }
        }
      },
      "confluence": {
        "type": "object",
        "description": "Optional Confluence publishing (disabled by default)",
        "additionalProperties": false,
        "properties": {
          "enabled": { "type": "boolean", "default": false },
          "domain":  { "type": "string", "description": "e.g. yourorg.atlassian.net" },
          "email":   { "type": "string" },
          "api_token": { "type": "string", "description": "Prefer env CONFLUENCE_API_TOKEN" },
          "space_key": { "type": "string" },
          "parent_page_id": { "type": "string", "description": "Confluence page ID for Memory Palace parent" }
        }
      }
    }
  }
}


path: requirements.txt
content: pyodbc>=4.0.0
PyYAML>=6.0


path: scripts/confluence_dream_publisher.py
content: """
scripts/confluence_dream_publisher.py — Post-dream Confluence wiki publisher

Runs AFTER post_dream_archiver.py (e.g., 4:30 AM).

What it does:
1. Reads wiki/main/syntheses/*.md from the OpenClaw wiki vault
2. Pushes each synthesis as a Confluence child page under Memory Palace parent
3. Creates page if it doesn't exist; updates if it does (keyed by title)

Requires confluence.enabled = true in config.yml.
Credentials via env: CONFLUENCE_API_TOKEN

Usage:
    python scripts/confluence_dream_publisher.py
    python scripts/confluence_dream_publisher.py --config /path/to/config.yml
    python scripts/confluence_dream_publisher.py --dry-run
"""

import sys
import os
import argparse
import re
import requests
from pathlib import Path
from base64 import b64encode

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

import yaml


def load_config(config_path: str) -> dict:
    with open(config_path) as f:
        return yaml.safe_load(f)


def md_to_confluence_storage(md_content: str) -> str:
    """
    Minimal markdown → Confluence storage format conversion.
    Handles headings, paragraphs, code blocks, bullets.
    For production, use a proper converter library.
    """
    lines = []
    in_code = False
    lang = "none"

    for line in md_content.splitlines():
        if line.startswith("```"):
            if not in_code:
                lang = line[3:].strip() or "none"
                lines.append(f'<ac:structured-macro ac:name="code"><ac:parameter ac:name="language">{lang}</ac:parameter><ac:plain-text-body><![CDATA[')
                in_code = True
            else:
                lines.append("]]></ac:plain-text-body></ac:structured-macro>")
                in_code = False
        elif in_code:
            lines.append(line)
        elif re.match(r"^#{1,6} ", line):
            level = len(re.match(r"^(#+)", line).group(1))
            text = line.lstrip("#").strip()
            lines.append(f"<h{level}>{text}</h{level}>")
        elif line.startswith("- ") or line.startswith("* "):
            lines.append(f"<li>{line[2:].strip()}</li>")
        elif line.startswith("|"):
            # Skip markdown tables — complex to convert; just use pre
            lines.append(f"<p><code>{line}</code></p>")
        elif line.strip():
            lines.append(f"<p>{line}</p>")
        else:
            lines.append("")

    return "\n".join(lines)


class ConfluencePublisher:
    """Minimal Confluence REST API client for dream wiki publishing."""

    def __init__(self, domain: str, email: str, api_token: str):
        self.base_url = f"https://{domain}/wiki/rest/api"
        creds = b64encode(f"{email}:{api_token}".encode()).decode()
        self.headers = {
            "Authorization": f"Basic {creds}",
            "Content-Type": "application/json",
            "Accept": "application/json",
        }

    def get_page_by_title(self, space_key: str, title: str) -> dict | None:
        """Search for a page by title in a space. Returns page dict or None."""
        r = requests.get(
            f"{self.base_url}/content",
            headers=self.headers,
            params={"spaceKey": space_key, "title": title, "expand": "version"},
        )
        r.raise_for_status()
        results = r.json().get("results", [])
        return results[0] if results else None

    def create_page(self, space_key: str, parent_id: str, title: str, body: str) -> dict:
        """Create a new child page under parent_id."""
        payload = {
            "type": "page",
            "title": title,
            "ancestors": [{"id": parent_id}],
            "space": {"key": space_key},
            "body": {"storage": {"value": body, "representation": "storage"}},
        }
        r = requests.post(f"{self.base_url}/content", headers=self.headers, json=payload)
        r.raise_for_status()
        return r.json()

    def update_page(self, page_id: str, title: str, body: str, version: int) -> dict:
        """Update an existing page."""
        payload = {
            "type": "page",
            "title": title,
            "version": {"number": version + 1},
            "body": {"storage": {"value": body, "representation": "storage"}},
        }
        r = requests.put(f"{self.base_url}/content/{page_id}", headers=self.headers, json=payload)
        r.raise_for_status()
        return r.json()


def run(config_path: str, dry_run: bool = False) -> None:
    cfg = load_config(config_path)
    cf_cfg = cfg.get("confluence", {})

    if not cf_cfg.get("enabled", False):
        print("confluence_dream_publisher: Confluence disabled in config. Set confluence.enabled = true to activate.")
        return

    domain = cf_cfg.get("domain", "")
    email = cf_cfg.get("email", "")
    api_token = os.environ.get("CONFLUENCE_API_TOKEN", cf_cfg.get("api_token", ""))
    space_key = cf_cfg.get("space_key", "")
    parent_page_id = cf_cfg.get("parent_page_id", "")

    if not all([domain, email, api_token, space_key, parent_page_id]):
        print("❌ Missing Confluence config: domain, email, CONFLUENCE_API_TOKEN, space_key, parent_page_id all required")
        return

    workspace_dir = Path(cfg["dreaming"]["workspace_dir"])
    syntheses_dir = workspace_dir / ".." / ".." / ".openclaw" / "wiki" / "main" / "syntheses"
    syntheses_dir = syntheses_dir.resolve()

    if not syntheses_dir.exists():
        print(f"⚠️  Syntheses directory not found: {syntheses_dir}")
        return

    synthesis_files = [f for f in syntheses_dir.glob("*.md") if f.name != "index.md"]
    print(f"confluence_dream_publisher: Found {len(synthesis_files)} synthesis files")

    if not synthesis_files:
        print("  No synthesis files to publish")
        return

    if dry_run:
        for f in synthesis_files:
            print(f"  [dry-run] would publish: {f.name}")
        return

    publisher = ConfluencePublisher(domain, email, api_token)
    published = 0
    updated = 0

    for f in synthesis_files:
        content = f.read_text(encoding="utf-8")
        # Extract title from frontmatter or first heading
        title_match = re.search(r"^title:\s*['\"]?(.+?)['\"]?\s*$", content, re.MULTILINE)
        h1_match = re.search(r"^# (.+)$", content, re.MULTILINE)
        title = (title_match.group(1) if title_match else None) or (h1_match.group(1) if h1_match else f.stem)

        # Strip frontmatter before converting
        body_md = re.sub(r"^---\n.*?\n---\n", "", content, flags=re.DOTALL).strip()
        body_cf = md_to_confluence_storage(body_md)

        existing = publisher.get_page_by_title(space_key, title)
        if existing:
            version = existing["version"]["number"]
            publisher.update_page(existing["id"], title, body_cf, version)
            print(f"  ✏️  Updated: {title}")
            updated += 1
        else:
            publisher.create_page(space_key, parent_page_id, title, body_cf)
            print(f"  ✅ Created: {title}")
            published += 1

    print(f"\n✅ Confluence publish complete: {published} created, {updated} updated")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Post-dream Confluence wiki publisher")
    parser.add_argument("--config", default="config/config.yml")
    parser.add_argument("--dry-run", action="store_true")
    args = parser.parse_args()

    run(args.config, dry_run=args.dry_run)


path: scripts/phase_signal_reconciler.py
content: """
scripts/phase_signal_reconciler.py — Sync dream phase signals from JSON files to SQL

OpenClaw owns the JSON state files (phase-signals.json, daily-ingestion.json,
short-term-recall.json). This script reads them after each dream cycle and writes
durable SQL copies — so phase signal history survives if the files are deleted or
the workspace is rebuilt.

Design principle:
  - OpenClaw reads/writes its JSON files unchanged (we never touch them)
  - This script reads the JSON files and UPSERTs into dreams.PhaseSignals
  - SQL is the backup/audit copy; JSON is the live state OpenClaw uses
  - Safe to run multiple times (idempotent UPSERT)

When to run:
  Add to crontab AFTER the dream cycle (e.g., 4:05 AM, between dream and archiver):
  5 8 * * * python /path/to/scripts/phase_signal_reconciler.py

Usage:
    python scripts/phase_signal_reconciler.py
    python scripts/phase_signal_reconciler.py --config /path/to/config.yml
    python scripts/phase_signal_reconciler.py --dry-run
"""

import sys
import os
import json
import argparse
from datetime import datetime, timezone
from pathlib import Path

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

import yaml
from src.sql_connector import SQLDreamerConnector


def load_config(config_path: str) -> dict:
    with open(config_path) as f:
        return yaml.safe_load(f)


def parse_phase_signals(workspace_dir: Path) -> list[dict]:
    """
    Parse memory/.dreams/phase-signals.json into SQL-ready rows.

    Returns list of dicts matching dreams.PhaseSignals columns.
    """
    path = workspace_dir / "memory" / ".dreams" / "phase-signals.json"
    if not path.exists():
        return []

    with open(path) as f:
        data = json.load(f)

    rows = []
    for key, entry in data.get("entries", {}).items():
        row = {
            "signal_key": key,
            "light_hits": entry.get("lightHits", 0),
            "rem_hits": entry.get("remHits", 0),
            "last_light_at": entry.get("lastLightAt"),
            "last_rem_at": entry.get("lastRemAt"),
            "updated_at": data.get("updatedAt", datetime.now(timezone.utc).isoformat()),
        }
        rows.append(row)

    return rows


def parse_daily_ingestion(workspace_dir: Path) -> list[dict]:
    """
    Parse memory/.dreams/daily-ingestion.json into a summary for SQL logging.
    Returns list of file entries as dicts.
    """
    path = workspace_dir / "memory" / ".dreams" / "daily-ingestion.json"
    if not path.exists():
        return []

    with open(path) as f:
        data = json.load(f)

    return [
        {"file_path": fp, "mtime_ms": info.get("mtimeMs"), "size": info.get("size")}
        for fp, info in data.get("files", {}).items()
    ]


def upsert_phase_signals(conn: SQLDreamerConnector, rows: list[dict], dry_run: bool) -> int:
    """
    UPSERT phase signal rows into dreams.PhaseSignals.
    Uses MERGE for idempotent upsert.
    Returns count of rows processed.
    """
    if not rows:
        return 0

    if dry_run:
        print(f"  [dry-run] would upsert {len(rows)} phase signal rows")
        return len(rows)

    sql = """
        MERGE dreams.PhaseSignals AS target
        USING (SELECT ? AS signal_key) AS source ON target.signal_key = source.signal_key
        WHEN MATCHED THEN
            UPDATE SET
                light_hits   = ?,
                rem_hits     = ?,
                last_light_at = ?,
                last_rem_at   = ?,
                updated_at    = ?
        WHEN NOT MATCHED THEN
            INSERT (signal_key, light_hits, rem_hits, last_light_at, last_rem_at, updated_at)
            VALUES (?, ?, ?, ?, ?, ?);
    """
    now = datetime.now(timezone.utc)

    processed = 0
    for row in rows:
        last_light = _parse_ts(row["last_light_at"])
        last_rem = _parse_ts(row["last_rem_at"])
        updated = _parse_ts(row["updated_at"]) or now

        params = (
            row["signal_key"],           # USING source.signal_key
            row["light_hits"],            # UPDATE light_hits
            row["rem_hits"],              # UPDATE rem_hits
            last_light,                   # UPDATE last_light_at
            last_rem,                     # UPDATE last_rem_at
            updated,                      # UPDATE updated_at
            row["signal_key"],            # INSERT signal_key
            row["light_hits"],            # INSERT light_hits
            row["rem_hits"],              # INSERT rem_hits
            last_light,                   # INSERT last_light_at
            last_rem,                     # INSERT last_rem_at
            updated,                      # INSERT updated_at
        )
        conn.execute(sql, params)
        processed += 1

    return processed


def _parse_ts(ts_str) -> datetime | None:
    """Parse ISO timestamp string to datetime, or return None."""
    if not ts_str:
        return None
    try:
        return datetime.fromisoformat(ts_str.replace("Z", "+00:00"))
    except (ValueError, AttributeError):
        return None


def run(config_path: str, dry_run: bool = False) -> None:
    cfg = load_config(config_path)
    workspace_dir = Path(cfg["dreaming"]["workspace_dir"])

    print(f"phase_signal_reconciler: {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}")
    print(f"  Workspace: {workspace_dir}")
    if dry_run:
        print("  MODE: dry-run")

    # Parse JSON state files
    phase_signals = parse_phase_signals(workspace_dir)
    daily_ingestion = parse_daily_ingestion(workspace_dir)

    print(f"\n  Phase signals found: {len(phase_signals)}")
    print(f"  Daily ingestion files tracked: {len(daily_ingestion)}")

    if not phase_signals:
        print("  ⚠️  No phase signals to sync — dream cycle may not have run yet")
        return

    with SQLDreamerConnector.from_config(config_path) as conn:
        count = upsert_phase_signals(conn, phase_signals, dry_run)
        print(f"  ✅ Phase signals synced to SQL: {count} rows")

        # Log daily ingestion summary as an activity record
        if not dry_run and daily_ingestion:
            for entry in daily_ingestion:
                sql = """
                    MERGE dreams.DreamCorpus AS target
                    USING (SELECT ? AS key_name, ? AS cycle_date) AS source
                        ON target.key_name = source.key_name AND target.cycle_date = source.cycle_date
                    WHEN NOT MATCHED THEN
                        INSERT (cycle_date, category, key_name, ingested_at)
                        VALUES (?, 'daily_ingestion_file', ?, ?);
                """
                today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
                conn.execute(sql, (
                    entry["file_path"], today,
                    today, entry["file_path"], datetime.now(timezone.utc)
                ))
            print(f"  ✅ Daily ingestion records synced: {len(daily_ingestion)} files")

    print("\n✅ phase_signal_reconciler complete")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Sync dream phase signals from JSON to SQL")
    parser.add_argument("--config", default="config/config.yml")
    parser.add_argument("--dry-run", action="store_true")
    args = parser.parse_args()
    run(args.config, dry_run=args.dry_run)


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)