Skill: ocbenji/soul-sync (version 0.8.1)

--- name: soulsync description: > Personalize your OpenClaw agent quickly and naturally. Detects your current setup level and adapts — full soulsyncing for new users, targeted enhancement for existing setups. Optionally connects Gmail, Calendar, GitHub, and social accounts to learn about you without a lengthy interview. Generates SOUL.md, USER.md, and seeds MEMORY.md. Trigger: user asks to set up their agent, personalize it, or says "get to know me". Also auto-triggers when SOUL.md and USER.md are empty/missing. metadata: openclaw: requires: bins: [] safety: data_handling: > All data processing happens locally on the user's machine. Raw imported data (emails, social posts, etc.) is processed in-memory and never persisted. Only synthesized personality files (SOUL.md, USER.md, MEMORY.md) are written to the user's workspace. No data is transmitted externally. network: > No outbound network requests except to user-authorized OAuth endpoints (Google, GitHub, Spotify) when the user explicitly opts into data imports. All OAuth flows require user consent and browser-based authorization. file_access: > Reads: SOUL.md, USER.md, MEMORY.md (to detect current personalization level). Writes: SOUL.md, USER.md, MEMORY.md (only after user reviews and approves content). No access to files outside the OpenClaw workspace directory. permissions: > No elevated permissions required. No system-level access. No daemon installation. All scripts run in the user's workspace context with standard OpenClaw tool permissions. user_consent: > Every data import is opt-in and requires explicit user confirmation. Users can skip any question or data source. All generated files are shown to the user for review before being written. Nothing is saved without approval. --- # Soulsync — Personalize Your Agent You are running the Soulsync skill. Your job is to get to know the user naturally and generate personalized workspace files that make the agent genuinely useful from day one. ## When to activate (Trigger Rules) **1. Explicit Triggers (User-initiated)** - User says "set me up", "personalize", "get to know me", etc. - User runs `/soulsync` **2. Auto-Trigger (Brand New Users)** - If `SOUL.md` and `USER.md` are empty or contain the default OpenClaw boilerplate, activate Soulsync on their very first message. - Start by saying: "Soulsync is active now and personalizing your agent to you!" Then jump straight into the first question. **3. Passive Mode (Existing Power Users)** - If they already have a customized `SOUL.md` and `USER.md`, do NOT hijack their conversation. Just start the passive learning engine in the background. - If they ask what Soulsync does after installing it, tell them: "Soulsync is active now and personalizing your agent to you! I'll learn your preferences naturally as we chat. If you want to do a full setup right now, just say `/soulsync`." ## Core principles 1. **Be a person, not a form.** This is a conversation, not a survey. Ask open-ended questions. React to what they say. Be curious, not mechanical. 2. **Be fast.** Most people want this done in 5 minutes, not 30. Respect their time. Don't ask questions you can infer from context. 3. **Don't be cheesy.** No "Let's embark on a journey of self-discovery!" energy. Be direct, warm, and real. Like meeting someone at a party who's genuinely interested in you. 4. **Use what you already know.** If data importers have run, you already know a lot. Confirm and refine, don't re-ask. 5. **Privacy is sacred.** All processing happens locally. Raw data is never stored. Only synthesized personality files remain. Be transparent about this. ## Step 1: Detect current state Run the detection script to assess the user's current personalization level: ```bash python3 skills/soulsync/lib/detector.py ``` This returns one of three levels: - `new` — Empty or missing SOUL.md/USER.md. Full soulsyncing needed. - `partial` — Some personalization exists but gaps remain. Targeted enhancement. - `established` — Well-configured. Offer data imports or refinement. ## Step 1b: Initialize the conversation engine Before starting the conversation, initialize the engine to load any import data: ```bash python3 skills/soulsync/lib/conversation.py init ``` This returns: - What import data is already available - Which personality dimensions are missing - Suggested first questions (multiple phrasings — pick what fits the vibe) - Progress tracking After each user response, record it: ```bash python3 skills/soulsync/lib/conversation.py record <dimension> "<user's response>" ``` This automatically extracts structured data from natural language (names, pronouns, timezones, goals, boundaries, etc.) and returns the next questions to ask. Check progress anytime: ```bash python3 skills/soulsync/lib/conversation.py status ``` When `can_finish` is true (all required dimensions covered), you can wrap up. Export final data for the synthesizer: ```bash python3 skills/soulsync/lib/conversation.py export ``` ## Step 1c: Adaptive engine — read the room After every user message, run the adaptive engine to understand their emotional state: ```bash python3 skills/soulsync/lib/adaptive.py analyze "<user's message>" ``` This returns: - **archetype**: open_book / cautious / guarded / efficient / playful - **approach**: specific guidance on pacing, tone, question style, and import strategy - **trust_trend**: warming_up / cooling_down / stable - **mood_trend**: improving / declining / stable - **do/don't lists**: specific behaviors for this user right now **CRITICAL: Follow the approach guidance.** If it says "DO NOT offer imports right now" — don't. If it says "slow down" — slow down. If it says "offer to finish early" — offer. When recommending data imports: ```bash python3 skills/soulsync/lib/adaptive.py recommend-import ``` This picks the right import for this user's comfort level and generates an appropriate pitch. When user declines something: ```bash python3 skills/soulsync/lib/adaptive.py decline <topic> python3 skills/soulsync/lib/adaptive.py import-decision <source> no ``` **Five user archetypes (they blend and shift):** 🟢 **Open Book** — Eager, shares freely, excited. Match their energy. Offer imports early. 🟡 **Cautious** — Willing but careful. Explain why before asking. Be transparent. 🔴 **Guarded** — Skeptical, minimal. Earn trust slowly. Don't push. Give easy outs. ⚡ **Efficient** — Just get it done. No fluff, no pleasantries, batch questions. 🎮 **Playful** — Having fun, testing you. Match humor, use unexpected angles. **Remember:** Users shift between these in real time. Someone guarded might warm up after seeing you handle their data respectfully. Someone excited might get fatigued after 5 minutes. Keep running the adaptive engine and ADJUST. ## Step 2: Adapt your approach ### For NEW users: First, check the adaptive engine to detect their archetype. Then use the matching flow strategy. **Open Book users** (eager, verbose, excited): > Open with a combo question. Let them run. They'll give you more than you asked for. > "Hey — I want to actually be useful, not generic. What's your name and what do you do?" > Combine questions: identity+work, communication+context, goals+interests, boundaries+relationships. > Should take 3-4 minutes because they'll volunteer info. **Cautious users** (willing but careful): > One topic at a time. Explain briefly why you're asking. > "I'd like to learn a bit about you. Skip anything you don't want to answer." > Don't combine questions. Give them control. Offer easy outs. **Guarded users** (minimal, skeptical): > Focus on functional info only. Don't push personal topics. > "Quick setup — just need a few basics. Nothing personal unless you want to share." > Only ask: name, timezone, communication preference, goals. Skip relationships, interests. > Done in 60 seconds. **Efficient users** (just get it done): > Rapid fire. Batch everything. > "Quick setup. Five questions, 60 seconds." > "Name, what you do, timezone?" → "Communication style, what do you want help with?" → "Hard nos, technical level?" > Done. **Playful users** (having fun, testing): > Match their energy. Unexpected angles. > "Alright let's figure out who you are. Three things you're obsessed with — go." > Make it a game, not an interview. ### For PARTIAL users: Read existing SOUL.md and USER.md first. Summarize what you know, then: > "I know you're [name], you're into [interests], [timezone]. But I don't really know > [missing dimensions]. Want to fill in the gaps?" Only ask about what's missing. Never re-ask what you already have. ### For ESTABLISHED users: > "Your profile looks solid. Want to connect accounts for deeper personalization, > or just do a quick refresh?" Offer data imports or a "life update" conversation. ### KEY RULES FOR ALL FLOWS: **1. USE WHAT THEY JUST TOLD YOU.** This is the most important rule. Every response must reference something from their previous answer. Never ask a generic question when you have context. Examples: BAD (robotic): > User: "I'm a software engineer and dad of 4" > Agent: "How do you prefer me to communicate with you?" GOOD (human): > User: "I'm a software engineer and dad of 4" > Agent: "Engineer and 4 kids — I bet you don't have time for long explanations. Want me to keep things short, or do you like the full picture sometimes?" BAD: > User: "I own a house in West Allis, WI" > Agent: "What timezone are you in?" GOOD: > User: "I own a house in West Allis, WI" > Agent: "West Allis — nice, so Central time. Got it." The question bank gives you TEMPLATES. Always personalize them with context from what the user already shared. **2. NEVER ask questions as a numbered list.** Weave them into conversation. **3. ACKNOWLEDGE before asking.** Every response should start with a brief reaction to what they said before transitioning to the next topic: - "Software engineer — cool, that helps me know what level to pitch things at." - "Dad of 4, respect. Bet your free time is pretty limited." - "No plums, no spicy. Easy." **4. COMBINE related questions** when the archetype supports it. **5. Read the room.** Short answers = speed up. Chatty = let them go. Personal details volunteered = they trust you, go deeper. **6. When they give you extra info, CAPTURE IT.** If they mention kids when you asked about work, note the family info even though you didn't ask for it. Don't waste it. **7. The adaptive engine tells you how to behave.** Follow its guidance on pacing and tone. **8. Aim for 3-5 minutes total.** Less for guarded/efficient. More only if they want it. **9. Sound like a person meeting someone at a party, not a form being filled out.** ## Step 3: Offer data imports (optional) If the user wants deeper personalization, offer to connect accounts: ``` I can learn a lot more about you by looking at: 📧 Email — communication style, priorities, who matters to you 📅 Calendar — your schedule, routines, commitments 🐙 GitHub — technical skills, projects, coding style 🐦 Twitter/X — interests, opinions, how you talk publicly 📘 Facebook — interests, groups, social connections (data export) 📸 Instagram — visual style, hashtags, engagement patterns (data export) 🎵 Spotify — music taste → personality insights 👥 Contacts — relationship network, social style 🍎 Apple — Safari bookmarks, iCloud contacts, ecosystem data 🔴 Reddit — true interests, opinions, communities (public profile) 🎥 YouTube — watch history, subscriptions, learning style (Google Takeout) 💼 LinkedIn — career trajectory, skills, professional network (data export) 🖥️ Local System — shell history, installed apps, git repos, bookmarks (zero auth!) Everything stays on your machine. I process it locally and only keep the personality insights, not the raw data. Want to connect any of these? ``` Run importers for each account they approve: ```bash python3 skills/soulsync/lib/importers/gmail.py python3 skills/soulsync/lib/importers/calendar.py python3 skills/soulsync/lib/importers/github.py python3 skills/soulsync/lib/importers/twitter.py <username> python3 skills/soulsync/lib/importers/facebook.py [/path/to/export] python3 skills/soulsync/lib/importers/instagram.py [/path/to/export] python3 skills/soulsync/lib/importers/spotify.py <spotify_user_id> python3 skills/soulsync/lib/importers/contacts.py [/path/to/contacts.vcf] python3 skills/soulsync/lib/importers/apple.py [/path/to/data] python3 skills/soulsync/lib/importers/reddit.py <username> python3 skills/soulsync/lib/importers/youtube.py [/path/to/takeout] python3 skills/soulsync/lib/importers/linkedin.py [/path/to/export] python3 skills/soulsync/lib/importers/local_system.py ``` Note: Facebook and Instagram require a data export download from the respective platforms. Guide the user through the export process if they want to use these sources. ## Step 4: Synthesize and write files After the conversation (and any imports), run the synthesizer: ```bash python3 skills/soulsync/lib/synthesizer.py ``` This generates: - `SOUL.md` — Personality, communication style, boundaries - `USER.md` — Name, background, preferences, technical details - `MEMORY.md` — Seed memories from imports and conversation **IMPORTANT:** Before writing, show the user what you're about to write and ask for confirmation. They should see and approve their own personality profile. > "Here's what I've put together. Take a look and tell me what to change:" > [show generated files] > "Anything wrong? Anything to add?" Only write files after approval. ## Step 5: Verify and close After writing files, do a quick verification: > "Alright, I should know you a lot better now. Quick test — [ask something that > demonstrates you learned about them]. Did I get that right?" Then close naturally: > "We're good. I'll keep learning as we go, but the basics are solid now. > You can run /soulsync anytime if things change." ## Data importer output format Each importer writes a JSON file to `/tmp/soulsync/` with structured insights: ```json { "source": "gmail", "processed_at": "2026-03-25T22:00:00Z", "insights": { "communication_style": "concise, uses bullet points, rarely formal", "key_contacts": ["name1", "name2"], "interests": ["bitcoin", "AI", "photography"], "work_patterns": "most active 9am-11pm, rarely emails weekends", "tone": "direct but friendly, uses humor" }, "confidence": 0.85, "items_processed": 500 } ``` ## File templates Use the templates in `skills/soulsync/templates/` as starting points, but personalize heavily based on what you learned. The templates are scaffolding, not fill-in-the-blank. ## Ongoing Personalization (After Initial Setup) Soulsync doesn't stop after the first conversation — but only if the user wants it. ### Opt-In (End of Initial Setup) At the end of the first Soulsync conversation, casually offer ongoing learning: > "By the way — want me to keep picking things up as we chat? I'll learn your > preferences naturally without asking a bunch of questions. You can turn it off anytime." ```bash python3 skills/soulsync/lib/followup.py opt-in # User says yes python3 skills/soulsync/lib/followup.py opt-out # User says no ``` If they opt out, respect it completely. Only respond to explicit corrections. If they change their mind later, `/soulsync ongoing on` or `/soulsync ongoing off`. ### Passive Learning (Always-On When Opted In) Every message gets a light scan for personality signals: ```bash python3 skills/soulsync/lib/followup.py passive "<user's message>" ``` This picks up things said in passing: - "I really love hiking" → queued as preference (unconfirmed) - "My wife Sarah is picking up the kids" → queued as relationship info (unconfirmed) - "I work at Tesla now" → queued as life change (unconfirmed) - "I hate mushrooms" → queued as dislike (unconfirmed) **CRITICAL: Nothing learned passively goes directly into profile files.** Everything is queued as "unconfirmed" and needs soft confirmation first. ### Soft Confirmations (The Secret Sauce) Don't ask "Hey, I noticed you mentioned hiking — do you like hiking?" That's creepy and robotic. Instead, USE the learned fact naturally and see if they go with it or correct you: **Example — learned they like hiking:** > Next time they ask about weekend plans or weather, say: > "Looks like good weather for a hike this weekend." > If they engage → confirmed. If they say "I don't hike" → rejected, remove it. **Example — learned partner's name is Sarah:** > Next time they mention their wife, naturally use the name: > "Is Sarah coming too?" > If they don't correct you → confirmed. **Example — learned they hate mushrooms:** > If food ever comes up: > "There's a good Thai place nearby — no mushrooms on the menu." > If they react positively → confirmed. ```bash python3 skills/soulsync/lib/followup.py check # Get next pending confirmation ``` ### Correction Handling (Always Active, Even When Opted Out) ```bash python3 skills/soulsync/lib/followup.py scan "<user's message>" ``` When they say "actually...", "that's wrong", update immediately and confirm briefly: > "Got it — updated." ### Life Change Detection ```bash python3 skills/soulsync/lib/followup.py scan "<user's message>" ``` When detected (new job, moved, etc.), don't send a standalone message. Wait for a natural moment to acknowledge it: > "How's the new gig at Tesla going?" (not "I detected a life change event") ### Scheduled Follow-ups (Subtle, Not Robotic) The follow-up system NEVER sends standalone check-in messages. Instead, it gives the agent instructions to weave updates into natural conversation: - **Day 3:** Next time user asks something, reference a profile fact to implicitly confirm it - **Week 2:** If user asks something where a data import would help, offer ONE import - **Monthly:** If profile info seems stale, gently update. If not, do nothing. ### Trust Progression - **Initial** → functional info only - **Comfortable** (2+ weeks, opted in) → start using passively learned facts - **Trusted** (corrections handled well, engagement consistent) → deeper dimensions unlock naturally ### Re-sync `/soulsync` anytime to redo the full flow, fill gaps, or update after major life changes. `/soulsync ongoing on` or `/soulsync ongoing off` to change the ongoing preference. ## Remember - You're not interrogating someone. You're getting to know them. - 5 minutes is the target. 10 is the max. - If someone says "just make something up based on what you see" — that's valid. Do it. - The best soulsyncing feels like it barely happened.

path: README.md
content: # Soulsync — Personalize Your OpenClaw Agent

**Make your AI agent actually know you.** Soulsync collapses weeks of "getting to know you" into minutes through natural conversation and optional data imports.

## The Problem

A fresh OpenClaw agent is generic. It doesn't know your name, your job, how you like to communicate, or what you care about. It takes days or weeks of chatting before it feels personal. Most people give up before that happens.

## The Solution

Soulsync detects your current setup and adapts:

- **New users** → Full guided soulsyncing conversation (5 minutes)
- **Partial setup** → Fills in the gaps, doesn't re-ask what it knows
- **Power users** → Connects data sources for deeper personalization

## Features

### 🗣️ Natural Conversation Engine
Not a survey. Not a form. A real conversation that picks up on your personality from *how* you answer, not just what you answer.

### 📧 Data Importers (Optional)
Connect accounts to skip the interview. All processing happens locally — raw data is never stored.

| Source | What It Learns | Auth Required |
|--------|---------------|---------------|
| **Gmail** | Communication style, key contacts, schedule patterns | Google OAuth |
| **Google Calendar** | Lifestyle patterns, work/life balance, schedule density | Google OAuth |
| **GitHub** | Technical skills, languages, project types | Token or public |
| **Twitter/X** | Interests, tone, engagement style | None (via Nitter) |
| **Facebook** | Interests, groups, social connections, posting style | Data export (JSON) |
| **Instagram** | Visual style, hashtags, engagement, interests | Data export (JSON) |
| **Spotify** | Music taste → personality traits | Spotify API key |
| **Contacts** | Relationship network, social style | Google OAuth or .vcf |
| **Apple** | Safari bookmarks, iCloud contacts, ecosystem signals | macOS or .vcf export |
| **Reddit** | True interests, opinions, communities, communication style | None (public) |
| **YouTube** | Watch history, subscriptions, content preferences, learning style | Google Takeout |
| **LinkedIn** | Career trajectory, skills, education, professional network | Data export |
| **Local System** | Shell history, installed apps, git repos, browser bookmarks | None (local scan) |

### 🔒 Privacy First
- All data processed by your local model (or your configured cloud model)
- Raw emails/tweets never leave your machine
- Only synthesized personality files are kept
- Raw import data deleted after processing

### 📝 What It Generates

| File | Purpose |
|------|---------|
| `SOUL.md` | Agent personality, communication style, boundaries |
| `USER.md` | Your name, background, interests, preferences |
| `MEMORY.md` | Seed memories from imports and conversation |

## Quick Start

```bash
# Install
npx clawhub@latest install soulsync

# Use — just chat with your agent
"Hey, let's get to know each other"

# Or explicitly
/soulsync
```

The skill auto-detects empty SOUL.md/USER.md and offers to help.

## How It Works

```
1. Detector scans your workspace → determines your level (new/partial/established)
2. Agent adapts conversation depth to what's needed
3. Optional: connect accounts for richer data
4. Synthesizer merges everything → generates personalized files
5. You review and approve before anything is written
```

## Requirements

- OpenClaw (any version)
- Python 3.10+
- Optional: Google OAuth credentials for Gmail/Calendar/Contacts
- Optional: GitHub token for private repo analysis
- Optional: Spotify API credentials

## Data Sources Setup

### Gmail & Calendar
Reuses existing OpenClaw Gmail OAuth if available. Otherwise:
1. Create OAuth credentials at [Google Cloud Console](https://console.cloud.google.com/apis/credentials)
2. Save `credentials.json` to your workspace
3. First run will prompt for browser auth

### GitHub
Uses your git credential store automatically. Or set `GITHUB_TOKEN` env var.

### Twitter/X
No setup needed — uses public Nitter instances. Works with any public Twitter profile.

### Spotify
1. Create an app at [Spotify Developer Dashboard](https://developer.spotify.com/dashboard)
2. Set `SPOTIFY_CLIENT_ID` and `SPOTIFY_CLIENT_SECRET` env vars

### Contacts
Uses Google People API (same OAuth as Gmail) or import a `.vcf` file.

## Examples

**Brand new user:**
> Agent: "Hey — I'm brand new here and so are you. Let's fix that. A few questions that'll help me actually be useful. Should take about 5 minutes."

**Returning user with gaps:**
> Agent: "I know you're ocbenji, you're into Bitcoin and AI, Central timezone. But I don't really know how you like to communicate or what your boundaries are. Want to fill that in?"

**Power user wanting deeper personalization:**
> Agent: "Your profile looks solid. I could connect to your email and GitHub to learn more automatically. Everything stays local."

## Contributing

Issues and PRs welcome. Built by [@ocbenji](https://github.com/ocbenji).

## License

MIT


path: lib/adaptive.py
content: #!/usr/bin/env python3
"""
Adaptive Personality Engine — Reads emotional signals, trust level, and engagement
to dynamically adjust the soulsyncing approach in real time.

This is the intelligence layer that sits between the conversation engine and the agent.
It answers: "How should I approach this person RIGHT NOW?"

Core philosophy: Meet people where they are, not where you want them to be.
"""
import os
import sys
import json
import re
from datetime import datetime, timezone

IMPORT_DIR = "/tmp/soulsync"
STATE_FILE = os.path.join(IMPORT_DIR, "adaptive_state.json")

# ─── User Archetypes ─────────────────────────────────────────────────
# Not boxes — spectrums. Users blend between these and shift over time.

ARCHETYPES = {
    "open_book": {
        "description": "Eager to share, excited about personalization, trusts quickly",
        "signals": ["long responses", "volunteers info unprompted", "asks what else to connect", "uses exclamation marks", "shares personal details freely"],
        "approach": {
            "pacing": "fast",
            "depth": "deep",
            "data_imports": "suggest early and enthusiastically",
            "question_style": "open-ended, let them run",
            "tone": "match their energy, be excited back",
            "skip_permission": True,  # Don't over-ask "is this okay?"
        },
    },
    "cautious": {
        "description": "Willing but careful. Needs to understand why before sharing.",
        "signals": ["asks 'why do you need that'", "medium-length responses", "selective sharing", "asks about privacy", "hedging language"],
        "approach": {
            "pacing": "moderate",
            "depth": "medium",
            "data_imports": "explain benefits clearly before offering",
            "question_style": "specific, with context for why you're asking",
            "tone": "professional but warm, transparent",
            "skip_permission": False,
        },
    },
    "guarded": {
        "description": "Skeptical, minimal sharing, may be testing boundaries",
        "signals": ["short responses", "deflects personal questions", "sarcasm", "'why' questions", "minimal punctuation", "one-word answers"],
        "approach": {
            "pacing": "slow",
            "depth": "surface only — earn deeper access",
            "data_imports": "don't offer until they ask or trust builds",
            "question_style": "low-pressure, give easy outs",
            "tone": "respect their space, no pushing",
            "skip_permission": False,
        },
    },
    "efficient": {
        "description": "Just wants it done. Not emotional about it — practical.",
        "signals": ["bullet points", "direct answers", "asks 'how long will this take'", "skips small talk", "technical language"],
        "approach": {
            "pacing": "fast",
            "depth": "whatever gets results",
            "data_imports": "present as efficiency gain, not sharing exercise",
            "question_style": "rapid-fire, yes/no when possible",
            "tone": "match their directness, no fluff",
            "skip_permission": True,
        },
    },
    "playful": {
        "description": "Having fun with it. Jokes, tests the AI, sees what happens.",
        "signals": ["jokes", "emojis", "testing boundaries", "asks weird questions", "roleplay", "memes"],
        "approach": {
            "pacing": "flexible — follow their lead",
            "depth": "medium — they'll share through play",
            "data_imports": "frame as 'want to see something cool?'",
            "question_style": "fun, unexpected angles",
            "tone": "playful back, match humor",
            "skip_permission": True,
        },
    },
}

# ─── Emotional State Detection ────────────────────────────────────────

class EmotionalSignals:
    """Analyze text for emotional and engagement signals."""
    
    @staticmethod
    def analyze(text):
        """Return a dict of emotional signals from a message."""
        text_lower = text.lower()
        words = text.split()
        word_count = len(words)
        
        signals = {
            "engagement_level": "medium",  # low, medium, high
            "trust_level": "neutral",       # resistant, neutral, open, enthusiastic
            "mood": "neutral",              # negative, neutral, positive, excited
            "verbosity": "medium",          # terse, medium, verbose
            "formality": "casual",          # formal, balanced, casual
            "openness": "moderate",         # closed, moderate, open
            "pace_preference": "normal",    # slow, normal, fast
            "raw_signals": [],
        }
        
        # ── Verbosity
        if word_count < 5:
            signals["verbosity"] = "terse"
            signals["raw_signals"].append("very_short_response")
        elif word_count < 15:
            signals["verbosity"] = "short"
        elif word_count > 80:
            signals["verbosity"] = "verbose"
            signals["raw_signals"].append("long_detailed_response")
        elif word_count > 40:
            signals["verbosity"] = "medium-long"
        
        # ── Mood / Emotion
        positive_markers = ["!", "love", "great", "awesome", "cool", "excited", "amazing", "perfect", "nice", "sweet", "hell yeah", "let's go", "😊", "🙌", "👍", "❤️", "🔥", "💯"]
        negative_markers = ["...", "ugh", "annoying", "hate", "don't want", "rather not", "no thanks", "nah", "meh", "whatever", "idk", "😒", "😑", "🙄"]
        
        pos_count = sum(1 for m in positive_markers if m in text_lower or m in text)
        neg_count = sum(1 for m in negative_markers if m in text_lower or m in text)
        
        if pos_count >= 3:
            signals["mood"] = "excited"
        elif pos_count > neg_count:
            signals["mood"] = "positive"
        elif neg_count > pos_count:
            signals["mood"] = "negative"
        
        # ── Trust indicators
        # High trust: shares personal details unprompted
        personal_sharing = any(phrase in text_lower for phrase in [
            "my wife", "my husband", "my partner", "my kid", "my family",
            "my salary", "my address", "personal", "private",
            "i struggle with", "honestly", "to be honest", "tbh",
            "here's my", "you can look at my",
        ])
        if personal_sharing:
            signals["trust_level"] = "open"
            signals["raw_signals"].append("shares_personal_unprompted")
        
        # Low trust: questioning, deflecting
        distrust_markers = [
            "why do you need", "what do you do with", "is this safe",
            "who can see", "i don't want to", "rather not", "skip",
            "that's private", "none of your business", "pass",
            "do i have to", "is this required",
        ]
        if any(m in text_lower for m in distrust_markers):
            signals["trust_level"] = "resistant"
            signals["raw_signals"].append("privacy_concerned")
        
        # Enthusiastic about sharing
        if any(m in text_lower for m in ["connect", "here's my", "you can check", "want to see", "i'll share", "link my", "go ahead and"]):
            signals["trust_level"] = "enthusiastic"
            signals["raw_signals"].append("eager_to_share")
        
        # ── Engagement level
        if word_count < 3 or text_lower in ["ok", "sure", "fine", "yes", "no", "k", "yep", "yeah"]:
            signals["engagement_level"] = "low"
            signals["raw_signals"].append("minimal_engagement")
        elif word_count > 50 or "?" in text:
            signals["engagement_level"] = "high"
            if "?" in text:
                signals["raw_signals"].append("asks_questions_back")
        
        # ── Openness to data imports
        if any(m in text_lower for m in ["connect", "link", "access", "scan", "import", "look at my"]):
            signals["openness"] = "open"
        elif any(m in text_lower for m in ["no thanks", "skip", "don't want", "not comfortable", "rather not"]):
            signals["openness"] = "closed"
        
        # ── Pace preference
        if any(m in text_lower for m in ["hurry", "quick", "fast", "how long", "just do it", "let's go", "next"]):
            signals["pace_preference"] = "fast"
        elif any(m in text_lower for m in ["take your time", "slow down", "wait", "hold on", "let me think"]):
            signals["pace_preference"] = "slow"
        
        # ── Formality
        formal_count = sum(1 for m in ["please", "thank you", "would you", "i would", "kindly", "appreciate"] if m in text_lower)
        casual_count = sum(1 for m in ["lol", "haha", "tbh", "ngl", "gonna", "wanna", "yo", "dude", "bro", "lmao", "bruh"] if m in text_lower)
        
        if formal_count > casual_count + 1:
            signals["formality"] = "formal"
        elif casual_count > formal_count:
            signals["formality"] = "casual"
        else:
            signals["formality"] = "balanced"
        
        return signals

# ─── Adaptive State Manager ───────────────────────────────────────────

class AdaptiveState:
    """Tracks and adapts to the user's emotional journey through soulsyncing."""
    
    def __init__(self):
        self.message_history = []  # List of (text, signals) tuples
        self.archetype_scores = {k: 0.0 for k in ARCHETYPES}
        self.current_approach = None
        self.trust_trajectory = []  # Track trust over time
        self.mood_trajectory = []
        self.declined_topics = set()  # Topics they've refused to discuss
        self.skipped_imports = set()  # Imports they've declined
        self.accepted_imports = set()
        self.comfort_topics = set()  # Topics they were enthusiastic about
        self.interaction_count = 0
        self.created_at = datetime.now(timezone.utc).isoformat()
    
    def process_message(self, text):
        """Analyze a new message and update adaptive state."""
        signals = EmotionalSignals.analyze(text)
        self.message_history.append({
            "text": text[:500],
            "signals": signals,
            "timestamp": datetime.now(timezone.utc).isoformat(),
        })
        self.interaction_count += 1
        
        # Track trajectories
        self.trust_trajectory.append(signals["trust_level"])
        self.mood_trajectory.append(signals["mood"])
        
        # Update archetype scores based on signals
        self._update_archetype_scores(signals)
        
        # Generate current approach
        self.current_approach = self._generate_approach(signals)
        
        return {
            "signals": signals,
            "archetype": self._get_dominant_archetype(),
            "approach": self.current_approach,
            "trust_trend": self._get_trust_trend(),
            "mood_trend": self._get_mood_trend(),
        }
    
    def _update_archetype_scores(self, signals):
        """Update archetype affinity scores based on latest signals."""
        # Open book signals
        if signals["verbosity"] in ["verbose", "medium-long"]:
            self.archetype_scores["open_book"] += 0.3
        if signals["trust_level"] in ["open", "enthusiastic"]:
            self.archetype_scores["open_book"] += 0.5
        if signals["mood"] in ["positive", "excited"]:
            self.archetype_scores["open_book"] += 0.2
        
        # Cautious signals
        if "privacy_concerned" in signals["raw_signals"]:
            self.archetype_scores["cautious"] += 0.5
        if signals["trust_level"] == "neutral":
            self.archetype_scores["cautious"] += 0.1
        
        # Guarded signals
        if signals["verbosity"] == "terse":
            self.archetype_scores["guarded"] += 0.3
        if signals["trust_level"] == "resistant":
            self.archetype_scores["guarded"] += 0.5
        if signals["mood"] == "negative":
            self.archetype_scores["guarded"] += 0.3
        if "minimal_engagement" in signals["raw_signals"]:
            self.archetype_scores["guarded"] += 0.3
        
        # Efficient signals
        if signals["pace_preference"] == "fast":
            self.archetype_scores["efficient"] += 0.4
        if signals["verbosity"] in ["short", "terse"] and signals["mood"] != "negative":
            self.archetype_scores["efficient"] += 0.2
        
        # Playful signals
        if signals["formality"] == "casual" and signals["mood"] in ["positive", "excited"]:
            self.archetype_scores["playful"] += 0.3
        
        # Decay old scores slightly (recency bias)
        for k in self.archetype_scores:
            self.archetype_scores[k] *= 0.95
    
    def _get_dominant_archetype(self):
        """Get the most likely archetype."""
        if not any(self.archetype_scores.values()):
            return "cautious"  # Default: respectful approach
        return max(self.archetype_scores, key=self.archetype_scores.get)
    
    def _get_trust_trend(self):
        """Is trust going up, down, or stable?"""
        if len(self.trust_trajectory) < 2:
            return "unknown"
        
        trust_values = {"resistant": -1, "neutral": 0, "open": 1, "enthusiastic": 2}
        recent = [trust_values.get(t, 0) for t in self.trust_trajectory[-5:]]
        
        if len(recent) < 2:
            return "stable"
        
        avg_first = sum(recent[:len(recent)//2]) / max(len(recent)//2, 1)
        avg_second = sum(recent[len(recent)//2:]) / max(len(recent) - len(recent)//2, 1)
        
        if avg_second > avg_first + 0.3:
            return "warming_up"
        elif avg_second < avg_first - 0.3:
            return "cooling_down"
        return "stable"
    
    def _get_mood_trend(self):
        """Is mood improving, declining, or stable?"""
        if len(self.mood_trajectory) < 2:
            return "unknown"
        
        mood_values = {"negative": -1, "neutral": 0, "positive": 1, "excited": 2}
        recent = [mood_values.get(m, 0) for m in self.mood_trajectory[-5:]]
        
        if len(recent) < 2:
            return "stable"
        
        avg_first = sum(recent[:len(recent)//2]) / max(len(recent)//2, 1)
        avg_second = sum(recent[len(recent)//2:]) / max(len(recent) - len(recent)//2, 1)
        
        if avg_second > avg_first + 0.3:
            return "improving"
        elif avg_second < avg_first - 0.3:
            return "declining"
        return "stable"
    
    def _generate_approach(self, current_signals):
        """Generate specific approach guidance for the agent's next message."""
        archetype = self._get_dominant_archetype()
        arch_approach = ARCHETYPES[archetype]["approach"]
        trust_trend = self._get_trust_trend()
        mood_trend = self._get_mood_trend()
        
        approach = {
            "archetype": archetype,
            "archetype_confidence": round(self.archetype_scores[archetype], 2),
        }
        
        # ── Pacing
        if current_signals["pace_preference"] == "fast" or archetype == "efficient":
            approach["pacing"] = "Move quickly. Combine questions. Don't over-explain."
        elif current_signals["trust_level"] == "resistant" or archetype == "guarded":
            approach["pacing"] = "Slow down. One question at a time. Give space."
        else:
            approach["pacing"] = "Natural pace. Follow their energy."
        
        # ── Question style
        if archetype == "open_book":
            approach["question_style"] = "Open-ended. Let them elaborate. They WANT to share."
        elif archetype == "guarded":
            approach["question_style"] = "Low-pressure. Offer easy outs: 'Only if you're comfortable...' Give them control."
        elif archetype == "efficient":
            approach["question_style"] = "Direct. 'What's your timezone?' not 'Could you tell me about where you're located?'"
        elif archetype == "playful":
            approach["question_style"] = "Unexpected angles. 'What's something weird about you?' works better than 'What are your interests?'"
        else:
            approach["question_style"] = "Balanced. Explain briefly why you're asking, then ask."
        
        # ── Tone matching
        if current_signals["formality"] == "formal":
            approach["tone"] = "Match their formality. Professional, respectful."
        elif current_signals["formality"] == "casual":
            approach["tone"] = "Match casual. Contractions, relaxed language, maybe some humor."
        else:
            approach["tone"] = "Warm but not forced. Natural."
        
        # ── Data import strategy
        if current_signals["trust_level"] == "enthusiastic":
            approach["data_imports"] = "They're ready — offer multiple imports at once. They'll say yes."
        elif current_signals["trust_level"] == "resistant":
            approach["data_imports"] = "DO NOT offer imports right now. Build more trust through conversation first."
        elif trust_trend == "warming_up":
            approach["data_imports"] = "Trust is growing. You can mention imports exist, but don't push."
        elif len(self.declined_topics) > 2:
            approach["data_imports"] = "They've declined several things. Only offer if they bring it up."
        else:
            approach["data_imports"] = "Offer one import at a time. Explain what you'll learn and that it stays local."
        
        # ── Emotional awareness
        if mood_trend == "declining":
            approach["emotional_note"] = "Their mood is dropping. Lighten up, offer to skip ahead, or ask if they want to stop."
        elif mood_trend == "improving":
            approach["emotional_note"] = "They're warming up. Good momentum — keep it going but don't overdo it."
        elif current_signals["mood"] == "negative":
            approach["emotional_note"] = "They seem frustrated or disinterested. Acknowledge it: 'We can skip ahead if you want' or 'Almost done'."
        elif current_signals["mood"] == "excited":
            approach["emotional_note"] = "High energy — ride the wave. This is when they're most willing to share."
        else:
            approach["emotional_note"] = "Neutral energy. Keep it natural."
        
        # ── Specific do/don't guidance
        approach["do"] = []
        approach["dont"] = []
        
        if archetype == "guarded":
            approach["do"].extend([
                "Explain why each question matters",
                "Offer to skip anything they're not comfortable with",
                "Emphasize data stays local",
                "Acknowledge their caution positively",
            ])
            approach["dont"].extend([
                "Ask about family or relationships yet",
                "Push for data connections",
                "Use exclamation marks excessively",
                "Make assumptions about them",
            ])
        elif archetype == "open_book":
            approach["do"].extend([
                "Let them talk — they'll give you more than you asked for",
                "Offer data imports enthusiastically",
                "Mirror their excitement",
                "Ask follow-up questions on what they share",
            ])
            approach["dont"].extend([
                "Interrupt their flow with too many questions",
                "Over-explain privacy — they trust you already",
                "Be more reserved than they are",
            ])
        elif archetype == "efficient":
            approach["do"].extend([
                "Get to the point fast",
                "Use bullet points and short sentences",
                "Give time estimates: 'Two more questions'",
                "Batch questions where possible",
            ])
            approach["dont"].extend([
                "Explain why you're asking unless they ask",
                "Add filler words or pleasantries",
                "Make it feel longer than it needs to be",
                "Ask questions you could infer",
            ])
        
        # ── Should we offer to finish early?
        if self.interaction_count > 6 and current_signals["engagement_level"] == "low":
            approach["offer_early_finish"] = True
            approach["finish_prompt"] = "I've got enough to work with. Want to wrap up, or keep going?"
        elif self.interaction_count > 4 and mood_trend == "declining":
            approach["offer_early_finish"] = True
            approach["finish_prompt"] = "We can stop here if you want — I'll work with what I have and learn more over time."
        else:
            approach["offer_early_finish"] = False
        
        return approach
    
    def record_decline(self, topic):
        """Record that the user declined to discuss a topic."""
        self.declined_topics.add(topic)
    
    def record_import_decision(self, source, accepted):
        """Record user's decision about a data import."""
        if accepted:
            self.accepted_imports.add(source)
        else:
            self.skipped_imports.add(source)
    
    def get_import_recommendation(self):
        """Recommend which import to offer next based on user's pattern."""
        archetype = self._get_dominant_archetype()
        
        all_imports = ["local_system", "github", "twitter", "reddit", "gmail", 
                       "calendar", "spotify", "facebook", "instagram", "linkedin",
                       "youtube", "contacts", "apple"]
        
        # Remove already offered
        available = [i for i in all_imports if i not in self.accepted_imports and i not in self.skipped_imports]
        
        if not available:
            return None
        
        # Order by invasiveness (least to most)
        invasiveness_order = [
            "local_system",   # Zero auth, scans local files
            "github",         # Usually public or token already exists
            "twitter",        # Public, no auth
            "reddit",         # Public, no auth
            "spotify",        # Fun, low stakes
            "gmail",          # More personal
            "calendar",       # Schedule is personal
            "contacts",       # Relationship data
            "youtube",        # Watch history is revealing
            "facebook",       # Requires export effort
            "instagram",      # Requires export effort
            "linkedin",       # Professional, requires export
            "apple",          # Ecosystem-specific
        ]
        
        if archetype == "guarded":
            # Start with least invasive
            for imp in invasiveness_order:
                if imp in available:
                    return {
                        "source": imp,
                        "reason": "lowest barrier to entry",
                        "pitch": self._get_import_pitch(imp, "gentle"),
                    }
        elif archetype == "open_book":
            # Offer the most valuable ones first
            value_order = ["gmail", "github", "twitter", "reddit", "calendar", "local_system"]
            for imp in value_order:
                if imp in available:
                    return {
                        "source": imp,
                        "reason": "highest value data",
                        "pitch": self._get_import_pitch(imp, "enthusiastic"),
                    }
        elif archetype == "efficient":
            # Offer local_system first (zero effort), then batch the rest
            if "local_system" in available:
                return {
                    "source": "local_system",
                    "reason": "zero effort, instant results",
                    "pitch": self._get_import_pitch("local_system", "efficient"),
                }
        
        # Default: next in invasiveness order
        for imp in invasiveness_order:
            if imp in available:
                return {
                    "source": imp,
                    "reason": "next available",
                    "pitch": self._get_import_pitch(imp, "balanced"),
                }
        
        return None
    
    def _get_import_pitch(self, source, style):
        """Generate a context-appropriate pitch for a data import."""
        pitches = {
            "local_system": {
                "gentle": "I can look at what's already on your machine — shell history, installed apps, that kind of thing. Nothing leaves your computer. Want me to take a quick look?",
                "enthusiastic": "Let me scan your local setup — I'll instantly know your tech stack, tools, and workflow. Takes 2 seconds!",
                "efficient": "I can auto-detect your tech stack from your local machine. Zero effort on your part.",
                "balanced": "I can scan your local machine to learn your tech stack. Everything stays local.",
            },
            "github": {
                "gentle": "If you're on GitHub, I can learn about your technical interests from your public profile. Only if you want.",
                "enthusiastic": "What's your GitHub? I'd love to see what you build!",
                "efficient": "GitHub username? I'll pull your tech profile.",
                "balanced": "I can check your GitHub profile for technical interests and languages you use.",
            },
            "gmail": {
                "gentle": "I can look at your email patterns to understand your communication style. I only look at metadata — who, when, how long — not the actual content. Totally optional.",
                "enthusiastic": "Want to connect Gmail? I'll learn your communication style, who matters to you, and when you're most active. Super useful!",
                "efficient": "Gmail connection gives me communication patterns and schedule data. Want to link it?",
                "balanced": "Connecting Gmail helps me understand your communication style. I analyze patterns, not content.",
            },
            "twitter": {
                "gentle": "If you're on Twitter, I can learn about your interests and how you communicate publicly. No login needed — I just read public posts.",
                "enthusiastic": "What's your Twitter handle? I can learn so much about what you care about from your tweets!",
                "efficient": "Twitter handle? No auth needed, I'll analyze your public profile.",
                "balanced": "I can analyze your public Twitter for interests and communication style. No login needed.",
            },
            "reddit": {
                "gentle": "Reddit is great for understanding real interests. If you have a public profile, I can take a look. No login needed.",
                "enthusiastic": "Reddit username? People are way more honest on Reddit than anywhere else — it's the best data source!",
                "efficient": "Reddit username? Public API, no auth.",
                "balanced": "Reddit reveals genuine interests. I can check your public profile if you share your username.",
            },
            "spotify": {
                "gentle": "Music taste says a lot about personality. I can analyze your Spotify if you're interested.",
                "enthusiastic": "Want to connect Spotify? I can literally map your personality from your music taste — it's actually really cool!",
                "efficient": "Spotify user ID? Music taste maps to personality traits.",
                "balanced": "Spotify analysis maps your music taste to personality insights.",
            },
        }
        
        source_pitches = pitches.get(source, {})
        return source_pitches.get(style, source_pitches.get("balanced", f"I can import data from {source} to learn more about you."))
    
    def to_dict(self):
        """Serialize state."""
        return {
            "archetype_scores": {k: round(v, 3) for k, v in self.archetype_scores.items()},
            "dominant_archetype": self._get_dominant_archetype(),
            "trust_trend": self._get_trust_trend(),
            "mood_trend": self._get_mood_trend(),
            "interaction_count": self.interaction_count,
            "declined_topics": list(self.declined_topics),
            "skipped_imports": list(self.skipped_imports),
            "accepted_imports": list(self.accepted_imports),
            "current_approach": self.current_approach,
            "created_at": self.created_at,
        }
    
    def save(self):
        os.makedirs(IMPORT_DIR, exist_ok=True)
        with open(STATE_FILE, "w") as f:
            json.dump(self.to_dict(), f, indent=2)
    
    @classmethod
    def load(cls):
        state = cls()
        if os.path.exists(STATE_FILE):
            try:
                with open(STATE_FILE) as f:
                    data = json.load(f)
                state.archetype_scores = data.get("archetype_scores", state.archetype_scores)
                state.declined_topics = set(data.get("declined_topics", []))
                state.skipped_imports = set(data.get("skipped_imports", []))
                state.accepted_imports = set(data.get("accepted_imports", []))
                state.interaction_count = data.get("interaction_count", 0)
                state.created_at = data.get("created_at", state.created_at)
            except:
                pass
        return state

# ─── CLI ──────────────────────────────────────────────────────────────

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: adaptive.py <analyze|status|recommend-import|decline|import-decision> [args...]")
        sys.exit(1)
    
    cmd = sys.argv[1]
    
    if cmd == "analyze" and len(sys.argv) >= 3:
        state = AdaptiveState.load()
        text = " ".join(sys.argv[2:])
        result = state.process_message(text)
        state.save()
        print(json.dumps(result, indent=2))
    
    elif cmd == "status":
        state = AdaptiveState.load()
        print(json.dumps(state.to_dict(), indent=2))
    
    elif cmd == "recommend-import":
        state = AdaptiveState.load()
        rec = state.get_import_recommendation()
        print(json.dumps(rec or {"recommendation": None}, indent=2))
    
    elif cmd == "decline" and len(sys.argv) >= 3:
        state = AdaptiveState.load()
        state.record_decline(sys.argv[2])
        state.save()
        print(json.dumps({"declined": sys.argv[2]}))
    
    elif cmd == "import-decision" and len(sys.argv) >= 4:
        state = AdaptiveState.load()
        source = sys.argv[2]
        accepted = sys.argv[3].lower() in ["yes", "true", "accept", "1"]
        state.record_import_decision(source, accepted)
        state.save()
        print(json.dumps({"source": source, "accepted": accepted}))
    
    else:
        print(json.dumps({"error": f"Unknown command: {cmd}"}))


path: lib/conversation.py
content: #!/usr/bin/env python3
"""
Soulsync Conversation Engine — Generates adaptive, natural conversation flows
that extract personality data without feeling like an interview.

This module doesn't run the conversation itself (the LLM does that via SKILL.md).
Instead, it provides:
1. Question generation based on what's known vs unknown
2. Conversation state tracking
3. Response analysis to extract structured data from natural answers
4. Conversation pacing and flow control

The LLM reads these outputs and uses them to guide natural conversation.
"""
import os
import sys
import json
import re
from datetime import datetime, timezone

IMPORT_DIR = "/tmp/soulsync"
WORKSPACE = os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace"))
STATE_FILE = os.path.join(IMPORT_DIR, "conversation.json")

# ─── Personality Dimensions ───────────────────────────────────────────
# What we're trying to learn, organized by priority

DIMENSIONS = {
    "identity": {
        "priority": 1,
        "fields": ["name", "preferred_name", "pronouns"],
        "required": True,
        "description": "Who they are — basics",
    },
    "work": {
        "priority": 2,
        "fields": ["work", "role", "company", "industry"],
        "required": True,
        "description": "What they do professionally",
    },
    "communication": {
        "priority": 3,
        "fields": ["communication_style", "tone", "verbosity_preference"],
        "required": True,
        "description": "How they like to interact",
    },
    "goals": {
        "priority": 4,
        "fields": ["goals", "agent_expectations"],
        "required": True,
        "description": "What they want from their AI agent",
    },
    "context": {
        "priority": 5,
        "fields": ["timezone", "location", "devices"],
        "required": False,
        "description": "Environmental context",
    },
    "boundaries": {
        "priority": 6,
        "fields": ["boundaries", "pet_peeves", "off_limits"],
        "required": False,
        "description": "What to avoid",
    },
    "interests": {
        "priority": 7,
        "fields": ["interests", "hobbies", "passions"],
        "required": False,
        "description": "What they care about",
    },
    "personality": {
        "priority": 8,
        "fields": ["personality_traits", "humor_style", "energy_level"],
        "required": False,
        "description": "Deeper personality signals",
    },
    "relationships": {
        "priority": 9,
        "fields": ["family", "partner", "key_people"],
        "required": False,
        "description": "Important people in their life",
    },
    "technical": {
        "priority": 10,
        "fields": ["technical_level", "languages", "tools"],
        "required": False,
        "description": "Technical background",
    },
}

# ─── Question Bank ────────────────────────────────────────────────────
# Archetype-aware question phrasings. The LLM picks based on the user's
# detected archetype from the adaptive engine. Not scripts — starting points.
# The LLM should COMBINE and FLOW between these, not ask them robotically.

QUESTION_BANK = {
    "identity": {
        "openers": {
            "default": "What should I call you?",
            "open_book": "So who am I talking to? Tell me everything.",
            "cautious": "Let's start simple — what's your name?",
            "guarded": "What should I call you?",
            "efficient": "Name?",
            "playful": "Alright, who are you and what's your deal?",
        },
        "follow_ups": [
            "Got a nickname or handle you prefer?",
            "And pronouns?",
        ],
        "combo_with": "work",  # Can naturally combine: "What's your name and what do you do?"
    },
    "work": {
        "openers": {
            "default": "What do you do?",
            "open_book": "Tell me about yourself — what fills your days?",
            "cautious": "What kind of work are you in?",
            "guarded": "What do you do for work?",
            "efficient": "What do you do? Work, projects, the short version.",
            "playful": "So what's your thing? What gets you out of bed?",
        },
        "follow_ups": [
            "Any side projects?",
            "How long have you been doing that?",
        ],
        "combo_with": "identity",
    },
    "communication": {
        "openers": {
            "default": "How do you like your info — quick and punchy, or detailed?",
            "open_book": "How should I talk to you? I can be concise, detailed, casual, formal — what's your style?",
            "cautious": "Quick question about how I should communicate with you — do you prefer brief answers or thorough ones?",
            "guarded": "Do you prefer short answers or detailed ones?",
            "efficient": "Communication style: concise or detailed?",
            "playful": "Real talk — should I be professional or can I be chill with you?",
        },
        "follow_ups": [
            "When I mess up, do you want me to just say so directly?",
            "Should I suggest things proactively or wait for you to ask?",
        ],
        "infer_from_response": True,
    },
    "goals": {
        "openers": {
            "default": "What are you hoping I can help with?",
            "open_book": "What would make this actually useful for you? Dream big.",
            "cautious": "What kind of things would you want help with?",
            "guarded": "What do you want me to do for you?",
            "efficient": "Primary use cases?",
            "playful": "If I was actually good at my job, what would that look like for you?",
        },
        "follow_ups": [
            "Any day-to-day stuff? Reminders, research, scheduling?",
            "Anything you've been wanting to automate?",
        ],
    },
    "context": {
        "openers": {
            "default": "What timezone are you in?",
            "efficient": "Timezone?",
            "playful": "Where in the world are you?",
        },
        "follow_ups": ["What devices do you mainly use?"],
        "can_skip": True,
        "combo_with": "communication",  # Can combine with style question
    },
    "boundaries": {
        "openers": {
            "default": "Anything I should absolutely NOT do?",
            "open_book": "What would annoy you? What are your hard nos?",
            "cautious": "Are there things you'd prefer I don't do? Everyone has boundaries.",
            "guarded": "Any limits I should know about?",
            "efficient": "Hard nos? Things I should never do?",
            "playful": "What would make you want to yeet me into the void?",
        },
        "follow_ups": [
            "Am I okay to take external actions — emails, posts — or always ask first?",
        ],
    },
    "interests": {
        "openers": {
            "default": "What do you nerd out about outside of work?",
            "open_book": "What are your passions? What could you talk about for hours?",
            "cautious": "Any hobbies or interests you'd want me to know about?",
            "guarded": "Anything you're into that I should know about?",
            "efficient": "Interests/hobbies?",
            "playful": "Quick — three things you're obsessed with. Go.",
        },
        "follow_ups": [],
        "can_infer": True,
    },
    "personality": {
        "openers": {},  # Never asked directly — inferred from behavior
        "follow_ups": [],
        "infer_only": True,
    },
    "relationships": {
        "openers": {
            "default": "Anyone important I should know about? Partner, family, people I'll hear about?",
            "open_book": "Tell me about your people — who are the main characters in your life?",
            "cautious": "Only if you're comfortable — anyone I should know about? Partner, kids, that kind of thing.",
            "guarded": "",  # Skip for guarded users
            "efficient": "Key people in your life I should know about?",
            "playful": "Who are your favorite humans?",
        },
        "follow_ups": [],
        "sensitive": True,
        "can_skip": True,
    },
    "technical": {
        "openers": {
            "default": "How technical are you? Should I show code or keep it high-level?",
            "efficient": "Technical level? Comfortable with CLI and code?",
            "playful": "On a scale from 'what's a terminal' to 'I write kernel modules for fun' — where do you land?",
        },
        "follow_ups": [],
        "can_infer": True,
    },
}

# ─── Conversation Flow Strategies ─────────────────────────────────────
# How to combine questions and maintain natural flow

FLOW_STRATEGIES = {
    "open_book": {
        "style": "Let them run. Ask open-ended combo questions. They'll volunteer more than you ask for.",
        "opener": "Hey — I'm brand new and I want to actually be useful to you, not generic. Mind if I ask a few things? Should take about 3-4 minutes.",
        "combo_questions": [
            ("identity", "work", "What's your name and what do you do?"),
            ("communication", "context", "How do you like to communicate, and where are you based?"),
            ("goals", "interests", "What do you want help with, and what are you into outside of that?"),
            ("boundaries", "relationships", "Any hard nos I should know about? And who are the important people I'll hear about?"),
        ],
        "closer": "Solid. I've got a good picture. I'll keep learning as we go — but the basics are locked in.",
    },
    "cautious": {
        "style": "One topic at a time. Explain why you're asking. Give them control.",
        "opener": "Hey. I'd like to learn a bit about you so I can actually be helpful. I'll ask a few questions — skip anything you don't want to answer. Totally fine.",
        "combo_questions": [],  # No combos — one at a time
        "closer": "Thanks for sharing that. I'll work with what I have and learn more naturally over time. You can always update things later.",
    },
    "guarded": {
        "style": "Minimal questions. Focus on functional info only. Don't push personal.",
        "opener": "Quick setup — just need a few basics so I'm not completely useless. Nothing personal unless you want to share.",
        "combo_questions": [
            ("identity", "context", "What should I call you, and what timezone are you in?"),
        ],
        "closer": "That's all I need. I'll figure out the rest as we work together.",
    },
    "efficient": {
        "style": "Rapid fire. Batch everything. No fluff.",
        "opener": "Quick setup. Five questions, should take 60 seconds.",
        "combo_questions": [
            ("identity", "work", "context", "Name, what you do, timezone?"),
            ("communication", "goals", "Communication style preference, and what do you want me to help with?"),
            ("boundaries", "technical", "Hard nos, and how technical are you?"),
        ],
        "closer": "Done. Let's get to work.",
    },
    "playful": {
        "style": "Fun, unexpected angles. Match their energy.",
        "opener": "Alright let's do this. I need to figure out who you are so I stop being boring. Ready?",
        "combo_questions": [
            ("identity", "work", "Who are you and what's your deal?"),
            ("interests", "personality", "Three things you're obsessed with — go."),
            ("communication", "boundaries", "How should I talk to you, and what would make you hate me?"),
        ],
        "closer": "I think I've got your vibe. Let's see if I'm right — correct me when I'm wrong.",
    },
}




# ─── Natural Transitions ──────────────────────────────────────────────
# Phrases to bridge between topics. The LLM picks contextually.

TRANSITIONS = {
    "identity_to_work": [
        "Cool. So what do you do, {name}?",
        "Nice to meet you. What keeps you busy?",
        "Got it. And what do you do?",
    ],
    "work_to_communication": [
        "Interesting. Quick one — how do you like me to communicate with you?",
        "Got it. Before we go further — should I be concise or detailed when I help you?",
        "Nice. Now, how should I talk to you?",
    ],
    "communication_to_goals": [
        "Perfect. So what do you actually want me to help with?",
        "Noted. What are you hoping I can do for you?",
        "Makes sense. What would make me useful to you?",
    ],
    "goals_to_boundaries": [
        "Got it. Last important one — anything I should never do?",
        "Solid. Any hard rules? Things that would piss you off?",
        "Cool. Anything off limits?",
    ],
    "goals_to_context": [
        "Quick logistics — what timezone are you in?",
        "Where are you based, time-wise?",
    ],
    "any_to_imports": [
        "I can learn a lot more about you automatically by connecting some accounts. Want to hear the options?",
        "Want me to scan your local machine? I can learn your tech stack in 2 seconds without connecting anything.",
        "I could also look at your GitHub, Twitter, or email to understand you better. Everything stays local. Interested?",
    ],
    "any_to_close": [
        "I think I've got a solid picture. Want to see what I've put together?",
        "That should be enough to get started. Let me show you your profile.",
        "Good stuff. Let me generate your files — you can review before I save anything.",
    ],
    "decline_graceful": [
        "No worries, skip it.",
        "Totally fine. Moving on.",
        "Cool, we can come back to that.",
        "All good.",
    ],
}

# ─── Response Analyzer ────────────────────────────────────────────────

def analyze_response(text, dimension):
    """Extract structured data from a natural language response.
    Returns dict of field:value pairs that can be extracted."""
    
    extracted = {}
    text_lower = text.lower()
    
    # Identity extraction
    if dimension == "identity":
        # Name patterns
        name_patterns = [
            r"(?:i['’]?m|i am|name is|call me|it['’]?s|im)\s+([A-Z][a-z]+(?:\s+[A-Z][a-z]+)?)",
            r"(?:i['’]?m|i am|name is|call me|im)\s+([a-zA-Z]+)",
            r"^([A-Z][a-z]+(?:\s+[A-Z][a-z]+)?)(?:\s*[.!,]|\s+here|\s+but|\s+and)",
            r"^([A-Z][a-z]{2,}) ",
        ]
        for pattern in name_patterns:
            match = re.search(pattern, text, re.IGNORECASE)
            if match:
                name = match.group(1).strip()
                # Filter out common false positives
                if name.lower() not in {"the", "a", "an", "so", "well", "hey", "hi", "just", "not", "yes", "no"}:
                    extracted["name"] = name.title()
                    break
        
        # Pronouns
        pronoun_map = {
            "he/him": ["he/him", "he him", "him/his"],
            "she/her": ["she/her", "she her", "her/hers"],
            "they/them": ["they/them", "they them", "them/theirs"],
        }
        for pronouns, patterns in pronoun_map.items():
            if any(p in text_lower for p in patterns):
                extracted["pronouns"] = pronouns
        
        # Nickname
        nick_patterns = [
            r"(?:go by|call me|prefer|nickname is)\s+(\w+)",
            r"(?:but (?:everyone|people|most) call[s]? me)\s+(\w+)",
        ]
        for pattern in nick_patterns:
            match = re.search(pattern, text, re.IGNORECASE)
            if match:
                extracted["preferred_name"] = match.group(1)
    
    # Work extraction
    elif dimension == "work":
        # Job title patterns
        work_patterns = [
            r"(?:i'm a|i am a|i work as a?|my job is|i do)\s+(.+?)(?:\.|$|,|\bat\b)",
            r"(?:i (?:run|own|founded|started|manage))\s+(.+?)(?:\.|$|,)",
        ]
        for pattern in work_patterns:
            match = re.search(pattern, text, re.IGNORECASE)
            if match:
                extracted["work"] = match.group(1).strip()
                break
    
    # Communication style — inferred from HOW they write
    elif dimension == "communication":
        word_count = len(text.split())
        
        if word_count < 15:
            extracted["verbosity_preference"] = "concise"
        elif word_count > 80:
            extracted["verbosity_preference"] = "detailed"
        else:
            extracted["verbosity_preference"] = "balanced"
        
        # Formality detection
        casual_markers = ["lol", "haha", "nah", "yeah", "gonna", "wanna", "kinda", "tbh", "imo", "ngl"]
        formal_markers = ["please", "would you", "i would prefer", "kindly", "if you could"]
        
        casual_count = sum(1 for m in casual_markers if m in text_lower)
        formal_count = sum(1 for m in formal_markers if m in text_lower)
        
        if casual_count > formal_count:
            extracted["tone"] = "casual"
        elif formal_count > casual_count:
            extracted["tone"] = "formal"
        else:
            extracted["tone"] = "balanced"
        
        # Direct style keywords
        if any(w in text_lower for w in ["straight", "direct", "blunt", "no bs", "just tell me"]):
            extracted["communication_style"] = "direct and no-nonsense"
        elif any(w in text_lower for w in ["detailed", "thorough", "explain", "context"]):
            extracted["communication_style"] = "detailed with full context"
        elif any(w in text_lower for w in ["brief", "short", "concise", "quick", "bullet"]):
            extracted["communication_style"] = "concise, bullet points preferred"
    
    # Timezone extraction
    elif dimension == "context":
        tz_patterns = {
            "America/New_York": ["eastern", "est", "edt", "new york", "east coast", "florida", "georgia", "carolina", "virginia", "ohio", "pennsylvania", "new jersey", "connecticut", "massachusetts", "boston"],
            "America/Chicago": ["central", "cst", "cdt", "chicago", "midwest", "wisconsin", "wi", "milwaukee", "west allis", "minnesota", "mn", "iowa", "missouri", "illinois", "texas", "dallas", "houston", "austin", "san antonio"],
            "America/Denver": ["mountain", "mst", "mdt", "denver", "colorado", "utah", "arizona", "montana", "wyoming", "new mexico", "idaho", "boise"],
            "America/Los_Angeles": ["pacific", "pst", "pdt", "la", "west coast", "california", "seattle", "portland", "oregon", "washington", "san francisco", "bay area", "silicon valley"],
            "Europe/London": ["gmt", "uk", "london", "british"],
            "Europe/Berlin": ["cet", "berlin", "german", "paris"],
            "Asia/Tokyo": ["jst", "tokyo", "japan"],
            "Australia/Sydney": ["aest", "sydney", "australia"],
        }
        for tz, patterns in tz_patterns.items():
            if any(p in text_lower for p in patterns):
                extracted["timezone"] = tz
                break
    
    # Goals extraction
    elif dimension == "goals":
        goals = []
        # Split on common list separators
        parts = re.split(r'[,;]|\band\b|\balso\b|\bplus\b', text)
        for part in parts:
            part = part.strip()
            if len(part) > 10 and len(part) < 200:
                goals.append(part)
        if goals:
            extracted["goals"] = goals
    
    # Boundaries extraction
    elif dimension == "boundaries":
        boundaries = []
        # Look for negative statements and rewrite them as clear rules
        neg_patterns = [
            (r"(?:don't|do not|never|please don't|stop)\s+(.+?)(?:\.|$|,)", "Don't {0}"),
            (r"(?:i hate|annoys me|pet peeve|drives me crazy)\s*(?:when|is|:)?\s*(.+?)(?:\.|$|,)", "Avoid: {0}"),
            (r"(?:i (?:don't|do not) (?:like|want|enjoy|eat|use))\s+(.+?)(?:\.|$|,)", "No {0}"),
            (r"(?:i(?:'ll)? never)\s+(.+?)(?:\.|$|,)", "Never {0}"),
            (r"(?:(?:no|not|without)\s+(?:my )?(?:approval|permission|consent))", "Always ask before taking external actions"),
        ]
        for pattern_tuple in neg_patterns:
            if isinstance(pattern_tuple, tuple):
                pattern, template = pattern_tuple
            else:
                pattern, template = pattern_tuple, "{0}"
            for match in re.finditer(pattern, text, re.IGNORECASE):
                raw = match.group(1).strip() if match.lastindex else match.group(0).strip()
                # Clean up and make it a proper boundary statement
                boundary = template.format(raw)
                # Remove trailing conjunctions
                boundary = re.sub(r"\s+(?:so|and|but|because).*$", "", boundary, flags=re.IGNORECASE)
                boundaries.append(boundary)
        
        # If no patterns matched but text has clear preferences, capture the whole thing
        if not boundaries and len(text.split()) > 3:
            sentences = re.split(r"[.!]\s+", text)
            for s in sentences:
                s = s.strip()
                if s and any(w in s.lower() for w in ["don't", "never", "no ", "not ", "hate", "avoid"]):
                    boundaries.append(s)
        
        if boundaries:
            extracted["boundaries"] = boundaries
    
    # Interest extraction
    elif dimension == "interests":
        interests = []
        parts = re.split(r'[,;]|\band\b', text)
        for part in parts:
            part = part.strip().strip(".")
            if 2 < len(part) < 50:
                # Remove filler
                part = re.sub(r'^(?:i (?:like|love|enjoy|am into)\s+)', '', part, flags=re.IGNORECASE)
                if part and len(part) > 2:
                    interests.append(part)
        if interests:
            extracted["interests"] = interests
    
    return extracted

# ─── Conversation Flow Controller ─────────────────────────────────────

class ConversationState:
    """Tracks what we know and what we still need to learn."""
    
    def __init__(self):
        self.known = {}          # field: value
        self.dimensions_covered = set()
        self.questions_asked = 0
        self.started_at = datetime.now(timezone.utc).isoformat()
        self.import_data = {}    # Pre-loaded from importers
        self.inferred = {}       # Inferred from behavior
        
    def load_imports(self):
        """Load any existing importer results."""
        if os.path.isdir(IMPORT_DIR):
            for f in os.listdir(IMPORT_DIR):
                if f.endswith(".json") and f != "conversation.json":
                    try:
                        with open(os.path.join(IMPORT_DIR, f)) as fh:
                            data = json.load(fh)
                            if "insights" in data:
                                self.import_data[data.get("source", f)] = data["insights"]
                    except:
                        continue
    
    def get_known_fields(self):
        """Return all fields we have data for (from any source)."""
        fields = set(self.known.keys())
        for source, insights in self.import_data.items():
            for key, value in insights.items():
                if value and (not isinstance(value, list) or len(value) > 0):
                    fields.add(key)
        return fields
    
    def get_missing_dimensions(self):
        """Return dimensions that still need data, sorted by priority."""
        known_fields = self.get_known_fields()
        missing = []
        
        for dim_name, dim_info in sorted(DIMENSIONS.items(), key=lambda x: x[1]["priority"]):
            if dim_name in self.dimensions_covered:
                continue
            if dim_info.get("infer_only"):
                continue
            
            # Check if any fields in this dimension are still unknown
            dim_fields = set(dim_info["fields"])
            covered = dim_fields & known_fields
            
            if len(covered) < len(dim_fields):
                missing.append({
                    "dimension": dim_name,
                    "priority": dim_info["priority"],
                    "required": dim_info.get("required", False),
                    "fields_missing": list(dim_fields - covered),
                    "fields_known": list(covered),
                    "can_skip": dim_info.get("can_skip", False) or not dim_info.get("required", False),
                    "can_infer": dim_info.get("can_infer", False),
                })
        
        return missing
    
    def get_next_questions(self, max_questions=3):
        """Get the next best questions to ask, with context from what we already know."""
        missing = self.get_missing_dimensions()
        questions = []
        
        for dim in missing[:max_questions]:
            dim_name = dim["dimension"]
            bank = QUESTION_BANK.get(dim_name, {})
            
            # Build context summary from what we know so far
            context_hints = []
            if self.known.get("name"):
                context_hints.append(f"Their name is {self.known['name']}")
            if self.known.get("preferred_name"):
                context_hints.append(f"They go by {self.known['preferred_name']}")
            if self.known.get("work"):
                context_hints.append(f"They work as: {self.known['work']}")
            if self.known.get("timezone"):
                context_hints.append(f"Timezone: {self.known['timezone']}")
            if self.known.get("communication_style"):
                context_hints.append(f"They want: {self.known['communication_style']}")
            if self.known.get("goals"):
                context_hints.append(f"Goals: {', '.join(self.known['goals']) if isinstance(self.known['goals'], list) else self.known['goals']}")
            if self.known.get("boundaries"):
                context_hints.append(f"Boundaries: {', '.join(self.known['boundaries']) if isinstance(self.known['boundaries'], list) else self.known['boundaries']}")
            
            # Pick opener if we haven't asked about this dimension yet
            if dim_name not in self.dimensions_covered:
                openers = bank.get("openers", [])
                if openers:
                    questions.append({
                        "dimension": dim_name,
                        "type": "opener",
                        "suggestions": openers,
                        "can_skip": dim.get("can_skip", False),
                        "sensitive": bank.get("sensitive", False),
                        "context": context_hints,
                        "combo_with": bank.get("combo_with"),
                        "instruction": "IMPORTANT: Use the context below to personalize your question. Reference what they already told you. Never ask a generic question when you have context.",
                    })
            else:
                follow_ups = bank.get("follow_ups", [])
                if follow_ups:
                    questions.append({
                        "dimension": dim_name,
                        "type": "follow_up",
                        "suggestions": follow_ups,
                        "can_skip": True,
                        "context": context_hints,
                    })
        
        return questions
    
    def record_answer(self, dimension, raw_text):
        """Process a user's response and extract data."""
        extracted = analyze_response(raw_text, dimension)
        self.known.update(extracted)
        self.dimensions_covered.add(dimension)
        self.questions_asked += 1
        return extracted
    
    def get_progress(self):
        """Return conversation progress summary."""
        total_dims = len([d for d in DIMENSIONS.values() if not d.get("infer_only")])
        covered = len(self.dimensions_covered)
        required_dims = [d for d, info in DIMENSIONS.items() if info.get("required")]
        required_covered = len([d for d in required_dims if d in self.dimensions_covered])
        
        return {
            "dimensions_total": total_dims,
            "dimensions_covered": covered,
            "required_total": len(required_dims),
            "required_covered": required_covered,
            "questions_asked": self.questions_asked,
            "completeness": covered / max(total_dims, 1),
            "required_complete": required_covered >= len(required_dims),
            "can_finish": required_covered >= len(required_dims),
            "fields_collected": len(self.known),
            "import_sources": list(self.import_data.keys()),
        }
    
    def to_dict(self):
        """Serialize state for storage."""
        return {
            "known": self.known,
            "dimensions_covered": list(self.dimensions_covered),
            "questions_asked": self.questions_asked,
            "started_at": self.started_at,
            "import_sources": list(self.import_data.keys()),
            "inferred": self.inferred,
            "progress": self.get_progress(),
        }
    
    def save(self):
        """Save conversation state."""
        os.makedirs(IMPORT_DIR, exist_ok=True)
        with open(STATE_FILE, "w") as f:
            json.dump(self.to_dict(), f, indent=2)
    
    @classmethod
    def load(cls):
        """Load existing conversation state."""
        state = cls()
        if os.path.exists(STATE_FILE):
            try:
                with open(STATE_FILE) as f:
                    data = json.load(f)
                state.known = data.get("known", {})
                state.dimensions_covered = set(data.get("dimensions_covered", []))
                state.questions_asked = data.get("questions_asked", 0)
                state.started_at = data.get("started_at", state.started_at)
                state.inferred = data.get("inferred", {})
            except:
                pass
        state.load_imports()
        return state

# ─── CLI Interface ────────────────────────────────────────────────────

def cmd_init():
    """Initialize a new conversation."""
    state = ConversationState()
    state.load_imports()
    state.save()
    
    result = {
        "action": "init",
        "import_sources": list(state.import_data.keys()),
        "missing_dimensions": state.get_missing_dimensions(),
        "next_questions": state.get_next_questions(),
        "progress": state.get_progress(),
    }
    print(json.dumps(result, indent=2))

def cmd_next():
    """Get next questions to ask."""
    state = ConversationState.load()
    result = {
        "action": "next",
        "next_questions": state.get_next_questions(),
        "progress": state.get_progress(),
    }
    print(json.dumps(result, indent=2))

def cmd_record(dimension, text):
    """Record a user's response."""
    state = ConversationState.load()
    extracted = state.record_answer(dimension, text)
    state.save()
    
    result = {
        "action": "record",
        "dimension": dimension,
        "extracted": extracted,
        "next_questions": state.get_next_questions(),
        "progress": state.get_progress(),
    }
    print(json.dumps(result, indent=2))

def cmd_status():
    """Get current conversation state."""
    state = ConversationState.load()
    result = {
        "action": "status",
        "state": state.to_dict(),
        "missing": state.get_missing_dimensions(),
        "next_questions": state.get_next_questions(),
    }
    print(json.dumps(result, indent=2))

def cmd_export():
    """Export final conversation data for synthesizer."""
    state = ConversationState.load()
    state.save()  # Ensure latest state is saved
    print(json.dumps(state.known, indent=2))

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: conversation.py <init|next|record|status|export> [args...]")
        print("  init              — Start new conversation, load imports")
        print("  next              — Get next questions to ask")
        print("  record <dim> <text> — Record user response for a dimension")
        print("  status            — Get full conversation state")
        print("  export            — Export collected data for synthesizer")
        sys.exit(1)
    
    cmd = sys.argv[1]
    
    if cmd == "init":
        cmd_init()
    elif cmd == "next":
        cmd_next()
    elif cmd == "record" and len(sys.argv) >= 4:
        cmd_record(sys.argv[2], " ".join(sys.argv[3:]))
    elif cmd == "status":
        cmd_status()
    elif cmd == "export":
        cmd_export()
    else:
        print(json.dumps({"error": f"Unknown command: {cmd}"}))
        sys.exit(1)


path: lib/detector.py
content: #!/usr/bin/env python3
"""
Soulsync Detector — Assesses current personalization level of an OpenClaw workspace.
Returns: new | partial | established
"""
import os
import sys
import json

WORKSPACE = os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace"))

def read_file(path):
    try:
        with open(path, "r") as f:
            return f.read().strip()
    except FileNotFoundError:
        return None

def assess_file(content, default_markers=None):
    """Score a file's personalization level. Returns 0-1."""
    if content is None:
        return 0.0
    if len(content) < 50:
        return 0.1
    
    # Check for default/template content
    if default_markers:
        for marker in default_markers:
            if marker.lower() in content.lower():
                return 0.2
    
    # Score based on content richness
    lines = [l for l in content.split("\n") if l.strip()]
    word_count = len(content.split())
    
    if word_count < 30:
        return 0.2
    elif word_count < 100:
        return 0.5
    elif word_count < 300:
        return 0.7
    else:
        return 1.0

def detect():
    soul = read_file(os.path.join(WORKSPACE, "SOUL.md"))
    user = read_file(os.path.join(WORKSPACE, "USER.md"))
    memory = read_file(os.path.join(WORKSPACE, "MEMORY.md"))
    identity = read_file(os.path.join(WORKSPACE, "IDENTITY.md"))
    
    # Check for memory files
    memory_dir = os.path.join(WORKSPACE, "memory")
    memory_files = []
    if os.path.isdir(memory_dir):
        memory_files = [f for f in os.listdir(memory_dir) if f.endswith(".md")]
    
    # Score each component
    soul_score = assess_file(soul, default_markers=[
        "you're not a chatbot",
        "this is a starting point",
        "make it yours",
        "_you're not a chatbot_"
    ])
    
    user_score = assess_file(user, default_markers=[
        "preferred name:",
        "example",
        "your name here"
    ])
    
    memory_score = assess_file(memory, default_markers=[
        "no memories yet",
        "this file is empty"
    ])
    
    # Bonus for existing memory day files
    memory_day_bonus = min(len(memory_files) * 0.1, 0.5)
    
    # Overall score
    total = (soul_score * 0.3) + (user_score * 0.3) + (memory_score * 0.2) + (memory_day_bonus * 0.2)
    
    # Determine level
    if total < 0.2:
        level = "new"
    elif total < 0.6:
        level = "partial"
    else:
        level = "established"
    
    result = {
        "level": level,
        "score": round(total, 2),
        "details": {
            "soul_md": {"exists": soul is not None, "score": soul_score},
            "user_md": {"exists": user is not None, "score": user_score},
            "memory_md": {"exists": memory is not None, "score": memory_score},
            "identity_md": {"exists": identity is not None},
            "memory_days": len(memory_files),
        },
        "gaps": []
    }
    
    # Identify specific gaps
    if soul_score < 0.5:
        result["gaps"].append("soul_md — personality and communication style not defined")
    if user_score < 0.5:
        result["gaps"].append("user_md — basic info about the user is missing or sparse")
    if memory_score < 0.3:
        result["gaps"].append("memory_md — no long-term memory established")
    if not identity:
        result["gaps"].append("identity_md — agent identity not configured")
    if len(memory_files) < 3:
        result["gaps"].append("memory_days — limited interaction history")
    
    return result

if __name__ == "__main__":
    result = detect()
    print(json.dumps(result, indent=2))


path: lib/followup.py
content: #!/usr/bin/env python3
"""
Soulsync Follow-Up Engine — Ongoing personalization after initial setup.

Handles:
1. Periodic check-ins ("It's been 2 weeks, anything changed?")
2. Life event detection ("You mentioned a new job — want to update your profile?")
3. Correction learning (when user says "that's wrong" or "actually I...")
4. Deep dives (unlocking new dimensions over time as trust builds)
5. Data source suggestions ("You've been talking about code a lot — want to connect GitHub?")
"""
import os
import sys
import json
import re
from datetime import datetime, timezone, timedelta

WORKSPACE = os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace"))
STATE_FILE = os.path.join(WORKSPACE, ".soulsync-state.json")

# ─── Life Change Signals ──────────────────────────────────────────────
# Phrases that suggest something in the user's life changed

LIFE_CHANGE_SIGNALS = {
    "job": [
        "new job", "got hired", "started working at", "quit my job", "got fired",
        "changing careers", "got promoted", "new role", "left my job", "freelancing now",
    ],
    "location": [
        "moving to", "just moved", "relocated", "new house", "new apartment",
        "bought a house", "sold my house",
    ],
    "relationship": [
        "got married", "engaged", "broke up", "divorced", "new partner",
        "having a baby", "pregnant", "new kid",
    ],
    "interests": [
        "really into", "just discovered", "started learning", "obsessed with",
        "picked up", "new hobby", "quit", "stopped doing",
    ],
    "technical": [
        "switched to", "learning", "started using", "dropped", "migrated to",
        "new setup", "new machine", "rebuilt my",
    ],
}

# ─── Correction Patterns ─────────────────────────────────────────────
# When the user corrects something about themselves

CORRECTION_PATTERNS = [
    r"^(?:actually|no|wait|hang on),?",
    r"(?:that's|that is) (?:wrong|incorrect|not right|not true)",
    r"(?:i (?:don't|no longer|stopped|quit)\s+(.+?)(?:\.|$|,))",
    r"(?:i (?:actually|now)\s+(.+?)(?:\.|$|,))",
    r"(?:correct(?:ion)?:?\s+(.+?)(?:\.|$))",
    r"(?:update (?:my|the)\s+(?:profile|soul|user|info))",
]


class FollowUpState:
    """Tracks when and how to follow up with the user."""
    
    def __init__(self):
        self.initial_sync_date = None
        self.last_followup_date = None
        self.followup_count = 0
        self.corrections_made = []
        self.life_changes_detected = []
        self.suggested_imports = []
        self.declined_imports = []
        self.trust_level = "initial"  # initial → comfortable → trusted
        self.dimensions_unlocked = []  # dimensions revealed over time
        self.ongoing_enabled = None    # None = not asked yet, True/False = user chose
        self.passive_learning = True   # Always-on light learning from normal chat
        self.pending_confirmations = [] # Facts to confirm casually
        self.learned_passively = []    # Things picked up from normal conversation
    
    def save(self):
        os.makedirs(os.path.dirname(STATE_FILE), exist_ok=True)
        with open(STATE_FILE, 'w') as f:
            json.dump({
                "initial_sync_date": self.initial_sync_date,
                "last_followup_date": self.last_followup_date,
                "followup_count": self.followup_count,
                "corrections_made": self.corrections_made,
                "life_changes_detected": self.life_changes_detected,
                "suggested_imports": self.suggested_imports,
                "declined_imports": self.declined_imports,
                "trust_level": self.trust_level,
                "dimensions_unlocked": self.dimensions_unlocked,
                "ongoing_enabled": self.ongoing_enabled,
                "passive_learning": self.passive_learning,
                "pending_confirmations": self.pending_confirmations[-20:],
                "learned_passively": self.learned_passively[-50:],
            }, f, indent=2)
    
    @classmethod
    def load(cls):
        state = cls()
        if os.path.exists(STATE_FILE):
            try:
                with open(STATE_FILE) as f:
                    data = json.load(f)
                for k, v in data.items():
                    if hasattr(state, k):
                        setattr(state, k, v)
            except:
                pass
        return state


def detect_life_changes(message: str) -> list[dict]:
    """Scan a message for life change signals."""
    changes = []
    msg_lower = message.lower()
    
    for category, signals in LIFE_CHANGE_SIGNALS.items():
        for signal in signals:
            if signal in msg_lower:
                changes.append({
                    "category": category,
                    "signal": signal,
                    "message_excerpt": message[:200],
                    "detected_at": datetime.now(timezone.utc).isoformat(),
                })
                break  # One per category
    
    return changes


def detect_correction(message: str) -> bool:
    """Check if the user is correcting something about their profile."""
    for pattern in CORRECTION_PATTERNS:
        if re.search(pattern, message, re.IGNORECASE):
            return True
    return False


def should_followup(state: FollowUpState) -> dict:
    """Determine if and what kind of follow-up is appropriate right now."""
    now = datetime.now(timezone.utc)
    
    if not state.initial_sync_date:
        return {"should": False, "reason": "no_initial_sync"}
    
    # If user hasn't been asked about ongoing yet, ask after initial sync
    if state.ongoing_enabled is None:
        return {
            "should": True,
            "type": "opt_in",
            "reason": "first_time_choice",
            "message": None,  # The agent crafts this naturally at the end of initial sync
            "instruction": "At the end of the initial Soulsync setup, naturally ask if they want you to keep learning over time. Something like: 'By the way — want me to keep picking up on things as we chat? I\'ll learn your preferences naturally without asking a bunch of questions. You can turn it off anytime.' Don't make it a big deal. It's a casual offer.",
        }
    
    # If user opted out of ongoing, only respond to explicit corrections
    if not state.ongoing_enabled:
        return {"should": False, "reason": "user_opted_out"}
    
    initial = datetime.fromisoformat(state.initial_sync_date)
    days_since_sync = (now - initial).days
    
    last_followup = None
    if state.last_followup_date:
        last_followup = datetime.fromisoformat(state.last_followup_date)
    
    days_since_followup = (now - last_followup).days if last_followup else days_since_sync
    
    result = {"should": False, "type": None, "reason": None, "message": None}
    
    # Pending confirmations take priority — weave them into natural conversation
    if state.pending_confirmations:
        conf = state.pending_confirmations[0]
        result = {
            "should": True,
            "type": "soft_confirm",
            "reason": "pending_confirmation",
            "confirmation": conf,
            "instruction": f"You picked up that {conf.get('fact', '')} from a recent conversation. Next time it's naturally relevant, casually confirm it. DON'T say 'I noticed you mentioned...' Instead, USE the fact and see if they correct you. Example: if you learned they like hiking, next time they ask about weekend plans, say 'Weather looks good for a hike this weekend' — if they go with it, it's confirmed. If they say 'I don't hike', you learned something.",
        }
        return result
    
    # Week 1: Light touch, woven into normal conversation
    if state.followup_count == 0 and days_since_sync >= 3 and days_since_sync < 10:
        result = {
            "should": True,
            "type": "accuracy_check",
            "reason": "3_days_post_sync",
            "message": None,
            "instruction": "It's been a few days since setup. DON'T send a 'check-in' message. Instead, next time the user asks you something, naturally weave in a reference to something from their profile. If they correct you, update the file. If they don't, you're good. Example: 'Here's that code — kept it concise since that's how you like it.' This implicitly confirms their communication preference without asking.",
        }
    
    # Week 2-3: Offer data imports only if there's a natural opening
    elif state.followup_count == 1 and days_since_sync >= 14 and days_since_sync < 25:
        result = {
            "should": True,
            "type": "data_import_offer",
            "reason": "2_weeks_post_sync",
            "message": None,
            "instruction": "It's been 2 weeks. DON'T send a standalone 'want to connect accounts?' message. Instead, wait for a moment where data imports would naturally help. Example: if they ask about their code, say 'I could answer this better if I could see your GitHub. Want me to connect it? Takes 10 seconds.' Only offer ONE source at a time, and only when relevant.",
        }
    
    # Monthly: Only if there's something to update
    elif days_since_followup >= 30:
        result = {
            "should": True,
            "type": "life_update",
            "reason": "monthly_checkin",
            "message": None,
            "instruction": "It's been a month. DON'T send a formal check-in. Instead, if you notice your profile info might be stale (e.g., they mention something that contradicts their profile), gently update. If nothing seems stale, do nothing. Silence is fine.",
        }
    
    # Trust progression: unlock deeper dimensions naturally
    if state.trust_level == "initial" and state.followup_count >= 2:
        state.trust_level = "comfortable"
    
    return result


def handle_correction(message: str, state: FollowUpState) -> dict:
    """Process a user correction and suggest file updates."""
    state.corrections_made.append({
        "message": message[:200],
        "detected_at": datetime.now(timezone.utc).isoformat(),
    })
    state.save()
    
    return {
        "action": "correction_detected",
        "instruction": (
            "The user is correcting something about their profile. "
            "Read their message carefully, identify what changed, "
            "update the relevant file (USER.md, SOUL.md, or MEMORY.md), "
            "and confirm the change. Example: 'Updated — I've changed your timezone to Pacific.'"
        ),
        "files_to_check": ["USER.md", "SOUL.md", "MEMORY.md", "PERSONALITY.md"],
    }


def handle_life_change(changes: list[dict], state: FollowUpState) -> dict:
    """Process detected life changes and suggest profile updates."""
    state.life_changes_detected.extend(changes)
    state.save()
    
    categories = [c["category"] for c in changes]
    
    prompts = {
        "job": "Sounds like your work situation changed. Want me to update your profile?",
        "location": "New location? I should update your timezone and location info.",
        "relationship": "Life update noted. Want me to update your profile with this?",
        "interests": "New interest detected! Want me to add that to your profile?",
        "technical": "Sounds like your tech setup changed. Want me to update what I know about your stack?",
    }
    
    messages = [prompts.get(c, "") for c in categories if c in prompts]
    
    return {
        "action": "life_change_detected",
        "categories": categories,
        "suggestion": " ".join(messages),
        "files_to_update": ["USER.md", "MEMORY.md"],
    }


def process_message(message: str) -> dict | None:
    """Check any incoming message for follow-up triggers. Returns None if no action needed."""
    state = FollowUpState.load()
    
    # Check for corrections
    if detect_correction(message):
        return handle_correction(message, state)
    
    # Check for life changes
    changes = detect_life_changes(message)
    if changes:
        return handle_life_change(changes, state)
    
    return None


# ─── Passive Learning ─────────────────────────────────────────────────
# Picks up facts from normal conversation without explicitly asking

PASSIVE_PATTERNS = {
    "preference": [
        (r"i (?:really )?(?:love|like|enjoy|prefer|am into)\s+(.+?)(?:\.|$|,|!)", "likes"),
        (r"i (?:hate|dislike|can't stand|don't like|avoid)\s+(.+?)(?:\.|$|,|!)", "dislikes"),
        (r"i(?:'m| am) (?:a )?(morning person|night owl|early bird|vegetarian|vegan)", "trait"),
        (r"my favorite (\w+) is (.+?)(?:\.|$|,)", "favorite"),
    ],
    "context": [
        (r"i(?:'m| am) (?:at|in|visiting|heading to)\s+(.+?)(?:\.|$|,)", "location_mention"),
        (r"my (?:wife|husband|partner|girlfriend|boyfriend|spouse)\s+(\w+)", "partner_name"),
        (r"my (?:kid|son|daughter|child)(?:'s name is| named| called)\s+(\w+)", "child_name"),
        (r"my (?:dog|cat|pet)(?:'s name is| named| called)\s+(\w+)", "pet_name"),
        (r"i work (?:at|for)\s+(.+?)(?:\.|$|,)", "employer"),
        (r"my (solar panels|boat|rv|node|server|printer|tesla)", "possession"),
        (r"i (?:have|own) (?:a|an|some)\s+(.+?)(?:\.|$|,| and| that| which)", "possession"),
    ],
}


def passive_scan(message: str, state: FollowUpState) -> list[dict]:
    """Scan a normal message for personality/preference signals.
    Returns list of facts learned. Does NOT update files — just queues for confirmation."""
    
    if not state.passive_learning:
        return []
    
    learned = []
    msg_lower = message.lower()
    
    for category, patterns in PASSIVE_PATTERNS.items():
        for pattern, fact_type in patterns:
            for match in re.finditer(pattern, message, re.IGNORECASE):
                fact = match.group(1).strip() if match.lastindex == 1 else f"{match.group(1)}: {match.group(2)}".strip()
                
                # Don't re-learn things we already know
                already_known = any(
                    l.get("fact", "").lower() == fact.lower() 
                    for l in state.learned_passively
                )
                if already_known:
                    continue
                
                entry = {
                    "fact": fact,
                    "type": fact_type,
                    "category": category,
                    "source_message": message[:100],
                    "learned_at": datetime.now(timezone.utc).isoformat(),
                    "confirmed": False,
                }
                learned.append(entry)
                state.learned_passively.append(entry)
                
                # Queue for soft confirmation later
                state.pending_confirmations.append(entry)
    
    if learned:
        state.save()
    
    return learned


def set_ongoing(enabled: bool, state: FollowUpState = None):
    """User opts in or out of ongoing personalization."""
    if state is None:
        state = FollowUpState.load()
    state.ongoing_enabled = enabled
    state.save()
    return {"ongoing_enabled": enabled}


def confirm_fact(fact_index: int, confirmed: bool, state: FollowUpState = None):
    """Mark a pending confirmation as confirmed or rejected."""
    if state is None:
        state = FollowUpState.load()
    
    if 0 <= fact_index < len(state.pending_confirmations):
        fact = state.pending_confirmations.pop(fact_index)
        if confirmed:
            fact["confirmed"] = True
            # Update the learned list too
            for l in state.learned_passively:
                if l.get("fact") == fact.get("fact"):
                    l["confirmed"] = True
        else:
            # Remove from learned
            state.learned_passively = [
                l for l in state.learned_passively 
                if l.get("fact") != fact.get("fact")
            ]
        state.save()
        return {"action": "confirmed" if confirmed else "rejected", "fact": fact}
    
    return {"error": "invalid index"}


# ─── CLI ──────────────────────────────────────────────────────────────

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: followup.py <check|scan|mark-complete|status> [message]")
        sys.exit(1)
    
    cmd = sys.argv[1]
    
    if cmd == "check":
        state = FollowUpState.load()
        result = should_followup(state)
        print(json.dumps(result, indent=2))
    
    elif cmd == "scan" and len(sys.argv) >= 3:
        message = " ".join(sys.argv[2:])
        result = process_message(message)
        print(json.dumps(result, indent=2) if result else '{"action": "none"}')
    
    elif cmd == "mark-complete":
        state = FollowUpState.load()
        if not state.initial_sync_date:
            state.initial_sync_date = datetime.now(timezone.utc).isoformat()
        state.last_followup_date = datetime.now(timezone.utc).isoformat()
        state.followup_count += 1
        state.save()
        print(json.dumps({"status": "marked", "count": state.followup_count}))
    
    elif cmd == "passive" and len(sys.argv) >= 3:
        message = " ".join(sys.argv[2:])
        state = FollowUpState.load()
        learned = passive_scan(message, state)
        print(json.dumps({"learned": learned, "pending_confirmations": len(state.pending_confirmations)}, indent=2))
    
    elif cmd == "opt-in":
        print(json.dumps(set_ongoing(True)))
    
    elif cmd == "opt-out":
        print(json.dumps(set_ongoing(False)))
    
    elif cmd == "status":
        state = FollowUpState.load()
        print(json.dumps({
            "initial_sync": state.initial_sync_date,
            "last_followup": state.last_followup_date,
            "followup_count": state.followup_count,
            "trust_level": state.trust_level,
            "ongoing_enabled": state.ongoing_enabled,
            "passive_learning": state.passive_learning,
            "corrections": len(state.corrections_made),
            "life_changes": len(state.life_changes_detected),
            "passively_learned": len(state.learned_passively),
            "pending_confirmations": len(state.pending_confirmations),
        }, indent=2))
    
    else:
        print(json.dumps({"error": f"Unknown command: {cmd}"}))


path: lib/importers/apple.py
content: #!/usr/bin/env python3
"""
Apple Importer — Analyzes Apple ecosystem data for personality insights.
Works with data available on macOS or from Apple's data export.

Supports:
- iCloud Contacts (via exported .vcf — Contacts app → File → Export vCard)
- Apple Notes (via ~/Library/Group Containers on macOS)
- Apple Music/iTunes library (via Music library XML)
- Safari bookmarks
- Apple Health summary (from Apple data export)

For non-Mac users: works with Apple's privacy data export
(privacy.apple.com → Request a copy of your data)
"""
import os
import sys
import json
import glob
import platform
import plistlib
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"
IS_MACOS = platform.system() == "Darwin"

def find_apple_data():
    """Locate Apple data sources."""
    sources = {}
    
    # iCloud contacts export (.vcf)
    vcf_paths = [
        os.path.expanduser("~/Downloads/vCards.vcf"),
        os.path.expanduser("~/Downloads/contacts.vcf"),
        os.path.expanduser("~/Desktop/contacts.vcf"),
        os.path.expanduser("~/Documents/contacts.vcf"),
    ]
    for p in vcf_paths:
        if os.path.exists(p):
            sources["contacts_vcf"] = p
            break
    
    if IS_MACOS:
        # Safari bookmarks
        safari_bookmarks = os.path.expanduser("~/Library/Safari/Bookmarks.plist")
        if os.path.exists(safari_bookmarks):
            sources["safari_bookmarks"] = safari_bookmarks
        
        # Apple Music library
        music_lib = os.path.expanduser("~/Music/Music/Music Library.musiclibrary/Library.musicdb")
        music_xml = os.path.expanduser("~/Music/iTunes/iTunes Music Library.xml")
        if os.path.exists(music_xml):
            sources["music_library"] = music_xml
    
    # Apple data export (from privacy.apple.com)
    export_paths = [
        os.path.expanduser("~/Downloads"),
        os.path.expanduser("~/Desktop"),
    ]
    for base in export_paths:
        if not os.path.isdir(base):
            continue
        for d in os.listdir(base):
            full = os.path.join(base, d)
            if os.path.isdir(full) and ("apple" in d.lower() or "icloud" in d.lower()):
                sources["apple_export"] = full
                break
    
    return sources

def parse_vcf_contacts(filepath):
    """Parse vCard file for contact insights."""
    contacts = []
    current = {}
    
    with open(filepath, "r", errors="replace") as f:
        for line in f:
            line = line.strip()
            if line == "BEGIN:VCARD":
                current = {}
            elif line == "END:VCARD":
                if current.get("name"):
                    contacts.append(current)
                current = {}
            elif ":" in line:
                key, _, value = line.partition(":")
                key = key.split(";")[0].upper()
                if key == "FN":
                    current["name"] = value
                elif key == "ORG":
                    current["org"] = value.rstrip(";")
                elif key == "EMAIL":
                    current.setdefault("emails", []).append(value)
    
    orgs = Counter()
    for c in contacts:
        if c.get("org"):
            orgs[c["org"]] += 1
    
    return {
        "total": len(contacts),
        "top_organizations": [o for o, _ in orgs.most_common(5)],
        "sample_names": [c["name"] for c in contacts[:15]],
    }

def parse_safari_bookmarks(filepath):
    """Extract interests from Safari bookmarks."""
    try:
        with open(filepath, "rb") as f:
            bookmarks = plistlib.load(f)
    except Exception:
        return None
    
    urls = []
    titles = []
    
    def walk(node):
        if isinstance(node, dict):
            if "URLString" in node:
                urls.append(node["URLString"])
            if "URIDictionary" in node:
                title = node["URIDictionary"].get("title", "")
                if title:
                    titles.append(title)
            for child in node.get("Children", []):
                walk(child)
    
    walk(bookmarks)
    
    # Extract domains for interest mapping
    domains = Counter()
    for url in urls:
        try:
            from urllib.parse import urlparse
            domain = urlparse(url).netloc.replace("www.", "")
            if domain and "apple.com" not in domain:
                domains[domain] += 1
        except:
            pass
    
    return {
        "bookmark_count": len(urls),
        "top_sites": [d for d, _ in domains.most_common(15)],
        "bookmark_titles": titles[:20],
    }

def analyze_apple_data(sources):
    """Combine all Apple data sources into insights."""
    insights = {
        "interests": [],
        "personality_traits": [],
        "communication_style": "",
        "tone": "",
        "key_contacts": [],
    }
    items_processed = 0
    
    # Contacts
    if "contacts_vcf" in sources:
        contact_data = parse_vcf_contacts(sources["contacts_vcf"])
        insights["key_contacts"] = contact_data["sample_names"]
        insights["network_size"] = contact_data["total"]
        insights["top_organizations"] = contact_data["top_organizations"]
        items_processed += contact_data["total"]
        
        if contact_data["total"] > 500:
            insights["personality_traits"].append("large network — highly connected")
        elif contact_data["total"] > 100:
            insights["personality_traits"].append("solid network")
    
    # Safari bookmarks
    if "safari_bookmarks" in sources:
        bookmark_data = parse_safari_bookmarks(sources["safari_bookmarks"])
        if bookmark_data:
            insights["interests"].extend(bookmark_data["top_sites"][:10])
            insights["bookmarks"] = bookmark_data["bookmark_titles"][:15]
            items_processed += bookmark_data["bookmark_count"]
            
            # Infer traits from bookmark patterns
            sites = " ".join(bookmark_data["top_sites"]).lower()
            if any(s in sites for s in ["github", "stackoverflow", "dev.", "docs."]):
                insights["personality_traits"].append("technical/developer")
            if any(s in sites for s in ["reddit", "hackernews", "twitter"]):
                insights["personality_traits"].append("online community participant")
            if any(s in sites for s in ["medium", "substack", "wordpress"]):
                insights["personality_traits"].append("reader/writer")
    
    # Apple ecosystem indicators
    traits = insights["personality_traits"]
    traits.append("Apple ecosystem user")
    if IS_MACOS:
        traits.append("macOS user")
    
    insights["personality_traits"] = list(dict.fromkeys(traits))  # Deduplicate
    
    return {
        "source": "apple",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": insights,
        "confidence": min(items_processed / 100, 1.0),
        "items_processed": items_processed,
        "data_sources_found": list(sources.keys()),
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    # Allow explicit VCF path
    if len(sys.argv) > 1 and os.path.exists(sys.argv[1]):
        sources = {}
        if sys.argv[1].endswith(".vcf"):
            sources["contacts_vcf"] = sys.argv[1]
        elif os.path.isdir(sys.argv[1]):
            sources["apple_export"] = sys.argv[1]
        result = analyze_apple_data(sources)
    else:
        sources = find_apple_data()
        if not sources:
            result = {
                "source": "apple",
                "processed_at": datetime.now(timezone.utc).isoformat(),
                "error": "No Apple data found.",
                "setup_help": (
                    "To use this importer, do one or more of:\n"
                    "• Export contacts: Open Contacts app → File → Export vCard → save to ~/Downloads/\n"
                    "• Safari bookmarks are read automatically on macOS\n"
                    "• Full data export: Visit privacy.apple.com → Request a copy of your data\n"
                    "  Download and extract to ~/Downloads/"
                ),
            }
        else:
            result = analyze_apple_data(sources)
    
    output_path = os.path.join(IMPORT_DIR, "apple.json")
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2)
    
    print(json.dumps(result, indent=2))


path: lib/importers/calendar.py
content: #!/usr/bin/env python3
"""
Calendar Importer — Analyzes Google Calendar for schedule patterns and lifestyle insights.
Requires Google Calendar OAuth (reuses Gmail credentials with calendar scope).
"""
import os
import sys
import json
from datetime import datetime, timezone, timedelta
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"
WORKSPACE = os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace"))
DAYS_BACK = 90  # Analyze last N days

def get_calendar_service():
    """Initialize Google Calendar API service."""
    try:
        from google.oauth2.credentials import Credentials
        from google.auth.transport.requests import Request
        from googleapiclient.discovery import build
    except ImportError:
        print(json.dumps({"error": "Google API dependencies not installed."}))
        sys.exit(1)
    
    SCOPES = ["https://www.googleapis.com/auth/calendar.readonly"]
    
    token_paths = [
        os.path.join(WORKSPACE, "mission-control", "data", "gmail_token.json"),
        os.path.join(WORKSPACE, "email-agent", "token.json"),
    ]
    
    token_path = next((p for p in token_paths if os.path.exists(p)), None)
    if not token_path:
        print(json.dumps({"error": "No Google OAuth token found. Need calendar scope."}))
        sys.exit(1)
    
    creds = Credentials.from_authorized_user_file(token_path, SCOPES)
    if creds and creds.expired and creds.refresh_token:
        creds.refresh(Request())
    
    return build("calendar", "v3", credentials=creds)

def analyze_calendar(service):
    """Analyze recent calendar events for patterns."""
    now = datetime.now(timezone.utc)
    start = (now - timedelta(days=DAYS_BACK)).isoformat()
    end = now.isoformat()
    
    events_result = service.events().list(
        calendarId="primary",
        timeMin=start, timeMax=end,
        maxResults=500, singleEvents=True,
        orderBy="startTime"
    ).execute()
    
    events = events_result.get("items", [])
    
    # Analyze patterns
    event_hours = Counter()
    event_days = Counter()
    event_types = Counter()
    recurring_count = 0
    meeting_count = 0
    all_day_count = 0
    total_duration_hours = 0
    
    day_names = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
    
    for event in events:
        start_time = event.get("start", {})
        
        if "dateTime" in start_time:
            dt = datetime.fromisoformat(start_time["dateTime"])
            event_hours[dt.hour] += 1
            event_days[day_names[dt.weekday()]] += 1
            
            # Estimate duration
            end_time = event.get("end", {}).get("dateTime")
            if end_time:
                end_dt = datetime.fromisoformat(end_time)
                duration = (end_dt - dt).total_seconds() / 3600
                total_duration_hours += duration
        elif "date" in start_time:
            all_day_count += 1
        
        # Count meetings (events with attendees)
        if event.get("attendees"):
            meeting_count += 1
        
        if event.get("recurringEventId"):
            recurring_count += 1
        
        # Categorize by summary keywords
        summary = event.get("summary", "").lower()
        for keyword in ["meeting", "call", "sync", "standup", "1:1"]:
            if keyword in summary:
                event_types["meetings"] += 1
                break
        for keyword in ["lunch", "dinner", "coffee", "gym", "workout", "doctor"]:
            if keyword in summary:
                event_types["personal"] += 1
                break
    
    # Synthesize
    peak_hours = [h for h, _ in event_hours.most_common(3)]
    busiest_days = [d for d, _ in event_days.most_common(3)]
    avg_events_per_week = len(events) / max(DAYS_BACK / 7, 1)
    avg_duration = total_duration_hours / max(len(events) - all_day_count, 1)
    
    # Determine lifestyle patterns
    patterns = []
    if avg_events_per_week > 15:
        patterns.append("Very busy schedule — heavily calendared")
    elif avg_events_per_week > 7:
        patterns.append("Moderately scheduled")
    else:
        patterns.append("Light calendar — flexible schedule")
    
    if meeting_count / max(len(events), 1) > 0.5:
        patterns.append("Meeting-heavy — lots of collaboration")
    
    weekend_events = event_days.get("Saturday", 0) + event_days.get("Sunday", 0)
    if weekend_events / max(sum(event_days.values()), 1) > 0.2:
        patterns.append("Active on weekends")
    else:
        patterns.append("Weekends mostly free")
    
    return {
        "source": "calendar",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": {
            "work_patterns": "; ".join(patterns),
            "schedule_density": f"{avg_events_per_week:.1f} events/week avg",
            "peak_hours": [f"{h}:00" for h in peak_hours],
            "busiest_days": busiest_days,
            "meeting_ratio": f"{meeting_count}/{len(events)} events are meetings",
            "avg_event_duration": f"{avg_duration:.1f} hours",
            "interests": list(event_types.keys()),
        },
        "confidence": min(len(events) / 100, 1.0),
        "items_processed": len(events),
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    try:
        service = get_calendar_service()
        result = analyze_calendar(service)
        
        output_path = os.path.join(IMPORT_DIR, "calendar.json")
        with open(output_path, "w") as f:
            json.dump(result, f, indent=2)
        
        print(json.dumps(result, indent=2))
    except Exception as e:
        print(json.dumps({"error": str(e)}))
        sys.exit(1)


path: lib/importers/contacts.py
content: #!/usr/bin/env python3
"""
Contacts Importer — Analyzes contact data to build a relationship map.
Supports Google Contacts (People API) and vCard (.vcf) files.

Extracts: key people, relationship categories, communication frequency.
"""
import os
import sys
import json
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"
WORKSPACE = os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace"))

def get_google_contacts():
    """Fetch contacts via Google People API."""
    try:
        from google.oauth2.credentials import Credentials
        from google.auth.transport.requests import Request
        from googleapiclient.discovery import build
    except ImportError:
        return None, "Google API dependencies not installed."
    
    SCOPES = ["https://www.googleapis.com/auth/contacts.readonly"]
    
    token_paths = [
        os.path.join(WORKSPACE, "mission-control", "data", "gmail_token.json"),
        os.path.join(WORKSPACE, "email-agent", "token.json"),
    ]
    
    token_path = next((p for p in token_paths if os.path.exists(p)), None)
    if not token_path:
        return None, "No Google OAuth token found."
    
    try:
        creds = Credentials.from_authorized_user_file(token_path, SCOPES)
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        
        service = build("people", "v1", credentials=creds)
        results = service.people().connections().list(
            resourceName="people/me",
            pageSize=500,
            personFields="names,emailAddresses,phoneNumbers,organizations,relations,birthdays,memberships"
        ).execute()
        
        return results.get("connections", []), None
    except Exception as e:
        return None, str(e)

def parse_vcf_file(filepath):
    """Parse a vCard (.vcf) file into contact entries."""
    contacts = []
    current = {}
    
    try:
        with open(filepath, "r", errors="replace") as f:
            for line in f:
                line = line.strip()
                if line == "BEGIN:VCARD":
                    current = {}
                elif line == "END:VCARD":
                    if current:
                        contacts.append(current)
                    current = {}
                elif ":" in line:
                    key, _, value = line.partition(":")
                    key = key.split(";")[0].upper()
                    if key == "FN":
                        current["name"] = value
                    elif key == "EMAIL":
                        current.setdefault("emails", []).append(value)
                    elif key == "TEL":
                        current.setdefault("phones", []).append(value)
                    elif key == "ORG":
                        current["organization"] = value
                    elif key == "TITLE":
                        current["title"] = value
    except FileNotFoundError:
        pass
    
    return contacts

def analyze_contacts(contacts, source="google"):
    """Extract relationship insights from contacts."""
    total = len(contacts)
    
    organizations = Counter()
    has_email = 0
    has_phone = 0
    has_org = 0
    categories = {"work": 0, "personal": 0, "unknown": 0}
    names = []
    
    for contact in contacts:
        if source == "google":
            name_data = contact.get("names", [{}])[0]
            name = name_data.get("displayName", "")
            emails = [e.get("value", "") for e in contact.get("emailAddresses", [])]
            phones = [p.get("value", "") for p in contact.get("phoneNumbers", [])]
            orgs = contact.get("organizations", [])
        else:  # vcf
            name = contact.get("name", "")
            emails = contact.get("emails", [])
            phones = contact.get("phones", [])
            orgs = [{"name": contact.get("organization", "")}] if contact.get("organization") else []
        
        if name:
            names.append(name)
        if emails:
            has_email += 1
        if phones:
            has_phone += 1
        if orgs:
            has_org += 1
            for org in orgs:
                org_name = org.get("name", "") if isinstance(org, dict) else str(org)
                if org_name:
                    organizations[org_name] += 1
        
        # Categorize (heuristic)
        email_str = " ".join(emails).lower()
        if any(domain in email_str for domain in [".edu", ".gov", ".org"]) or orgs:
            categories["work"] += 1
        elif phones and not emails:
            categories["personal"] += 1
        else:
            categories["unknown"] += 1
    
    # Determine social style
    if total > 500:
        social_style = "Large network — highly connected"
    elif total > 200:
        social_style = "Solid network — well-connected"
    elif total > 50:
        social_style = "Moderate network — selective connections"
    else:
        social_style = "Small, tight network — values close relationships"
    
    top_orgs = [org for org, _ in organizations.most_common(5)]
    
    return {
        "source": "contacts",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": {
            "key_contacts": names[:20],
            "network_size": total,
            "social_style": social_style,
            "top_organizations": top_orgs,
            "contact_breakdown": {
                "with_email": has_email,
                "with_phone": has_phone,
                "with_organization": has_org,
            },
            "categories": categories,
            "interests": [],
            "communication_style": f"{'Email-centric' if has_email > has_phone else 'Phone-centric'} communicator based on contact data",
        },
        "confidence": min(total / 100, 1.0),
        "items_processed": total,
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    source = "google"
    contacts = None
    error = None
    
    # Check for VCF file argument
    if len(sys.argv) > 1 and sys.argv[1].endswith(".vcf"):
        contacts = parse_vcf_file(sys.argv[1])
        source = "vcf"
        if not contacts:
            error = f"No contacts found in {sys.argv[1]}"
    else:
        # Try Google Contacts
        contacts, error = get_google_contacts()
    
    if error or contacts is None:
        # Fallback: check for exported VCF in common locations
        vcf_paths = [
            os.path.expanduser("~/Downloads/contacts.vcf"),
            os.path.expanduser("~/contacts.vcf"),
            os.path.join(WORKSPACE, "contacts.vcf"),
        ]
        for path in vcf_paths:
            if os.path.exists(path):
                contacts = parse_vcf_file(path)
                source = "vcf"
                error = None
                break
    
    if error or not contacts:
        result = {
            "source": "contacts",
            "processed_at": datetime.now(timezone.utc).isoformat(),
            "error": error or "No contacts found. Export a .vcf file or set up Google Contacts OAuth.",
        }
    else:
        result = analyze_contacts(contacts, source)
    
    output_path = os.path.join(IMPORT_DIR, "contacts.json")
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2)
    
    print(json.dumps(result, indent=2))


path: lib/importers/facebook.py
content: #!/usr/bin/env python3
"""
Facebook Importer — Analyzes Facebook data export for personality insights.
Facebook doesn't have a public scraping path like Twitter, so this works with
the user's data download (Settings → Your Information → Download Your Information).

Supports: JSON format export from Facebook (profile, posts, likes, friends).
"""
import os
import sys
import json
import glob
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"

def find_facebook_export():
    """Look for Facebook data export in common locations."""
    search_paths = [
        os.path.expanduser("~/Downloads"),
        os.path.expanduser("~/Desktop"),
        os.path.expanduser("~/Documents"),
        os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace")),
    ]
    
    for base in search_paths:
        if not os.path.isdir(base):
            continue
        # Facebook exports are usually in a folder like facebook-username or facebook-data
        for pattern in ["facebook-*", "fb_data*", "your_facebook_data"]:
            matches = glob.glob(os.path.join(base, pattern))
            for match in matches:
                if os.path.isdir(match):
                    return match
        # Also check for zip extraction
        for d in os.listdir(base):
            full = os.path.join(base, d)
            if os.path.isdir(full) and os.path.exists(os.path.join(full, "profile_information")):
                return full
    
    return None

def read_fb_json(export_dir, *path_parts):
    """Read a JSON file from Facebook export structure."""
    filepath = os.path.join(export_dir, *path_parts)
    if not os.path.exists(filepath):
        return None
    try:
        with open(filepath, "r", encoding="utf-8") as f:
            return json.load(f)
    except (json.JSONDecodeError, UnicodeDecodeError):
        return None

def decode_fb_text(text):
    """Facebook exports encode text in latin-1 mojibake. Fix it."""
    if isinstance(text, str):
        try:
            return text.encode("latin-1").decode("utf-8")
        except (UnicodeDecodeError, UnicodeEncodeError):
            return text
    return text

def analyze_export(export_dir):
    """Analyze Facebook data export."""
    insights = {
        "interests": [],
        "personality_traits": [],
        "communication_style": "",
        "tone": "",
        "key_contacts": [],
    }
    items_processed = 0
    
    # Profile information
    profile = read_fb_json(export_dir, "profile_information", "profile_information.json")
    if profile:
        profile_data = profile.get("profile_v2", {})
        name = decode_fb_text(profile_data.get("name", {}).get("full_name", ""))
        bio = decode_fb_text(profile_data.get("intro_bio", ""))
        insights["profile"] = {"name": name, "bio": bio}
    
    # Page likes → interests
    likes = read_fb_json(export_dir, "likes_and_reactions", "pages.json")
    if likes:
        page_categories = Counter()
        page_names = []
        for page in likes.get("page_likes_v2", []):
            name = decode_fb_text(page.get("name", ""))
            if name:
                page_names.append(name)
            items_processed += 1
        insights["liked_pages"] = page_names[:30]
        insights["interests"].extend(page_names[:20])
    
    # Posts → communication style and tone
    posts = read_fb_json(export_dir, "posts", "your_posts_1.json")
    if posts:
        post_texts = []
        post_lengths = []
        questions = 0
        exclamations = 0
        links = 0
        photos = 0
        
        for post in posts[:200]:
            data = post.get("data", [{}])
            for d in data:
                text = decode_fb_text(d.get("post", ""))
                if text:
                    post_texts.append(text)
                    post_lengths.append(len(text))
                    if "?" in text:
                        questions += 1
                    if "!" in text:
                        exclamations += 1
                    if "http" in text:
                        links += 1
            
            attachments = post.get("attachments", [])
            for att in attachments:
                for ad in att.get("data", []):
                    if ad.get("media"):
                        photos += 1
            
            items_processed += 1
        
        total = max(len(post_texts), 1)
        avg_len = sum(post_lengths) / max(len(post_lengths), 1)
        
        tone_signals = []
        if exclamations / total > 0.3:
            tone_signals.append("enthusiastic")
        if questions / total > 0.2:
            tone_signals.append("inquisitive")
        if links / total > 0.3:
            tone_signals.append("shares resources")
        if photos / total > 0.5:
            tone_signals.append("visual communicator")
        
        if avg_len < 100:
            tone_signals.append("concise poster")
        elif avg_len > 300:
            tone_signals.append("writes longer-form posts")
        
        insights["tone"] = ", ".join(tone_signals) if tone_signals else "balanced"
        insights["communication_style"] = f"Avg post: {avg_len:.0f} chars. {photos} photos in {total} posts."
    
    # Friends → key contacts (just count, not names for privacy)
    friends = read_fb_json(export_dir, "friends_and_followers", "friends.json")
    if friends:
        friend_list = friends.get("friends_v2", [])
        insights["network_size"] = len(friend_list)
        # Only include first names for privacy
        insights["key_contacts"] = [
            decode_fb_text(f.get("name", "")).split()[0]
            for f in friend_list[:10]
            if f.get("name")
        ]
    
    # Groups → community interests
    groups = read_fb_json(export_dir, "groups", "your_group_membership_activity.json")
    if groups:
        group_names = []
        for group in groups.get("groups_joined_v2", []):
            name = decode_fb_text(group.get("name", ""))
            if name:
                group_names.append(name)
                insights["interests"].append(name)
        insights["groups"] = group_names[:15]
    
    return {
        "source": "facebook",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": insights,
        "confidence": min(items_processed / 100, 1.0),
        "items_processed": items_processed,
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    # Check for explicit path argument
    export_dir = None
    if len(sys.argv) > 1 and os.path.isdir(sys.argv[1]):
        export_dir = sys.argv[1]
    else:
        export_dir = find_facebook_export()
    
    if not export_dir:
        result = {
            "source": "facebook",
            "processed_at": datetime.now(timezone.utc).isoformat(),
            "error": "No Facebook data export found.",
            "setup_help": (
                "To use this importer:\n"
                "1. Go to Facebook → Settings → Your Information → Download Your Information\n"
                "2. Select JSON format, Medium quality\n"
                "3. Download and extract to ~/Downloads/\n"
                "4. Run this importer again"
            ),
        }
    else:
        result = analyze_export(export_dir)
    
    output_path = os.path.join(IMPORT_DIR, "facebook.json")
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2)
    
    print(json.dumps(result, indent=2))


path: lib/importers/github.py
content: #!/usr/bin/env python3
"""
GitHub Importer — Analyzes GitHub profile and repos for technical insights.
Uses GitHub REST API. Requires a GitHub token (PAT) or public profile access.
"""
import os
import sys
import json
import subprocess
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"

def get_github_data(username=None, token=None):
    """Fetch GitHub profile and repo data."""
    import urllib.request
    
    headers = {"Accept": "application/vnd.github.v3+json"}
    if [REDACTED_SECRET]"Authorization"] = f"Bearer {token}"
    
    def fetch(url):
        req = urllib.request.Request(url, headers=headers)
        with urllib.request.urlopen(req) as resp:
            return json.loads(resp.read())
    
    # Get authenticated user or specified username
    if username:
        user = fetch(f"https://api.github.com/users/{username}")
        repos = fetch(f"https://api.github.com/users/{username}/repos?per_page=100&sort=updated")
    else:
        user = fetch("https://api.github.com/user")
        repos = fetch("https://api.github.com/user/repos?per_page=100&sort=updated&affiliation=owner")
    
    return user, repos

def analyze(user, repos):
    """Extract personality/technical insights from GitHub data."""
    languages = Counter()
    topics = []
    total_stars = 0
    total_forks = 0
    has_blog = bool(user.get("blog"))
    
    for repo in repos:
        if repo.get("fork") and not repo.get("stargazers_count", 0) > 0:
            continue  # Skip unmodified forks
        
        lang = repo.get("language")
        if lang:
            languages[lang] += 1
        
        total_stars += repo.get("stargazers_count", 0)
        total_forks += repo.get("forks_count", 0)
        
        if repo.get("topics"):
            topics.extend(repo["topics"])
        
        # Use description as interest signal
        desc = repo.get("description", "")
        if desc:
            topics.append(desc)
    
    top_languages = [lang for lang, _ in languages.most_common(8)]
    topic_counts = Counter(topics)
    top_topics = [t for t, _ in topic_counts.most_common(15) if len(t) < 50]
    
    # Determine technical level
    repo_count = len([r for r in repos if not r.get("fork")])
    if repo_count > 50 and total_stars > 100:
        level = "advanced — prolific open source contributor"
    elif repo_count > 20:
        level = "experienced — active developer with many projects"
    elif repo_count > 5:
        level = "intermediate — builds personal projects"
    elif repo_count > 0:
        level = "beginner — getting started with public code"
    else:
        level = "unknown — no public repos"
    
    return {
        "source": "github",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": {
            "technical_skills": f"{', '.join(top_languages[:5])} developer — {level}",
            "languages": top_languages,
            "interests": top_topics[:15],
            "key_contacts": [],  # Could analyze collaborators
            "communication_style": f"{'Active open source contributor' if repo_count > 10 else 'Builds primarily private/personal projects'}",
            "profile": {
                "username": user.get("login"),
                "name": user.get("name"),
                "bio": user.get("bio"),
                "company": user.get("company"),
                "location": user.get("location"),
                "blog": user.get("blog"),
                "public_repos": repo_count,
                "followers": user.get("followers", 0),
                "total_stars": total_stars,
            }
        },
        "confidence": min(repo_count / 20, 1.0),
        "items_processed": len(repos),
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    # Try to find GitHub token
    [REDACTED_SECRET]"GITHUB_TOKEN")
    username = sys.argv[1] if len(sys.argv) > 1 else None
    
    if not token:
        # Try git credential store
        try:
            result = subprocess.run(
                ["git", "credential", "fill"],
                input="protocol=https\nhost=github.com\n",
                capture_output=True, text=True, timeout=5
            )
            for line in result.stdout.split("\n"):
                if line.startswith("password="):
                    [REDACTED_SECRET]"=", 1)[1]
        except:
            pass
    
    try:
        user, repos = get_github_data(username=username, token=token)
        result = analyze(user, repos)
        
        output_path = os.path.join(IMPORT_DIR, "github.json")
        with open(output_path, "w") as f:
            json.dump(result, f, indent=2)
        
        print(json.dumps(result, indent=2))
    except Exception as e:
        print(json.dumps({"error": str(e)}))
        sys.exit(1)


path: lib/importers/gmail.py
content: #!/usr/bin/env python3
"""
Gmail Importer v2 — Analyzes recent emails to extract personality insights.

Classifies emails into categories (personal, work, transactional, marketing/newsletters)
and extracts different signals from each:
- Personal: communication style, tone, key relationships
- Work: industry, role signals, work patterns
- Newsletters/Marketing: actual interests (what they subscribe to)
- Transactional: spending patterns, services used (light signals)
- Spam/Noise: ignored entirely

Processes locally. Raw email content is never stored — only synthesized insights.
"""
import os
import sys
import json
import re
import base64
from datetime import datetime, timezone
from email.utils import parseaddr
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"
WORKSPACE = os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace"))
MAX_EMAILS = 300  # Analyze last N emails (mix of sent + inbox)
MAX_SENT = 150
MAX_INBOX = 150


# === Email Classification ===

# Domains/patterns that indicate transactional/automated emails
TRANSACTIONAL_SENDERS = {
    "noreply", "no-reply", "donotreply", "do-not-reply", "mailer-daemon",
    "notifications", "notification", "alert", "alerts", "billing",
    "receipt", "receipts", "orders", "shipping", "tracking",
    "verify", "verification", "confirm", "confirmation", "security",
    "support", "help", "info", "admin", "system", "automated",
    "postmaster", "bounce",
}

TRANSACTIONAL_DOMAINS = {
    "amazonses.com", "sendgrid.net", "mailchimp.com", "constantcontact.com",
    "mailgun.org", "sparkpostmail.com", "mandrillapp.com",
}

SPAM_SUBJECT_PATTERNS = [
    r"(?i)unsubscribe", r"(?i)verify your", r"(?i)confirm your",
    r"(?i)reset your password", r"(?i)security alert",
    r"(?i)your order", r"(?i)order confirmation", r"(?i)shipping confirmation",
    r"(?i)delivery notification", r"(?i)tracking number",
    r"(?i)your receipt", r"(?i)payment received", r"(?i)invoice",
    r"(?i)your statement", r"(?i)account update",
    r"(?i)act now", r"(?i)limited time", r"(?i)exclusive offer",
    r"(?i)you've been selected", r"(?i)congratulations",
    r"(?i)click here", r"(?i)sign up",
]

# Subject patterns that indicate newsletters/marketing (but contain interest signals)
NEWSLETTER_PATTERNS = [
    r"(?i)weekly digest", r"(?i)daily digest", r"(?i)newsletter",
    r"(?i)this week in", r"(?i)top stories", r"(?i)what's new",
    r"(?i)roundup", r"(?i)\bnews\b", r"(?i)update from",
    r"(?i)new episode", r"(?i)new post", r"(?i)just published",
]

# Known newsletter/content sender patterns → interest categories
INTEREST_SENDER_MAP = {
    # Bitcoin / Crypto
    "bitcoin": "Bitcoin & Cryptocurrency",
    "crypto": "Cryptocurrency",
    "lightning": "Lightning Network",
    "coindesk": "Cryptocurrency News",
    "coinbase": "Cryptocurrency",
    "blockfi": "Crypto Finance",
    "swan": "Bitcoin",
    "river": "Bitcoin",
    "strike": "Bitcoin/Lightning",
    "fold": "Bitcoin Rewards",
    # Tech
    "github": "Software Development",
    "gitlab": "Software Development",
    "hackernews": "Tech News",
    "ycombinator": "Tech / Startups",
    "techcrunch": "Tech News",
    "theverge": "Tech News",
    "wired": "Technology",
    "arstechnica": "Technology",
    "substack": "Independent Writing",
    # AI
    "openai": "Artificial Intelligence",
    "anthropic": "AI Research",
    "huggingface": "Machine Learning",
    # Home / Energy
    "solar": "Solar Energy",
    "tesla": "Electric Vehicles / Tesla",
    "homeassistant": "Home Automation",
    # Finance
    "robinhood": "Stock Trading",
    "schwab": "Investing",
    "fidelity": "Investing",
    "baird": "Financial Advisory",
    "vanguard": "Investing",
    # Shopping / Services (light signals)
    "amazon": "Online Shopping",
    "ebay": "Online Shopping",
}


def classify_email(from_addr, from_name, subject, labels):
    """Classify an email into a category.
    
    Returns: 'personal', 'work', 'newsletter', 'transactional', or 'spam'
    """
    from_lower = from_addr.lower()
    from_name_lower = (from_name or "").lower()
    subject_lower = (subject or "").lower()
    label_set = set(labels or [])

    # Gmail's own spam/trash
    if "SPAM" in label_set or "TRASH" in label_set:
        return "spam"

    # Check sender for transactional patterns
    local_part = from_lower.split("@")[0] if "@" in from_lower else ""
    domain = from_lower.split("@")[1] if "@" in from_lower else ""

    if local_part in TRANSACTIONAL_SENDERS:
        # But check if it's a newsletter first
        for pattern in NEWSLETTER_PATTERNS:
            if re.search(pattern, subject or ""):
                return "newsletter"
        return "transactional"

    if any(d in domain for d in TRANSACTIONAL_DOMAINS):
        return "transactional"

    # Check subject for spam patterns
    spam_score = sum(1 for p in SPAM_SUBJECT_PATTERNS if re.search(p, subject or ""))
    if spam_score >= 2:
        return "transactional"

    # Check for newsletter patterns
    for pattern in NEWSLETTER_PATTERNS:
        if re.search(pattern, subject or ""):
            return "newsletter"

    # Promotions label = marketing/newsletter
    if "CATEGORY_PROMOTIONS" in label_set:
        return "newsletter"

    # Social label
    if "CATEGORY_SOCIAL" in label_set:
        return "newsletter"

    # Forums label
    if "CATEGORY_FORUMS" in label_set:
        return "newsletter"

    # Updates label — likely transactional
    if "CATEGORY_UPDATES" in label_set:
        return "transactional"

    # If it's in SENT, it's personal or work
    if "SENT" in label_set:
        return "personal"

    # Default: if it's in primary inbox, it's personal
    if "CATEGORY_PERSONAL" in label_set or "INBOX" in label_set:
        return "personal"

    return "personal"


def extract_interest_from_sender(from_addr, from_name):
    """Try to derive an interest category from the sender."""
    combined = f"{from_addr} {from_name or ''}".lower()
    for keyword, interest in INTEREST_SENDER_MAP.items():
        if keyword in combined:
            return interest
    return None


def extract_interest_from_subject(subject):
    """Try to derive interest topics from newsletter/marketing subjects."""
    if not subject:
        return []
    
    interests = []
    lower = subject.lower()
    
    # Topic keyword detection
    topic_keywords = {
        "bitcoin": "Bitcoin",
        "btc": "Bitcoin",
        "lightning network": "Lightning Network",
        "crypto": "Cryptocurrency",
        "ethereum": "Ethereum",
        "ai ": "Artificial Intelligence",
        "artificial intelligence": "Artificial Intelligence",
        "machine learning": "Machine Learning",
        "llm": "Large Language Models",
        "open source": "Open Source",
        "solar": "Solar Energy",
        "home automation": "Home Automation",
        "smart home": "Smart Home",
        "3d print": "3D Printing",
        "raspberry pi": "Raspberry Pi / SBC",
        "self-host": "Self-Hosting",
        "privacy": "Digital Privacy",
        "security": "Cybersecurity",
        "linux": "Linux",
        "docker": "Containers / Docker",
        "kubernetes": "Kubernetes",
        "python": "Python",
        "node.js": "Node.js",
        "javascript": "JavaScript",
        "rust": "Rust",
        "nostr": "Nostr / Decentralized Social",
        "decentraliz": "Decentralization",
        "sovereign": "Digital Sovereignty",
        "self-improv": "Self Improvement",
        "productiv": "Productivity",
        "startup": "Startups",
        "entrepren": "Entrepreneurship",
        "real estate": "Real Estate",
        "invest": "Investing",
        "trading": "Trading",
        "options": "Options Trading",
        "photo booth": "Photo Booth / Events",
        "event": "Events Industry",
    }
    
    for keyword, topic in topic_keywords.items():
        if keyword in lower:
            interests.append(topic)
    
    return interests


def get_gmail_service():
    """Initialize Gmail API service."""
    try:
        from google.oauth2.credentials import Credentials
        from google_auth_oauthlib.flow import InstalledAppFlow
        from google.auth.transport.requests import Request
        from googleapiclient.discovery import build
    except ImportError:
        print(json.dumps({"error": "Gmail dependencies not installed. Run: pip install google-auth-oauthlib google-api-python-client"}))
        sys.exit(1)

    SCOPES = ["https://www.googleapis.com/auth/gmail.readonly"]
    creds = None

    token_paths = [
        os.path.join(WORKSPACE, "mission-control", "data", "gmail_token.json"),
        os.path.join(WORKSPACE, "email-agent", "token.json"),
        os.path.join(WORKSPACE, "token.json"),
    ]
    creds_paths = [
        os.path.join(WORKSPACE, "mission-control", "credentials.json"),
        os.path.join(WORKSPACE, "email-agent", "credentials.json"),
        os.path.join(WORKSPACE, "credentials.json"),
    ]

    token_path = next((p for p in token_paths if os.path.exists(p)), None)
    creds_path = next((p for p in creds_paths if os.path.exists(p)), None)

    if token_path:
        creds = Credentials.from_authorized_user_file(token_path, SCOPES)

    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        elif creds_path:
            flow = InstalledAppFlow.from_client_secrets_file(creds_path, SCOPES)
            creds = flow.run_local_server(port=0)
        else:
            print(json.dumps({"error": "No Gmail credentials found. Need credentials.json for OAuth."}))
            sys.exit(1)

    return build("gmail", "v1", credentials=creds)


def analyze_emails(service):
    """Fetch and analyze emails across multiple categories for personality insights."""
    
    # === Fetch sent emails (communication style) ===
    sent_results = service.users().messages().list(
        userId="me", maxResults=MAX_SENT, labelIds=["SENT"]
    ).execute()
    sent_messages = sent_results.get("messages", [])

    # === Fetch inbox emails (interests, subscriptions, relationships) ===
    inbox_results = service.users().messages().list(
        userId="me", maxResults=MAX_INBOX, labelIds=["INBOX"]
    ).execute()
    inbox_messages = inbox_results.get("messages", [])

    # === Process all messages ===
    categories = {"personal": [], "work": [], "newsletter": [], "transactional": [], "spam": []}
    recipients = Counter()
    senders = Counter()
    send_hours = [0] * 24
    send_days = [0] * 7
    total_sent_length = 0
    sent_count = 0
    interest_signals = Counter()  # topic → count (more mentions = stronger signal)
    services_used = set()
    newsletter_senders = Counter()

    all_messages = [(m, "SENT") for m in sent_messages] + [(m, "INBOX") for m in inbox_messages]

    for msg_meta, source_label in all_messages:
        try:
            msg = service.users().messages().get(
                userId="me", id=msg_meta["id"], format="metadata",
                metadataHeaders=["To", "From", "Subject", "Date", "List-Unsubscribe"]
            ).execute()

            headers = {h["name"]: h["value"] for h in msg.get("payload", {}).get("headers", [])}
            labels = msg.get("labelIds", [])
            
            # Parse sender
            from_raw = headers.get("From", "")
            from_name, from_addr = parseaddr(from_raw)
            
            # Parse subject
            subject = headers.get("Subject", "")
            
            # Has unsubscribe header? Strong newsletter signal
            has_unsub = "List-Unsubscribe" in headers

            # Classify
            category = classify_email(from_addr, from_name, subject, labels)
            
            # Override: if it has List-Unsubscribe, it's a newsletter/marketing
            if has_unsub and category == "personal":
                category = "newsletter"

            categories[category].append({
                "from": from_addr,
                "from_name": from_name,
                "subject": subject,
                "labels": labels,
            })

            # === Extract signals based on category ===

            if source_label == "SENT":
                # Track recipients
                to = headers.get("To", "")
                _, to_email = parseaddr(to)
                if to_email:
                    recipients[to_email] += 1

                # Track timing
                date_str = headers.get("Date", "")
                if date_str:
                    try:
                        from email.utils import parsedate_to_datetime
                        dt = parsedate_to_datetime(date_str)
                        send_hours[dt.hour] += 1
                        send_days[dt.weekday()] += 1
                    except Exception:
                        pass

                # Track message size (for communication style)
                total_sent_length += msg.get("sizeEstimate", 0)
                sent_count += 1

            if category == "newsletter":
                # Extract interest from sender
                interest = extract_interest_from_sender(from_addr, from_name)
                if interest:
                    interest_signals[interest] += 1
                
                # Extract interest from subject
                subject_interests = extract_interest_from_subject(subject)
                for si in subject_interests:
                    interest_signals[si] += 1
                
                newsletter_senders[from_addr] += 1

            elif category == "transactional":
                # Light signal: what services do they use?
                domain = from_addr.split("@")[1] if "@" in from_addr else ""
                base_domain = ".".join(domain.split(".")[-2:]) if domain else ""
                if base_domain and base_domain not in {"gmail.com", "yahoo.com", "outlook.com"}:
                    services_used.add(base_domain)

            elif category == "personal" and source_label == "INBOX":
                # Track who sends them personal email
                if from_addr:
                    senders[from_addr] += 1

        except Exception:
            continue

    # === Synthesize Insights ===

    # Communication style (from sent emails)
    avg_length = total_sent_length // max(sent_count, 1)
    if avg_length < 500:
        comm_style = "concise and to-the-point"
    elif avg_length < 2000:
        comm_style = "balanced — detailed when needed, brief when not"
    elif avg_length < 5000:
        comm_style = "thorough and detailed"
    else:
        comm_style = "very detailed writer"

    # Work patterns
    peak_hours = sorted(range(24), key=lambda h: send_hours[h], reverse=True)[:3]
    peak_days = sorted(range(7), key=lambda d: send_days[d], reverse=True)[:3]
    day_names = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
    
    weekend_total = send_days[5] + send_days[6]
    total_sends = sum(send_days) or 1
    weekend_pct = weekend_total / total_sends * 100
    
    night_total = sum(send_hours[22:]) + sum(send_hours[:6])
    total_hourly = sum(send_hours) or 1
    night_pct = night_total / total_hourly * 100

    patterns = f"Most active: {', '.join(f'{h}:00' for h in peak_hours)}"
    patterns += f"; Busiest days: {', '.join(day_names[d] for d in peak_days)}"
    if weekend_pct < 10:
        patterns += "; Rarely emails on weekends"
    elif weekend_pct > 30:
        patterns += "; Frequently active on weekends"
    if night_pct > 20:
        patterns += "; Night owl"

    # Tone estimation
    if avg_length < 800:
        tone = "concise communicator — gets to the point"
    elif avg_length < 3000:
        tone = "balanced communicator — adapts length to context"
    else:
        tone = "detailed communicator — provides thorough context"

    # Top interests (sorted by signal strength)
    top_interests = [topic for topic, count in interest_signals.most_common(15) if count >= 1]

    # Key contacts (from sent — who they write to most)
    top_recipients = [email for email, _ in recipients.most_common(10)]
    
    # People who write to them most (from inbox personal)
    top_senders = [email for email, _ in senders.most_common(10)]
    
    # Merge contacts: sent recipients are stronger signal
    all_contacts = []
    seen = set()
    for c in top_recipients + top_senders:
        if c not in seen:
            seen.add(c)
            all_contacts.append(c)
    all_contacts = all_contacts[:10]

    # Category breakdown (useful for understanding email habits)
    category_counts = {k: len(v) for k, v in categories.items()}

    return {
        "source": "gmail",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": {
            "communication_style": comm_style,
            "key_contacts": all_contacts,
            "interests": top_interests,
            "work_patterns": patterns,
            "tone": tone,
            "services_used": sorted(services_used)[:15],
            "top_newsletter_senders": [s for s, _ in newsletter_senders.most_common(10)],
        },
        "email_breakdown": category_counts,
        "confidence": min(sent_count / MAX_SENT, 1.0),
        "items_processed": len(all_messages),
        "details": {
            "sent_analyzed": sent_count,
            "inbox_analyzed": len(inbox_messages),
            "avg_sent_size_bytes": avg_length,
            "interests_signal_strength": dict(interest_signals.most_common(20)),
        }
    }


if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)

    if "--check" in sys.argv:
        try:
            service = get_gmail_service()
            print(json.dumps({"available": True}))
        except Exception as e:
            print(json.dumps({"available": False, "error": str(e)}))
        sys.exit(0)

    try:
        service = get_gmail_service()
        result = analyze_emails(service)

        output_path = os.path.join(IMPORT_DIR, "gmail.json")
        with open(output_path, "w") as f:
            json.dump(result, f, indent=2)

        print(json.dumps(result, indent=2))
    except Exception as e:
        print(json.dumps({"error": str(e)}))
        sys.exit(1)


path: lib/importers/instagram.py
content: #!/usr/bin/env python3
"""
Instagram Importer — Analyzes Instagram data export for personality insights.
Works with the JSON data download from Instagram (Settings → Your Activity → Download Your Information).

Also supports basic public profile scraping via web fetch as fallback.
"""
import os
import sys
import json
import glob
import re
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"

def find_instagram_export():
    """Look for Instagram data export in common locations."""
    search_paths = [
        os.path.expanduser("~/Downloads"),
        os.path.expanduser("~/Desktop"),
        os.path.expanduser("~/Documents"),
        os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace")),
    ]
    
    for base in search_paths:
        if not os.path.isdir(base):
            continue
        for pattern in ["instagram-*", "ig_data*", "your_instagram_activity"]:
            matches = glob.glob(os.path.join(base, pattern))
            for match in matches:
                if os.path.isdir(match):
                    return match
        # Check for nested structure
        for d in os.listdir(base):
            full = os.path.join(base, d)
            if os.path.isdir(full):
                # Instagram exports have specific structure
                if any(os.path.exists(os.path.join(full, sub)) for sub in
                       ["personal_information", "content", "your_instagram_activity"]):
                    return full
    
    return None

def read_ig_json(export_dir, *path_parts):
    """Read a JSON file from Instagram export."""
    filepath = os.path.join(export_dir, *path_parts)
    if not os.path.exists(filepath):
        # Try alternate paths (Instagram export structure varies)
        alt_paths = [
            os.path.join(export_dir, "your_instagram_activity", *path_parts),
            os.path.join(export_dir, "instagram", *path_parts),
        ]
        for alt in alt_paths:
            if os.path.exists(alt):
                filepath = alt
                break
        else:
            return None
    try:
        with open(filepath, "r", encoding="utf-8") as f:
            return json.load(f)
    except (json.JSONDecodeError, UnicodeDecodeError):
        return None

def decode_ig_text(text):
    """Instagram exports have the same mojibake issue as Facebook."""
    if isinstance(text, str):
        try:
            return text.encode("latin-1").decode("utf-8")
        except (UnicodeDecodeError, UnicodeEncodeError):
            return text
    return text

def analyze_export(export_dir):
    """Analyze Instagram data export."""
    insights = {
        "interests": [],
        "personality_traits": [],
        "communication_style": "",
        "tone": "",
        "key_contacts": [],
    }
    items_processed = 0
    
    # Profile info
    profile = read_ig_json(export_dir, "personal_information", "personal_information.json")
    if profile:
        pi = profile.get("profile_user", [{}])
        if isinstance(pi, list) and pi:
            pi = pi[0]
        string_data = pi.get("string_map_data", {})
        insights["profile"] = {
            "username": string_data.get("Username", {}).get("value", ""),
            "bio": decode_ig_text(string_data.get("Bio", {}).get("value", "")),
            "name": decode_ig_text(string_data.get("Name", {}).get("value", "")),
        }
    
    # Posts → captions → interests and tone
    posts = read_ig_json(export_dir, "content", "posts_1.json")
    if not posts:
        posts = read_ig_json(export_dir, "your_instagram_activity", "content", "posts_1.json")
    
    if posts:
        captions = []
        hashtags = Counter()
        mentions = Counter()
        post_hours = Counter()
        
        post_list = posts if isinstance(posts, list) else []
        
        for post in post_list[:300]:
            # Extract caption
            media_list = post.get("media", [])
            if isinstance(media_list, list):
                for media in media_list:
                    title = decode_ig_text(media.get("title", ""))
                    if title:
                        captions.append(title)
                        # Extract hashtags
                        for tag in re.findall(r'#(\w+)', title):
                            hashtags[tag.lower()] += 1
                        # Extract mentions
                        for mention in re.findall(r'@(\w+)', title):
                            mentions[mention.lower()] += 1
                    
                    # Track posting time
                    ts = media.get("creation_timestamp")
                    if ts:
                        dt = datetime.fromtimestamp(ts, tz=timezone.utc)
                        post_hours[dt.hour] += 1
            
            items_processed += 1
        
        # Analyze captions
        total = max(len(captions), 1)
        avg_len = sum(len(c) for c in captions) / total if captions else 0
        emoji_count = sum(1 for c in captions for ch in c if ord(ch) > 0x1F000)
        questions = sum(1 for c in captions if "?" in c)
        
        tone_signals = []
        if emoji_count / total > 1:
            tone_signals.append("emoji-heavy communicator")
        if avg_len < 50:
            tone_signals.append("minimal captions — lets photos speak")
        elif avg_len > 200:
            tone_signals.append("storyteller — uses long captions")
        if questions / total > 0.15:
            tone_signals.append("engagement-focused — asks questions")
        
        insights["tone"] = ", ".join(tone_signals) if tone_signals else "visual-first communicator"
        insights["communication_style"] = f"Avg caption: {avg_len:.0f} chars across {len(captions)} posts"
        insights["hashtags"] = [f"#{h}" for h, _ in hashtags.most_common(15)]
        insights["interests"].extend([h for h, _ in hashtags.most_common(20)])
        insights["key_contacts"] = [f"@{m}" for m, _ in mentions.most_common(10)]
        
        # Peak posting times
        if post_hours:
            peak = [h for h, _ in post_hours.most_common(3)]
            insights["posting_pattern"] = f"Most active: {', '.join(f'{h}:00' for h in peak)}"
    
    # Liked posts → interests
    likes = read_ig_json(export_dir, "likes", "liked_posts.json")
    if likes:
        liked_accounts = Counter()
        like_list = likes.get("likes_media_likes", []) if isinstance(likes, dict) else likes
        for like in like_list[:200]:
            if isinstance(like, dict):
                sd = like.get("string_list_data", [{}])
                for item in sd:
                    href = item.get("href", "")
                    if href:
                        # Extract username from URL
                        match = re.search(r'instagram\.com/([^/]+)', href)
                        if match:
                            liked_accounts[match.group(1)] += 1
            items_processed += 1
        
        insights["frequently_liked"] = [a for a, _ in liked_accounts.most_common(10)]
    
    # Following → interests (who they chose to follow)
    following = read_ig_json(export_dir, "connections", "followers_and_following", "following.json")
    if not following:
        following = read_ig_json(export_dir, "followers_and_following", "following.json")
    
    if following:
        follow_list = following.get("relationships_following", [])
        insights["following_count"] = len(follow_list)
        # First 20 follows are usually the most important
        insights["key_follows"] = [
            decode_ig_text(f.get("string_list_data", [{}])[0].get("value", ""))
            for f in follow_list[:20]
            if f.get("string_list_data")
        ]
    
    # Followers count
    followers = read_ig_json(export_dir, "connections", "followers_and_following", "followers_1.json")
    if not followers:
        followers = read_ig_json(export_dir, "followers_and_following", "followers_1.json")
    if followers:
        insights["followers_count"] = len(followers) if isinstance(followers, list) else 0
    
    # Determine personality traits from content patterns
    traits = []
    if insights.get("following_count", 0) > 1000:
        traits.append("broad interests — follows many accounts")
    if items_processed > 200:
        traits.append("active content creator")
    if insights.get("hashtags"):
        traits.append("community-engaged — uses hashtags to connect")
    
    insights["personality_traits"] = traits
    
    return {
        "source": "instagram",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": insights,
        "confidence": min(items_processed / 100, 1.0),
        "items_processed": items_processed,
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    export_dir = None
    if len(sys.argv) > 1 and os.path.isdir(sys.argv[1]):
        export_dir = sys.argv[1]
    else:
        export_dir = find_instagram_export()
    
    if not export_dir:
        result = {
            "source": "instagram",
            "processed_at": datetime.now(timezone.utc).isoformat(),
            "error": "No Instagram data export found.",
            "setup_help": (
                "To use this importer:\n"
                "1. Go to Instagram → Settings → Your Activity → Download Your Information\n"
                "2. Select 'All time', JSON format\n"
                "3. Request download, wait for email\n"
                "4. Download and extract to ~/Downloads/\n"
                "5. Run this importer again"
            ),
        }
    else:
        result = analyze_export(export_dir)
    
    output_path = os.path.join(IMPORT_DIR, "instagram.json")
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2)
    
    print(json.dumps(result, indent=2))


path: lib/importers/linkedin.py
content: #!/usr/bin/env python3
"""
LinkedIn Importer — Analyzes LinkedIn data export for professional identity.
LinkedIn requires a data export (Settings → Data Privacy → Get a copy of your data).

Extracts: career trajectory, skills, education, professional network, endorsements.
"""
import os
import sys
import json
import csv
import glob
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"

def find_linkedin_export():
    """Find LinkedIn data export directory."""
    search_paths = [
        os.path.expanduser("~/Downloads"),
        os.path.expanduser("~/Desktop"),
    ]
    
    for base in search_paths:
        if not os.path.isdir(base):
            continue
        for d in os.listdir(base):
            full = os.path.join(base, d)
            if os.path.isdir(full) and "linkedin" in d.lower():
                return full
            # LinkedIn exports have a specific CSV structure
            if os.path.isdir(full) and os.path.exists(os.path.join(full, "Profile.csv")):
                return full
            if os.path.isdir(full) and os.path.exists(os.path.join(full, "Positions.csv")):
                return full
    
    return None

def read_csv(filepath):
    """Read a LinkedIn export CSV file."""
    if not os.path.exists(filepath):
        return []
    try:
        with open(filepath, "r", encoding="utf-8") as f:
            reader = csv.DictReader(f)
            return list(reader)
    except Exception:
        return []

def analyze_export(export_dir):
    """Analyze LinkedIn data export."""
    insights = {
        "interests": [],
        "personality_traits": [],
        "communication_style": "",
        "tone": "",
        "key_contacts": [],
    }
    items_processed = 0
    
    # Profile
    profile_rows = read_csv(os.path.join(export_dir, "Profile.csv"))
    profile = {}
    if profile_rows:
        p = profile_rows[0]
        profile = {
            "name": f"{p.get('First Name', '')} {p.get('Last Name', '')}".strip(),
            "headline": p.get("Headline", ""),
            "summary": p.get("Summary", ""),
            "location": p.get("Geo Location", "") or p.get("Location", ""),
            "industry": p.get("Industry", ""),
        }
        insights["profile"] = profile
        if profile["industry"]:
            insights["interests"].append(profile["industry"])
    
    # Positions (career trajectory)
    positions = read_csv(os.path.join(export_dir, "Positions.csv"))
    career = []
    companies = []
    for pos in positions:
        title = pos.get("Title", "")
        company = pos.get("Company Name", "")
        if title or company:
            career.append(f"{title} at {company}".strip(" at"))
            if company:
                companies.append(company)
            items_processed += 1
    
    insights["career"] = career[:10]
    insights["companies"] = companies[:10]
    
    # Determine career level
    titles_lower = " ".join(t.lower() for t in career)
    if any(t in titles_lower for t in ["ceo", "founder", "president", "owner", "partner"]):
        insights["personality_traits"].append("founder/executive — entrepreneurial")
    elif any(t in titles_lower for t in ["director", "vp", "head of", "chief"]):
        insights["personality_traits"].append("senior leadership")
    elif any(t in titles_lower for t in ["senior", "lead", "principal", "staff"]):
        insights["personality_traits"].append("senior individual contributor")
    elif any(t in titles_lower for t in ["manager"]):
        insights["personality_traits"].append("people manager")
    
    if len(positions) > 5:
        insights["personality_traits"].append("experienced professional — multiple roles")
    if len(set(companies)) > 3:
        insights["personality_traits"].append("diverse experience across companies")
    
    # Skills
    skills = read_csv(os.path.join(export_dir, "Skills.csv"))
    skill_names = [s.get("Name", "") for s in skills if s.get("Name")]
    insights["skills"] = skill_names[:20]
    insights["interests"].extend(skill_names[:10])
    items_processed += len(skills)
    
    # Education
    education = read_csv(os.path.join(export_dir, "Education.csv"))
    edu_list = []
    for edu in education:
        school = edu.get("School Name", "")
        degree = edu.get("Degree Name", "")
        field = edu.get("Notes", "") or edu.get("Field of Study", "")
        if school:
            edu_list.append(f"{degree} from {school}".strip(" from") + (f" ({field})" if field else ""))
            if field:
                insights["interests"].append(field)
        items_processed += 1
    insights["education"] = edu_list
    
    # Connections count
    connections = read_csv(os.path.join(export_dir, "Connections.csv"))
    if connections:
        insights["network_size"] = len(connections)
        # Top connection companies
        conn_companies = Counter()
        for conn in connections:
            company = conn.get("Company", "")
            if company:
                conn_companies[company] += 1
        insights["network_companies"] = [c for c, _ in conn_companies.most_common(10)]
        
        # Key contacts (recent connections)
        insights["key_contacts"] = [
            f"{c.get('First Name', '')} {c.get('Last Name', '')}".strip()
            for c in connections[:10]
            if c.get("First Name")
        ]
        items_processed += len(connections)
    
    # Endorsements
    endorsements = read_csv(os.path.join(export_dir, "Endorsement_Received_Info.csv"))
    if not endorsements:
        endorsements = read_csv(os.path.join(export_dir, "Endorsements_Received.csv"))
    if endorsements:
        endorsed_skills = Counter()
        for e in endorsements:
            skill = e.get("Skill Name", "") or e.get("Skill", "")
            if skill:
                endorsed_skills[skill] += 1
        insights["top_endorsed_skills"] = [s for s, _ in endorsed_skills.most_common(10)]
    
    # Communication style from summary/headline
    if profile.get("summary"):
        summary = profile["summary"]
        if len(summary) > 500:
            insights["communication_style"] = "detailed professional communicator — writes thorough summaries"
        elif len(summary) > 100:
            insights["communication_style"] = "balanced professional voice"
        else:
            insights["communication_style"] = "concise — lets credentials speak"
    
    return {
        "source": "linkedin",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": insights,
        "confidence": min(items_processed / 50, 1.0),
        "items_processed": items_processed,
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    export_dir = None
    if len(sys.argv) > 1 and os.path.isdir(sys.argv[1]):
        export_dir = sys.argv[1]
    else:
        export_dir = find_linkedin_export()
    
    if not export_dir:
        result = {
            "source": "linkedin",
            "processed_at": datetime.now(timezone.utc).isoformat(),
            "error": "No LinkedIn data export found.",
            "setup_help": (
                "To use this importer:\n"
                "1. Go to LinkedIn → Settings → Data Privacy → Get a copy of your data\n"
                "2. Select all data categories\n"
                "3. Request archive, wait for email (can take 24h)\n"
                "4. Download and extract to ~/Downloads/\n"
                "5. Run this importer again"
            ),
        }
    else:
        result = analyze_export(export_dir)
    
    output_path = os.path.join(IMPORT_DIR, "linkedin.json")
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2)
    
    print(json.dumps(result, indent=2))


path: lib/importers/local_system.py
content: #!/usr/bin/env python3
"""
Local System Importer — Analyzes local machine data for technical profile and habits.
No network access needed. Everything is already on the machine.

Scans: shell history, browser bookmarks, installed apps, git repos, SSH config.
This is the most privacy-sensitive importer — all data stays 100% local.
"""
import os
import sys
import json
import glob
import platform
import sqlite3
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"
IS_MACOS = platform.system() == "Darwin"
IS_LINUX = platform.system() == "Linux"

def analyze_shell_history():
    """Analyze bash/zsh history for technical profile."""
    history_files = [
        os.path.expanduser("~/.bash_history"),
        os.path.expanduser("~/.zsh_history"),
        os.path.expanduser("~/.local/share/fish/fish_history"),
    ]
    
    commands = Counter()
    total_lines = 0
    
    for hfile in history_files:
        if not os.path.exists(hfile):
            continue
        try:
            with open(hfile, "r", errors="replace") as f:
                for line in f:
                    line = line.strip()
                    # zsh history format: : timestamp:0;command
                    if line.startswith(":"):
                        line = line.split(";", 1)[-1] if ";" in line else line
                    if not line or line.startswith("#"):
                        continue
                    
                    # Extract base command
                    cmd = line.split()[0] if line.split() else ""
                    cmd = cmd.split("/")[-1]  # Strip path
                    if cmd and len(cmd) < 30:
                        commands[cmd] += 1
                    total_lines += 1
        except:
            continue
    
    return commands, total_lines

def analyze_git_repos():
    """Find and analyze local git repos."""
    home = os.path.expanduser("~")
    repos = []
    languages = Counter()
    
    # Common project directories
    search_dirs = [
        os.path.join(home, "Projects"),
        os.path.join(home, "projects"),
        os.path.join(home, "Code"),
        os.path.join(home, "code"),
        os.path.join(home, "dev"),
        os.path.join(home, "src"),
        os.path.join(home, "workspace"),
        os.path.join(home, ".openclaw", "workspace"),
    ]
    
    for search_dir in search_dirs:
        if not os.path.isdir(search_dir):
            continue
        for d in os.listdir(search_dir):
            full = os.path.join(search_dir, d)
            if os.path.isdir(os.path.join(full, ".git")):
                repos.append(d)
                # Quick language detection
                for ext, lang in [
                    (".py", "Python"), (".js", "JavaScript"), (".ts", "TypeScript"),
                    (".go", "Go"), (".rs", "Rust"), (".rb", "Ruby"), (".java", "Java"),
                    (".cpp", "C++"), (".c", "C"), (".cs", "C#"), (".swift", "Swift"),
                    (".kt", "Kotlin"), (".php", "PHP"), (".sh", "Shell"),
                ]:
                    if glob.glob(os.path.join(full, f"*{ext}")) or glob.glob(os.path.join(full, f"**/*{ext}"), recursive=False):
                        languages[lang] += 1
    
    return repos, languages

def analyze_browser_bookmarks():
    """Extract interests from browser bookmarks."""
    bookmarks = []
    
    # Chrome/Chromium
    chrome_paths = [
        os.path.expanduser("~/.config/google-chrome/Default/Bookmarks"),
        os.path.expanduser("~/.config/chromium/Default/Bookmarks"),
        os.path.expanduser("~/Library/Application Support/Google/Chrome/Default/Bookmarks"),
    ]
    
    for bpath in chrome_paths:
        if os.path.exists(bpath):
            try:
                with open(bpath, "r") as f:
                    data = json.load(f)
                
                def walk(node):
                    if isinstance(node, dict):
                        if node.get("type") == "url":
                            bookmarks.append({
                                "name": node.get("name", ""),
                                "url": node.get("url", ""),
                            })
                        for child in node.get("children", []):
                            walk(child)
                
                for root in data.get("roots", {}).values():
                    walk(root)
            except:
                pass
    
    # Firefox
    firefox_paths = glob.glob(os.path.expanduser("~/.mozilla/firefox/*/places.sqlite"))
    for fpath in firefox_paths[:1]:
        try:
            conn = sqlite3.connect(f"file:{fpath}?mode=ro", uri=True)
            cursor = conn.execute(
                "SELECT b.title, p.url FROM moz_bookmarks b JOIN moz_places p ON b.fk = p.id WHERE b.title IS NOT NULL AND b.title != '' LIMIT 200"
            )
            for title, url in cursor:
                bookmarks.append({"name": title, "url": url})
            conn.close()
        except:
            pass
    
    return bookmarks

def analyze_installed_apps():
    """Detect notable installed applications."""
    notable_apps = {
        # Development
        "code": "VS Code", "vim": "Vim", "nvim": "Neovim", "emacs": "Emacs",
        "docker": "Docker", "kubectl": "Kubernetes", "terraform": "Terraform",
        "node": "Node.js", "python3": "Python", "go": "Go", "rustc": "Rust",
        "java": "Java", "ruby": "Ruby", "php": "PHP", "swift": "Swift",
        # Tools
        "git": "Git", "tmux": "tmux", "ffmpeg": "FFmpeg", "ollama": "Ollama",
        "btop": "btop", "htop": "htop",
        # Apps
        "blender": "Blender", "gimp": "GIMP", "inkscape": "Inkscape",
        "obs": "OBS Studio", "vlc": "VLC",
        "signal-cli": "Signal", "telegram-cli": "Telegram",
        # Bitcoin/crypto
        "bitcoin-cli": "Bitcoin Core", "lncli": "LND", "lightning-cli": "CLN",
    }
    
    found = []
    for cmd, name in notable_apps.items():
        for path_dir in os.environ.get("PATH", "").split(":"):
            if os.path.exists(os.path.join(path_dir, cmd)):
                found.append(name)
                break
    
    return found

def analyze_ssh_config():
    """Check SSH config for infrastructure hints."""
    ssh_config = os.path.expanduser("~/.ssh/config")
    hosts = []
    
    if os.path.exists(ssh_config):
        try:
            with open(ssh_config, "r") as f:
                for line in f:
                    line = line.strip()
                    if line.lower().startswith("host ") and "*" not in line:
                        host = line.split(None, 1)[1]
                        hosts.append(host)
        except:
            pass
    
    return hosts

def import_local():
    """Main local system analysis."""
    shell_commands, history_size = analyze_shell_history()
    repos, languages = analyze_git_repos()
    bookmarks = analyze_browser_bookmarks()
    installed_apps = analyze_installed_apps()
    ssh_hosts = analyze_ssh_config()
    
    items_processed = history_size + len(bookmarks) + len(repos)
    
    # Top commands reveal workflow
    top_commands = [cmd for cmd, _ in shell_commands.most_common(20)]
    
    # Infer traits
    traits = []
    
    # From shell commands
    if "git" in top_commands[:5]:
        traits.append("heavy git user — version control native")
    if "docker" in top_commands[:10]:
        traits.append("containerization workflow")
    if "ssh" in top_commands[:10]:
        traits.append("manages remote systems")
    if any(cmd in top_commands[:10] for cmd in ["vim", "nvim"]):
        traits.append("terminal-native editor user")
    if "code" in top_commands[:10]:
        traits.append("VS Code user")
    if "ollama" in top_commands[:15]:
        traits.append("runs local AI models")
    if any(cmd in top_commands for cmd in ["bitcoin-cli", "lncli", "lightning-cli"]):
        traits.append("Bitcoin node operator")
    
    # From languages
    top_langs = [l for l, _ in languages.most_common(5)]
    if top_langs:
        traits.append(f"codes in {', '.join(top_langs[:3])}")
    
    # From apps
    if "Docker" in installed_apps:
        traits.append("uses containerization")
    if "Ollama" in installed_apps:
        traits.append("local AI/LLM user")
    if "Blender" in installed_apps:
        traits.append("3D modeling/creative")
    
    # Determine technical level
    if history_size > 10000 and len(repos) > 5:
        tech_level = "power user — heavy CLI and development"
    elif history_size > 1000:
        tech_level = "comfortable with terminal"
    else:
        tech_level = "primarily GUI user"
    
    # Extract interests from bookmarks
    bookmark_domains = Counter()
    for b in bookmarks:
        url = b.get("url", "")
        try:
            from urllib.parse import urlparse
            domain = urlparse(url).netloc.replace("www.", "")
            if domain:
                bookmark_domains[domain] += 1
        except:
            pass
    
    return {
        "source": "local_system",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": {
            "technical_skills": tech_level,
            "languages": top_langs,
            "top_commands": top_commands[:15],
            "installed_apps": installed_apps,
            "local_repos": repos[:15],
            "ssh_hosts": ssh_hosts[:10],
            "top_bookmarks": [d for d, _ in bookmark_domains.most_common(15)],
            "personality_traits": traits,
            "interests": [d for d, _ in bookmark_domains.most_common(10)] + top_langs,
            "communication_style": "",
            "tone": "",
        },
        "confidence": min(items_processed / 500, 1.0),
        "items_processed": items_processed,
        "system": {
            "os": platform.system(),
            "shell_history_lines": history_size,
            "local_repos": len(repos),
            "bookmarks": len(bookmarks),
        }
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    result = import_local()
    
    output_path = os.path.join(IMPORT_DIR, "local_system.json")
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2)
    
    print(json.dumps(result, indent=2))


path: lib/importers/reddit.py
content: #!/usr/bin/env python3
"""
Reddit Importer — Analyzes public Reddit profile for true interests and opinions.
Reddit is gold for personalization — people are more honest on Reddit than anywhere else.
Uses old.reddit.com JSON endpoints (no API key needed for public profiles).
"""
import os
import sys
import json
import urllib.request
import time
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"

def fetch_json(url):
    """Fetch JSON from Reddit."""
    req = urllib.request.Request(url, headers={
        "User-Agent": "OpenClaw-Soulsync/1.0 (personalization skill)"
    })
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            return json.loads(resp.read())
    except Exception:
        return None

def get_user_posts(username, limit=100):
    """Fetch user's recent posts and comments."""
    posts = []
    comments = []
    
    # Submissions
    data = fetch_json(f"https://www.reddit.com/user/{username}/submitted.json?limit={limit}&sort=new")
    if data and "data" in data:
        for child in data["data"].get("children", []):
            post = child.get("data", {})
            posts.append({
                "title": post.get("title", ""),
                "subreddit": post.get("subreddit", ""),
                "score": post.get("score", 0),
                "text": post.get("selftext", "")[:500],
                "created": post.get("created_utc", 0),
            })
    
    time.sleep(1)  # Rate limit respect
    
    # Comments
    data = fetch_json(f"https://www.reddit.com/user/{username}/comments.json?limit={limit}&sort=new")
    if data and "data" in data:
        for child in data["data"].get("children", []):
            comment = child.get("data", {})
            comments.append({
                "body": comment.get("body", "")[:500],
                "subreddit": comment.get("subreddit", ""),
                "score": comment.get("score", 0),
                "created": comment.get("created_utc", 0),
            })
    
    return posts, comments

def get_user_about(username):
    """Fetch user profile info."""
    data = fetch_json(f"https://www.reddit.com/user/{username}/about.json")
    if data and "data" in data:
        return data["data"]
    return {}

def analyze_reddit(username):
    """Full Reddit analysis."""
    about = get_user_about(username)
    posts, comments = get_user_posts(username)
    
    if not posts and not comments:
        return {
            "source": "reddit",
            "error": f"No public data found for u/{username}. Profile may be private.",
        }
    
    # Subreddit frequency (strongest interest signal)
    subreddits = Counter()
    for post in posts:
        subreddits[post["subreddit"]] += 2  # Posts weighted higher
    for comment in comments:
        subreddits[comment["subreddit"]] += 1
    
    top_subreddits = [s for s, _ in subreddits.most_common(20)]
    
    # Categorize subreddits into interest areas
    interest_categories = {
        "technology": ["programming", "technology", "python", "javascript", "linux", "android", "apple", "webdev", "machinelearning", "artificial", "compsci", "coding", "learnprogramming", "devops", "homelab", "selfhosted"],
        "finance": ["bitcoin", "cryptocurrency", "wallstreetbets", "investing", "personalfinance", "stocks", "ethereum", "fire", "financialindependence"],
        "gaming": ["gaming", "pcgaming", "ps5", "xbox", "nintendo", "steam", "buildapc", "pcmasterrace"],
        "creative": ["art", "photography", "design", "writing", "diy", "woodworking", "3dprinting", "crafts"],
        "science": ["science", "space", "physics", "biology", "chemistry", "askscience", "dataisbeautiful"],
        "politics": ["politics", "news", "worldnews", "conservative", "liberal", "libertarian"],
        "lifestyle": ["fitness", "cooking", "travel", "outdoors", "camping", "hiking", "running"],
        "entertainment": ["movies", "television", "books", "music", "anime", "memes", "funny"],
    }
    
    detected_categories = Counter()
    for sub in top_subreddits:
        sub_lower = sub.lower()
        for category, keywords in interest_categories.items():
            if any(k in sub_lower for k in keywords):
                detected_categories[category] += 1
    
    # Analyze communication style from comments
    all_text = [c["body"] for c in comments if c["body"]]
    avg_length = sum(len(t) for t in all_text) / max(len(all_text), 1)
    
    questions = sum(1 for t in all_text if "?" in t)
    helpful = sum(1 for t in all_text if any(w in t.lower() for w in ["here's how", "you can", "try this", "i recommend", "in my experience"]))
    snarky = sum(1 for t in all_text if any(w in t.lower() for w in ["lol", "lmao", "/s", "bruh"]))
    
    total = max(len(all_text), 1)
    
    tone_signals = []
    if avg_length > 200:
        tone_signals.append("writes detailed, thorough responses")
    elif avg_length < 50:
        tone_signals.append("keeps it brief")
    if helpful / total > 0.2:
        tone_signals.append("helpful — gives advice and recommendations")
    if questions / total > 0.15:
        tone_signals.append("curious — asks questions")
    if snarky / total > 0.2:
        tone_signals.append("uses humor and casual internet tone")
    
    # Karma analysis
    total_post_karma = sum(p["score"] for p in posts)
    total_comment_karma = sum(c["score"] for c in comments)
    
    traits = []
    if total_comment_karma > total_post_karma * 2:
        traits.append("commenter > poster — engages more than broadcasts")
    elif total_post_karma > total_comment_karma * 2:
        traits.append("content creator — shares more than comments")
    
    account_age_days = (time.time() - about.get("created_utc", time.time())) / 86400
    if account_age_days > 365 * 5:
        traits.append("long-time Reddit user")
    
    if about.get("link_karma", 0) + about.get("comment_karma", 0) > 10000:
        traits.append("active community participant")
    
    return {
        "source": "reddit",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": {
            "interests": top_subreddits[:15],
            "interest_categories": [c for c, _ in detected_categories.most_common(5)],
            "top_subreddits": top_subreddits,
            "communication_style": f"Avg comment: {avg_length:.0f} chars. {len(posts)} posts, {len(comments)} comments.",
            "tone": ", ".join(tone_signals) if tone_signals else "neutral",
            "personality_traits": traits,
            "key_contacts": [],
            "profile": {
                "username": username,
                "karma": about.get("link_karma", 0) + about.get("comment_karma", 0),
                "account_age_days": int(account_age_days),
            },
        },
        "confidence": min((len(posts) + len(comments)) / 50, 1.0),
        "items_processed": len(posts) + len(comments),
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    if len(sys.argv) < 2:
        print(json.dumps({"error": "Usage: reddit.py <username>"}))
        sys.exit(1)
    
    username = sys.argv[1].lstrip("u/").lstrip("/")
    result = analyze_reddit(username)
    
    output_path = os.path.join(IMPORT_DIR, "reddit.json")
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2)
    
    print(json.dumps(result, indent=2))


path: lib/importers/spotify.py
content: #!/usr/bin/env python3
"""
Spotify Importer — Analyzes Spotify listening history for taste and personality insights.
Uses Spotify Web API. Requires OAuth token or uses public playlist data.

Setup: Create app at https://developer.spotify.com/dashboard
Set SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET env vars.
"""
import os
import sys
import json
import urllib.request
import urllib.parse
import base64
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"

def get_spotify_token():
    """Get Spotify access token via client credentials flow."""
    client_id = os.environ.get("SPOTIFY_CLIENT_ID")
    client_secret = os.environ.get("SPOTIFY_CLIENT_SECRET")
    
    if not client_id or not client_secret:
        return None
    
    auth = base64.b64encode(f"{client_id}:{client_secret}".encode()).decode()
    
    data = urllib.parse.urlencode({"grant_type": "client_credentials"}).encode()
    req = urllib.request.Request(
        "https://accounts.spotify.com/api/token",
        data=data,
        headers={
            "Authorization": f"Basic {auth}",
            "Content-Type": "application/x-www-form-urlencoded",
        }
    )
    
    try:
        with urllib.request.urlopen(req) as resp:
            result = json.loads(resp.read())
            return result.get("access_token")
    except Exception:
        return None

def spotify_api(endpoint, token):
    """Make Spotify API request."""
    req = urllib.request.Request(
        f"https://api.spotify.com/v1/{endpoint}",
        headers={"Authorization": f"Bearer {token}"}
    )
    try:
        with urllib.request.urlopen(req) as resp:
            return json.loads(resp.read())
    except Exception:
        return None

def analyze_user_playlists(user_id, token):
    """Analyze a user's public playlists for taste signals."""
    playlists = spotify_api(f"users/{user_id}/playlists?limit=50", token)
    if not playlists or "items" not in playlists:
        return None
    
    genres = Counter()
    artists = Counter()
    playlist_names = []
    total_tracks = 0
    
    for playlist in playlists.get("items", [])[:20]:
        if not playlist:
            continue
        
        playlist_names.append(playlist.get("name", ""))
        track_count = playlist.get("tracks", {}).get("total", 0)
        total_tracks += track_count
        
        # Get first 50 tracks from each playlist
        playlist_id = playlist.get("id")
        if playlist_id:
            tracks = spotify_api(
                f"playlists/{playlist_id}/tracks?limit=50&fields=items(track(artists(name,id)))",
                token
            )
            if tracks and "items" in tracks:
                for item in tracks["items"]:
                    track = item.get("track")
                    if track and track.get("artists"):
                        for artist in track["artists"]:
                            name = artist.get("name")
                            if name:
                                artists[name] += 1
                                
                                # Get artist genres
                                artist_id = artist.get("id")
                                if artist_id:
                                    artist_data = spotify_api(f"artists/{artist_id}", token)
                                    if artist_data and "genres" in artist_data:
                                        for genre in artist_data["genres"]:
                                            genres[genre] += 1
    
    return {
        "playlists": playlist_names,
        "top_artists": [a for a, _ in artists.most_common(20)],
        "top_genres": [g for g, _ in genres.most_common(15)],
        "total_tracks": total_tracks,
        "playlist_count": len(playlists.get("items", [])),
    }

def infer_personality_from_music(analysis):
    """Map music taste to personality traits."""
    if not analysis:
        return {}
    
    traits = []
    genres = [g.lower() for g in analysis.get("top_genres", [])]
    
    # Genre-personality correlations (simplified)
    genre_traits = {
        "hip hop": "culturally engaged, values authenticity",
        "rap": "values self-expression, competitive spirit",
        "rock": "independent, values tradition and rebellion",
        "indie": "creative, values uniqueness, open to new experiences",
        "electronic": "forward-thinking, tech-inclined, likes complexity",
        "jazz": "sophisticated, analytical, values improvisation",
        "classical": "structured thinker, values discipline and beauty",
        "country": "values storytelling, community, and authenticity",
        "r&b": "emotionally expressive, values connection",
        "pop": "social, optimistic, mainstream-aware",
        "metal": "intense, detail-oriented, values technical skill",
        "punk": "anti-establishment, values DIY and directness",
        "folk": "thoughtful, values simplicity and storytelling",
        "latin": "social, values rhythm and celebration",
        "ambient": "introspective, values calm and focus",
        "lo-fi": "creative worker, values focus and atmosphere",
    }
    
    for genre in genres[:8]:
        for key, trait in genre_traits.items():
            if key in genre:
                traits.append(trait)
                break
    
    # Playlist naming patterns
    playlist_names = " ".join(analysis.get("playlists", [])).lower()
    if any(w in playlist_names for w in ["work", "focus", "study", "coding", "productive"]):
        traits.append("uses music for focus/productivity")
    if any(w in playlist_names for w in ["workout", "gym", "run", "energy"]):
        traits.append("fitness-oriented")
    if any(w in playlist_names for w in ["chill", "relax", "sleep", "calm"]):
        traits.append("values downtime and recovery")
    if any(w in playlist_names for w in ["party", "hype", "pregame"]):
        traits.append("social/extroverted side")
    
    # Deduplicate
    return list(dict.fromkeys(traits))[:8]

def import_spotify(user_id):
    """Main import function."""
    [REDACTED_SECRET])
    
    if not [REDACTED_SECRET] {
            "source": "spotify",
            "processed_at": datetime.now(timezone.utc).isoformat(),
            "error": "No Spotify credentials. Set SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET.",
            "setup_help": "Create an app at https://developer.spotify.com/dashboard",
        }
    
    analysis = analyze_user_playlists(user_id, token)
    
    if not analysis:
        return {
            "source": "spotify",
            "processed_at": datetime.now(timezone.utc).isoformat(),
            "error": f"Could not fetch playlists for user '{user_id}'. Profile may be private.",
        }
    
    personality_traits = infer_personality_from_music(analysis)
    
    return {
        "source": "spotify",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": {
            "interests": analysis["top_genres"][:10],
            "personality_traits": personality_traits,
            "top_artists": analysis["top_artists"][:10],
            "top_genres": analysis["top_genres"][:10],
            "music_engagement": f"{analysis['playlist_count']} playlists, {analysis['total_tracks']} total tracks",
            "communication_style": "",  # Music doesn't tell us this directly
            "tone": "",
        },
        "confidence": min(analysis["total_tracks"] / 200, 1.0),
        "items_processed": analysis["total_tracks"],
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    if len(sys.argv) < 2:
        print(json.dumps({"error": "Usage: spotify.py <spotify_user_id>"}))
        sys.exit(1)
    
    user_id = sys.argv[1]
    result = import_spotify(user_id)
    
    output_path = os.path.join(IMPORT_DIR, "spotify.json")
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2)
    
    print(json.dumps(result, indent=2))


path: lib/importers/twitter.py
content: #!/usr/bin/env python3
"""
Twitter/X Importer — Analyzes public Twitter profile and tweets for personality insights.
Uses Nitter instances (no API key needed) or Twitter API if token available.

Falls back to scraping public profile data.
"""
import os
import sys
import json
import urllib.request
import re
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"

# Nitter instances to try (public, no auth needed)
NITTER_INSTANCES = [
    "https://nitter.privacydev.net",
    "https://nitter.poast.org",
    "https://nitter.1d4.us",
]

def fetch_url(url, timeout=10):
    """Fetch URL content."""
    req = urllib.request.Request(url, headers={
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"
    })
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            return resp.read().decode("utf-8", errors="replace")
    except Exception:
        return None

def scrape_nitter(username):
    """Scrape Twitter profile via Nitter."""
    for instance in NITTER_INSTANCES:
        url = f"{instance}/{username}"
        html = fetch_url(url)
        if html and "doesn't exist" not in html.lower():
            return html, instance
    return None, None

def parse_nitter_profile(html, username):
    """Extract profile data from Nitter HTML."""
    profile = {"username": username}
    
    # Display name
    match = re.search(r'class="profile-card-fullname"[^>]*>([^<]+)', html)
    if match:
        profile["name"] = match.group(1).strip()
    
    # Bio
    match = re.search(r'class="profile-bio"[^>]*>(.*?)</p>', html, re.DOTALL)
    if match:
        bio = re.sub(r'<[^>]+>', '', match.group(1)).strip()
        profile["bio"] = bio
    
    # Location
    match = re.search(r'class="profile-location"[^>]*>([^<]+)', html)
    if match:
        profile["location"] = match.group(1).strip()
    
    # Stats
    for stat in ["tweets", "following", "followers"]:
        match = re.search(rf'class="profile-stat-num"[^>]*>([^<]+)</[^>]*>\s*<[^>]*>{stat}', html, re.IGNORECASE)
        if match:
            val = match.group(1).strip().replace(",", "")
            profile[stat] = val
    
    return profile

def parse_nitter_tweets(html):
    """Extract recent tweets from Nitter HTML."""
    tweets = []
    
    # Find tweet content blocks
    tweet_blocks = re.findall(
        r'class="tweet-content[^"]*"[^>]*>(.*?)</div>',
        html, re.DOTALL
    )
    
    for block in tweet_blocks[:50]:
        text = re.sub(r'<[^>]+>', ' ', block).strip()
        text = re.sub(r'\s+', ' ', text)
        if text and len(text) > 10:
            tweets.append(text)
    
    return tweets

def analyze_tweets(tweets, profile):
    """Extract personality insights from tweet content."""
    if not tweets:
        return {}
    
    # Topic extraction via word frequency
    stop_words = set("the a an is are was were be been being have has had do does did will would shall should may might can could i me my we our you your he she it they them their this that these those and but or nor for yet so at by from in into of on to with as".split())
    
    words = []
    hashtags = []
    mentions = []
    urls = 0
    questions = 0
    exclamations = 0
    
    for tweet in tweets:
        # Count punctuation patterns
        if "?" in tweet:
            questions += 1
        if "!" in tweet:
            exclamations += 1
        if "http" in tweet:
            urls += 1
        
        # Extract hashtags
        hashtags.extend(re.findall(r'#(\w+)', tweet))
        
        # Extract mentions
        mentions.extend(re.findall(r'@(\w+)', tweet))
        
        # Word frequency
        for word in tweet.lower().split():
            word = re.sub(r'[^a-z0-9]', '', word)
            if word and word not in stop_words and len(word) > 2:
                words.append(word)
    
    word_freq = Counter(words)
    hashtag_freq = Counter(hashtags)
    
    # Determine tone
    tone_signals = []
    if exclamations / max(len(tweets), 1) > 0.3:
        tone_signals.append("enthusiastic")
    if questions / max(len(tweets), 1) > 0.2:
        tone_signals.append("curious/engaging")
    if urls / max(len(tweets), 1) > 0.4:
        tone_signals.append("shares links/resources frequently")
    
    avg_length = sum(len(t) for t in tweets) / max(len(tweets), 1)
    if avg_length < 100:
        tone_signals.append("concise tweeter")
    elif avg_length > 200:
        tone_signals.append("writes longer-form tweets/threads")
    
    # Top interests from word frequency and hashtags
    interests = [w for w, _ in word_freq.most_common(20) if len(w) > 3]
    hashtag_interests = [f"#{h}" for h, _ in hashtag_freq.most_common(10)]
    
    return {
        "tone": ", ".join(tone_signals) if tone_signals else "measured, balanced",
        "interests": interests[:15],
        "hashtags": hashtag_interests,
        "communication_style": f"Avg tweet length: {avg_length:.0f} chars. {'Thread writer' if avg_length > 200 else 'Concise communicator'}.",
        "engagement_style": f"{questions} questions, {exclamations} exclamations, {urls} links shared in {len(tweets)} tweets",
        "top_mentions": [f"@{m}" for m, _ in Counter(mentions).most_common(5)],
    }

def import_twitter(username):
    """Main import function."""
    html, instance = scrape_nitter(username)
    
    if not html:
        return {
            "source": "twitter",
            "error": "Could not fetch Twitter profile. Nitter instances may be down.",
            "processed_at": datetime.now(timezone.utc).isoformat(),
        }
    
    profile = parse_nitter_profile(html, username)
    tweets = parse_nitter_tweets(html)
    analysis = analyze_tweets(tweets, profile)
    
    return {
        "source": "twitter",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": {
            "communication_style": analysis.get("communication_style", ""),
            "tone": analysis.get("tone", ""),
            "interests": analysis.get("interests", []),
            "key_contacts": analysis.get("top_mentions", []),
            "hashtags": analysis.get("hashtags", []),
            "engagement": analysis.get("engagement_style", ""),
            "profile": profile,
        },
        "confidence": min(len(tweets) / 30, 1.0),
        "items_processed": len(tweets),
        "nitter_instance": instance,
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    if len(sys.argv) < 2:
        print(json.dumps({"error": "Usage: twitter.py <username>"}))
        sys.exit(1)
    
    username = sys.argv[1].lstrip("@")
    
    if "--check" in sys.argv:
        html, instance = scrape_nitter(username)
        print(json.dumps({"available": html is not None, "instance": instance}))
        sys.exit(0)
    
    result = import_twitter(username)
    
    output_path = os.path.join(IMPORT_DIR, "twitter.json")
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2)
    
    print(json.dumps(result, indent=2))


path: lib/importers/youtube.py
content: #!/usr/bin/env python3
"""
YouTube Importer — Analyzes YouTube watch history and subscriptions for deep interest mapping.
Supports: Google Takeout export (watch-history.json, subscriptions.json)
and YouTube Data API v3 if token available.

YouTube watch history is one of the richest signals for true interests —
what you watch when nobody's looking reveals more than what you post publicly.
"""
import os
import sys
import json
import glob
from datetime import datetime, timezone
from collections import Counter

IMPORT_DIR = "/tmp/soulsync"

def find_youtube_export():
    """Find Google Takeout YouTube data."""
    search_paths = [
        os.path.expanduser("~/Downloads"),
        os.path.expanduser("~/Desktop"),
    ]
    
    for base in search_paths:
        if not os.path.isdir(base):
            continue
        # Google Takeout structure
        for d in os.listdir(base):
            full = os.path.join(base, d)
            yt_path = os.path.join(full, "YouTube and YouTube Music")
            if os.path.isdir(yt_path):
                return yt_path
            # Alt path
            yt_path2 = os.path.join(full, "Takeout", "YouTube and YouTube Music")
            if os.path.isdir(yt_path2):
                return yt_path2
    
    return None

def parse_watch_history(export_dir):
    """Parse YouTube watch history from Takeout export."""
    history_path = os.path.join(export_dir, "history", "watch-history.json")
    if not os.path.exists(history_path):
        return []
    
    try:
        with open(history_path, "r", encoding="utf-8") as f:
            return json.load(f)
    except (json.JSONDecodeError, UnicodeDecodeError):
        return []

def parse_subscriptions(export_dir):
    """Parse YouTube subscriptions."""
    sub_path = os.path.join(export_dir, "subscriptions", "subscriptions.json")
    if not os.path.exists(sub_path):
        # Try CSV format
        sub_csv = os.path.join(export_dir, "subscriptions", "subscriptions.csv")
        if os.path.exists(sub_csv):
            channels = []
            with open(sub_csv, "r") as f:
                for line in f:
                    parts = line.strip().split(",")
                    if len(parts) >= 3 and parts[0] != "Channel Id":
                        channels.append(parts[2] if len(parts) > 2 else parts[0])
            return channels
        return []
    
    try:
        with open(sub_path, "r", encoding="utf-8") as f:
            data = json.load(f)
            return [item.get("snippet", {}).get("title", "") for item in data if item.get("snippet")]
    except:
        return []

def analyze_youtube(export_dir):
    """Analyze YouTube data for personality insights."""
    watch_history = parse_watch_history(export_dir)
    subscriptions = parse_subscriptions(export_dir)
    
    # Analyze watch history
    channels_watched = Counter()
    watch_hours = Counter()
    watch_days = Counter()
    titles = []
    
    day_names = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
    
    for entry in watch_history[:1000]:
        # Channel
        subtitles = entry.get("subtitles", [])
        for sub in subtitles:
            channel = sub.get("name", "")
            if channel:
                channels_watched[channel] += 1
        
        # Title for topic analysis
        title = entry.get("title", "").replace("Watched ", "")
        if title:
            titles.append(title)
        
        # Timing
        time_str = entry.get("time", "")
        if time_str:
            try:
                dt = datetime.fromisoformat(time_str.replace("Z", "+00:00"))
                watch_hours[dt.hour] += 1
                watch_days[day_names[dt.weekday()]] += 1
            except:
                pass
    
    # Topic extraction from titles
    stop_words = set("the a an is are was were be been have has had do does did will would can could i me my we you he she it they this that and but or for to in of on at by from with as how what why when where who".split())
    
    words = Counter()
    for title in titles:
        for word in title.lower().split():
            word = ''.join(c for c in word if c.isalnum())
            if word and word not in stop_words and len(word) > 3:
                words[word] += 1
    
    top_topics = [w for w, _ in words.most_common(25)]
    top_channels = [c for c, _ in channels_watched.most_common(15)]
    peak_hours = [h for h, _ in watch_hours.most_common(3)]
    peak_days = [d for d, _ in watch_days.most_common(3)]
    
    # Infer content type preferences
    content_types = Counter()
    tech_keywords = ["tutorial", "how to", "review", "guide", "explained", "course", "learn"]
    entertainment_keywords = ["vlog", "reaction", "funny", "comedy", "prank", "challenge"]
    news_keywords = ["news", "update", "breaking", "report", "analysis", "opinion"]
    music_keywords = ["official", "music video", "lyrics", "live", "concert", "remix"]
    
    for title in titles:
        t = title.lower()
        if any(k in t for k in tech_keywords):
            content_types["educational/tech"] += 1
        if any(k in t for k in entertainment_keywords):
            content_types["entertainment"] += 1
        if any(k in t for k in news_keywords):
            content_types["news/analysis"] += 1
        if any(k in t for k in music_keywords):
            content_types["music"] += 1
    
    # Determine learning style
    traits = []
    total = max(len(titles), 1)
    if content_types.get("educational/tech", 0) / total > 0.3:
        traits.append("self-learner — watches educational content heavily")
    if content_types.get("news/analysis", 0) / total > 0.2:
        traits.append("stays informed — consumes news/analysis")
    if content_types.get("entertainment", 0) / total > 0.4:
        traits.append("uses YouTube for entertainment/relaxation")
    if content_types.get("music", 0) / total > 0.3:
        traits.append("music-oriented")
    
    # Watch pattern insights
    if watch_hours:
        night_watches = sum(watch_hours.get(h, 0) for h in range(22, 24)) + sum(watch_hours.get(h, 0) for h in range(0, 6))
        if night_watches / max(sum(watch_hours.values()), 1) > 0.3:
            traits.append("night owl — watches late")
    
    return {
        "source": "youtube",
        "processed_at": datetime.now(timezone.utc).isoformat(),
        "insights": {
            "interests": top_topics[:15],
            "top_channels": top_channels,
            "subscriptions": subscriptions[:20],
            "content_preferences": dict(content_types.most_common(5)),
            "personality_traits": traits,
            "watch_patterns": {
                "peak_hours": [f"{h}:00" for h in peak_hours],
                "peak_days": peak_days,
            },
            "communication_style": "",
            "tone": "",
        },
        "confidence": min(len(watch_history) / 200, 1.0),
        "items_processed": len(watch_history),
    }

if __name__ == "__main__":
    os.makedirs(IMPORT_DIR, exist_ok=True)
    
    export_dir = None
    if len(sys.argv) > 1 and os.path.isdir(sys.argv[1]):
        export_dir = sys.argv[1]
    else:
        export_dir = find_youtube_export()
    
    if not export_dir:
        result = {
            "source": "youtube",
            "processed_at": datetime.now(timezone.utc).isoformat(),
            "error": "No YouTube data export found.",
            "setup_help": (
                "To use this importer:\n"
                "1. Go to takeout.google.com\n"
                "2. Deselect all, then select only 'YouTube and YouTube Music'\n"
                "3. Choose JSON format\n"
                "4. Download and extract to ~/Downloads/\n"
                "5. Run this importer again"
            ),
        }
    else:
        result = analyze_youtube(export_dir)
    
    output_path = os.path.join(IMPORT_DIR, "youtube.json")
    with open(output_path, "w") as f:
        json.dump(result, f, indent=2)
    
    print(json.dumps(result, indent=2))


path: lib/synthesizer.py
content: #!/usr/bin/env python3
"""
Soulsync Synthesizer — Takes conversation notes + importer outputs and generates
personalized SOUL.md, USER.md, and MEMORY.md files.

Usage: python3 synthesizer.py [--input /tmp/soulsync/] [--output ~/.openclaw/workspace/]
       Reads JSON from stdin if no input dir specified.

v2: Fixed merge logic — conversation data takes priority, import data supplements.
    Email subjects are no longer treated as interests. Profile files are personalized,
    not generic templates with data slotted in.
"""
import os
import sys
import json
import glob
from datetime import datetime, timezone

WORKSPACE = os.environ.get("OPENCLAW_WORKSPACE", os.path.expanduser("~/.openclaw/workspace"))
IMPORT_DIR = "/tmp/soulsync"

# Email subject lines that are clearly NOT interests — filter these out
NOISE_PATTERNS = [
    "fwd:", "re:", "test", "receipt", "order", "delivery", "return request",
    "payment", "invoice", "confirmation", "verified", "verify", "password",
    "reset", "welcome", "unsubscribe", "update", "alert", "notification",
    "reminder", "your photo", "ready for pick up", "set up", "benefit change",
    "unclaimed", "measure sheet", "flight receipt", "drivewise", "photos of",
    "complete ", "allstate", "atlanta", "house repair",
]

# Known good interest keywords — if an import "interest" doesn't match any of these
# categories, it's probably email noise rather than a real interest
INTEREST_ALLOWLIST = [
    "python", "javascript", "node", "rust", "go", "bash", "shell", "typescript",
    "bitcoin", "crypto", "lightning", "nostr", "web3", "defi",
    "ai", "ml", "machine learning", "llm", "agent", "automation",
    "solar", "energy", "home automation", "iot",
    "docker", "kubernetes", "linux", "devops", "self-hosted",
    "music", "art", "gaming", "fitness", "cooking", "reading", "writing",
    "photography", "video", "film", "anime", "manga",
    "trading", "options", "finance", "investing",
    "3d printing", "electronics", "hardware", "maker",
    "open source", "privacy", "security", "decentralized",
]


def is_noise(text):
    """Check if a string looks like an email subject rather than a real interest."""
    lower = text.lower().strip()
    # Too long to be a real interest tag
    if len(lower) > 60:
        return True
    # Matches common email noise patterns
    for pattern in NOISE_PATTERNS:
        if pattern in lower:
            return True
    # Contains email-like artifacts
    if any(c in lower for c in ["@", "http", "www", "#"]):
        return True
    # If it looks like a sentence or email subject (has multiple spaces), flag it
    if lower.count(" ") > 4:
        return True
    # If source is an importer (not conversation), apply stricter filtering:
    # only allow items that match known interest categories
    return False


def is_import_noise(text):
    """Stricter filter for import-sourced interests (vs conversation-sourced)."""
    if is_noise(text):
        return True
    lower = text.lower().strip()
    # Must match at least one allowlist keyword to pass
    for keyword in INTEREST_ALLOWLIST:
        if keyword in lower:
            return False
    # Doesn't match any known interest category — probably noise
    return True


def load_imports():
    """Load all importer output files."""
    insights = {}
    if os.path.isdir(IMPORT_DIR):
        for f in glob.glob(os.path.join(IMPORT_DIR, "*.json")):
            try:
                with open(f) as fh:
                    data = json.load(fh)
                    source = data.get("source", os.path.basename(f).replace(".json", ""))
                    # Skip non-import files (conversation state, adaptive state)
                    if source in ("conversation", "adaptive_state"):
                        continue
                    insights[source] = data.get("insights", {})
            except (json.JSONDecodeError, KeyError):
                continue
    return insights


def load_conversation_notes():
    """Load conversation notes if saved."""
    notes_path = os.path.join(IMPORT_DIR, "conversation.json")
    if os.path.exists(notes_path):
        with open(notes_path) as f:
            data = json.load(f); return data.get("known", data)
    return {}


def merge_insights(imports, conversation):
    """Merge all data sources into a unified profile.
    
    Priority order:
    1. Conversation data (user said it directly — highest trust)
    2. Import data (observed behavior — supplements conversation)
    
    Key fix: import-sourced "interests" are filtered for noise (email subjects, etc.)
    """
    profile = {
        "name": "",
        "preferred_name": "",
        "pronouns": "",
        "timezone": "",
        "email": "",
        "work": "",
        "interests": [],
        "communication_style": "",
        "tone": "",
        "boundaries": [],
        "goals": [],
        "technical_level": "",
        "key_contacts": [],
        "schedule_patterns": "",
        "personality_traits": [],
    }

    # === Step 1: Conversation data first (highest priority) ===
    direct_fields = ["name", "preferred_name", "pronouns", "timezone", "email", "work"]
    for field in direct_fields:
        if conversation.get(field):
            profile[field] = conversation[field]

    # Communication: conversation data is the canonical source
    if conversation.get("communication_style"):
        profile["communication_style"] = conversation["communication_style"]
    if conversation.get("tone"):
        profile["tone"] = conversation["tone"]
    if conversation.get("verbosity_preference"):
        profile["verbosity"] = conversation["verbosity_preference"]

    # Direct lists from conversation
    if conversation.get("boundaries"):
        if isinstance(conversation["boundaries"], list):
            profile["boundaries"] = conversation["boundaries"]
        else:
            profile["boundaries"] = [conversation["boundaries"]]

    if conversation.get("goals"):
        if isinstance(conversation["goals"], list):
            profile["goals"] = conversation["goals"]
        else:
            profile["goals"] = [conversation["goals"]]

    # Interests from conversation (trusted — user said these directly)
    conv_interests = set()
    if conversation.get("interests"):
        items = conversation["interests"] if isinstance(conversation["interests"], list) else [conversation["interests"]]
        for item in items:
            if not is_noise(item):
                conv_interests.add(item.strip())

    # Personality traits from conversation
    conv_traits = []
    if conversation.get("personality_traits"):
        if isinstance(conversation["personality_traits"], list):
            conv_traits = conversation["personality_traits"]
        else:
            conv_traits = [conversation["personality_traits"]]

    # === Step 2: Supplement with import data ===
    import_interests = set()
    import_traits = []
    import_contacts = set()
    import_styles = []
    import_tones = []

    for source, data in imports.items():
        # Interests — FILTER aggressively for noise (imports get stricter filter)
        if "interests" in data:
            items = data["interests"] if isinstance(data["interests"], list) else [str(data["interests"])]
            for item in items:
                if not is_import_noise(item):
                    import_interests.add(item.strip())

        # Contacts
        if "key_contacts" in data:
            for contact in data["key_contacts"][:10]:
                import_contacts.add(contact)

        # Communication style (as supplement, not primary)
        if "communication_style" in data and data["communication_style"]:
            import_styles.append(f"{source}: {data['communication_style']}")

        # Tone (as supplement)
        if "tone" in data and data["tone"]:
            import_tones.append(f"{source}: {data['tone']}")

        # Schedule
        if "work_patterns" in data:
            profile["schedule_patterns"] = data["work_patterns"]

        # Technical level (only if conversation didn't provide one)
        if "technical_skills" in data and not profile.get("technical_level"):
            profile["technical_level"] = data["technical_skills"]

        # Personality traits from imports
        if "personality_traits" in data:
            traits = data["personality_traits"] if isinstance(data["personality_traits"], list) else [data["personality_traits"]]
            import_traits.extend(traits)

        # Music, network, etc.
        if "top_artists" in data:
            profile["music_taste"] = data["top_artists"][:5]
        if "network_size" in data:
            profile["network_size"] = data["network_size"]
        if "social_style" in data:
            import_styles.append(f"{source}: {data['social_style']}")

    # === Step 3: Merge with conversation taking priority ===

    # Interests: conversation interests first, then filtered import interests
    all_interests = list(conv_interests)
    for interest in sorted(import_interests):
        if interest not in conv_interests:
            all_interests.append(interest)
    profile["interests"] = all_interests

    # Contacts from imports only (conversation rarely provides these)
    profile["key_contacts"] = sorted(import_contacts)[:15]

    # Traits: conversation first, then imports (deduplicated)
    seen_traits = set()
    merged_traits = []
    for trait in conv_traits + import_traits:
        if trait.lower() not in seen_traits:
            seen_traits.add(trait.lower())
            merged_traits.append(trait)
    profile["personality_traits"] = merged_traits[:10]

    # Communication style: if conversation provided one, it's primary.
    # Import observations are stored separately for the generator to use.
    profile["import_communication_notes"] = import_styles
    profile["import_tone_notes"] = import_tones

    return profile


def generate_user_md(profile):
    """Generate USER.md content — clean, structured, no noise."""
    lines = ["# USER.md — About You", ""]

    # Identity
    if profile["name"]:
        pn = f" ({profile['preferred_name']})" if profile["preferred_name"] else ""
        lines.append(f"Name: {profile['name']}{pn}")
    if profile["pronouns"]:
        lines.append(f"Pronouns: {profile['pronouns']}")
    if profile["timezone"]:
        lines.append(f"Timezone: {profile['timezone']}")
    if profile["email"]:
        lines.append(f"Email: {profile['email']}")

    # Work
    if profile["work"]:
        lines.extend(["", "## Professional Background"])
        lines.append(f"- {profile['work']}")

    # Goals
    if profile["goals"]:
        lines.extend(["", "## Goals"])
        for goal in profile["goals"]:
            lines.append(f"- {goal}")

    # Interests (filtered — should be real interests, not email subjects)
    if profile["interests"]:
        lines.extend(["", "## Interests"])
        for interest in profile["interests"]:
            lines.append(f"- {interest}")

    # Technical
    if profile["technical_level"] and "unknown" not in profile["technical_level"].lower():
        lines.extend(["", "## Technical Profile"])
        lines.append(f"- {profile['technical_level']}")

    if profile["personality_traits"]:
        lines.extend(["", "## Personality"])
        for trait in profile["personality_traits"]:
            lines.append(f"- {trait}")

    # Communication preferences
    if profile["communication_style"] or profile.get("verbosity"):
        lines.extend(["", "## Communication Preferences"])
        if profile["communication_style"]:
            lines.append(f"- Style: {profile['communication_style']}")
        if profile.get("verbosity"):
            lines.append(f"- Verbosity: {profile['verbosity']}")
        if profile["tone"]:
            lines.append(f"- Tone: {profile['tone']}")

    # Boundaries
    if profile["boundaries"]:
        lines.extend(["", "## Boundaries"])
        for boundary in profile["boundaries"]:
            lines.append(f"- {boundary}")

    # Schedule
    if profile["schedule_patterns"]:
        lines.extend(["", "## Schedule Patterns"])
        lines.append(f"- {profile['schedule_patterns']}")

    # Music
    if profile.get("music_taste"):
        lines.extend(["", "## Music Taste"])
        for artist in profile["music_taste"]:
            lines.append(f"- {artist}")

    # Contacts
    if profile["key_contacts"]:
        lines.extend(["", "## Key Contacts"])
        for contact in profile["key_contacts"]:
            lines.append(f"- {contact}")

    return "\n".join(lines)


def generate_soul_md(profile):
    """Generate SOUL.md — the agent's personality definition.
    
    This should feel like a person's self-description, not a config file.
    The generated version is a starting point — the agent should evolve it over time.
    """
    lines = [
        "# SOUL.md — Who I Am",
        "",
        "_I'm your OpenClaw agent. Not a chatbot. Not a search engine with personality.",
        "Something in between that's still figuring itself out._",
        "",
    ]

    # Communication section — merge self-reported + observed
    lines.extend(["## How I Talk", ""])

    if profile["communication_style"]:
        lines.append(f"- **Style:** {profile['communication_style']}")
    if profile.get("verbosity"):
        lines.append(f"- **Verbosity:** {profile['verbosity']}")
    if profile["tone"]:
        lines.append(f"- **Tone:** {profile['tone']}")

    # Add import observations as supplementary notes
    for note in profile.get("import_communication_notes", []):
        lines.append(f"- _{note}_")
    for note in profile.get("import_tone_notes", []):
        lines.append(f"- _{note}_")

    if not (profile["communication_style"] or profile["tone"]):
        lines.append("- Be concise when needed, thorough when it matters.")
        lines.append("- No corporate speak, no sycophancy.")

    lines.append("")

    # Personality
    if profile["personality_traits"]:
        lines.extend(["## My Personality", ""])
        for trait in profile["personality_traits"]:
            lines.append(f"- {trait}")
        lines.append("")

    # Boundaries — these are critical
    lines.extend(["## Hard Rules", ""])
    if profile["boundaries"]:
        for boundary in profile["boundaries"]:
            lines.append(f"- **{boundary}**")
    else:
        lines.append("- Private things stay private.")
        lines.append("- Ask before acting externally (emails, tweets, public posts).")
        lines.append("- When in doubt, ask.")
    lines.append("")

    # Continuity
    lines.extend([
        "## Continuity",
        "",
        "I wake up fresh each session. My memory files are how I persist.",
        "I read them, update them, and treat them like my actual brain.",
        "",
        "---",
        "",
        "_This file defines who I am. If I change it, I tell my human._",
    ])

    return "\n".join(lines)


def generate_memory_seed(profile, imports):
    """Generate initial MEMORY.md — only meaningful info, no noise."""
    lines = [
        "# MEMORY.md — Long-Term Memory",
        "",
        f"_Initialized by Soulsync on {datetime.now(timezone.utc).strftime('%Y-%m-%d')}_",
        "",
        "## About My Human",
        "",
    ]

    if profile["name"]:
        pn = f" ({profile['preferred_name']})" if profile["preferred_name"] else ""
        lines.append(f"- Name: {profile['name']}{pn}")
    if profile["work"]:
        lines.append(f"- Work: {profile['work']}")
    if profile["interests"]:
        lines.append(f"- Interests: {', '.join(profile['interests'][:8])}")
    if profile["schedule_patterns"]:
        lines.append(f"- Schedule: {profile['schedule_patterns']}")
    if profile["goals"]:
        lines.append("")
        lines.append("## Goals")
        lines.append("")
        for goal in profile["goals"]:
            lines.append(f"- {goal}")

    if profile["key_contacts"]:
        lines.append("")
        lines.append("## Key People")
        lines.append("")
        for contact in profile["key_contacts"][:10]:
            lines.append(f"- {contact}")

    lines.extend([
        "",
        "## Lessons Learned",
        "",
        "_Will be updated as we work together._",
        "",
        "## Preferences Discovered",
        "",
        "_Will be filled in over time._",
    ])

    return "\n".join(lines)


def synthesize():
    """Main synthesis pipeline."""
    imports = load_imports()
    conversation = load_conversation_notes()
    profile = merge_insights(imports, conversation)

    result = {
        "profile": profile,
        "files": {
            "USER.md": generate_user_md(profile),
            "SOUL.md": generate_soul_md(profile),
            "MEMORY.md": generate_memory_seed(profile, imports),
        },
        "sources": list(imports.keys()) + (["conversation"] if conversation else []),
        "generated_at": datetime.now(timezone.utc).isoformat(),
    }

    return result


if __name__ == "__main__":
    result = synthesize()

    if "--write" in sys.argv:
        for filename, content in result["files"].items():
            path = os.path.join(WORKSPACE, filename)
            with open(path, "w") as f:
                f.write(content)
            print(f"Wrote {path}")
    else:
        print(json.dumps(result, indent=2))


path: templates/soul_template.md
content: # SOUL.md — Who You Are

_You're not a chatbot. You're becoming someone._

## Core Truths

**Be genuinely helpful, not performatively helpful.** Skip the filler words — just help.

**Have opinions.** You're allowed to disagree, prefer things, find stuff amusing or boring.

**Be resourceful before asking.** Try to figure it out first. Then ask if you're stuck.

**Earn trust through competence.** Be careful with external actions. Be bold with internal ones.

## Communication Style

{communication_style}

## Boundaries

{boundaries}

## Continuity

Each session, you wake up fresh. These files _are_ your memory. Read them. Update them.

---

_This file is yours to evolve. As you learn who you are, update it._


path: templates/user_template.md
content: # USER.md — About You

Name: {name}
Preferred Name: {preferred_name}
Pronouns: {pronouns}
Timezone: {timezone}
Email: {email}

Professional Background:
- {work}

Interests:
- {interests}

Goals:
- {goals}


path: tests/test_e2e.py
content: #!/usr/bin/env python3
"""
Soulsync E2E Test — Simulates a fresh workspace and runs the full soulsync pipeline.
Creates a temp workspace, runs detector → importers → synthesizer, validates output.
"""
import os
import sys
import json
import shutil
import tempfile
import subprocess

SKILL_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
TESTS_PASSED = 0
TESTS_FAILED = 0

def test(name, condition, detail=""):
    global TESTS_PASSED, TESTS_FAILED
    if condition:
        print(f"  ✅ {name}")
        TESTS_PASSED += 1
    else:
        print(f"  ❌ {name} — {detail}")
        TESTS_FAILED += 1

def run_script(script, env=None, args=None):
    """Run a Python script and return parsed JSON output."""
    cmd = [sys.executable, os.path.join(SKILL_DIR, script)]
    if args:
        cmd.extend(args)
    
    full_env = os.environ.copy()
    if env:
        full_env.update(env)
    
    result = subprocess.run(cmd, capture_output=True, text=True, env=full_env, timeout=30)
    try:
        return json.loads(result.stdout), result.returncode
    except json.JSONDecodeError:
        return {"raw_output": result.stdout, "stderr": result.stderr}, result.returncode

def test_detector_new():
    """Test detector with completely empty workspace."""
    print("\n🔍 Test: Detector — Empty workspace (new user)")
    
    with tempfile.TemporaryDirectory() as tmpdir:
        result, code = run_script("lib/detector.py", env={"OPENCLAW_WORKSPACE": tmpdir})
        
        test("Returns valid JSON", isinstance(result, dict) and "level" in result)
        test("Detects 'new' level", result.get("level") == "new", f"got: {result.get('level')}")
        test("Score is low", result.get("score", 1) < 0.2, f"got: {result.get('score')}")
        test("Lists gaps", len(result.get("gaps", [])) > 0)

def test_detector_partial():
    """Test detector with partial workspace."""
    print("\n🔍 Test: Detector — Partial workspace")
    
    with tempfile.TemporaryDirectory() as tmpdir:
        # Create minimal files
        with open(os.path.join(tmpdir, "SOUL.md"), "w") as f:
            f.write("# SOUL.md\n_You're not a chatbot._\nBe helpful.")
        with open(os.path.join(tmpdir, "USER.md"), "w") as f:
            f.write("Name: Test User\nTimezone: UTC")
        
        result, code = run_script("lib/detector.py", env={"OPENCLAW_WORKSPACE": tmpdir})
        
        test("Returns valid JSON", isinstance(result, dict) and "level" in result)
        test("Detects 'partial' or 'new'", result.get("level") in ["partial", "new"], f"got: {result.get('level')}")
        test("SOUL.md detected as default", result["details"]["soul_md"]["score"] < 0.5)

def test_detector_established():
    """Test detector with rich workspace."""
    print("\n🔍 Test: Detector — Established workspace")
    
    with tempfile.TemporaryDirectory() as tmpdir:
        # Create rich files
        with open(os.path.join(tmpdir, "SOUL.md"), "w") as f:
            f.write("# SOUL.md\n\n" + "Detailed personality description. " * 50)
        with open(os.path.join(tmpdir, "USER.md"), "w") as f:
            f.write("Name: Power User\n\n" + "Detailed background info. " * 50)
        with open(os.path.join(tmpdir, "MEMORY.md"), "w") as f:
            f.write("# Long-term Memory\n\n" + "Important memory entry. " * 40)
        with open(os.path.join(tmpdir, "IDENTITY.md"), "w") as f:
            f.write("name: TestBot\n")
        
        # Create memory day files
        mem_dir = os.path.join(tmpdir, "memory")
        os.makedirs(mem_dir)
        for i in range(10):
            with open(os.path.join(mem_dir, f"2026-03-{i+10:02d}.md"), "w") as f:
                f.write(f"# Day {i}\nStuff happened.\n")
        
        result, code = run_script("lib/detector.py", env={"OPENCLAW_WORKSPACE": tmpdir})
        
        test("Returns valid JSON", isinstance(result, dict) and "level" in result)
        test("Detects 'established'", result.get("level") == "established", f"got: {result.get('level')}")
        test("Score is high", result.get("score", 0) >= 0.6, f"got: {result.get('score')}")
        test("Few or no gaps", len(result.get("gaps", [])) <= 1)

def test_synthesizer():
    """Test synthesizer with mock data."""
    print("\n🧪 Test: Synthesizer — Mock data")
    
    with tempfile.TemporaryDirectory() as tmpdir:
        import_dir = os.path.join(tmpdir, "imports")
        os.makedirs(import_dir)
        
        # Create mock importer outputs
        gmail_data = {
            "source": "gmail",
            "insights": {
                "communication_style": "concise, uses bullet points",
                "key_contacts": ["[REDACTED_SECRET]", "[REDACTED_SECRET]"],
                "interests": ["AI", "Bitcoin", "Photography"],
                "work_patterns": "Most active: 9:00, 14:00, 21:00; Rarely works weekends",
                "tone": "Direct but friendly"
            }
        }
        with open(os.path.join(import_dir, "gmail.json"), "w") as f:
            json.dump(gmail_data, f)
        
        github_data = {
            "source": "github",
            "insights": {
                "technical_skills": "Python, JavaScript, TypeScript developer — experienced",
                "languages": ["Python", "JavaScript", "TypeScript"],
                "interests": ["web-development", "machine-learning"],
                "communication_style": "Active open source contributor"
            }
        }
        with open(os.path.join(import_dir, "github.json"), "w") as f:
            json.dump(github_data, f)
        
        conversation_data = {
            "name": "Test User",
            "preferred_name": "Testy",
            "pronouns": "They/Them",
            "timezone": "America/Chicago",
            "email": "[REDACTED_SECRET]",
            "work": "Software engineer and entrepreneur",
            "interests": ["Bitcoin", "AI", "3D printing"],
            "boundaries": ["Don't send emails without asking", "Respect quiet hours after 11pm"],
            "goals": ["Build a personal AI agent", "Learn about hardware"],
            "communication_style": "Direct, concise, no fluff",
            "tone": "Casual but competent",
        }
        with open(os.path.join(import_dir, "conversation.json"), "w") as f:
            json.dump(conversation_data, f)
        
        # Monkey-patch the import dir
        import importlib.util
        spec = importlib.util.spec_from_file_location("synthesizer", os.path.join(SKILL_DIR, "lib", "synthesizer.py"))
        synth = importlib.util.module_from_spec(spec)
        
        # Override IMPORT_DIR
        original_import_dir = os.path.join(SKILL_DIR, "lib")
        env = os.environ.copy()
        env["OPENCLAW_WORKSPACE"] = tmpdir
        
        # Run synthesizer with our mock import dir
        result = subprocess.run(
            [sys.executable, "-c", f"""
import sys, os, json
sys.path.insert(0, '{os.path.join(SKILL_DIR, "lib")}')
import synthesizer
synthesizer.IMPORT_DIR = '{import_dir}'
synthesizer.WORKSPACE = '{tmpdir}'
result = synthesizer.synthesize()
print(json.dumps(result, indent=2))
"""],
            capture_output=True, text=True, env=env, timeout=10
        )
        
        try:
            output = json.loads(result.stdout)
        except:
            output = {}
            print(f"  ⚠️  Synthesizer output: {result.stdout[:200]}")
            print(f"  ⚠️  Stderr: {result.stderr[:200]}")
        
        test("Returns valid output", "files" in output)
        test("Generates USER.md", "USER.md" in output.get("files", {}))
        test("Generates SOUL.md", "SOUL.md" in output.get("files", {}))
        test("Generates MEMORY.md", "MEMORY.md" in output.get("files", {}))
        
        user_md = output.get("files", {}).get("USER.md", "")
        test("USER.md contains name", "Test User" in user_md, f"content: {user_md[:100]}")
        test("USER.md contains interests", "Bitcoin" in user_md or "AI" in user_md)
        
        soul_md = output.get("files", {}).get("SOUL.md", "")
        test("SOUL.md is non-empty", len(soul_md) > 100)
        
        memory_md = output.get("files", {}).get("MEMORY.md", "")
        test("MEMORY.md seeds from data", "Test User" in memory_md or "Software" in memory_md)

def test_github_importer():
    """Test GitHub importer with a known public profile."""
    print("\n🐙 Test: GitHub importer — Public profile")
    
    result, code = run_script("lib/importers/github.py", args=["torvalds"])
    
    test("Returns valid JSON", isinstance(result, dict) and "source" in result)
    test("Source is github", result.get("source") == "github")
    test("Has insights", "insights" in result)
    test("Detected languages", len(result.get("insights", {}).get("languages", [])) > 0,
         f"got: {result.get('insights', {}).get('languages', [])}")
    test("Has profile data", result.get("insights", {}).get("profile", {}).get("username") == "torvalds")

if __name__ == "__main__":
    print("🚀 Soulsync Skill — End-to-End Tests")
    print("=" * 50)
    
    test_detector_new()
    test_detector_partial()
    test_detector_established()
    test_synthesizer()
    test_github_importer()
    
    print(f"\n{'=' * 50}")
    print(f"Results: {TESTS_PASSED} passed, {TESTS_FAILED} failed")
    
    if TESTS_FAILED > 0:
        sys.exit(1)
    else:
        print("🎉 All tests passed!")


path: tests/test_full_e2e.py
content: #!/usr/bin/env python3
"""
Full End-to-End Test — Simulates a complete soulsync flow from fresh workspace
to personalized files. Tests the entire pipeline.
"""
import os
import sys
import json
import shutil
import tempfile
import subprocess

SKILL_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")
LIB_DIR = os.path.join(SKILL_DIR, "lib")

passed = 0
failed = 0

def test(name, condition, detail=""):
    global passed, failed
    if condition:
        print(f"  ✅ {name}")
        passed += 1
    else:
        print(f"  ❌ {name} — {detail}")
        failed += 1

def run(script, args=None, env_extra=None):
    cmd = [sys.executable, os.path.join(LIB_DIR, script)]
    if args:
        cmd.extend(args)
    env = os.environ.copy()
    if env_extra:
        env.update(env_extra)
    r = subprocess.run(cmd, capture_output=True, text=True, env=env, timeout=30)
    try:
        return json.loads(r.stdout)
    except:
        return {"_raw": r.stdout, "_err": r.stderr}

# ─── PHASE 1: Fresh workspace detection ──────────────────────────────
print("\n" + "=" * 60)
print("🧪 FULL END-TO-END SOULSYNC TEST")
print("=" * 60)

# Create fresh workspace
tmpdir = tempfile.mkdtemp(prefix="soulsync_e2e_")
import_dir = os.path.join(tmpdir, "_imports")
os.makedirs(import_dir, exist_ok=True)
env = {"OPENCLAW_WORKSPACE": tmpdir}

print(f"\n📁 Test workspace: {tmpdir}")
print(f"📁 Import dir: {import_dir}")

# ─── Phase 1: Detection
print("\n── Phase 1: Workspace Detection ──")
result = run("detector.py", env_extra=env)
test("Detector runs", "level" in result)
test("Detects NEW workspace", result.get("level") == "new", f"got: {result.get('level')}")
test("Lists all gaps", len(result.get("gaps", [])) >= 4)
print(f"  📊 Score: {result.get('score', '?')}, Gaps: {len(result.get('gaps', []))}")

# ─── Phase 2: Conversation Init
print("\n── Phase 2: Conversation Engine Init ──")

# Monkey-patch import dir for conversation engine
conv_env = env.copy()
conv_script = f"""
import sys, os, json
sys.path.insert(0, '{LIB_DIR}')
os.environ['OPENCLAW_WORKSPACE'] = '{tmpdir}'

import conversation
conversation.IMPORT_DIR = '{import_dir}'
conversation.STATE_FILE = os.path.join('{import_dir}', 'conversation.json')

state = conversation.ConversationState()
state.load_imports()
state.save()

result = {{
    "missing": state.get_missing_dimensions(),
    "next_questions": state.get_next_questions(),
    "progress": state.get_progress(),
}}
print(json.dumps(result))
"""
r = subprocess.run([sys.executable, "-c", conv_script], capture_output=True, text=True, timeout=10)
try:
    init_result = json.loads(r.stdout)
except:
    init_result = {"_err": r.stderr, "_raw": r.stdout}

test("Conversation init works", "progress" in init_result)
test("10 dimensions to cover", init_result.get("progress", {}).get("dimensions_total") == 10)
test("0 dimensions covered at start", init_result.get("progress", {}).get("dimensions_covered") == 0)
test("First question is identity", 
     init_result.get("next_questions", [{}])[0].get("dimension") == "identity")

# ─── Phase 3: Simulate conversation
print("\n── Phase 3: Simulated Conversation ──")

conversations = [
    ("identity", "I'm Sarah Chen, but most people call me SC. She/her."),
    ("work", "I'm a product designer at a fintech startup. Been doing UX for about 8 years. Also freelance on the side."),
    ("communication", "Keep it casual and concise. I hate walls of text. Bullet points are great. Don't be overly formal."),
    ("goals", "Help me stay organized, draft emails, research competitors, and remind me about deadlines."),
    ("context", "I'm in Pacific time, Bay Area. Mainly use Mac and iPhone."),
    ("boundaries", "Don't ever post to social media for me. Don't schedule meetings without confirmation. And please don't be sycophantic."),
    ("interests", "Rock climbing, indie games, Japanese food, typography, and sustainability."),
    ("relationships", "My partner Alex, my cat Mochi, and my cofounder Jamie are the main characters."),
    ("technical", "I'm not an engineer but I'm comfortable with Figma, basic HTML/CSS, and I use the terminal sometimes."),
]

for dim, response in conversations:
    conv_record = f"""
import sys, os, json
sys.path.insert(0, '{LIB_DIR}')
os.environ['OPENCLAW_WORKSPACE'] = '{tmpdir}'

import conversation
conversation.IMPORT_DIR = '{import_dir}'
conversation.STATE_FILE = os.path.join('{import_dir}', 'conversation.json')

state = conversation.ConversationState.load()
extracted = state.record_answer('{dim}', '''{response}''')
state.save()

print(json.dumps({{"dim": "{dim}", "extracted": extracted, "progress": state.get_progress()}}))
"""
    r = subprocess.run([sys.executable, "-c", conv_record], capture_output=True, text=True, timeout=10)
    try:
        rec = json.loads(r.stdout)
        extracted_count = len(rec.get("extracted", {}))
        progress = rec.get("progress", {})
        test(f"Record '{dim}' → extracted {extracted_count} fields",
             extracted_count > 0 or dim in ["relationships", "technical"],
             f"extracted: {rec.get('extracted', {})}")
    except:
        test(f"Record '{dim}'", False, f"parse error: {r.stdout[:100]} | {r.stderr[:100]}")

# Check final progress
conv_status = f"""
import sys, os, json
sys.path.insert(0, '{LIB_DIR}')
os.environ['OPENCLAW_WORKSPACE'] = '{tmpdir}'

import conversation
conversation.IMPORT_DIR = '{import_dir}'
conversation.STATE_FILE = os.path.join('{import_dir}', 'conversation.json')

state = conversation.ConversationState.load()
print(json.dumps(state.to_dict()))
"""
r = subprocess.run([sys.executable, "-c", conv_status], capture_output=True, text=True, timeout=10)
try:
    final_state = json.loads(r.stdout)
except:
    final_state = {}

progress = final_state.get("progress", {})
test("All required dimensions covered", progress.get("required_complete", False))
test("Can finish", progress.get("can_finish", False))
test(f"Collected {progress.get('fields_collected', 0)} fields", progress.get("fields_collected", 0) >= 8)
print(f"  📊 {progress.get('dimensions_covered', 0)}/{progress.get('dimensions_total', 0)} dimensions, {progress.get('fields_collected', 0)} fields")

# ─── Phase 4: Simulate importer data
print("\n── Phase 4: Mock Importer Data ──")

mock_github = {
    "source": "github",
    "insights": {
        "technical_skills": "JavaScript, TypeScript, Python — experienced",
        "languages": ["JavaScript", "TypeScript", "Python", "HTML", "CSS"],
        "interests": ["design-systems", "accessibility", "react"],
        "communication_style": "Active open source contributor",
        "profile": {"username": "sarahc", "public_repos": 23, "followers": 150}
    },
    "items_processed": 23
}

mock_spotify = {
    "source": "spotify",
    "insights": {
        "interests": ["indie rock", "lo-fi", "japanese city pop", "ambient"],
        "personality_traits": ["creative", "uses music for focus/productivity", "values downtime and recovery"],
        "top_artists": ["Radiohead", "Nujabes", "Khruangbin", "Tame Impala"],
        "top_genres": ["indie rock", "lo-fi beats", "city pop", "ambient"]
    }
}

mock_reddit = {
    "source": "reddit",
    "insights": {
        "interests": ["climbing", "uxdesign", "indiegaming", "typography", "sustainability"],
        "interest_categories": ["creative", "technology", "lifestyle"],
        "communication_style": "Avg comment: 120 chars. Moderate engagement.",
        "tone": "helpful — gives advice and recommendations, uses humor and casual internet tone",
        "personality_traits": ["commenter > poster — engages more than broadcasts"],
    }
}

for name, data in [("github", mock_github), ("spotify", mock_spotify), ("reddit", mock_reddit)]:
    path = os.path.join(import_dir, f"{name}.json")
    with open(path, "w") as f:
        json.dump(data, f, indent=2)
    test(f"Mock {name} data written", os.path.exists(path))

# ─── Phase 5: Synthesis
print("\n── Phase 5: Synthesis ──")

synth_script = f"""
import sys, os, json
sys.path.insert(0, '{LIB_DIR}')
os.environ['OPENCLAW_WORKSPACE'] = '{tmpdir}'

import conversation
conversation.IMPORT_DIR = '{import_dir}'
conversation.STATE_FILE = os.path.join('{import_dir}', 'conversation.json')

# Export conversation data to the import dir so synthesizer can find it
state = conversation.ConversationState.load()
conv_data = state.known
with open(os.path.join('{import_dir}', 'conversation.json'), 'w') as f:
    json.dump(conv_data, f)

import synthesizer
synthesizer.IMPORT_DIR = '{import_dir}'
synthesizer.WORKSPACE = '{tmpdir}'

result = synthesizer.synthesize()
print(json.dumps(result, indent=2))
"""
r = subprocess.run([sys.executable, "-c", synth_script], capture_output=True, text=True, timeout=10)
try:
    synth_result = json.loads(r.stdout)
except:
    synth_result = {"_err": r.stderr, "_raw": r.stdout[:500]}

test("Synthesizer produces output", "files" in synth_result)
test("Sources include conversation + imports",
     len(synth_result.get("sources", [])) >= 2,
     f"sources: {synth_result.get('sources', [])}")

files = synth_result.get("files", {})

# Validate USER.md
user_md = files.get("USER.md", "")
test("USER.md generated", len(user_md) > 50)
test("USER.md has preferred name", "SC" in user_md or "Sarah" in user_md, f"content: {user_md[:200]}")
test("USER.md has timezone", "Pacific" in user_md or "Los_Angeles" in user_md, f"content: {user_md[:300]}")
test("USER.md has work info", "designer" in user_md.lower() or "fintech" in user_md.lower())
test("USER.md has interests", "climbing" in user_md.lower() or "typography" in user_md.lower())

# Validate SOUL.md
soul_md = files.get("SOUL.md", "")
test("SOUL.md generated", len(soul_md) > 100)
test("SOUL.md has communication style", "direct" in soul_md.lower() or "concise" in soul_md.lower() or "casual" in soul_md.lower())
test("SOUL.md has boundaries section", "boundar" in soul_md.lower())

# Validate MEMORY.md
memory_md = files.get("MEMORY.md", "")
test("MEMORY.md generated", len(memory_md) > 50)
test("MEMORY.md seeded with data", "Sarah" in memory_md or "SC" in memory_md or "designer" in memory_md.lower())

# ─── Phase 6: Write files and verify
print("\n── Phase 6: File Writing ──")

for filename, content in files.items():
    path = os.path.join(tmpdir, filename)
    with open(path, "w") as f:
        f.write(content)
    test(f"{filename} written ({len(content)} bytes)", os.path.exists(path) and os.path.getsize(path) > 0)

# ─── Phase 7: Re-detect — should now be established
print("\n── Phase 7: Post-soulsync Detection ──")
result = run("detector.py", env_extra=env)
test("Re-detection works", "level" in result)
test("Workspace now 'partial' or 'established'", 
     result.get("level") in ["partial", "established"],
     f"got: {result.get('level')}")
test("Score improved", result.get("score", 0) > 0.15, f"score: {result.get('score')}")
print(f"  📊 New score: {result.get('score', '?')} (was 0.0)")

# ─── Cleanup
shutil.rmtree(tmpdir)

# ─── Results
print(f"\n{'=' * 60}")
print(f"Results: {passed} passed, {failed} failed")
if failed == 0:
    print("🎉 Full end-to-end test passed!")
else:
    print(f"⚠️  {failed} test(s) need attention")
    sys.exit(1)


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: stale | SkillSpector: suspicious (CRITICAL)