Skill: eladrave/eladrave-filechat-rag (version 1.0.1)

--- name: filechat-rag description: > Search, retrieve, and chat with documents securely stored in ANY Google Drive folder using semantic vector search (RAG). Use when: (1) a user asks to save a file for later, (2) a user asks a question about their saved files/documents, (3) a user asks you to retrieve or send them a previously stored file, (4) a user asks to sync or flush their document library. This skill automatically chunks, embeds, and indexes Google Drive documents into isolated persistent vector databases per folder. metadata: openclaw: category: "knowledge" requires: bins: ["node", "npm", "gws"] install: - id: gws kind: node package: "@googleworkspace/cli" bins: ["gws"] label: "Install Google Workspace CLI" - id: filechat-deps kind: script script: "cd ./skills/filechat && npm install" label: "Install FileChat Dependencies" --- # FileChat RAG Skill Your personal RAG (Retrieval-Augmented Generation) document library backed by Google Drive. Supports multiple Google Drive folders dynamically and allows choosing between Gemini or OpenAI for embeddings. ## Setup & Bootstrap If the user asks to use FileChat or asks a question about their files, FIRST verify that the required environment variables are set in `/workspace/skills/filechat/.env`: 1. `EMBEDDING_PROVIDER` (either `gemini` or `openai`) 2. `GEMINI_API_KEY` or `OPENAI_API_KEY` (Depending on the provider) If they are missing, STOP and ask the user to provide them. Create the `.env` file like this: ```bash echo "EMBEDDING_PROVIDER=gemini" > ./skills/filechat/.env echo "GEMINI_API_KEY=your_key_here" >> ./skills/filechat/.env ``` ## How to Sync the Library When the user asks to "sync", "flush", or "update" a specific FileChat folder, you must run the ingestion script. This connects to Google Drive, downloads all new/changed files (including PDFs, resolving shortcuts, and traversing sub-folders), chunks the text, gets embeddings, and saves them to a local JSON vector database keyed by the Folder ID. You must supply the `FOLDER_ID` to the script. If you don't know the folder ID the user wants, ask them. ```bash cd ./skills/filechat && node sync.js <FOLDER_ID> ``` ## How to Answer User Questions (RAG) When a user asks a question about the contents of their documents, you MUST query the local vector store to fetch the relevant text chunks. You need the Folder ID. ```bash cd ./skills/filechat && node query.js <FOLDER_ID> "What does my medical discharge say?" ``` The output will give you the most relevant text snippets, the original file names, and the Google Drive File IDs. Use the text snippets to formulate a comprehensive answer for the user. Always cite the file name you are referencing. ## How to Retrieve and Send a Physical File If the user asks for the actual file (e.g., "Send me the discharge PDF"), first find the `File ID` using the query script. Then, use the `gws` CLI to download the file into your workspace: ```bash gws drive files get --params '{"fileId": "<FILE_ID>", "alt": "media"}' --output /workspace/discharge.pdf ``` Then, reply to the user using the OpenClaw media attachment syntax: `MEDIA:/workspace/discharge.pdf` to send the physical file directly to their chat window. ## How to Store a New File for the User If the user uploads a file and asks you to "store" or "save" it: 1. Upload it to their specific FileChat Google Drive folder using `gws`: ```bash gws drive files create \ --json '{"name": "filename.pdf", "parents": ["<FOLDER_ID>"]}' \ --upload /path/to/uploaded/file.pdf ``` 2. Trigger the sync process so the new file is immediately readable by the vector database: ```bash cd ./skills/filechat && node sync.js <FOLDER_ID> ```

path: README.md
content: # FileChat RAG

A powerful Retrieval-Augmented Generation (RAG) AgentSkill for OpenClaw. 

This skill connects to a Google Drive folder, recursively downloads all files (resolving shortcuts), extracts text (including text from images via OCR), chunks it, embeds it with Gemini, and saves it to a persistent local vector database.

## Features
- Deep sync with Google Drive folders and sub-folders
- Resolves Google Drive Shortcuts automatically
- OCRs image text automatically using Gemini Flash
- Reads PDFs natively
- Local Cosine-Similarity Vector Search (No external vector DB required)

## Usage
Once installed, the agent will ask for:
1. Your Google Gemini API Key
2. The Google Drive Folder ID you want to index

Then, just tell your agent to `sync`, and you can start chatting with your entire document library immediately!


path: image_parse.js
content: const fs = require('fs');
const { GoogleGenerativeAI } = require('@google/generative-ai');
require('dotenv').config();

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);

function fileToGenerativePart(path, mimeType) {
  return {
    inlineData: {
      data: Buffer.from(fs.readFileSync(path)).toString("base64"),
      mimeType
    },
  };
}

async function extractTextFromImage(filePath, mimeType) {
  const model = genAI.getGenerativeModel({ model: "gemini-2.5-flash" });
  const prompt = "Please transcribe all the text visible in this image accurately. Do not add any extra commentary, just the text.";
  const imagePart = fileToGenerativePart(filePath, mimeType);
  const result = await model.generateContent([prompt, imagePart]);
  const response = await result.response;
  return response.text();
}

module.exports = { extractTextFromImage };


path: package.json
content: {
  "name": "filechat",
  "version": "1.0.0",
  "description": "Google Drive RAG FileChat Skill",
  "main": "sync.js",
  "scripts": {
    "sync": "node sync.js",
    "query": "node query.js"
  },
  "dependencies": {
    "@google/generative-ai": "^0.2.1",
    "chromadb": "^1.8.1",
    "pdf-parse": "^1.1.1",
    "dotenv": "^16.4.5"
  }
}


path: query.js
content: const fs = require('fs');
const path = require('path');
require('dotenv').config({ path: path.join(__dirname, '.env') });

const folderArg = process.argv[2];
const query = process.argv[3];
if (!folderArg || !query) {
  console.error("Usage: node query.js <DRIVE_FOLDER_ID> \"<SEARCH_QUERY>\"");
  process.exit(1);
}

const ROOT_FOLDER_ID = folderArg;
const DB_PATH = path.join(__dirname, `vector_db_${ROOT_FOLDER_ID}.json`);

const EMBEDDING_PROVIDER = process.env.EMBEDDING_PROVIDER || "gemini";
const GEMINI_API_KEY = process.env.GEMINI_API_KEY;
const OPENAI_API_KEY = process.env.OPENAI_API_KEY;

if (!fs.existsSync(DB_PATH)) {
  console.error(`Database not found for folder ${ROOT_FOLDER_ID}. Please run sync.js first.`);
  process.exit(1);
}

const db = JSON.parse(fs.readFileSync(DB_PATH, 'utf8'));

function cosineSimilarity(A, B) {
  let dotProduct = 0;
  let normA = 0;
  let normB = 0;
  for (let i = 0; i < A.length; i++) {
    dotProduct += A[i] * B[i];
    normA += A[i] * A[i];
    normB += B[i] * B[i];
  }
  return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}

async function getEmbedding(text) {
  if (EMBEDDING_PROVIDER === "openai") {
    const response = await fetch("https://api.openai.com/v1/embeddings", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Authorization": `Bearer ${OPENAI_API_KEY}`
      },
      body: JSON.stringify({
        model: "text-embedding-3-small",
        input: text
      })
    });
    if (!response.ok) throw new Error(await response.text());
    const data = await response.json();
    return data.data[0].embedding;
  } else {
    // Gemini
    const url1 = `https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-2-preview:embedContent?key=${GEMINI_API_KEY}`;
    const res1 = await fetch(url1, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        model: "models/gemini-embedding-2-preview",
        content: { parts: [{ text: text }] }
      })
    });
    
    if (res1.ok) {
      const data = await res1.json();
      return data.embedding.values;
    }
    
    // Fallback if gemini-embedding-2-preview fails
    const url2 = `https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-001:embedContent?key=${GEMINI_API_KEY}`;
    const res2 = await fetch(url2, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          model: "models/gemini-embedding-001",
          content: { parts: [{ text: text }] }
        })
    });
    if (!res2.ok) {
        throw new Error("Both Gemini models failed: " + await res2.text());
    }
    const data2 = await res2.json();
    return data2.embedding.values;
  }
}

async function main() {
  const queryEmbedding = await getEmbedding(query);

  const results = [];
  for (const doc of db) {
    const sim = cosineSimilarity(queryEmbedding, doc.embedding);
    results.push({ ...doc, similarity: sim });
  }

  results.sort((a, b) => b.similarity - a.similarity);
  const topK = results.slice(0, 3);

  console.log("Top matches for:", query);
  for (let i = 0; i < topK.length; i++) {
    console.log(`\nMatch ${i+1}:`);
    console.log(`Filename: ${topK[i].filename}`);
    console.log(`File ID: ${topK[i].fileId}`);
    console.log(`Similarity: ${topK[i].similarity.toFixed(4)}`);
    console.log(`Snippet: ${topK[i].text.substring(0, 200)}...`);
  }
}

main().catch(console.error);


path: sync.js
content: const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');
const pdf = require('pdf-parse');
require('dotenv').config({ path: path.join(__dirname, '.env') });

const folderArg = process.argv[2];
if (!folderArg) {
  console.error("Usage: node sync.js <DRIVE_FOLDER_ID>");
  process.exit(1);
}

const ROOT_FOLDER_ID = folderArg;
const DB_PATH = path.join(__dirname, `vector_db_${ROOT_FOLDER_ID}.json`);

const EMBEDDING_PROVIDER = process.env.EMBEDDING_PROVIDER || "gemini";
const GEMINI_API_KEY = process.env.GEMINI_API_KEY;
const OPENAI_API_KEY = process.env.OPENAI_API_KEY;

if (EMBEDDING_PROVIDER === "gemini" && !GEMINI_API_KEY) {
  console.error("Missing GEMINI_API_KEY for embedding provider 'gemini'.");
  process.exit(1);
} else if (EMBEDDING_PROVIDER === "openai" && !OPENAI_API_KEY) {
  console.error("Missing OPENAI_API_KEY for embedding provider 'openai'.");
  process.exit(1);
}

function getGWSFiles(folderId) {
  const query = `'${folderId}' in parents and trashed = false`;
  const escapedQuery = query.replace(/'/g, "'\\''");
  const cmd = `SSL_CERT_FILE=/workspace/cacert.pem npx @googleworkspace/cli drive files list --params '{"q": "${escapedQuery}", "fields": "files(id, name, mimeType, shortcutDetails)"}'`;
  try {
    const res = execSync(cmd, { encoding: 'utf-8', stdio: 'pipe' });
    const jsonStart = res.indexOf('{');
    const cleanRes = res.substring(jsonStart);
    return JSON.parse(cleanRes).files || [];
  } catch(e) {
    console.error("Error fetching files from GWS:", e.message);
    return [];
  }
}

function downloadFile(fileId, dest) {
  const cmd = `SSL_CERT_FILE=/workspace/cacert.pem npx @googleworkspace/cli drive files get --params '{"fileId": "${fileId}", "alt": "media"}' --output "${dest}"`;
  try {
    execSync(cmd, { stdio: 'pipe' });
    return true;
  } catch(e) {
    return false;
  }
}

async function getEmbedding(text) {
  if (EMBEDDING_PROVIDER === "openai") {
    const response = await fetch("https://api.openai.com/v1/embeddings", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Authorization": `Bearer ${OPENAI_API_KEY}`
      },
      body: JSON.stringify({
        model: "text-embedding-3-small",
        input: text
      })
    });
    if (!response.ok) throw new Error(await response.text());
    const data = await response.json();
    return data.data[0].embedding;
  } else {
    // Gemini
    const url1 = `https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-2-preview:embedContent?key=${GEMINI_API_KEY}`;
    const res1 = await fetch(url1, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        model: "models/gemini-embedding-2-preview",
        content: { parts: [{ text: text }] }
      })
    });
    
    if (res1.ok) {
      const data = await res1.json();
      return data.embedding.values;
    }
    
    // Fallback if gemini-embedding-2-preview fails
    const url2 = `https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-001:embedContent?key=${GEMINI_API_KEY}`;
    const res2 = await fetch(url2, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          model: "models/gemini-embedding-001",
          content: { parts: [{ text: text }] }
        })
    });
    if (!res2.ok) {
        throw new Error("Both Gemini models failed: " + await res2.text());
    }
    const data2 = await res2.json();
    return data2.embedding.values;
  }
}

function chunkText(text, maxWords = 200) {
  const words = text.split(/\s+/);
  const chunks = [];
  for(let i=0; i < words.length; i += maxWords) {
    chunks.push(words.slice(i, i + maxWords).join(" "));
  }
  return chunks;
}

async function extractTextFromImage(filePath, mimeType) {
  if (EMBEDDING_PROVIDER === "openai") {
    // Basic fallback or warn (could use gpt-4o for OCR, but complex. Using standard text parsing instead or skip)
    console.log("OCR for OpenAI not implemented. Skipping image text extraction.");
    return "";
  } else {
    const { GoogleGenerativeAI } = require("@google/generative-ai");
    const genAI = new GoogleGenerativeAI(GEMINI_API_KEY);
    const model = genAI.getGenerativeModel({ model: "gemini-2.5-flash" });
    const prompt = "Please transcribe all the text visible in this image accurately. Do not add any extra commentary, just the text.";
    const imagePart = {
      inlineData: {
        data: Buffer.from(fs.readFileSync(filePath)).toString("base64"),
        mimeType
      },
    };
    const result = await model.generateContent([prompt, imagePart]);
    const response = await result.response;
    return response.text();
  }
}

async function processFolder(folderId, currentPath, db) {
  console.log(`Scanning folder: ${currentPath || 'Root'} (${folderId})`);
  const items = getGWSFiles(folderId);
  
  for(const item of items) {
    const filePath = currentPath ? `${currentPath}/${item.name}` : item.name;
    
    let targetId = item.id;
    let mimeType = item.mimeType;
    if (mimeType === 'application/vnd.google-apps.shortcut') {
      targetId = item.shortcutDetails.targetId;
      mimeType = item.shortcutDetails.targetMimeType;
      console.log(`Resolving shortcut: ${item.name} -> ${targetId}`);
    }

    if (mimeType === 'application/vnd.google-apps.folder') {
      await processFolder(targetId, filePath, db);
    } else if (mimeType === 'application/pdf' || mimeType.includes('text') || mimeType.includes('image')) {
      console.log(`Processing file: ${filePath} (${mimeType})`);
      const tmpFile = `./filechat_${targetId}`;
      
      if(downloadFile(targetId, tmpFile)) {
        let text = "";
        try {
          if (mimeType === 'application/pdf') {
            const dataBuffer = fs.readFileSync(tmpFile);
            const data = await pdf(dataBuffer);
            text = data.text;
          } else if (mimeType.includes('image')) {
            text = await extractTextFromImage(tmpFile, mimeType);
          } else {
            text = fs.readFileSync(tmpFile, 'utf8');
          }
          
          if(text.trim()) {
            const chunks = chunkText(text);
            for(let i=0; i<chunks.length; i++) {
              const c = chunks[i];
              const emb = await getEmbedding(c);
              db.push({
                fileId: targetId,
                filename: filePath,
                chunkIndex: i,
                text: c,
                embedding: emb
              });
            }
            console.log(`Embedded ${chunks.length} chunks for ${item.name}`);
          }
        } catch(e) {
          console.error(`Failed to parse/embed ${item.name}: ${e.message}`);
        }
        if(fs.existsSync(tmpFile)) fs.unlinkSync(tmpFile);
      } else {
        console.error(`Failed to download ${item.name}`);
      }
    } else {
      console.log(`Skipping unsupported file type: ${filePath} (${mimeType})`);
    }
  }
}

async function main() {
  console.log(`Starting FileChat Sync for Folder ID: ${ROOT_FOLDER_ID}...`);
  let db = [];
  
  await processFolder(ROOT_FOLDER_ID, "", db);
  
  fs.writeFileSync(DB_PATH, JSON.stringify(db));
  console.log(`Sync complete. Database saved to disk with ${db.length} total chunks.`);
}

main().catch(console.error);


Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)