Skill: crimsondevil333333/sheetsmith (version 1.0.1)

--- name: sheetsmith description: Pandas-powered CSV & Excel management for quick previews, summaries, filtering, transforming, and format conversions. Use this skill whenever you need to inspect spreadsheet files, compute column-level summaries, apply queries or expressions, or export cleansed data to a new CSV/TSV/XLSX output without rewriting pandas every time. --- # Sheetsmith ## Overview Sheetsmith is a lightweight pandas wrapper that keeps the focus on working with CSV/Excel files: previewing, describing, filtering, transforming, and converting them in one place. The CLI lives at `skills/sheetsmith/scripts/sheetsmith.py`, and it automatically loads any CSV/TSV/Excel file, reports structural metadata, runs pandas expressions, and writes the results back safely. ## Quick start 1. Place the spreadsheet (CSV, TSV, or XLS/XLSX) inside the workspace or reference it via a full path. 2. Run `python3 skills/sheetsmith/scripts/sheetsmith.py <command> <path>` with the command described below. 3. When you modify data, either provide `--output new-file` to save a copy or pass `--inplace` to overwrite the source file. 4. Check `references/usage.md` for extra sample commands and tips. ## Commands ### summary Prints row/column counts, dtype breakdowns, columns with missing data, and head/tail previews. Use `--rows` to control how many rows are shown after the summary and `--tail` to preview the tail instead of the head. ### describe Runs `pandas.DataFrame.describe(include='all')` (customizable with `--include`) so you instantly see numeric statistics, cardinality, and frequency information. Supply `--percentiles` to add additional percentile lines. ### preview Shows a quick tabulated peek at the first (`--rows`) or last (`--tail`) rows so you can sanity-check column order or formatting before taking actions. ### filter Enter a pandas query string via `--query` (e.g., `state == 'CA' and population > 1e6`). The command can either print the filtered rows or, when you also pass `--output`, write the filtered table to a new CSV/TSV/XLSX file. Add `--sample` to inspect a random subset instead of the entire result. ### transform Compose new columns, rename or drop existing ones, and immediately inspect the resulting table. Provide one or more `--expr` expressions such as `total = quantity * price`. Use `--rename old:new` and `--drop column` to reshape the table, and persist changes via `--output` or `--inplace`. The preview version (without writing) reuses the same `--rows`/`--tail` flags as the other commands. ### convert Convert between supported formats (CSV/TSV/Excel). Always specify `--output` with the desired extension, and the helper will detect the proper writer (Excel uses `openpyxl`, CSV preserves the comma separator by default, TSV uses tabs). This is the simplest way to normalize data before running other commands. ## Workflow rules - Always keep a copy of the raw file or write to a new path; the script will only overwrite the original when you explicitly demand `--inplace`. - Use the same CLI for both exploration (`summary`, `preview`, `describe`) and editing (`filter`, `transform`). The `--output` flag works for filter/transform so you can easily branch results. - Behind the scenes, the script relies on pandas + `tabulate` for Markdown previews and supports Excel/CSV/TSV, so ensure those dependencies are present (pandas, openpyxl, xlrd, tabulate are installed via apt on this system). - Use `references/usage.md` for extended examples (multi-step cleaning, dataset comparison, expression tips) when the basic command descriptions above are not enough. ## References - **Usage guidelines:** `references/usage.md` (contains ready-to-copy commands, expression patterns, and dataset cleanup recipes). ## Resources - **GitHub:** https://github.com/CrimsonDevil333333/sheetsmith - **ClawHub:** https://www.clawhub.ai/skills/sheetsmith

path: README.md
content: # Sheetsmith

Sheetsmith is the pandas-based CSV/TSV/Excel assistant for OpenClaw. It gives you one CLI for:

- Inspecting spreadsheets (`summary`, `describe`, `preview`)
- Filtering or sampling rows via pandas queries (`filter`)
- Transforming columns with expressions, renames, drops, and safe writes (`transform`)
- Converting between CSV, TSV, and Excel formats (`convert`)

The script lives at `skills/sheetsmith/scripts/sheetsmith.py` and is the single source of truth for all operations, so you never have to re-write pandas boilerplate.

## Dependencies (already installed)

Sheetsmith relies on the Debian-packaged stack:

- `python3-pandas` (dataframes + Excel/CSV readers)
- `python3-openpyxl` + `python3-xlrd` (Excel file support)
- `python3-tabulate` (pretty Markdown previews)

Because these are installed system-wide, you can run the CLI without building a virtualenv.

## Usage

1. **Place the file** somewhere under the workspace (e.g., `workspace/inputs/my-data.xlsx`).
2. **Run a command:** `python3 skills/sheetsmith/scripts/sheetsmith.py <command> <path>` plus any flags listed below.
3. **Inspect the output** (Markdown tables are provided for previews) and, when you write data, use `--output new.xlsx` or `--inplace` to persist it.

### Common commands

| Command | Description | Example |
|---------|-------------|---------|
| `summary` | Print shape/dtypes, missing-value info, and a Markdown preview of the first/last rows | `... summary inputs/sales.csv --rows 5 --tail` |
| `describe` | Run `DataFrame.describe()` with optional `--include`/`--percentiles` | `... describe data.xlsx --include all --percentiles 10 90` |
| `preview` | Show only head/tail rows without any analysis | `... preview report.tsv --tail --rows 3` |
| `filter` | Apply a pandas query string and optionally sample/write results | `... filter data.csv --query "state == 'CA'" --output outputs/ca.csv` |
| `transform` | Add/rename/drop columns via pandas expressions, then preview or save | `... transform data.csv --expr "density = population / area" --rename active:is_active --output outputs/with-density.csv` |
| `convert` | Re-export to CSV/TSV/XLSX by specifying `--output` with the desired extension | `... convert raw.xlsx --output clean.csv` |

### Filtering & transforming tips

- Query strings use pandas syntax: `"region == 'EMEA' and sales >= 1e5"`.
- Use `--sample <n>` on `filter` to inspect a random subset without overwhelming the session.
- Provide `--expr` expressions (star `column = formula`) to create calculated columns, then rename/dropping as needed before writing.
- Write to a new file with `--output`; `--inplace` only overwrites when you explicitly request it.

### Automation & repeated work

You can chain commands manually:

1. `filter` the rows you need
2. `transform` to add/rename/drop columns
3. `convert` to output the final format

Every step shares the same CLI, so scripts and workflows stay consistent.

## Handling files from humans/bots

1. **Receive the attachment** (CSV/Excel) and save it into the workspace (I usually place it under `workspace/inputs/`).
2. **Run Sheetsmith** pointing to that path. For example:
   ```bash
   python3 skills/sheetsmith/scripts/sheetsmith.py summary workspace/inputs/inbox.xlsx --rows 5
   ```
3. **Share results** back in chat. If a modified file is needed, use `--output` to write it into `workspace/outputs/` and upload that file (I can send it back via Telegram or WhatsApp).

If you want me to keep a log of every dataset I touched, I can update `memory` entries as part of the workflow.

## Testing

Run the unit tests with:

```bash
python3 -m unittest discover skills/sheetsmith/tests
```

They exercise the summary/preview workflows, `filter`, and `transform` commands using `tests/data/test.csv`, so you can trust the CLI on small but representative data.

## Publishing & development notes

- Skill metadata: `SKILL.md` explains triggers and workflows.
- Additional reference: `references/usage.md` contains a cheat sheet plus troubleshooting notes.
- Packaging script: `python3 $(npm root -g)/openclaw/skills/skill-creator/scripts/package_skill.py skills/sheetsmith` creates `sheetsmith.skill` for ClawHub or release bundles.

## Links

- **GitHub:** https://github.com/CrimsonDevil333333/sheetsmith
- **ClawHub:** https://www.clawhub.ai/skills/sheetsmith


path: references/usage.md
content: # Sheetsmith usage reference

## Command cheat sheet

| Command | Purpose | Example |
|--------|---------|---------|
| `summary` | High-level diagnostics (shape, dtypes, missing data) plus a Markdown preview. | `python3 skills/sheetsmith/scripts/sheetsmith.py summary data/customers.csv --rows 5` |
| `describe` | Run `pandas.DataFrame.describe()` (supports `--include` or `--percentiles`). | `... describe reports.csv --percentiles 10 50 90` |
| `preview` | Just show head/tail without analyzing. | `... preview workbook.xlsx --tail` |
| `filter` | Keep rows that match a pandas query string; optional `--output`/`--sample`. | `... filter sales.csv --query "region=='EMEA'" --output filtered/emea.csv` |
| `transform` | Create new columns, drop/rename fields, and write filtered results. | `... transform ledger.csv --expr "net = credit - debit" --rename net:net_balance --output ledger/with-net.csv` |
| `convert` | Export the same table to CSV/TSV/XLSX. | `... convert dataset.xlsx --output dataset.csv` |

*Tip:* The CLI guesses format from the file extension. Use `.tsv` or `.txt` for tab-delimited text, `.csv` for comma, and `.xlsx`/`.xls` for Excel.

## Expression & filter patterns

- Arithmetic expressions: `--expr "total = quantity * price"`.
- Boolean flags: `--expr "is_active = status == 'active'"`.
- Drop columns you no longer need with `--drop colA colB` prior to saving a smaller file.
- Rename columns on the fly via `--rename old:new another:replacement`.
- Use `--query` to filter rows (`"score >= 80 and country == 'IN'"`). If you only want a sample, pair `--sample 20` with `--query` so the stored dataset remains manageable.

## Workflow tips

1. Always preview before writing when dealing with unfamiliar schemas.
2. Use `--output` to branch data copies; once satisfied, you can prune the old file or move the copy into place.
3. For Excel sheets, pass `--sheet "Sheet1"` to read a specific tab.
4. Combine commands when needed: first `filter` to restrict rows, then `transform` with `--output` to add calculated columns, and finally `convert` to export as CSV for a teammate.
5. Because previews use Markdown tables, they look best when the terminal supports monospace output.

## Troubleshooting

- **Missing dependency `tabulate`**: install `python3-tabulate` via apt or pip so previews render clean tables.
- **Excel export error**: ensure `openpyxl` is installed (`python3-openpyxl` on Debian).
- **Sheet not found**: pass the sheet name or 0-based index with `--sheet`. If it still fails, open the workbook in LibreOffice or Excel to confirm the tab name.
- **`ValueError: Unsupported export format`**: double-check the target file extension (`.csv`, `.tsv`, `.xlsx`).
- **`--inplace` not saving**: the script only overwrites when `--inplace` is paired with `transform`; for `filter`, use `--output`.


path: scripts/sheetsmith.py
content: #!/usr/bin/env python3
"""Utility for analyzing and editing CSV/Excel files with pandas."""
from __future__ import annotations

import argparse
import io
import sys
import textwrap
from pathlib import Path
from typing import Iterable, Sequence

import pandas as pd

PREVIEW_ROWS = 5


def detect_path(path: Path) -> str:
    suffix = path.suffix.lower()
    if suffix in {".csv", ".tsv", ".txt", ".xlsx", ".xls", ".xlsm", ".xlsb"}:
        return suffix
    raise ValueError(f"Unsupported spreadsheet format: {suffix}")


def load_dataframe(path: Path, sheet: str | None, delimiter: str, encoding: str) -> pd.DataFrame:
    suffix = path.suffix.lower()
    if suffix in {".xlsx", ".xls", ".xlsm", ".xlsb"}:
        return pd.read_excel(path, sheet_name=sheet)
    sep = "\t" if suffix == ".tsv" else delimiter
    return pd.read_csv(path, sep=sep, encoding=encoding)


def save_dataframe(df: pd.DataFrame, path: Path, engine: str | None = None) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    suffix = path.suffix.lower()
    if suffix in {".xlsx", ".xls"}:
        df.to_excel(path, index=False, engine=engine or "openpyxl")
    elif suffix == ".tsv" or suffix == ".txt":
        df.to_csv(path, index=False, sep="\t")
    elif suffix == ".csv":
        df.to_csv(path, index=False)
    else:
        raise ValueError(f"Unsupported export format: {suffix}")


def format_missing(df: pd.DataFrame) -> Sequence[tuple[str, float]]:
    missing = df.isna().mean()
    return [(col, pct) for col, pct in missing.items() if pct > 0]


def print_summary(df: pd.DataFrame) -> None:
    print(f"Shape: {df.shape[0]} rows × {df.shape[1]} columns")
    dtype_counts = df.dtypes.value_counts()
    print("Column types:")
    for dtype, count in dtype_counts.items():
        print(f"  {dtype}: {count}")
    missing = format_missing(df)
    if missing:
        print("Columns with missing values (percent):")
        for col, pct in sorted(missing, key=lambda item: item[1], reverse=True):
            print(f"  {col}: {pct:.1%}")
    else:
        print("No missing values detected.")
    print("Top cardinality (sample values):")
    for col in df.columns[:5]:
        unique = df[col].nunique(dropna=True)
        sample = df[col].dropna().unique()[:3]
        print(f"  {col}: {unique} unique, sample {list(sample)}")


def show_info(df: pd.DataFrame) -> None:
    buffer = io.StringIO()
    df.info(buf=buffer, memory_usage="deep")
    print(buffer.getvalue().rstrip())


def show_preview(df: pd.DataFrame, rows: int, tail: bool) -> None:
    print("Preview:")
    target = df.tail(rows) if tail else df.head(rows)
    print(target.to_markdown(index=False))


def run_summary(args: argparse.Namespace) -> None:
    df = load_dataframe(Path(args.path), args.sheet, args.delimiter, args.encoding)
    print_summary(df)
    if args.rows:
        show_preview(df, args.rows, args.tail)


def run_describe(args: argparse.Namespace) -> None:
    df = load_dataframe(Path(args.path), args.sheet, args.delimiter, args.encoding)
    include = args.include or "all"
    describe = df.describe(include=include, datetime_is_numeric=True)
    if args.percentiles:
        describe = df.describe(include=include, percentiles=[p / 100 for p in args.percentiles],
                               datetime_is_numeric=True)
    print(describe)


def run_preview(args: argparse.Namespace) -> None:
    df = load_dataframe(Path(args.path), args.sheet, args.delimiter, args.encoding)
    show_preview(df, args.rows, args.tail)


def run_filter(args: argparse.Namespace) -> None:
    df = load_dataframe(Path(args.path), args.sheet, args.delimiter, args.encoding)
    if not args.query:
        raise SystemExit("--query expression is required for filter")
    result = df.query(args.query, engine="python")
    if args.sample:
        result = result.sample(min(args.sample, len(result)))
    if args.output:
        save_dataframe(result, Path(args.output))
        print(f"Filtered output written to {args.output}")
    else:
        show_preview(result, args.rows or PREVIEW_ROWS, args.tail)


def run_transform(args: argparse.Namespace) -> None:
    df = load_dataframe(Path(args.path), args.sheet, args.delimiter, args.encoding)
    if not args.expr:
        raise SystemExit("At least one --expr is required for transform")
    for expr in args.expr:
        df.eval(expr, inplace=True, engine="python")
    if args.drop:
        df.drop(columns=args.drop, errors="ignore", inplace=True)
    if args.rename:
        rename_map = dict(pair.split(":", 1) for pair in args.rename)
        df.rename(columns=rename_map, inplace=True)
    if args.output or args.inplace:
        dest = Path(args.output) if args.output else Path(args.path)
        save_dataframe(df, dest)
        print(f"Transformations saved to {dest}")
    else:
        show_preview(df, args.rows or PREVIEW_ROWS, args.tail)


def run_convert(args: argparse.Namespace) -> None:
    df = load_dataframe(Path(args.path), args.sheet, args.delimiter, args.encoding)
    target = Path(args.output)
    save_dataframe(df, target, engine=args.engine)
    print(f"Converted {args.path} → {target}")


def main() -> int:
    parser = argparse.ArgumentParser(
        description="Sheetsmith: inspect, summarize, and edit CSV/Excel files with pandas",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    parent = argparse.ArgumentParser(add_help=False)
    parent.add_argument("--sheet", help="Excel sheet name or index")
    parent.add_argument("--delimiter", "-d", default=",",
                        help="Default CSV separator (ignored for Excel)")
    parent.add_argument("--encoding", default="utf-8", help="File encoding for text files")

    subparsers = parser.add_subparsers(dest="command", required=True)

    summary_parser = subparsers.add_parser("summary", parents=[parent], help="High-level summary")
    summary_parser.add_argument("path", help="Path to CSV/Excel file")
    summary_parser.add_argument("--rows", type=int, help="Preview rows after summary")
    summary_parser.add_argument("--tail", action="store_true", help="Show tail instead of head")
    summary_parser.set_defaults(func=run_summary)

    describe_parser = subparsers.add_parser("describe", parents=[parent], help="Pandas describe report")
    describe_parser.add_argument("path", help="Path to CSV/Excel file")
    describe_parser.add_argument("--include", choices=["all", "object", "number", "datetime"],
                                 help="Which dtypes to include in describe")
    describe_parser.add_argument("--percentiles", type=int, nargs="*",
                                 help="Additional percentile values (0-100)")
    describe_parser.set_defaults(func=run_describe)

    preview_parser = subparsers.add_parser("preview", parents=[parent], help="Quick head/tail preview")
    preview_parser.add_argument("path", help="Path to file")
    preview_parser.add_argument("--rows", type=int, default=5)
    preview_parser.add_argument("--tail", action="store_true")
    preview_parser.set_defaults(func=run_preview)

    filter_parser = subparsers.add_parser("filter", parents=[parent], help="Filter rows with pandas query")
    filter_parser.add_argument("path", help="Source file")
    filter_parser.add_argument("--query", required=True, help="pandas query, e.g., 'state == \"CA\"'")
    filter_parser.add_argument("--rows", type=int, help="Preview rows when not writing")
    filter_parser.add_argument("--tail", action="store_true")
    filter_parser.add_argument("--sample", type=int, help="Return a random sample of the matches")
    filter_parser.add_argument("--output", help="Write filtered rows to this path")
    filter_parser.set_defaults(func=run_filter)

    transform_parser = subparsers.add_parser("transform", parents=[parent],
                                             help="Add/rename/drop columns via pandas expressions")
    transform_parser.add_argument("path", help="Source file")
    transform_parser.add_argument("--expr", action="append",
                                  help="Expression, e.g., 'total=quantity*price'",)
    transform_parser.add_argument("--drop", nargs="*", help="Columns to drop")
    transform_parser.add_argument("--rename", nargs="*",
                                  help="Rename mappings (old:new). Example: --rename foo:bar baz:q1")
    transform_parser.add_argument("--output", help="Save transformed dataframe to this path")
    transform_parser.add_argument("--inplace", action="store_true",
                                  help="Overwrite the source file")
    transform_parser.add_argument("--rows", type=int, help="Show preview rows when not writing")
    transform_parser.add_argument("--tail", action="store_true")
    transform_parser.set_defaults(func=run_transform)

    convert_parser = subparsers.add_parser("convert", parents=[parent], help="Convert between formats")
    convert_parser.add_argument("path", help="Source file")
    convert_parser.add_argument("--output", required=True, help="Target file (csv, tsv, xlsx)")
    convert_parser.add_argument("--engine", help="Excel engine override (openpyxl/xlsxwriter)")
    convert_parser.set_defaults(func=run_convert)

    args = parser.parse_args()
    try:
        args.func(args)
    except Exception as exc:  # pragma: no cover - best effort
        print(f"Error: {exc}", file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: tests/data/test.csv
content: name,state,population,area,active
Alpha,CA,100,10,yes
Beta,NY,200,20,no
Gamma,CA,150,15,yes


path: tests/test_sheetsmith.py
content: import csv
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
import unittest

SCRIPT_PATH = Path(__file__).resolve().parents[1] / "scripts" / "sheetsmith.py"
SAMPLE_DATA = Path(__file__).resolve().parents[1] / "tests" / "data" / "test.csv"


def run_cli(args, workspace):
    return subprocess.run(
        [sys.executable, str(SCRIPT_PATH)] + args,
        capture_output=True,
        text=True,
        cwd=workspace,
    )


class SheetsmithTests(unittest.TestCase):
    def test_summary_outputs_shape_and_preview(self):
        with tempfile.TemporaryDirectory() as tmpdir:
            workspace = Path(tmpdir)
            dest = workspace / "data.csv"
            shutil.copy2(SAMPLE_DATA, dest)

            result = run_cli(["summary", str(dest), "--rows", "2"], workspace)
            self.assertEqual(result.returncode, 0)
            self.assertIn("Shape:", result.stdout)
            self.assertIn("Preview:", result.stdout)

    def test_filter_and_transform_write_files(self):
        with tempfile.TemporaryDirectory() as tmpdir:
            workspace = Path(tmpdir)
            source = workspace / "data.csv"
            shutil.copy2(SAMPLE_DATA, source)

            filtered = workspace / "filtered.csv"
            result = run_cli([
                "filter",
                str(source),
                "--query",
                "state == 'CA'",
                "--output",
                str(filtered),
            ], workspace)
            self.assertEqual(result.returncode, 0)
            self.assertTrue(filtered.exists())
            with open(filtered, newline="", encoding="utf-8") as stream:
                reader = list(csv.DictReader(stream))
                self.assertEqual(len(reader), 2)

            transformed = workspace / "with-density.csv"
            result = run_cli([
                "transform",
                str(filtered),
                "--expr",
                "density = population / area",
                "--output",
                str(transformed),
            ], workspace)
            self.assertEqual(result.returncode, 0)
            with open(transformed, newline="", encoding="utf-8") as stream:
                reader = csv.DictReader(stream)
                self.assertIn("density", reader.fieldnames)
                rows = list(reader)
                self.assertTrue(all(float(row["density"]) > 0 for row in rows))


if __name__ == "__main__":
    unittest.main()


Security scan verdicts — ClawScan: clean | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)