npm package: @maydotinc/s3-syncer
Description: Full-featured CLI to sync local directories to S3-compatible buckets via GitHub Actions or locally. You may also pull files from a remote target.
Latest version: 0.1.6

# @maydotinc/s3-syncer

Sync local folders to S3-compatible storage such as AWS S3, Cloudflare R2, and MinIO.

`s3-syncer` is designed for real app repos and monorepos:
- reusable `s3-syncer.json` config
- interactive setup and one-off sync flows
- dry runs and validation before mutating remote state
- streaming uploads and hashing for large asset sets
- bounded concurrency and retries for production reliability
- `pull` support with incremental diffing and dry-run previews

## Quick Start

Most teams can start with:

```bash
npx @maydotinc/s3-syncer sync
```

If `s3-syncer.json` is missing, the CLI offers to:
- create a reusable config, or
- run a one-time sync without writing config

If credentials are missing, it can:
- load them from `.env` or `--env-file`, or
- prompt for them interactively

## Requirements

- Node.js 20+
- S3-compatible credentials with:
  - `ListObjectsV2`
  - `PutObject`
  - `GetObject` for `pull`
  - optionally `DeleteObject` when `delete: true`

## Common Commands

```bash
# Interactive setup
npx @maydotinc/s3-syncer init --config-only

# Validate config, env placeholders, credentials, and read access
npx @maydotinc/s3-syncer validate --env

# Preview uploads/deletes without mutating remote state
npx @maydotinc/s3-syncer sync --dry-run --env

# Run an actual sync
npx @maydotinc/s3-syncer sync --env

# Watch local files and rerun sync on changes
npx @maydotinc/s3-syncer sync --watch --env

# Preview a pull without downloading files
npx @maydotinc/s3-syncer pull --dry-run -o ./pulled --env

# Pull only changed/new files
npx @maydotinc/s3-syncer pull -o ./pulled --env
```

## Core Features

### Sync

- fingerprints local files
- lists remote objects
- compares local hashes to remote ETags
- uploads changed/new files
- optionally deletes stale remote files
- retries transient failures
- ends with a summary instead of aborting on the first bad file

### Pull

- lists remote objects under a target/prefix
- compares remote files against local files in the output directory
- downloads only `new` or `changed` files
- skips `unchanged` files
- supports `--dry-run`
- honors target `exclude` rules

### Production controls

Each target can define:
- `exclude`
- `concurrency`
- `retry`
- `maxFileSize`
- `watch`
- `envFile`
- `${ENV_VAR}` placeholders for bucket, endpoint, and credentials

## Guides

Start here for the level of depth you need:

- [Commands and config reference](./docs/commands-and-config.md)
- [Notifications](./docs/notifications.md)
- [Watch Mode](./docs/watch-mode.md)
- [Monorepo setup (basic)](./docs/monorepo-basic-setup.md)
- [Advanced setup (single target)](./docs/monorepo-advanced-setup.md)
- [Agent Skills](./docs/agent-skills.md)

## Agent Skills

This repo includes portable Agent Skills under [`skills/`](./skills):
- `s3-syncer-integration`
- `shared-cdn-assets`
- `s3-syncer-troubleshooting`

They package the integration, architecture, and troubleshooting workflows that agents tend to get wrong when they only see a generic README.

## Minimal Config Example

```json
{
  "targets": [
    {
      "directory": "public",
      "bucket": "my-assets",
      "region": "auto",
      "endpoint": "https://<account>.r2.cloudflarestorage.com",
      "prefix": "",
      "delete": false,
      "exclude": ["**/*.map", ".next/cache/**"],
      "concurrency": {
        "fingerprint": 8,
        "upload": 4,
        "delete": 8
      },
      "retry": {
        "maxAttempts": 3,
        "baseDelayMs": 250,
        "maxDelayMs": 4000
      },
      "maxFileSize": {
        "warnBytes": 25000000,
        "errorBytes": 100000000
      },
      "watch": {
        "debounceMs": 500
      }
    }
  ],
  "branch": "main",
  "notifications": {
    "slack": false,
    "discord": false
  }
}
```

## Environment Variables

Required for normal sync/pull usage unless target-level credential placeholders are used:

```bash
AWS_S3_ACCESS_KEY_ID=your_access_key
AWS_S3_SECRET_ACCESS_KEY=your_secret_key
AWS_S3_ENDPOINT=
```

Optional:
- `SLACK_WEBHOOK_URL`
- `DISCORD_WEBHOOK_URL`

## Notes

- `sync` and `pull` both use MD5 vs remote ETag for change detection today.
- If remote objects were uploaded outside `s3-syncer` with multipart uploads or provider-specific ETag behavior, some unchanged files may still be treated as changed.
- Hashing and uploads are streamed, so large files are not fully buffered in memory.

## Development

```bash
git clone https://github.com/maydotinc/s3-syncer.git
cd s3-syncer
pnpm install
pnpm test
pnpm typecheck
pnpm build
```

## License

[MIT](./LICENSE)