npm package: @profullstack/places
Description: CLI tool to scrape business place data from ValueSERP API
Latest version: 1.0.2

# Places Data Script

A powerful CLI tool and programmatic API for scraping business place data from the ValueSERP API with support for pagination, detailed information fetching, and export to JSON and CSV formats.

## Features

- 🔧 **Interactive Setup** - Easy configuration wizard for API keys
- 🔄 **Smart Pagination** - Automatically handles pagination until all results are fetched
- 📊 **Dual Output** - Exports data to both JSON and CSV formats
- 🔁 **Retry Logic** - Automatic retry with exponential backoff for failed requests
- 🎯 **Deduplication** - Removes duplicate entries by data_cid
- ⚡ **Rate Limiting** - Configurable delays to respect API limits
- 📝 **Progress Tracking** - Real-time progress updates during scraping
- 🔌 **Programmatic API** - Use as a module in your own applications

## Installation

### Using pnpm (recommended)

```bash
pnpm install
```

### Using npm

```bash
npm install
```

### Global Installation

```bash
pnpm install -g .
# or
npm install -g .
```

## CLI Usage

### Setup

Configure your API key and preferences:

```bash
places setup
```

### Search

Search for places and export data:

```bash
places search --query "pizza" --location "New Jersey,United States"
```

#### Options

- `-q, --query <term>` - Search query (required)
- `-l, --location <location>` - Location as "City,State" (required)
- `-o, --output <name>` - Custom output filename prefix
- `-k, --api-key <key>` - Override saved API key
- `-d, --delay <ms>` - Delay between API calls (default: 500ms)

#### Examples

```bash
# Basic search
places search -q "pizza" -l "New Jersey,United States"

# Custom output name
places search -q "sushi" -l "Los Angeles,California" -o "sushi-restaurants"

# Slower rate limiting
places search -q "pizza" -l "New York,New York" -d 1000

# One-time API key override
places search -q "pizza" -l "Chicago,Illinois" -k "TEMP_API_KEY"
```

## Programmatic API

Use Places Data Script as a module in your own applications:

```javascript
import PlacesDataScript from 'places-data-script';

// Create instance with options
const scraper = new PlacesDataScript({
  apiKey: 'YOUR_API_KEY',
  delay: 500,
  verbose: true
});

// Search and export
const results = await scraper.search({
  query: 'pizza',
  location: 'New Jersey,United States',
  output: 'my-results', // optional
  export: true // optional, default true
});

console.log(`Found ${results.count} places`);
console.log(`JSON: ${results.export.json}`);
console.log(`CSV: ${results.export.csv}`);
```

### API Methods

#### `search(params)`

Search for places and optionally export results.

```javascript
const results = await scraper.search({
  query: 'pizza',
  location: 'New Jersey,United States',
  output: 'custom-name', // optional
  export: true // optional, default true
});
```

#### `fetchPlaceIds(params)`

Fetch only the list of place IDs without details.

```javascript
const ids = await scraper.fetchPlaceIds({
  query: 'pizza',
  location: 'New Jersey,United States'
});
```

#### `fetchDetails(dataCids)`

Fetch details for specific place IDs.

```javascript
const details = await scraper.fetchDetails(['id1', 'id2', 'id3']);
```

#### `export(data, query, location, output)`

Export data to JSON and CSV files.

```javascript
const result = await scraper.export(
  placeData,
  'pizza',
  'New Jersey,United States',
  'my-export'
);
```

### Individual Function Exports

You can also import individual functions:

```javascript
import {
  fetchAllPlaces,
  fetchAllPlaceDetails,
  exportData,
  saveToJSON,
  saveToCSV,
  loadConfig,
  saveConfig
} from 'places-data-script';
```

## Output Format

### JSON

```json
[
  {
    "data_cid": "1866378315924306344",
    "name": "Joe's Pizza",
    "phone": "+1 555-0123",
    "website": "https://joespizza.com",
    "address": "123 Main St, Newark, NJ 07102",
    "latitude": 40.7357,
    "longitude": -74.1724,
    "hours": "Mon-Sun: 11:00 AM - 10:00 PM",
    "review_count": 245,
    "rating": 4.5
  }
]
```

### CSV

```csv
data_cid,name,phone,website,address,latitude,longitude,hours,review_count,rating
1866378315924306344,Joe's Pizza,+1 555-0123,https://joespizza.com,"123 Main St, Newark, NJ 07102",40.7357,-74.1724,"Mon-Sun: 11:00 AM - 10:00 PM",245,4.5
```

## Configuration

Configuration is stored in `~/.config/places/settings.json`:

```json
{
  "apiKey": "YOUR_VALUESERP_API_KEY",
  "defaultDelay": 500
}
```

### Configuration Priority

1. Command-line `--api-key` flag (highest)
2. Constructor options (programmatic API)
3. Saved config in `~/.config/places/settings.json`
4. Default embedded API key (lowest)

## Development

### Install Dependencies

```bash
pnpm install
```

### Run Tests

The project uses **Vitest** for testing with comprehensive unit tests for all modules.

```bash
# Run all tests
pnpm test

# Run tests in watch mode
pnpm test:watch

# Run tests with coverage report
pnpm test:coverage
```

**Test Coverage:**
- ✅ Config module (17 tests) - Configuration management and fallbacks
- ✅ Export module (13 tests) - JSON/CSV export and deduplication
- ✅ API module (11 tests) - API calls, pagination, retry logic, rate limiting
- ✅ Programmatic API (20 tests) - PlacesDataScript class and all methods

**Total: 61 tests passing**

### Lint Code

```bash
pnpm lint
```

### Format Code

```bash
pnpm format
```

### Link for Local Development

```bash
pnpm link
```

## Documentation

- [Architecture](./docs/architecture.md) - Technical architecture and design decisions
- [API Documentation](./docs/README.md) - Detailed API documentation
- [Workflow Diagram](./docs/workflow.puml) - Visual representation of the data scraping process

## Project Structure

```
places/
├── bin/
│   └── places.js          # CLI entry point
├── lib/
│   ├── index.js           # Programmatic API
│   ├── config.js          # Configuration management
│   ├── api.js             # API client functions
│   └── export.js          # JSON/CSV export functions
├── docs/
│   ├── README.md          # API documentation
│   ├── architecture.md    # Architecture documentation
│   └── workflow.puml      # PlantUML workflow diagram
├── test/
│   └── *.test.js          # Test files
├── package.json
└── README.md              # This file
```

## License

MIT

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.