npm package: @peaseernest/jsondatabase
Description: Lightweight, Concurrency-Controlled JSON Persistence
Author: Pease Ernest / Ernest Tech House Co-operation
Latest version: 1.0.0

# 💾 jsondatabase

**Lightweight, Concurrency-Controlled JSON Persistence**

Built by: Pease Ernest / Ernest Tech House Co-operation

[![Node.js Version](https://img.shields.io/badge/node-%3E%3D14.0.0-brightgreen)](https://nodejs.org/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Workers](https://img.shields.io/badge/workers-50-orange)]()

---

## 📋 Table of Contents

1. [Overview](#-overview)
2. [Installation](#-installation)
3. [Quick Start](#-quick-start)
4. [Complete API Reference](#-complete-api-reference)
5. [Configuration Guide](#-configuration-guide)
6. [Usage Examples](#-usage-examples)
7. [Performance & Limitations](#-performance--limitations)
8. [Architecture](#-architecture)
9. [Best Practices](#-best-practices)
10. [Troubleshooting](#-troubleshooting)

---

## 🚀 Overview

**jsondatabase** is a specialized, embedded JSON file store designed for small to medium-sized projects that require stability and predictable I/O throughput.

### What Makes It Different?

Unlike heavyweight databases (SQLite, MongoDB) which prioritize complex querying and indexing, jsondatabase trades those features for:

- ✅ **Simple implementation** - Zero learning curve
- ✅ **Absolute control** over file I/O concurrency
- ✅ **Zero dependencies** - Pure Node.js
- ✅ **Human-readable storage** - Plain JSON files
- ✅ **50 Worker threads** - Massive parallel processing capability
- ✅ **Automatic backups** - Never lose data

### Perfect For:

- 🎯 Indie developers building MVPs
- 🎯 Small serverless functions
- 🎯 Utility scripts and automation
- 🎯 Config/settings storage
- 🎯 Cache systems
- 🎯 Data collection services
- 🎯 Applications where database complexity is overkill

### Not Recommended For:

- ❌ Large datasets (>50MB)
- ❌ Complex queries with joins
- ❌ High-frequency trading systems
- ❌ Systems requiring ACID transactions
- ❌ Real-time collaborative editing

---

## 📦 Installation

### NPM Installation

```bash
npm install jsondatabase
```

### From Source

```bash
git clone https://github.com/ernest-tech-house-co-operation/jsondatabase.git
cd jsondatabase
npm install
```

### Requirements

- Node.js >= 14.0.0
- ES Modules support

---

## 🛠️ Quick Start

### Basic Example

```javascript
import jsondatabase from 'jsondatabase';

// 1. Initialize database
const db = jsondatabase({
  name: 'mydata.json',      // Database filename
  dataid: 'MyApp',          // Root key in JSON
  databackup: true,         // Enable auto-backup
  writing: 2,               // Max 2 writes/sec
  reading: 6                // Max 6 reads/sec
});

// 2. Add data (async/await required!)
await db.add('user_ernest', {
  name: 'Ernest P.',
  email: 'ernest@example.com',
  balance: 600000000000000
});

// 3. Read data
const user = await db.get('user_ernest');
console.log(user.name); // "Ernest P."

// 4. Update nested fields
await db.setPath('user_ernest', 'balance', 1000000000000000);

// 5. Check existence
if (await db.has('user_ernest')) {
  console.log('User exists!');
}

// 6. Delete data
await db.delete('old_record');

// 7. Cleanup (important!)
await db.shutdown();
```

### ⚠️ Critical: Always Use `await`

**ALL database operations return Promises!** You MUST use `await`:

```javascript
// ❌ WRONG - Won't work!
const user = db.get('user_123');
console.log(user.name); // undefined (Promise not resolved)

// ✅ CORRECT
const user = await db.get('user_123');
console.log(user.name); // Works!
```

---

## 📚 Complete API Reference

### Initialization

#### `jsondatabase(config)`

Creates a new database instance.

**Parameters:**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `name` | string | ✅ Yes | - | Database filename (e.g., 'data.json') |
| `dataid` | string | ✅ Yes | - | Root key for data structure |
| `databackup` | boolean | No | `true` | Enable automatic backups |
| `writing` | number | No | `2` | Max write ops/sec (1-50) |
| `reading` | number | No | `6` | Max read ops/sec (1-50) |
| `logLevel` | string | No | `'info'` | Logging: silent/error/warn/info/debug |

**Example:**

```javascript
const db = jsondatabase({
  name: 'userdata.json',
  dataid: 'UserDatabase',
  databackup: true,
  writing: 5,
  reading: 20,
  logLevel: 'debug'
});
```

---

### Core Operations

#### `add(key, value)` → `Promise<boolean>`

Add or update a key-value pair.

**Parameters:**
- `key` (string): Unique identifier
- `value` (any): JSON-serializable value

**Returns:** `true` on success

**Example:**

```javascript
// Add simple value
await db.add('setting_theme', 'dark');

// Add object
await db.add('user_123', {
  name: 'Alice',
  age: 30,
  interests: ['coding', 'music']
});

// Add array
await db.add('scores', [100, 95, 88]);

// Add number
await db.add('counter', 42);
```

---

#### `get(key)` → `Promise<any>`

Retrieve a value by key.

**Parameters:**
- `key` (string): Key to retrieve

**Returns:** The stored value, or `undefined` if key doesn't exist

**Example:**

```javascript
const user = await db.get('user_123');
console.log(user.name); // 'Alice'

const missing = await db.get('nonexistent');
console.log(missing); // undefined
```

---

#### `setPath(key, path, value)` → `Promise<boolean>`

Update a nested field using dot notation.

**Parameters:**
- `key` (string): Key containing the object
- `path` (string): Dot-separated path (e.g., 'user.profile.age')
- `value` (any): Value to set

**Returns:** `true` on success

**Throws:** Error if key doesn't exist

**Example:**

```javascript
// Setup initial data
await db.add('user_123', {
  profile: {
    name: 'Alice',
    address: {
      city: 'New York'
    }
  },
  settings: {
    notifications: true
  }
});

// Update nested field
await db.setPath('user_123', 'profile.address.city', 'Los Angeles');

// Create new nested path
await db.setPath('user_123', 'profile.bio.description', 'Software engineer');

// Update deep nested value
await db.setPath('user_123', 'settings.notifications', false);

// Verify changes
const user = await db.get('user_123');
console.log(user.profile.address.city); // 'Los Angeles'
```

---

#### `delete(key)` → `Promise<boolean>`

Delete a key from the database.

**Parameters:**
- `key` (string): Key to delete

**Returns:** `true` if key existed, `false` if it didn't

**Example:**

```javascript
// Delete existing key
const deleted = await db.delete('user_123');
console.log(deleted); // true

// Try to delete non-existent key
const notFound = await db.delete('nonexistent');
console.log(notFound); // false
```

---

#### `has(key)` → `Promise<boolean>`

Check if a key exists.

**Parameters:**
- `key` (string): Key to check

**Returns:** `true` if exists, `false` otherwise

**Example:**

```javascript
if (await db.has('user_123')) {
  console.log('User exists');
  const user = await db.get('user_123');
} else {
  console.log('User not found');
}
```

---

#### `getAllKeys()` → `Promise<string[]>`

Get array of all keys in the database.

**Returns:** Array of key strings

**Example:**

```javascript
const keys = await db.getAllKeys();
console.log(keys); // ['user_123', 'user_456', 'setting_theme']

// Process all keys
for (const key of keys) {
  const value = await db.get(key);
  console.log(`${key}:`, value);
}
```

---

#### `getAll()` → `Promise<Object>`

Get a copy of all data in the database.

**Returns:** Object containing all key-value pairs

**Example:**

```javascript
const allData = await db.getAll();
console.log(allData);
// {
//   user_123: { name: 'Alice', age: 30 },
//   user_456: { name: 'Bob', age: 25 },
//   setting_theme: 'dark'
// }

// Safe to modify (it's a copy)
allData.user_123.name = 'Changed';
// Original database unchanged
```

---

#### `clear()` → `Promise<boolean>`

Remove all data from the database.

**Returns:** `true` on success

**Example:**

```javascript
// Clear everything
await db.clear();

// Verify
const keys = await db.getAllKeys();
console.log(keys.length); // 0
```

---

### Utility Methods

#### `getStats()` → `Object`

Get current database statistics (synchronous).

**Returns:** Statistics object

**Example:**

```javascript
const stats = db.getStats();
console.log(stats);
// {
//   queueSize: { read: 2, write: 1, total: 3 },
//   activeWorkers: 5,
//   availableWorkers: 45,
//   totalOperations: {
//     totalReads: 150,
//     totalWrites: 75,
//     totalOperations: 225
//   }
// }
```

---

#### `shutdown()` → `Promise<void>`

Gracefully shutdown the database.

**Important:** Always call before exiting your application!

**Example:**

```javascript
// At application exit
process.on('SIGINT', async () => {
  console.log('Shutting down...');
  await db.shutdown();
  process.exit(0);
});
```

---

## ⚙️ Configuration Guide

### Performance Tuning

```javascript
// Low traffic application
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  writing: 2,   // 2 writes per second
  reading: 6    // 6 reads per second
});

// Medium traffic application
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  writing: 10,  // 10 writes per second
  reading: 30   // 30 reads per second
});

// High traffic application (max)
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  writing: 50,  // 50 writes per second (maximum)
  reading: 50   // 50 reads per second (maximum)
});
```

### Logging Levels

```javascript
// Production: Silent (no logs)
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  logLevel: 'silent'
});

// Production: Errors only
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  logLevel: 'error'
});

// Development: Info logging
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  logLevel: 'info'
});

// Debugging: Verbose output
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  logLevel: 'debug'
});
```

---

## 💡 Usage Examples

### Example 1: User Management System

```javascript
import jsondatabase from 'jsondatabase';

const db = jsondatabase({
  name: 'users.json',
  dataid: 'UserSystem',
  writing: 5,
  reading: 20
});

// Register new user
async function registerUser(username, email, password) {
  // Check if user exists
  if (await db.has(`user_${username}`)) {
    throw new Error('Username already taken');
  }
  
  // Create user
  await db.add(`user_${username}`, {
    username,
    email,
    password, // In production, hash this!
    createdAt: new Date().toISOString(),
    profile: {
      bio: '',
      avatar: null
    },
    settings: {
      notifications: true,
      theme: 'light'
    }
  });
  
  console.log(`User ${username} registered`);
}

// Update user profile
async function updateProfile(username, bio, avatar) {
  await db.setPath(`user_${username}`, 'profile.bio', bio);
  await db.setPath(`user_${username}`, 'profile.avatar', avatar);
  console.log(`Profile updated for ${username}`);
}

// Get user settings
async function getUserSettings(username) {
  const user = await db.get(`user_${username}`);
  return user?.settings;
}

// Usage
await registerUser('alice', 'alice@example.com', 'pass123');
await updateProfile('alice', 'Software engineer', 'avatar.jpg');
const settings = await getUserSettings('alice');
console.log(settings); // { notifications: true, theme: 'light' }

await db.shutdown();
```

---

### Example 2: Shopping Cart System

```javascript
import jsondatabase from 'jsondatabase';

const db = jsondatabase({
  name: 'shop.json',
  dataid: 'ShopData'
});

// Add product to catalog
async function addProduct(id, name, price, stock) {
  await db.add(`product_${id}`, {
    name,
    price,
    stock,
    createdAt: new Date().toISOString()
  });
}

// Create shopping cart
async function createCart(userId) {
  await db.add(`cart_${userId}`, {
    items: [],
    total: 0,
    createdAt: new Date().toISOString()
  });
}

// Add item to cart
async function addToCart(userId, productId, quantity) {
  const cart = await db.get(`cart_${userId}`);
  const product = await db.get(`product_${productId}`);
  
  if (!product) {
    throw new Error('Product not found');
  }
  
  if (product.stock < quantity) {
    throw new Error('Insufficient stock');
  }
  
  // Update cart
  cart.items.push({
    productId,
    name: product.name,
    price: product.price,
    quantity
  });
  
  cart.total += product.price * quantity;
  
  await db.add(`cart_${userId}`, cart);
  
  // Update stock
  await db.setPath(`product_${productId}`, 'stock', product.stock - quantity);
}

// Get cart summary
async function getCartSummary(userId) {
  const cart = await db.get(`cart_${userId}`);
  return {
    itemCount: cart.items.length,
    total: cart.total,
    items: cart.items
  };
}

// Usage
await addProduct(1, 'Laptop', 999.99, 10);
await addProduct(2, 'Mouse', 29.99, 50);

await createCart('user_123');
await addToCart('user_123', 1, 1);
await addToCart('user_123', 2, 2);

const summary = await getCartSummary('user_123');
console.log(summary);
// {
//   itemCount: 2,
//   total: 1059.97,
//   items: [...]
// }

await db.shutdown();
```

---

### Example 3: Configuration Manager

```javascript
import jsondatabase from 'jsondatabase';

class ConfigManager {
  constructor() {
    this.db = jsondatabase({
      name: 'config.json',
      dataid: 'AppConfig',
      writing: 2,
      reading: 10
    });
  }
  
  async init() {
    // Set default config if not exists
    if (!(await this.db.has('app_settings'))) {
      await this.db.add('app_settings', {
        server: {
          port: 3000,
          host: 'localhost'
        },
        database: {
          connectionString: 'mongodb://localhost:27017',
          poolSize: 10
        },
        features: {
          enableCache: true,
          enableLogging: true
        }
      });
    }
  }
  
  async get(path) {
    const config = await this.db.get('app_settings');
    return this._getNestedValue(config, path);
  }
  
  async set(path, value) {
    await this.db.setPath('app_settings', path, value);
  }
  
  async getAll() {
    return await this.db.get('app_settings');
  }
  
  _getNestedValue(obj, path) {
    return path.split('.').reduce((current, key) => current?.[key], obj);
  }
  
  async close() {
    await this.db.shutdown();
  }
}

// Usage
const config = new ConfigManager();
await config.init();

// Get specific config
const port = await config.get('server.port');
console.log(port); // 3000

// Update config
await config.set('server.port', 8080);
await config.set('features.enableCache', false);

// Get all config
const allConfig = await config.getAll();
console.log(allConfig);

await config.close();
```

---

### Example 4: Task Queue System

```javascript
import jsondatabase from 'jsondatabase';

class TaskQueue {
  constructor() {
    this.db = jsondatabase({
      name: 'tasks.json',
      dataid: 'TaskQueue',
      writing: 10,
      reading: 20
    });
  }
  
  async addTask(taskId, data) {
    await this.db.add(`task_${taskId}`, {
      id: taskId,
      status: 'pending',
      data,
      createdAt: new Date().toISOString(),
      startedAt: null,
      completedAt: null,
      error: null
    });
  }
  
  async startTask(taskId) {
    await this.db.setPath(`task_${taskId}`, 'status', 'running');
    await this.db.setPath(`task_${taskId}`, 'startedAt', new Date().toISOString());
  }
  
  async completeTask(taskId, result) {
    await this.db.setPath(`task_${taskId}`, 'status', 'completed');
    await this.db.setPath(`task_${taskId}`, 'completedAt', new Date().toISOString());
    await this.db.setPath(`task_${taskId}`, 'result', result);
  }
  
  async failTask(taskId, error) {
    await this.db.setPath(`task_${taskId}`, 'status', 'failed');
    await this.db.setPath(`task_${taskId}`, 'error', error);
  }
  
  async getPendingTasks() {
    const keys = await this.db.getAllKeys();
    const tasks = [];
    
    for (const key of keys) {
      const task = await this.db.get(key);
      if (task.status === 'pending') {
        tasks.push(task);
      }
    }
    
    return tasks;
  }
  
  async getTaskStatus(taskId) {
    const task = await this.db.get(`task_${taskId}`);
    return task?.status;
  }
  
  async close() {
    await this.db.shutdown();
  }
}

// Usage
const queue = new TaskQueue();

// Add tasks
await queue.addTask('task_1', { action: 'send_email', to: 'user@example.com' });
await queue.addTask('task_2', { action: 'process_image', file: 'photo.jpg' });

// Process task
await queue.startTask('task_1');
// ... do work ...
await queue.completeTask('task_1', { sent: true });

// Check status
const status = await queue.getTaskStatus('task_1');
console.log(status); // 'completed'

// Get pending tasks
const pending = await queue.getPendingTasks();
console.log(pending.length); // 1

await queue.close();
```

---

## 🚨 Performance & Limitations

### File Size Guidelines

| File Size | Status | Performance | Recommendation |
|-----------|--------|-------------|----------------|
| < 1MB | ✅ Excellent | Fast | Ideal for most use cases |
| 1-10MB | ✅ Good | Normal | Recommended maximum |
| 10-50MB | ⚠️ Caution | Degraded | Consider alternatives |
| > 50MB | ❌ Not Recommended | Severe slowdown | Use real database |

### Performance Benchmarks

Typical performance on modern hardware:

```
Write Operations:    50-100 ops/sec
Read Operations:     200-400 ops/sec
Path Updates:        40-80 ops/sec
Mixed Operations:    100-200 ops/sec
```

### Concurrency

- **50 Worker Threads**: Massive parallel processing
- **Rate Limiting**: Prevents file system overload
- **Round-Robin Scheduling**: Fair allocation between reads/writes
- **Queue Management**: Automatic backpressure handling

### Known Limitations

1. **Not Atomic**: No transactional guarantees
2. **No Rollback**: Manual backup restore required
3. **No Querying**: Direct key lookup only (no `.find()`, `.filter()`)
4. **Memory Usage**: Entire file loaded for each operation
5. **Large Numbers**: Store numbers > 2^53 as strings
6. **Node.js Only**: Requires `fs` and `worker_threads`

---

## 🏗️ Architecture

### Component Overview

```
┌─────────────────────────────────────────┐
│         jsondatabase Instance           │
├─────────────────────────────────────────┤
│  - Configuration                        │
│  - Public API Methods                   │
└─────────────────┬───────────────────────┘
                  │
         ┌────────┼────────┐
         │        │        │
         ▼        ▼        ▼
   ┌─────────┐ ┌──────────┐ ┌─────────────┐
   │  File   │ │  Task    │ │   Worker    │
   │ Manager │ │Scheduler │ │    Pool     │
   └─────────┘ └──────────┘ └─────────────┘
       │            │              │
       │            │              │
       ▼            ▼              ▼
   ┌─────────┐ ┌──────────┐ ┌─────────────┐
   │  JSON   │ │Read/Write│ │ 50 Worker   │
   │  File   │ │  Queues  │ │  Threads    │
   └─────────┘ └──────────┘ └─────────────┘
```

### File Manager
- Handles all disk I/O operations
- Creates/validates JSON structure
- Manages backup files
- Provides file size info

### Task Scheduler
- Round-robin scheduling (write → read → write)
- Rate limiting (ops per second)
- Queue management
- Statistics tracking

### Worker Pool
- 50 independent worker threads
- Automatic worker replacement on failure
- Task distribution
- Timeout handling

---

## 🎯 Best Practices

### 1. Always Use Async/Await

```javascript
// ❌ BAD - Promises not handled
db.add('key', 'value');
const value = db.get('key');

// ✅ GOOD - Proper async handling
await db.add('key', 'value');
const value = await db.get('key');
```

### 2. Always Shutdown Gracefully

```javascript
// ❌ BAD - Potential data loss
process.exit(0);

// ✅ GOOD - Clean shutdown
await db.shutdown();
process.exit(0);

// ✅ BEST - Handle signals
process.on('SIGINT', async () => {
  await db.shutdown();
  process.exit(0);
});
```

### 3. Use Try-Catch for Error Handling

```javascript
try {
  await db.setPath('user_123', 'profile.name', 'Alice');
} catch (error) {
  if (error.message.includes('does not exist')) {
    // Create user first
    await db.add('user_123', { profile: { name: 'Alice' } });
  } else {
    throw error;
  }
}
```

### 4. Batch Related Operations

```javascript
// ❌ BAD - Many small operations
for (let i = 0; i < 100; i++) {
  await db.add(`item_${i}`, { value: i });
}

// ✅ BETTER - Batch preparation
const items = {};
for (let i = 0; i < 100; i++) {
  items[`item_${i}`] = { value: i };
}
// Then write once (if your use case allows)
```

### 5. Check Existence Before Updates

```javascript
// ❌ BAD - Might throw error
await db.setPath('user_123', 'profile.name', 'Alice');

// ✅ GOOD - Check first
if (await db.has('user_123')) {
  await db.setPath('user_123', 'profile.name', 'Alice');
} else {
  await db.add('user_123', { profile: { name: 'Alice' } });
}
```

### 6. Use Meaningful Keys

```javascript
// ❌ BAD - Unclear naming
await db.add('123', userData);

// ✅ GOOD - Descriptive keys
await db.add('user_123', userData);
await db.add('order_2024_001', orderData);
await db.add('config_app_settings', configData);
```

### 7. Store Large Numbers as Strings

```javascript
// ❌ BAD - Precision loss for large numbers
await db.add('balance', 999999999999999999);

// ✅ GOOD - Store as string
await db.add('balance', '999999999999999999');
```

---

## 🔧 Troubleshooting

### Common Issues

#### Issue: "Database validation failed"

```javascript
// Cause: Corrupted JSON file
// Solution: Delete file and reinitialize, or restore from backup

import fs from 'fs';
if (fs.existsSync('data.json.backup')) {
  fs.copyFileSync('data.json.backup', 'data.json');
}
```

#### Issue: Operations returning undefined

```javascript
// ❌ Problem: Missing await
const user = db.get('user_123');
console.log(user.name); // undefined - user is a Promise!

// ✅ Solution: Use await
const user = await db.get('user_123');
console.log(user.name); // Works!
```

#### Issue: "Key does not exist" error

```javascript
// ❌ Problem: Trying to update non-existent key
await db.setPath('user_999', 'name', 'Alice'); // Error!

// ✅ Solution: Check existence first
if (await db.has('user_999')) {
  await db.setPath('user_999', 'name', 'Alice');
} else {
  await db.add('user_999', { name: 'Alice' });
}
```

#### Issue: Slow performance

```javascript
// Possible causes:
// 1. File too large (>50MB)
// 2. Too many operations per second
// 3. Disk I/O bottleneck

// Solutions:
// - Split data into multiple databases
// - Increase rate limits
// - Use SSD instead of HDD
// - Consider a real database for large datasets
```

---

## 📝 Additional Resources

### Running Tests

```bash
npm test
```

### Running Examples

```bash
# Demo showcase
npm run demo

# Performance benchmark
npm run benchmark
```

### Project Structure

```
jsondatabase/
├── src/
│   ├── index.js          # Main entry point
│   ├── worker-pool.js    # 50 Worker thread pool
│   ├── worker.js         # Worker implementation
│   ├── scheduler.js      # Round-robin scheduler
│   ├── file-manager.js   # File I/O manager
│   └── utils/
│       └── logger.js     # Logging utility
├── examples/
│   ├── demo.js           # Feature demo
│   └── benchmark.js      # Performance tests
├── test/
│   └── test.js           # Test suite
├── package.json
└── README.md
```

### Contributing

Contributions welcome! Please:
1. Fork the repository
2. Create a feature branch
3. Add tests for new features
4. Submit a pull request

### License

MIT License - See LICENSE file

### Support

- GitHub Issues: Report bugs and request features
- Email: support@ernesttech.com

---

## 🙏 Acknowledgments

Built with ❤️ for indie developers who need something simple, reliable, and just right for small projects.

**Happy coding! 💻**

---

*Built by Pease Ernest / Ernest Tech House Co-operation*