npm package: @jemikalajah/scriba
Description: Hybrid logging library for NodeJS services with request/response interception and S3 storage
Author: Philip Jemikalajah
Latest version: 1.0.2

# Scriba

A hybrid logging library for NodeJS services with request/response interception and S3 storage. Built following the recommended approach of using internal libraries for request/response interception and dedicated workers for S3 storage.

## 🚀 Features

- **Request/Response Interception**: Automatically captures HTTP requests and responses
- **Structured Logging**: JSON-based log events with trace IDs and correlation
- **Multiple Transport Support**: Console, Kafka, SQS, Redis
- **S3 Storage**: Dedicated worker for batch uploading to S3
- **Express Middleware**: Easy integration with Express.js applications
- **Trace Management**: AsyncLocalStorage-based context management
- **Batch Processing**: Efficient batching for queue-based transports
- **Error Handling**: Comprehensive error logging and handling

## 📦 Installation

```bash
npm install @jemikalajah/scriba
```

## 🏗️ Architecture

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Your Service  │───▶│  Scriba Lib     │───▶│   Queue (SQS/   │
│                 │    │                 │    │    Kafka)       │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                       │
                                                       ▼
                                              ┌─────────────────┐
                                              │  S3 Worker      │
                                              │                 │
                                              └─────────────────┘
                                                       │
                                                       ▼
                                              ┌─────────────────┐
                                              │   S3 Storage    │
                                              │                 │
                                              └─────────────────┘
```

## 🚀 Quick Start

### Installation

```bash
npm install @jemikalajah/scriba
```

### Basic Usage

```typescript
import { createLogger } from 'scriba';

const logger = createLogger('my-service');

// Basic logging
await logger.info('User logged in', { userId: '123' });
await logger.error('Database connection failed', { error: 'timeout' });
```

## ⚙️ Production Configuration

Scriba supports multiple deployment configurations. Choose the setup that fits your infrastructure:

### Option 1: SQS + S3 (Recommended for AWS)

```bash
# Environment variables
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
S3_BUCKET=your-logs-bucket
SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/your-queue
ENABLE_QUEUE=true
QUEUE_TYPE=sqs
```

### Option 2: Kafka + S3 (For high-throughput scenarios)

```bash
# Environment variables
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
S3_BUCKET=your-logs-bucket
KAFKA_BROKERS=localhost:9092,kafka2:9092
KAFKA_TOPIC=application-logs
ENABLE_QUEUE=true
QUEUE_TYPE=kafka
```

### Option 3: Console Only (Development/Testing)

```bash
# Environment variables
ENABLE_CONSOLE=true
ENABLE_QUEUE=false
LOG_LEVEL=debug
```

### Configuration Files

Copy the appropriate example configuration:

```bash
# For SQS + S3
cp examples/env.sqs.example .env

# For Kafka + S3  
cp examples/env.kafka.example .env

# For console only
cp examples/env.console-only.example .env
```

### Express.js Integration

```typescript
import express from 'express';
import { createLogger, createExpressInterceptor } from 'scriba';

const app = express();
const logger = createLogger('my-api-service');
const interceptor = createExpressInterceptor(logger);

// Add middleware
app.use(interceptor.middleware());
app.use(interceptor.errorHandler());

// Your routes...
app.get('/users/:id', async (req, res) => {
  await logger.info('User profile accessed', { userId: req.params.id });
  res.json({ user: 'data' });
});
```

## ⚙️ Configuration

### Logger Configuration

```typescript
import { LogConfig } from 'scriba';

const config: Partial<LogConfig> = {
  environment: 'production',
  logLevel: 'info',
  enableConsole: true,
  enableQueue: true,
  queueConfig: {
    type: 'sqs',
    batchSize: 100,
    batchTimeout: 30000,
  },
  sqsConfig: {
    region: 'us-east-1',
    queueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/scriba-logs',
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },
};

const logger = createLogger('my-service', config);
```

### S3 Worker Configuration

```bash
# Environment variables for S3 worker
S3_BUCKET=scriba-logs
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
S3_PREFIX=logs
S3_COMPRESSION=true
S3_RETENTION_DAYS=30
```

### Custom S3 Path and File Naming

Scriba supports flexible S3 path and file naming configurations:

```typescript
const s3Config: S3Config = {
  bucket: 'scriba-logs',
  region: 'us-east-1',
  prefix: 'logs',
  compression: true,
  
  // Custom path options
  customPath: 'my-app/logs', // Simple custom path
  // OR
  customPath: (serviceName, timestamp) => 
    `apps/${serviceName}/logs/${moment(timestamp).format('YYYY-MM')}`, // Function-based path
  
  // Custom file naming
  customFileName: (timestamp, serviceName, eventCount) => 
    `${serviceName}-${moment(timestamp).format('YYYY-MM-DD-HH')}-${eventCount}events.jsonl.gz`,
  
  // Control path components
  includeDateInPath: false,    // Disable date in path
  includeHourInPath: false,    // Disable hour in path  
  includeServiceInPath: false, // Disable service name in path
};
```

#### Default Structure (if no custom config):
```
s3://bucket/prefix/service-name/YYYY/MM/DD/HH/timestamp.jsonl.gz
```

#### Example Custom Structures:
```
# Simple custom path
s3://bucket/my-app/logs/timestamp.jsonl.gz

# Function-based path
s3://bucket/apps/my-service/logs/2025-08/timestamp.jsonl.gz

# Custom file naming
s3://bucket/logs/my-service/2025-08-28-10-100events.jsonl.gz

# Flat structure
s3://bucket/production-logs/app-logs.jsonl.gz
```

# Queue configuration
QUEUE_TYPE=sqs
SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/scriba-logs
BATCH_SIZE=100
BATCH_TIMEOUT=30000
```

## 📋 API Reference

### Logger Class

#### Methods

- `debug(message: string, metadata?: LogMetadata): Promise<void>`
- `info(message: string, metadata?: LogMetadata): Promise<void>`
- `warn(message: string, metadata?: LogMetadata): Promise<void>`
- `error(message: string, metadata?: LogMetadata): Promise<void>`
- `logRequest(req: Request): Promise<void>`
- `logResponse(res: Response, responseTime: number): Promise<void>`
- `logError(error: Error, context?: Record<string, any>): Promise<void>`
- `batchLog(events: LogEvent[]): Promise<void>`
- `close(): Promise<void>`

### ExpressInterceptor Class

#### Methods

- `middleware(): RequestHandler`
- `errorHandler(): ErrorRequestHandler`

### Trace Management

```typescript
import { 
  getTraceId, 
  getRequestId, 
  getUserId, 
  setLogMetadata 
} from 'scriba';

// Get current context
const traceId = getTraceId();
const requestId = getRequestId();
const userId = getUserId();

// Set additional metadata
setLogMetadata({ 
  operation: 'user.create',
  source: 'api' 
});
```

## 🔧 Running the S3 Worker

### Development

```bash
npm run worker:dev
```

### Production

```bash
npm run worker:start
```

### Docker

```dockerfile
FROM node:18-alpine

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

COPY dist ./dist

CMD ["npm", "run", "worker:start"]
```

## 📊 Log Event Structure

```typescript
interface LogEvent {
  traceId: string;
  serviceName: string;
  timestamp: string;
  level: 'info' | 'error' | 'warn' | 'debug';
  message: string;
  metadata?: Record<string, any>;
  requestId?: string;
  userId?: string;
  correlationId?: string;
}
```

### Request Log Event

```typescript
interface RequestLogEvent extends LogEvent {
  type: 'request';
  method: string;
  url: string;
  headers?: Record<string, string>;
  body?: any;
  query?: Record<string, any>;
  params?: Record<string, any>;
  ip?: string;
  userAgent?: string;
}
```

### Response Log Event

```typescript
interface ResponseLogEvent extends LogEvent {
  type: 'response';
  statusCode: number;
  statusMessage: string;
  headers?: Record<string, string>;
  body?: any;
  responseTime: number;
  requestId: string;
}
```

## 🛠️ Development

### Setup

```bash
git clone <repository>
cd scriba
npm install
```

### Build

```bash
npm run build
```

### Test

```bash
npm test
```

### Demos

```bash
# Run Express demo
npm run demo:express
```

## 🔒 Security

- AWS credentials are loaded from environment variables
- SQS/Kafka connections use secure protocols
- Log data is sanitized before storage
- Trace IDs are generated using UUID v4

## 📈 Performance

- Async logging to prevent blocking
- Batch processing for queue transports
- Compression for S3 storage
- Connection pooling for database operations
- Memory-efficient streaming for large log files

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

## 📄 License

MIT License - see LICENSE file for details

## 🆘 Support

For support and questions:
- Create an issue on GitHub
- Check the documentation
- Review the demos

## 🔄 Changelog

### v1.0.0
- Initial release
- Express.js integration
- SQS and Kafka support
- S3 worker for batch processing
- Trace management with AsyncLocalStorage