npm package: @nm-logger/logger
Description: Express JSON logger with daily success/error/external logs, S3 append uploads, and configurable baseDir.
Latest version: 1.2.11

# @nm-logger/logger

Minimal JSON logger for Express + Axios with optional S3 sync.

## Features

- Logs handled Express requests as **success**
- Logs Express errors as **error**
- Logs Axios calls as **external**
- Optional **database error** logs as `db_error`
- Local daily files under configurable `baseDir`:
  - `YYYY/MM/DD/daily_logs_success.json`
  - `YYYY/MM/DD/daily_logs_error.json`
  - `YYYY/MM/DD/daily_logs_external.json`
  - `YYYY/MM/DD/daily_logs_db_error.json`
- S3: one JSON file per day per category (append by read + merge + put)
- Previous day's local folder is removed when the date changes
- `employee_id` taken from `req.user.employee_id || req.user.emp_code || req.user.id` (used internally only, **not** written to JSON)
- Each log entry includes:
  - `url`, `method`, `type` (internal/external/db_error)
  - `status_code`
  - `user_agent`
  - `project_version` (from Logger options)
  - `error`
  - optional `response` (for external errors)
  - `count` (when consecutive identical entries are compacted)
- External API logging:
  - Logs **response body only for error-like calls** (4xx/5xx or when a custom `externalErrorDetector` marks it as error)
  - Long responses are truncated with `maxResponseLength`

## Install

```bash
npm install @nm-logger/logger
```

## Basic usage (Express + Axios)

```js
const express = require("express");
const axios = require("axios");
const Logger = require("@nm-logger/logger");

const app = express();

const nm_logger = new Logger(
  {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: "us-east-1",
    bucket: "project",
  },
  {
    baseDir: "/var/log/nm-logger",
    maskFields: ["aadhaar", "pan"],

    // Optional metadata for logs
    projectVersion: "Project-1.0.0",

    // Background S3 uploader (set to false if you prefer cron; see below)
    enableDailyWatcher: true,

    // Interval for background S3 uploads when enableDailyWatcher = true
    uploadIntervalMs: 60 * 1000, // 1 minute

    // How many characters of external response to keep (after masking)
    maxResponseLength: 4000,

    // If true (default), include external response body only for error-like calls
    logExternalResponseOnError: true,

    // Control logging of successful external calls
    // - true  (default): log all external calls
    // - false: log only error-like calls (4xx/5xx or externalErrorDetector = true)
    logExternalSuccess: true,

    // Optional semantic error detector for 2xx/3xx external responses
    // Example below in "External error-only logging"
    externalErrorDetector: null,
  }
);

// Express middlewares
app.use(express.json());
app.use(nm_logger.requestLoggerMiddleware());
app.use("/api", require("./routes"));
app.use(nm_logger.expressMiddleware());

// Attach Axios interceptor
nm_logger.attachAxiosLogger(axios);

app.listen(3000, () => {
  console.log("Server listening on 3000");
});
```

### Folder permissions

Make sure the base directory exists and is writable by your Node process:

```bash
mkdir -p /var/log/nm-logger
chmod -R 777 /var/log/nm-logger
```

## External error-only logging (recommended for high load)

If your external API returns HTTP 200 but the **body** contains error info, you can:

- Skip logging successful calls completely
- Still log error-like responses with full (masked + truncated) body

Example:

```js
const nm_logger = new Logger(
  {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: "us-east-1",
    bucket: "project",
  },
  {
    baseDir: "/var/log/nm-logger",
    maskFields: ["aadhaar", "pan"],

    enableDailyWatcher: false, // use cron for S3 (see next section)
    logExternalSuccess: false, // do NOT log success responses

    externalErrorDetector: ({ response }) => {
      try {
        // Adjust this according to your external API response format
        if (response && response.success === false) return true;
        if (response && typeof response.status === "string" &&
            response.status.toLowerCase() === "error") return true;
        if (response && (response.error || response.error_code)) return true;
      } catch (e) {
        return false;
      }
      return false;
    },
  }
);
```

With this config:

- Only external calls that look like **errors** are logged (4xx/5xx or `externalErrorDetector` = true).
- For these, the masked + truncated `response` body is stored in JSON.

## S3 upload modes

There are two ways to sync daily logs to S3.

### Mode 1: Background timer (default)

If `enableDailyWatcher: true` (or omitted), the logger will:

- Start an internal `setInterval` in the same Node process
- Every `uploadIntervalMs`:
  - Read local daily log JSONs
  - Merge them with S3 files
  - Upload merged logs back to S3
  - Clean up previous day's local folder

This is simple but adds some extra work inside your API process.

### Mode 2: Cron-based upload (recommended for production)

To keep your API process as light as possible, you can:

1. **Disable the internal watcher in your API**
2. **Use a separate cron script** to upload logs to S3

#### 1. API server: disable watcher

```js
const nm_logger = new Logger(
  {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: "us-east-1",
    bucket: "project",
  },
  {
    baseDir: "/var/log/nm-logger",
    maskFields: ["aadhaar", "pan"],
    enableDailyWatcher: false,  // 🔴 no setInterval in API process
  }
);

app.use(nm_logger.requestLoggerMiddleware());
app.use("/api", routes);
app.use(nm_logger.expressMiddleware());
nm_logger.attachAxiosLogger(axios);
```

This way, the API only writes local JSON files.

### Mode 2b: node-cron inside Node (no system crontab)

If you are already using [`node-cron`](https://www.npmjs.com/package/node-cron) in your Node project, you can avoid Linux `crontab` entirely and schedule uploads from your app code.

1. Disable the internal watcher in your API (same as before):

```js
const nm_logger = new Logger(
  {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: "us-east-1",
    bucket: "project",
  },
  {
    baseDir: "/var/log/nm-logger",
    maskFields: ["aadhaar", "pan"],
    enableDailyWatcher: false,  // no setInterval
  }
);
```

2. Use `node-cron` in your Node app to trigger uploads:

```js
const cron = require("node-cron");
const Logger = require("@nm-logger/logger");

const nm_logger = new Logger(
  {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: "us-east-1",
    bucket: "project",
  },
  {
    baseDir: "/var/log/nm-logger",
    maskFields: ["aadhaar", "pan"],
    enableDailyWatcher: false,
  }
);

// Every 5 minutes
cron.schedule("*/5 * * * *", async () => {
  try {
    await nm_logger.uploadDailyLogsOnce();
    console.log("[nm-logger] ✅ Daily logs uploaded to S3 (node-cron)");
  } catch (err) {
    console.error("[nm-logger] ❌ Upload failed (node-cron):", err);
  }
});
```

This way you:
- Do **not** depend on OS-level `crontab`.
- Still avoid `setInterval`-based uploads inside the main request flow.
- Keep all scheduling logic inside Node where you are already comfortable.

#### 3. System Crontab entry (optional, Linux)

If you prefer OS-level cron instead of `node-cron`, you can still use a separate script as before.

Create a separate script (for example in your backend root):

```js
// upload-logs.js
require("dotenv").config();
const Logger = require("@nm-logger/logger");

async function main() {
  const logger = new Logger(
    {
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
      region: "us-east-1",
      bucket: "project",
    },
    {
      baseDir: "/var/log/nm-logger",
      maskFields: ["aadhaar", "pan"],
      enableDailyWatcher: false, // no internal timers here either
    }
  );

  try {
    await logger.uploadDailyLogsOnce();
    console.log("[nm-logger] ✅ Daily logs uploaded to S3");
  } catch (err) {
    console.error("[nm-logger] ❌ Failed to upload daily logs:", err);
    process.exitCode = 1;
  }
}

main();
```

On your server (Linux):

```bash
crontab -e
```

Add a line like:

```cron
*/5 * * * * /usr/bin/node /path/to/upload-logs.js >> /var/log/nm-logger-cron.log 2>&1
```

- This will **not** be created automatically by the package.
- You must add it manually using `crontab -e`.
- Make sure the `node` path (`/usr/bin/node`) and script path (`/path/to/upload-logs.js`) are correct for your server.

## Notes

- The logger does **not** create or manage cron jobs automatically; you must configure `node-cron` or `crontab` yourself if you choose cron mode.
- For lower CPU and disk usage on busy servers:
  - Prefer `logExternalSuccess: false` + `externalErrorDetector` to log only problematic external calls.
  - Use cron-based upload (Mode 2 / Mode 2b) so S3 sync runs in a separate short-lived context.