npm package: @hamak/smart-data-dico
Description: Collaborative data dictionary management system — model, document, and govern your data landscape
Latest version: 1.1.1

# Smart Data Dictionary

A collaborative data dictionary management system for modeling, documenting, and governing your organization's data landscape. Built on a microkernel plugin architecture for modularity, with file-based persistence (YAML/Git) designed for both technical and non-technical users.

## Core Concepts

### Packages

Packages organize entities into logical groups reflecting your system architecture (microservices, modules, bounded contexts). They are hierarchical -- packages can contain sub-packages, forming a tree that mirrors your application structure.

- Navigable via nested URLs: `/packages/eShop/ordering/entities/Order`
- Each package has a dashboard with entity counts, relationship counts, and metadata
- Package names follow kebab-case convention

### Entities & Attributes

Entities are the primary modeling unit -- they represent data structures (tables, documents, events, value objects). Each entity has:

- **Attributes** with types (string, number, integer, boolean, datetime, enum, object, array), constraints (min/max length, pattern, precision), and metadata
- **Nested attributes** for object and array types (e.g. `shippingAddress.postalCode`)
- **Status lifecycle**: draft -> submitted -> approved (or returned)
- **UUID-based identity** for stable cross-references

### Relationships

Relationships connect entities across packages. They are stored at the package level and define:

- **Source and target** entity (by UUID) with cardinality (one/many)
- **Navigation property names** for path traversal
- **Metadata** on the relationship itself

### Stereotypes

Stereotypes define metadata schemas that apply to specific element types (entity, attribute, or package). When assigned, they auto-populate required and optional metadata fields, ensuring consistency.

**Predefined stereotypes:**

| Stereotype | Applies To | Required Fields |
|-----------|-----------|-----------------|
| Aggregate Root | Entity | bounded-context |
| Reference Data | Entity | cacheable (flag) |
| Domain Event | Entity | event-source |
| Value Object | Entity | immutable (flag) |
| PII | Attribute | pii-category |
| Indexed | Attribute | - |
| Deprecated | Attribute | deprecated-since |

**Metadata types**: string, number, boolean, date, **flag** (semantic boolean rendered as toggle), **rule** (text + severity: info/warning/error).

### Perspectives

A Perspective captures a business view over a subset of the data model scoped to a business process (e.g. "Ordering", "Billing"). It is the core differentiating concept of the application.

**How it works:**

1. Select **root entities** -- the starting points of the business process
2. The system performs **BFS traversal** through relationships to discover all reachable entities
3. Each entity is identified by its **relationship path from root** (e.g. `Order/orderItems/product`)
4. The same entity reached via different paths is treated as separate usages

**Path grammar** -- uniform `/` navigation for all levels:
```
Order                              -- root entity
Order/totalAmount                  -- attribute on root
Order/orderItems                   -- relationship -> OrderItem
Order/orderItems/quantity          -- attribute on OrderItem
Order/orderItems/product/sku       -- deep chain -> attribute
Order/shippingAddress/postalCode   -- attribute via relationship
Order/billingAddress/postalCode    -- same attribute, different path
```

**PerspectiveNode** annotations on any path:
- **`traverse: false`** -- frontier node (include but stop BFS here)
- **`exclude: true`** -- prune this path entirely
- **`metadata`** -- path-scoped annotations (not overrides -- new metadata specific to this perspective usage)

### Review Workflow

Entities go through a status lifecycle for team governance:

```
draft --> submitted --> approved
                   \-> returned (with comments) --> submitted
```

- **Comments** can target specific attributes (e.g. "Missing validation rule on discountRate")
- **Status transitions** are role-gated (editors submit, admins approve/return)
- Comments are stored as sidecar YAML files per entity

### Quality Metrics

The quality dashboard tracks documentation completeness:

- **Description coverage** -- % of entities/attributes with descriptions
- **Metadata compliance** -- % of entities with correct stereotype metadata
- **Relationship coverage** -- % of entities with at least one relationship
- **Per-entity scoring** -- weighted composite (30% description, 30% attribute descriptions, 20% relationships, 20% stereotype compliance)
- **Drill-down** from package level to individual entity gaps

## User Journeys

### Journey 1: Model a New Domain (Data Architect)

> *Alice documents the data model for an e-commerce platform.*

1. **Create packages** -- `eShop > Ordering`, `Catalog`, `Billing` via the package dashboard
2. **Define entities** -- `Order`, `OrderLine`, `Customer` with typed attributes and constraints
3. **Define relationships** -- `Order` to `OrderLine` (one-to-many) at the package level
4. **Assign stereotypes** -- Mark `Order` as `aggregate-root`, auto-populating `bounded-context` metadata
5. **Add metadata** -- Flag `Customer.email` as PII, add business rules
6. **Visualize** -- Open the Cytoscape.js interactive graph, verify the model, save the layout
7. **Save & publish** -- Save (commit to workspace), publish (push to shared)

### Journey 2: Define a Perspective (Business Analyst)

> *Bob captures the billing-specific view of the data model.*

1. **Create perspective** -- "Billing" with `Invoice` and `Payment` as root entities
2. **Review resolved paths** -- BFS walks relationships, discovers `Order`, `Customer`, `LineItem`
3. **Set frontiers** -- Mark `Product` as frontier (`traverse: false`) -- include but don't follow further
4. **Exclude paths** -- Exclude `Order/audit` (not relevant to billing)
5. **Add annotations** -- On `Order/billingAddress/postalCode`, add validation rule "Must match payment country"
6. **Visualize** -- Graph overlay highlights perspective members, dims others

### Journey 3: Review & Approve (Data Steward)

> *Carol reviews changes submitted by the team.*

1. **See pending entities** -- Order is in `submitted` status
2. **Add comments** -- "Missing description on id field" targeting specific attribute
3. **Return for revision** -- Status changes to `returned`, author gets notification
4. **Approve** -- After fixes, approve. Status moves to `approved`
5. **Check quality** -- Quality dashboard shows 90% for order-service, up from 80%

### Journey 4: Explore & Discover (Developer)

> *Dave needs to understand the data model for implementation.*

1. **Browse** -- Navigate `/packages/order-service` to see the dashboard with stats
2. **Search** -- Search "order" returns 27 results across entities, attributes, metadata, relationships, packages
3. **Filter** -- Faceted search: filter by type=relationship, service=order-service, stereotype=aggregate-root
4. **Impact analysis** -- Check "Impact" tab on Order: 2 relationships, 1 perspective reference
5. **Export** -- Export order-service as JSON Schema for code generation, or Markdown for documentation

### Journey 5: Version & Collaborate (Team Lead)

> *Eve manages versioning aligned with application releases.*

1. **Create workspace** -- Creates `workspace/billing-v2` (git branch with friendly name)
2. **Monitor status** -- Navbar shows "MAIN 8" (branch + unsaved count), dropdown shows ahead/behind
3. **Save & publish** -- Save page shows changed files, commit with message, push to remote
4. **Switch workspaces** -- Workspaces page lists branches, switch with one click
5. **Merge** -- Merge page selects source workspace, previews diff, merges

## Architecture

### Monorepo

```
smart-data-dico/
  backend/              Express + TypeScript (controllers > services > models)
  frontend/             React 18 + Vite + TypeScript + Tailwind CSS + DaisyUI
  data-dictionaries/    YAML entity files, relationships, perspectives, stereotypes
  Dockerfile            Multi-stage production build
  docker-compose.yml    One-command deployment
```

### Microkernel Plugin Architecture

The frontend uses `@hamak/app-framework` with 11 plugins:

| Plugin | Responsibility |
|--------|---------------|
| **store** | Redux state management (10 domain slices) |
| **shell** | Layout, theming (synced with DaisyUI) |
| **auth** | JWT authentication with Auth0 |
| **data-dictionary** | Entity, package, relationship CRUD |
| **visualization** | Interactive Cytoscape.js graph |
| **search** | Full-text and faceted search |
| **version-control** | Save/publish, history |
| **perspective** | Perspective management and resolution |
| **remote-fs** | File operations via `@hamak/filesystem-server-impl` |
| **remote-git** | Git operations via `@hamak/ui-remote-git-fs-impl` |
| **notification** | Toast notifications |

### Data Model

```
Package
  +-- Entity (uuid, name, description, stereotype, status, attributes[], metadata[])
  |     +-- Attribute (uuid, name, type, required, constraints, metadata[])
  +-- Relationship (uuid, source, target, cardinality, metadata[])
  +-- Sub-packages

Stereotype (id, name, appliesTo, metadataDefinitions[])
Perspective (uuid, name, rootEntities[], nodes[], maxDepth)
  +-- PerspectiveNode (path, traverse?, exclude?, metadata[])
  +-- ResolvedNode (entityUuid, path, hopDistance, isRoot, isFrontier)
```

### Persistence

```
data-dictionaries/
  microservices/{package}/
    {uuid}_{name}.yaml          -- entity files
    {uuid}.comments.yaml        -- review comments (sidecar)
    relationships.yaml          -- package-level relationships
    metadata.yaml               -- package metadata
  perspectives/{uuid}.yaml      -- perspective definitions
  stereotypes.yaml              -- stereotype definitions
  diagrams/{id}.json            -- saved diagram layouts
```

All files are git-tracked for versioning, branching, and collaboration.

### API Surface

| Area | Endpoints |
|------|-----------|
| Entities | CRUD on `/api/services/:service/entities/:entity` |
| Packages | CRUD on `/api/packages`, hierarchy, path navigation |
| Relationships | CRUD on `/api/packages/:pkg/relationships` |
| Stereotypes | CRUD on `/api/stereotypes` |
| Perspectives | CRUD + resolve + graph on `/api/perspectives` |
| Search | `GET /api/search?q=...&type=...&service=...&stereotype=...` |
| Impact | `GET /api/entities/:uuid/impact` |
| Review | Submit/approve/return + comments on entities |
| Import | `POST /api/import/json-schema`, `POST /api/import/sql-ddl` |
| Export | `GET /api/export/json-schema/:service`, `GET /api/export/markdown/:service` |
| Quality | `GET /api/quality/report` |
| Git | Full git operations via `/api/git/dictionaries/*` (framework routes) |
| Version | Commit, history, revert via `/api/commit`, `/api/history` |

## Getting Started

### Prerequisites

- Node.js v18+
- npm

### Quick Start

```bash
# Install dependencies
cd backend && npm install
cd ../frontend && npm install

# Start (two terminals)
cd backend && npm run dev     # port 3001
cd frontend && npm run dev    # port 3000
```

- Frontend: http://localhost:3000
- Backend API: http://localhost:3001
- API docs: http://localhost:3001/api-docs

### Docker

```bash
docker-compose up
# App available at http://localhost:3001
```

### Dev Credentials

| User | Password | Role |
|------|----------|------|
| admin | admin123 | ADMIN (full access) |
| editor | editor123 | EDITOR (create/edit) |
| viewer | viewer123 | VIEWER (read-only) |

### Configuration Profiles

Set via `PROFILE` environment variable:

| Profile | Auth | Git | Use Case |
|---------|------|-----|----------|
| **local** | None | Local only | Single user, desktop |
| **team** | Basic/JWT | Remote | Small team sharing a repo |
| **server** | Auth0/SSO | Remote | Organization-wide deployment |

## Technologies

| Layer | Stack |
|-------|-------|
| Backend | Node.js, Express, TypeScript (ESM), YAML, Git |
| Frontend | React 18, Vite, TypeScript, Tailwind CSS, DaisyUI, Redux |
| Framework | @hamak/app-framework (microkernel, DI, plugins) |
| Visualization | Cytoscape.js (dagre + fcose layouts) |
| Auth | JWT + Auth0 (mock mode for dev) |
| Deployment | Docker (multi-stage build) |