npm package: @hauska-sdk/retrieval
Description: CNS Protocol Retrieval SDK - IPFS document retrieval with gated access
Author: Hauska SDK Team
Latest version: 0.1.0

# @hauska-sdk/retrieval

CNS Protocol Retrieval SDK - IPFS document retrieval with gated access, encryption, watermarking, and access logging.

## Installation

```bash
npm install @hauska-sdk/retrieval
```

## Quick Start

```typescript
import { RetrievalSDK } from "@hauska-sdk/retrieval";
import { VDASDK } from "@hauska-sdk/vda";
import { PostgreSQLStorageAdapter } from "@hauska-sdk/adapters-storage-postgres";
import { WalletManager } from "@hauska-sdk/wallet";
import { Pool } from "pg";

// Initialize VDA SDK (required for access verification)
const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
});

const vdaSDK = new VDASDK({
  storageAdapter: new PostgreSQLStorageAdapter({
    pool,
    autoMigrate: true,
  }),
  walletManager: new WalletManager(),
});

// Initialize Retrieval SDK
const retrievalSDK = new RetrievalSDK({
  pinata: {
    pinataJwt: process.env.PINATA_JWT!,
    pinataGateway: "https://gateway.pinata.cloud",
  },
  vdaSdk: vdaSDK,
  logHook: (level, message, data) => {
    console.log(`[${level}] ${message}`, data);
  },
});

// Upload a document
const file = Buffer.from("Document content");
const uploadResult = await retrievalSDK.uploadDocument(file, {
  encrypt: true,
  vdaId: "vda-123",
  metadata: {
    name: "property-deed.pdf",
    keyvalues: {
      propertyId: "prop-123",
    },
  },
});

console.log(`Document uploaded: ${uploadResult.cid}`);

// Retrieve a document with access verification
const document = await retrievalSDK.retrieveDocument(
  uploadResult.cid,
  "0x1234567890123456789012345678901234567890",
  {
    decrypt: true,
    watermark: true, // Watermark for access pass viewers
  }
);

console.log(`Document retrieved: ${document.size} bytes`);
```

## Features

- ✅ **IPFS Document Storage** - Upload and retrieve documents via Pinata
- ✅ **Encryption/Decryption** - AES-256-GCM encryption for secure document storage
- ✅ **Gated Access Control** - VDA-based access verification before document retrieval
- ✅ **Permission Enforcement** - Fine-grained permissions (view, download, write, annotate)
- ✅ **PDF Watermarking** - Automatic watermarking for access pass viewers
- ✅ **Access Logging** - Track document access for audit trails and analytics
- ✅ **Gateway Proxy** - Express/Fastify middleware for HTTP document access
- ✅ **Retry Logic** - Automatic retry for network failures
- ✅ **Logging Hooks** - Optional monitoring and debugging support

## Gated Access Flow

The Retrieval SDK enforces access control using VDAs (Verified Digital Assets). Here's how it works:

```
1. Document Upload
   └─> Upload to IPFS (Pinata)
   └─> Store CID in VDA metadata (ipfsCid)

2. Document Retrieval Request
   └─> Verify VDA Access
       ├─> Check direct ownership
       ├─> Check access passes
       ├─> Verify permissions
       └─> Check expiry/revocation

3. Access Granted
   └─> Fetch from IPFS
   └─> Decrypt (if encrypted)
   └─> Apply watermark (if access via access pass)
   └─> Log access
   └─> Return document

4. Access Denied
   └─> Return 402 (Payment Required) or 403 (Forbidden)
```

### Access Types

- **Direct Ownership** - Wallet owns the VDA directly
- **Access Pass** - Wallet has a valid, non-expired, non-revoked access pass
- **No Access** - Returns 402 Payment Required

### Permission Levels

- `view` - Can view the document
- `download` - Can download the document
- `write` - Can modify the document
- `annotate` - Can add annotations

## API Reference

### RetrievalSDK

Main SDK class that provides all document retrieval functionality.

#### Constructor

```typescript
new RetrievalSDK(config: RetrievalSDKConfig)
```

**Configuration:**

```typescript
interface RetrievalSDKConfig {
  pinata: PinataConfig;                    // Required: Pinata configuration
  vdaSdk: VDASDK;                          // Required: VDA SDK instance
  pinataClient?: PinataClient;             // Optional: Custom Pinata client
  accessVerification?: VDAAccessVerificationService; // Optional: Custom access verification
  accessLogging?: AccessLoggingConfig;     // Optional: Access logging configuration
  logHook?: LogHook;                       // Optional: Logging hook
  getEncryptionKey?: (cid: string, wallet: string) => Promise<Buffer | undefined>; // Optional: Encryption key provider
  getVDAIdFromCID?: (cid: string) => Promise<string | null>; // Optional: Custom VDA ID lookup
}
```

#### Methods

##### `uploadDocument(file: Buffer, options?: UploadDocumentOptions): Promise<UploadDocumentResult>`

Upload a document to IPFS via Pinata.

```typescript
const result = await retrievalSDK.uploadDocument(file, {
  encrypt: true,                    // Encrypt before upload
  encryptionKey: key,               // Optional: Custom encryption key
  pin: true,                         // Pin to Pinata
  metadata: {
    name: "document.pdf",
    keyvalues: {
      vdaId: "vda-123",
      spoke: "real-estate",
    },
  },
  vdaId: "vda-123",                  // Associate with VDA
  spoke: "real-estate",
});

console.log(`CID: ${result.cid}`);
console.log(`Encrypted: ${result.encrypted}`);
if (result.encryptionKey) {
  // Store encryption key securely
  await storeEncryptionKey(result.cid, result.encryptionKey);
}
```

##### `fetchDocument(cid: string, options?: FetchDocumentOptions): Promise<FetchDocumentResult>`

Fetch a document from IPFS via Pinata gateway.

```typescript
const document = await retrievalSDK.fetchDocument("QmTest123", {
  decrypt: true,
  encryptionKey: await getEncryptionKey("QmTest123"),
});

console.log(`Size: ${document.size} bytes`);
console.log(`Content Type: ${document.contentType}`);
console.log(`Decrypted: ${document.decrypted}`);
```

##### `verifyAccess(cid: string, wallet: string, requiredPermissions?: PermissionType[]): Promise<VDAAccessVerificationResult>`

Verify if a wallet has access to a document.

```typescript
const result = await retrievalSDK.verifyAccess(
  "QmTest123",
  "0x1234567890123456789012345678901234567890",
  ["view", "download"]
);

if (result.hasAccess) {
  console.log(`Access type: ${result.accessType}`); // "direct" or "access-pass"
  console.log(`VDA ID: ${result.vdaId}`);
} else {
  console.log(`Access denied: ${result.reason}`);
}
```

##### `retrieveDocument(cid: string, wallet: string, options?: RetrieveDocumentOptions): Promise<FetchDocumentResult>`

Retrieve a document with full access verification, optional watermarking, and access logging.

```typescript
const document = await retrievalSDK.retrieveDocument(
  "QmTest123",
  "0x1234567890123456789012345678901234567890",
  {
    requiredPermissions: ["view"],
    decrypt: true,
    watermark: true,                 // Watermark for access pass viewers
    operation: "view",
  }
);
```

**Options:**

```typescript
interface RetrieveDocumentOptions {
  requiredPermissions?: PermissionType[];  // Default: ["view"]
  decrypt?: boolean;                       // Default: false
  watermark?: boolean;                     // Default: false (only for access pass viewers)
  operation?: string;                     // Operation type for logging
}
```

##### `getAccessLogs(cid: string, options?: { limit?: number; offset?: number }): Promise<AccessLogQueryResult>`

Get access logs for a document (requires access logging configuration).

```typescript
const logs = await retrievalSDK.getAccessLogs("QmTest123", {
  limit: 50,
  offset: 0,
});

console.log(`Total accesses: ${logs.total}`);
logs.logs.forEach((log) => {
  console.log(`${log.wallet} accessed at ${new Date(log.timestamp)}`);
});
```

##### `getAccessLogsByWallet(wallet: string, options?: { limit?: number; offset?: number }): Promise<AccessLogQueryResult>`

Get access logs for a wallet.

```typescript
const logs = await retrievalSDK.getAccessLogsByWallet(
  "0x1234567890123456789012345678901234567890",
  { limit: 100 }
);
```

## Pinata Integration

The Retrieval SDK uses [Pinata](https://pinata.cloud) for IPFS storage and retrieval.

### Setup

1. **Create a Pinata account** at https://pinata.cloud
2. **Generate a JWT token** in the Pinata dashboard
3. **Configure the SDK** with your JWT:

```typescript
const retrievalSDK = new RetrievalSDK({
  pinata: {
    pinataJwt: process.env.PINATA_JWT!,
    pinataGateway: "https://gateway.pinata.cloud", // Optional: Custom gateway
  },
  vdaSdk: vdaSDK,
});
```

### Pinata Features Used

- **File Upload** - `pinFileToIPFS()` for document storage
- **File Retrieval** - Gateway for document fetching
- **Metadata** - Store VDA ID and document metadata
- **Pinning** - Ensure documents remain available

### Custom Gateway

You can use a custom Pinata gateway or public IPFS gateway:

```typescript
const retrievalSDK = new RetrievalSDK({
  pinata: {
    pinataJwt: process.env.PINATA_JWT!,
    pinataGateway: "https://ipfs.io", // Public gateway
  },
  vdaSdk: vdaSDK,
});
```

## Code Examples

### Complete Document Upload Flow

```typescript
import { RetrievalSDK } from "@hauska-sdk/retrieval";
import { VDASDK } from "@hauska-sdk/vda";
import fs from "fs";

// Initialize SDKs
const vdaSDK = new VDASDK({ /* ... */ });
const retrievalSDK = new RetrievalSDK({
  pinata: { pinataJwt: process.env.PINATA_JWT! },
  vdaSdk: vdaSDK,
});

// 1. Mint a VDA for the document
const vda = await vdaSDK.mint({
  assetType: "deed",
  address: "123 Main St, Schertz, TX 78154",
  ownerWallet: ownerWallet,
  spoke: "real-estate",
});

// 2. Read document file
const file = fs.readFileSync("property-deed.pdf");

// 3. Upload document to IPFS
const uploadResult = await retrievalSDK.uploadDocument(file, {
  encrypt: true,                    // Encrypt for security
  vdaId: vda.id,                    // Associate with VDA
  metadata: {
    name: "property-deed.pdf",
    keyvalues: {
      propertyId: "prop-123",
      documentType: "deed",
    },
  },
  spoke: "real-estate",
});

// 4. Update VDA with IPFS CID
await vdaSDK.mint({
  ...vda.metadata,
  ipfsCid: uploadResult.cid,
});

// 5. Store encryption key securely
await storeEncryptionKey(uploadResult.cid, uploadResult.encryptionKey!);

console.log(`Document uploaded: ${uploadResult.cid}`);
```

### Gated Document Retrieval

```typescript
// Retrieve document with access verification
const document = await retrievalSDK.retrieveDocument(
  "QmTest123",
  userWallet,
  {
    requiredPermissions: ["view"],
    decrypt: true,
    watermark: true,                 // Watermark for access pass viewers
  }
);

// Save document
fs.writeFileSync("retrieved-document.pdf", document.content);
```

### Express Gateway Proxy

```typescript
import express from "express";
import { createGatewayRoute } from "@hauska-sdk/retrieval";
import { RetrievalSDK } from "@hauska-sdk/retrieval";

const app = express();
const retrievalSDK = new RetrievalSDK({ /* ... */ });

// Create gated document route
const documentRoute = createGatewayRoute(
  {
    accessVerificationService: retrievalSDK.getAccessVerification(),
    pinataClient: retrievalSDK.getPinataClient(),
    extractWallet: (req) => req.headers["x-wallet-address"] as string,
    getEncryptionKey: async (cid, wallet) => {
      return await getEncryptionKey(cid);
    },
  },
  {
    requiredPermissions: ["view"],
    autoDecrypt: true,
  }
);

app.use("/documents", documentRoute);

// Access document: GET /documents/:cid
// Header: x-wallet-address: 0x...
```

### Access Logging

```typescript
import { AccessLoggingService } from "@hauska-sdk/retrieval";
import { PostgreSQLStorageAdapter } from "@hauska-sdk/adapters-storage-postgres";

// Configure access logging
const retrievalSDK = new RetrievalSDK({
  pinata: { pinataJwt: process.env.PINATA_JWT! },
  vdaSdk: vdaSDK,
  accessLogging: {
    storageAdapter: new PostgreSQLAccessLogAdapter({ /* ... */ }),
    analyticsService: {
      trackAccess: async (log) => {
        // Send to analytics service
        await analytics.track("document_access", log);
      },
    },
  },
});

// Access logs are automatically recorded when using retrieveDocument()
const document = await retrievalSDK.retrieveDocument(cid, wallet);

// Query access logs
const logs = await retrievalSDK.getAccessLogs(cid);
console.log(`Document accessed ${logs.total} times`);
```

### Watermarking

```typescript
// Watermarking is automatic for access pass viewers
const document = await retrievalSDK.retrieveDocument(
  cid,
  wallet,
  {
    watermark: true, // Only watermarks for access pass viewers, not owners
  }
);

// Watermark includes:
// - Viewer wallet address
// - Timestamp
// - Access pass ID
```

## Encryption

The Retrieval SDK supports AES-256-GCM encryption for secure document storage.

### Encrypting Documents

```typescript
// Upload with encryption
const result = await retrievalSDK.uploadDocument(file, {
  encrypt: true,                    // Auto-generates encryption key
});

// Or provide your own key
const key = generateEncryptionKey();
const result = await retrievalSDK.uploadDocument(file, {
  encrypt: true,
  encryptionKey: key,
});

// Store encryption key securely
await storeEncryptionKey(result.cid, result.encryptionKey || key);
```

### Decrypting Documents

```typescript
// Provide encryption key for decryption
const document = await retrievalSDK.fetchDocument(cid, {
  decrypt: true,
  encryptionKey: await getEncryptionKey(cid),
});
```

### Encryption Key Management

```typescript
// Store encryption key (example)
async function storeEncryptionKey(cid: string, key: Buffer): Promise<void> {
  // Store in secure key management service
  await keyManagementService.store(cid, key.toString("hex"));
}

// Retrieve encryption key
async function getEncryptionKey(cid: string): Promise<Buffer | undefined> {
  const keyHex = await keyManagementService.get(cid);
  return keyHex ? Buffer.from(keyHex, "hex") : undefined;
}

// Use with SDK
const retrievalSDK = new RetrievalSDK({
  pinata: { pinataJwt: process.env.PINATA_JWT! },
  vdaSdk: vdaSDK,
  getEncryptionKey: async (cid, wallet) => {
    // Only return key if wallet has access
    const hasAccess = await retrievalSDK.verifyAccess(cid, wallet);
    if (hasAccess.hasAccess) {
      return await getEncryptionKey(cid);
    }
    return undefined;
  },
});
```

## Gateway Proxy

The Retrieval SDK provides Express/Fastify middleware for HTTP document access.

### Basic Setup

```typescript
import express from "express";
import { createGatewayRoute } from "@hauska-sdk/retrieval";

const app = express();
const retrievalSDK = new RetrievalSDK({ /* ... */ });

// Create gated document route
app.use(
  "/documents",
  ...createGatewayRoute(
    {
      accessVerificationService: retrievalSDK.getAccessVerification(),
      pinataClient: retrievalSDK.getPinataClient(),
      extractWallet: (req) => req.headers["x-wallet-address"] as string,
    },
    {
      requiredPermissions: ["view"],
    }
  )
);

// Access: GET /documents/:cid
// Header: x-wallet-address: 0x...
```

### Custom Wallet Extraction

```typescript
import { createGatewayRoute, defaultExtractWallet } from "@hauska-sdk/retrieval";

// Extract wallet from JWT token
function extractWalletFromJWT(req: Request): string | undefined {
  const token = req.headers.authorization?.replace("Bearer ", "");
  if (token) {
    const decoded = jwt.verify(token, process.env.JWT_SECRET!);
    return decoded.wallet;
  }
  return undefined;
}

app.use(
  "/documents",
  ...createGatewayRoute(
    {
      accessVerificationService: retrievalSDK.getAccessVerification(),
      pinataClient: retrievalSDK.getPinataClient(),
      extractWallet: extractWalletFromJWT,
    },
    { requiredPermissions: ["view"] }
  )
);
```

### Operation-Specific Permissions

```typescript
import { createOperationMiddleware } from "@hauska-sdk/retrieval";

// View endpoint (requires "view" permission)
app.get(
  "/documents/:cid/view",
  createOperationMiddleware(config, "view"),
  createDocumentRoute(config)
);

// Download endpoint (requires "download" permission)
app.get(
  "/documents/:cid/download",
  createOperationMiddleware(config, "download"),
  createDocumentRoute(config)
);
```

## Troubleshooting

### "Document not found" Error

**Problem:** CID doesn't exist on IPFS or Pinata gateway.

**Solutions:**
1. Verify the CID is correct
2. Check if document was pinned to Pinata
3. Verify Pinata gateway is accessible
4. Check network connectivity

```typescript
// Verify CID exists
try {
  const metadata = await pinataClient.getFileMetadata(cid);
  if (!metadata) {
    console.error("Document not found in Pinata");
  }
} catch (error) {
  console.error("Error checking document:", error);
}
```

### "Access denied" Error

**Problem:** Wallet doesn't have access to the document.

**Solutions:**
1. Verify wallet owns the VDA or has a valid access pass
2. Check access pass hasn't expired
3. Check access pass hasn't been revoked
4. Verify required permissions are granted

```typescript
// Debug access verification
const result = await retrievalSDK.verifyAccess(cid, wallet);
console.log("Access result:", {
  hasAccess: result.hasAccess,
  accessType: result.accessType,
  reason: result.reason,
  vdaId: result.vdaId,
});
```

### "Encryption key required" Error

**Problem:** Document is encrypted but no encryption key provided.

**Solutions:**
1. Provide encryption key in fetch options
2. Configure `getEncryptionKey` in SDK config
3. Verify encryption key is correct

```typescript
// Provide encryption key
const document = await retrievalSDK.fetchDocument(cid, {
  decrypt: true,
  encryptionKey: await getEncryptionKey(cid),
});
```

### "VDA ID not found for CID" Error

**Problem:** VDA ID cannot be resolved from CID.

**Solutions:**
1. Ensure VDA has `ipfsCid` metadata set
2. Provide `getVDAIdFromCID` function in config
3. Verify Pinata metadata includes `vdaId` keyvalue

```typescript
// Custom VDA ID lookup
const retrievalSDK = new RetrievalSDK({
  pinata: { pinataJwt: process.env.PINATA_JWT! },
  vdaSdk: vdaSDK,
  getVDAIdFromCID: async (cid) => {
    // Query database for VDA with this CID
    const vda = await db.query("SELECT id FROM vdas WHERE ipfs_cid = $1", [cid]);
    return vda?.id || null;
  },
});
```

### Network/Retry Errors

**Problem:** Network failures when fetching from IPFS gateway.

**Solutions:**
1. SDK automatically retries (3 attempts with exponential backoff)
2. Check network connectivity
3. Try different gateway
4. Verify Pinata service status

```typescript
// Use custom gateway with retry
const retrievalSDK = new RetrievalSDK({
  pinata: {
    pinataJwt: process.env.PINATA_JWT!,
    pinataGateway: "https://ipfs.io", // Public gateway fallback
  },
  vdaSdk: vdaSDK,
});
```

### Watermarking Not Working

**Problem:** Watermarks not appearing on PDFs.

**Solutions:**
1. Verify document is a PDF (watermarking only works for PDFs)
2. Ensure `watermark: true` is set in retrieveDocument options
3. Verify access is via access pass (owners don't get watermarked)
4. Check PDF is valid and not corrupted

```typescript
// Verify PDF before watermarking
import { isPDF } from "@hauska-sdk/retrieval";

const document = await retrievalSDK.fetchDocument(cid);
if (isPDF(document.content)) {
  // Watermarking will work
  const watermarked = await retrievalSDK.retrieveDocument(cid, wallet, {
    watermark: true,
  });
}
```

## Performance Tips

1. **Use encryption keys efficiently:**
   ```typescript
   // Cache encryption keys
   const keyCache = new Map<string, Buffer>();
   const getEncryptionKey = async (cid: string) => {
     if (keyCache.has(cid)) {
       return keyCache.get(cid);
     }
     const key = await fetchEncryptionKey(cid);
     keyCache.set(cid, key);
     return key;
   };
   ```

2. **Batch access log queries:**
   ```typescript
   // Query multiple CIDs at once
   const cids = ["Qm1", "Qm2", "Qm3"];
   const logs = await Promise.all(
     cids.map((cid) => retrievalSDK.getAccessLogs(cid))
   );
   ```

3. **Use pagination for large result sets:**
   ```typescript
   const logs = await retrievalSDK.getAccessLogs(cid, {
     limit: 50,
     offset: 0,
   });
   ```

## License

MIT