open-index/open-npm huggingface dataset:
npm package: @lde/pipeline-console-reporter
Description: Console progress reporter for @lde/pipeline
Latest version: 0.19.10

# Pipeline Console Reporter

Console progress reporter for [@lde/pipeline](../pipeline). Displays real-time pipeline progress with spinners, colours and timing information.

## Usage

```typescript
import { Pipeline } from '@lde/pipeline';
import { ConsoleReporter } from '@lde/pipeline-console-reporter';

await new Pipeline({
  datasetSelector: selector,
  stages,
  writers,
  reporter: new ConsoleReporter(),
}).run();
```
npm package: @lde/pipeline-shacl-validator
Description: SHACL validation for @lde/pipeline
Latest version: 0.10.7

# @lde/pipeline-shacl-validator

SHACL validation for [`@lde/pipeline`](../pipeline).

Validates RDF quads produced by pipeline stages against [SHACL shapes](https://www.w3.org/TR/shacl/),
writing per-dataset report files in SHACL validation report format.
Shapes can be provided in any RDF serialization (Turtle, JSON-LD, N-Triples etc.).

## Usage

```typescript
import { Pipeline, Stage, SparqlConstructExecutor } from '@lde/pipeline';
import { ShaclValidator } from '@lde/pipeline-shacl-validator';

const validator = new ShaclValidator({
  shapesFile: './shapes.ttl',
  reportDir: './validation',
});

const pipeline = new Pipeline({
  // ...
  stages: [
    new Stage({
      name: 'transform',
      executors: new SparqlConstructExecutor({ query: '...' }),
      validation: {
        validator,
        onInvalid: 'write', // 'write' | 'skip' | 'halt'
      },
    }),
  ],
});

await pipeline.run();
```

### `onInvalid` options

| Value     | Behaviour                                                        |
| --------- | ---------------------------------------------------------------- |
| `'write'` | Write quads to the output even if validation fails **(default)** |
| `'skip'`  | Discard invalid quads silently                                   |
| `'halt'`  | Throw an error, stopping the pipeline                            |

### Report files

Validation violations are written to `<reportDir>/<dataset-iri>.validation.<ext>`
as SHACL validation report triples. The output format defaults to Turtle (`.ttl`)
and can be changed with the `reportFormat` option:

```typescript
const validator = new ShaclValidator({
  shapesFile: './shapes.ttl',
  reportDir: './validation',
  reportFormat: 'N-Triples', // 'Turtle' (default) | 'N-Triples' | 'N-Quads'
});
```
npm package: @lde/sparql-importer
Description: Import RDF data dumps to a local SPARQL endpoint. This is useful when:
Latest version: 0.6.0

# SPARQL Importer

Import RDF data dumps to a local SPARQL endpoint. This is useful when:

- the dataset does not offer a SPARQL endpoint itself
- or when it does but the endpoint is too slow or unreliable, particularly for more complex SPARQL queries.
npm package: @lde/sparql-qlever
Description: An adapter for the [QLever](https://github.com/ad-freiburg/qlever) SPARQL server.
Latest version: 0.14.2

# SPARQL QLever

An adapter for the [QLever](https://github.com/ad-freiburg/qlever) SPARQL server.

## Index caching

Building a QLever index is slow. To avoid rebuilding it on every pipeline run, the importer caches a single index and reuses it when the source data hasn't changed. On subsequent runs, indexing is skipped when the source file matches and hasn't been re-downloaded.

Only **one** index is cached at a time. In a multi-dataset pipeline, each dataset overwrites the previous index. On re-run, the last-indexed dataset gets a cache hit while the others rebuild.

Caching is enabled by default. Disable it by passing `cacheIndex: false` to `createQlever()` or the `Importer` constructor (e.g. driven by a `QLEVER_CACHE_INDEX=false` environment variable).

## Configuration

`createQlever()` accepts `indexOptions` and `serverOptions` to tune QLever's index builder and server respectively.

### Server options (`serverOptions`)

Passed to `qlever-server` at startup.

| Option                  | Description                                      | Default |
| ----------------------- | ------------------------------------------------ | ------- |
| `memory-max-size`       | Maximum memory for query processing and caching. | `'4G'`  |
| `default-query-timeout` | Default query timeout.                           | `'30s'` |
| `cache-max-size`        | Maximum cache size for query results.            | —       |

Example:

```ts
const { importer, server } = createQlever({
  mode: 'docker',
  image: 'adfreiburg/qlever:latest',
  serverOptions: {
    'memory-max-size': '12G',
    'default-query-timeout': '120s',
  },
});
```

### Index options (`indexOptions`)

Passed to `qlever-index` during import.

| Option                          | Description                                                                                   | Default     |
| ------------------------------- | --------------------------------------------------------------------------------------------- | ----------- |
| `ascii-prefixes-only`           | Enable faster parsing for well-behaved TTL files.                                             | `true`      |
| `num-triples-per-batch`         | Triples per batch; lower values reduce memory usage.                                          | `3_000_000` |
| `stxxl-memory`                  | Memory budget for sorting during the index build.                                             | `'10G'`     |
| `parse-parallel`                | Parse input in parallel.                                                                      | `true`      |
| `only-pso-and-pos-permutations` | Build only PSO and POS permutations. Faster, but queries with predicate variables won't work. | `false`     |