open-index/open-npm huggingface dataset:
npm package: @spark-connect-js/connect
Description: Generated TypeScript types and service stubs from Spark Connect protobuf definitions
Latest version: 0.3.0

# @spark-connect-js/connect

Generated TypeScript types and service definitions from the [Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html) protobuf spec.

> [!NOTE]
> This project is in early development. APIs may change between releases.

```bash
npm install @spark-connect-js/connect
```

```typescript
import { Plan, Relation, Expression, DataType } from "@spark-connect-js/connect";
import { SparkConnectService } from "@spark-connect-js/connect";
```

Used by [`@spark-connect-js/core`](https://www.npmjs.com/package/@spark-connect-js/core) and [`@spark-connect-js/node`](https://www.npmjs.com/package/@spark-connect-js/node). Most users don't need to install this directly.

## Documentation

See the [spark-connect-js](https://github.com/prustic/spark-connect-js) repository.

## License

[Apache-2.0](https://github.com/prustic/spark-connect-js/blob/main/LICENSE)
npm package: @spark-connect-js/core
Description: Pure TypeScript core: logical DataFrame API and Spark Connect plan builder — no runtime dependencies
Latest version: 0.3.0

# @spark-connect-js/core

DataFrame API and logical plan builder for [Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html), in pure TypeScript with zero runtime dependencies.

> [!NOTE]
> This project is in early development. APIs may change between releases.

```bash
npm install @spark-connect-js/core
```

```typescript
import { SparkSession, col, sum, desc, type Transport } from "@spark-connect-js/core";

const spark = new SparkSession(transport);

const df = spark
  .table("events")
  .filter(col("ts").gt(lit("2025-01-01")))
  .groupBy("category")
  .agg(sum("amount").alias("total"))
  .sort(desc("total"));
```

Provides `DataFrame`, `Column`, `SparkSession`, `Catalog`, `WindowSpec`, `DataFrameWriter`, `GroupedData`, and 248 built-in functions. Plans are serialized to Spark Connect protobuf but no I/O happens here. Bring your own `Transport` implementation, or use [`@spark-connect-js/node`](https://www.npmjs.com/package/@spark-connect-js/node) which wires everything together.

## Documentation

See the [spark-connect-js](https://github.com/prustic/spark-connect-js) repository.

## License

[Apache-2.0](https://github.com/prustic/spark-connect-js/blob/main/LICENSE)
npm package: @spark-connect-js/node
Description: Node.js runtime adapter: gRPC transport, Arrow decoding, Buffer handling, async streaming for Spark Connect
Latest version: 0.3.0

# @spark-connect-js/node

TypeScript client for [Apache Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html). Talks to Spark over gRPC and decodes results with Apache Arrow.

> [!NOTE]
> This project is in early development. APIs may change between releases.

```bash
npm install @spark-connect-js/node
```

```typescript
import { connect, col, sum, desc } from "@spark-connect-js/node";

const spark = connect("sc://localhost:15002");

const result = await spark
  .table("employees")
  .filter(col("age").gt(30))
  .groupBy("dept")
  .agg(sum("salary").alias("total"))
  .sort(desc("total"))
  .collect();

await spark.stop();
```

Requires Node.js >= 22 and a running [Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html) server (Spark 4.0+).

## Documentation

See the [spark-connect-js](https://github.com/prustic/spark-connect-js) repository for full documentation, examples, and contributing guidelines.

## License

[Apache-2.0](https://github.com/prustic/spark-connect-js/blob/main/LICENSE)