Repository: huggingface/huggingface.js
Pull request #1260: Integrate Nscale-cloud into HuggingFace inference
Author: login: nbarr07
Labels: name: inference-providers
description: integration of a new or existing Inference Provider
Created: 2025-03-07T15:10:00Z | Merged: 2025-04-24T08:03:06Z

# Integrate Nscale provider for HuggingFace Inference

This PR adds support for Nscale inference to the HuggingFace inference API. 

Note that our inference service is not publicly live yet but will be soon - this draft PR is for review and preparation.

- Implemented standard provider integration for nscale
- Added support for text-generation, conversational, and text-to-image tasks
- Included tests following the established patterns

The tests were all passing when I tried. Any feedback are welcomed!


Diff:
diff --git a/packages/inference/README.md b/packages/inference/README.md
index 241ba7a523..88d0328883 100644
--- a/packages/inference/README.md
+++ b/packages/inference/README.md
@@ -52,6 +52,7 @@ Currently, we support the following providers:
 - [Hyperbolic](https://hyperbolic.xyz)
 - [Nebius](https://studio.nebius.ai)
 - [Novita](https://novita.ai/?utm_source=github_huggingface&utm_medium=github_readme&utm_campaign=link)
+- [Nscale](https://nscale.com)
 - [Replicate](https://replicate.com)
 - [Sambanova](https://sambanova.ai)
 - [Together](https://together.xyz)
@@ -79,6 +80,7 @@ Only a subset of models are supported when requesting third-party providers. You
 - [Fireworks AI supported models](https://huggingface.co/api/partners/fireworks-ai/models)
 - [Hyperbolic supported models](https://huggingface.co/api/partners/hyperbolic/models)
 - [Nebius supported models](https://huggingface.co/api/partners/nebius/models)
+- [Nscale supported models](https://huggingface.co/api/partners/nscale/models)
 - [Replicate supported models](https://huggingface.co/api/partners/replicate/models)
 - [Sambanova supported models](https://huggingface.co/api/partners/sambanova/models)
 - [Together supported models](https://huggingface.co/api/partners/together/models)
diff --git a/packages/inference/src/lib/getProviderHelper.ts b/packages/inference/src/lib/getProviderHelper.ts
index c9c85d29b3..060cddffb7 100644
--- a/packages/inference/src/lib/getProviderHelper.ts
+++ b/packages/inference/src/lib/getProviderHelper.ts
@@ -8,6 +8,7 @@ import * as HFInference from "../providers/hf-inference";
 import * as Hyperbolic from "../providers/hyperbolic";
 import * as Nebius from "../providers/nebius";
 import * as Novita from "../providers/novita";
+import * as Nscale from "../providers/nscale";
 import * as OpenAI from "../providers/openai";
 import type {
 	AudioClassificationTaskHelper,
@@ -109,6 +110,10 @@ export const PROVIDERS: Record<InferenceProvider, Partial<Record<InferenceTask,
 		conversational: new Novita.NovitaConversationalTask(),
 		"text-generation": new Novita.NovitaTextGenerationTask(),
 	},
+	nscale: {
+		"text-to-image": new Nscale.NscaleTextToImageTask(),
+		conversational: new Nscale.NscaleConversationalTask(),
+	},
 	openai: {
 		conversational: new OpenAI.OpenAIConversationalTask(),
 	},
diff --git a/packages/inference/src/providers/consts.ts b/packages/inference/src/providers/consts.ts
index 7e6b3aa372..fd1e322531 100644
--- a/packages/inference/src/providers/consts.ts
+++ b/packages/inference/src/providers/consts.ts
@@ -28,6 +28,7 @@ export const HARDCODED_MODEL_INFERENCE_MAPPING: Record<
 	hyperbolic: {},
 	nebius: {},
 	novita: {},
+	nscale: {},
 	openai: {},
 	replicate: {},
 	sambanova: {},
diff --git a/packages/inference/src/providers/nscale.ts b/packages/inference/src/providers/nscale.ts
new file mode 100644
index 0000000000..4560ed8c25
--- /dev/null
+++ b/packages/inference/src/providers/nscale.ts
@@ -0,0 +1,79 @@
+/**
+ * See the registered mapping of HF model ID => Nscale model ID here:
+ *
+ * https://huggingface.co/api/partners/nscale-cloud/models
+ *
+ * This is a publicly available mapping.
+ *
+ * If you want to try to run inference for a new model locally before it's registered on huggingface.co,
+ * you can add it to the dictionary "HARDCODED_MODEL_ID_MAPPING" in consts.ts, for dev purposes.
+ *
+ * - If you work at Nscale and want to update this mapping, please use the model mapping API we provide on huggingface.co
+ * - If you're a community member and want to add a new supported HF model to Nscale, please open an issue on the present repo
+ * and we will tag Nscale team members.
+ *
+ * Thanks!
+ */
+import type { TextToImageInput } from "@huggingface/tasks";
+import { InferenceOutputError } from "../lib/InferenceOutputError";
+import type { BodyParams } from "../types";
+import { omit } from "../utils/omit";
+import { BaseConversationalTask, TaskProviderHelper, type TextToImageTaskHelper } from "./providerHelper";
+
+const NSCALE_API_BASE_URL = "https://inference.api.nscale.com";
+
+interface NscaleCloudBase64ImageGeneration {
+	data: Array<{
+		b64_json: string;
+	}>;
+}
+
+export class NscaleConversationalTask extends BaseConversationalTask {
+	constructor() {
+		super("nscale", NSCALE_API_BASE_URL);
+	}
+}
+
+export class NscaleTextToImageTask extends TaskProviderHelper implements TextToImageTaskHelper {
+	constructor() {
+		super("nscale", NSCALE_API_BASE_URL);
+	}
+
+	preparePayload(params: BodyParams<TextToImageInput>): Record<string, unknown> {
+		return {
+			...omit(params.args, ["inputs", "parameters"]),
+			...params.args.parameters,
+			response_format: "b64_json",
+			prompt: params.args.inputs,
+			model: params.model,
+		};
+	}
+
+	makeRoute(): string {
+		return "v1/images/generations";
+	}
+
+	async getResponse(
+		response: NscaleCloudBase64ImageGeneration,
+		url?: string,
+		headers?: HeadersInit,
+		outputType?: "url" | "blob"
+	): Promise<string | Blob> {
+		if (
+			typeof response === "object" &&
+			"data" in response &&
+			Array.isArray(response.data) &&
+			response.data.length > 0 &&
+			"b64_json" in response.data[0] &&
+			typeof response.data[0].b64_json === "string"
+		) {
+			const base64Data = response.data[0].b64_json;
+			if (outputType === "url") {
+				return `data:image/jpeg;base64,${base64Data}`;
+			}
+			return fetch(`data:image/jpeg;base64,${base64Data}`).then((res) => res.blob());
+		}
+
+		throw new InferenceOutputError("Expected Nscale text-to-image response format");
+	}
+}
diff --git a/packages/inference/src/types.ts b/packages/inference/src/types.ts
index 1e465c00fd..032bd41901 100644
--- a/packages/inference/src/types.ts
+++ b/packages/inference/src/types.ts
@@ -47,6 +47,7 @@ export const INFERENCE_PROVIDERS = [
 	"hyperbolic",
 	"nebius",
 	"novita",
+	"nscale",
 	"openai",
 	"replicate",
 	"sambanova",
diff --git a/packages/inference/test/InferenceClient.spec.ts b/packages/inference/test/InferenceClient.spec.ts
index f59b2cf2eb..2105902bc0 100644
--- a/packages/inference/test/InferenceClient.spec.ts
+++ b/packages/inference/test/InferenceClient.spec.ts
@@ -1690,4 +1690,65 @@ describe.skip("InferenceClient", () => {
 		},
 		TIMEOUT
 	);
+	describe.concurrent(
+		"Nscale",
+		() => {
+			const client = new InferenceClient(env.HF_NSCALE_KEY ?? "dummy");
+
+			HARDCODED_MODEL_INFERENCE_MAPPING["nscale"] = {
+				"meta-llama/Llama-3.1-8B-Instruct": {
+					hfModelId: "meta-llama/Llama-3.1-8B-Instruct",
+					providerId: "nscale",
+					status: "live",
+					task: "conversational",
+				},
+				"black-forest-labs/FLUX.1-schnell": {
+					hfModelId: "black-forest-labs/FLUX.1-schnell",
+					providerId: "flux-schnell",
+					status: "live",
+					task: "text-to-image",
+				},
+			};
+
+			it("chatCompletion", async () => {
+				const res = await client.chatCompletion({
+					model: "meta-llama/Llama-3.1-8B-Instruct",
+					provider: "nscale",
+					messages: [{ role: "user", content: "Complete this sentence with words, one plus one is equal " }],
+				});
+				if (res.choices && res.choices.length > 0) {
+					const completion = res.choices[0].message?.content;
+					expect(completion).toContain("two");
+				}
+			});
+			it("chatCompletion stream", async () => {
+				const stream = client.chatCompletionStream({
+					model: "meta-llama/Llama-3.1-8B-Instruct",
+					provider: "nscale",
+					messages: [{ role: "user", content: "Say 'this is a test'" }],
+					stream: true,
+				}) as AsyncGenerator<ChatCompletionStreamOutput>;
+				let fullResponse = "";
+				for await (const chunk of stream) {
+					if (chunk.choices && chunk.choices.length > 0) {
+						const content = chunk.choices[0].delta?.content;
+						if (content) {
+							fullResponse += content;
+						}
+					}
+				}
+				expect(fullResponse).toBeTruthy();
+				expect(fullResponse.length).toBeGreaterThan(0);
+			});
+			it("textToImage", async () => {
+				const res = await client.textToImage({
+					model: "black-forest-labs/FLUX.1-schnell",
+					provider: "nscale",
+					inputs: "An astronaut riding a horse",
+				});
+				expect(res).toBeInstanceOf(Blob);
+			});
+		},
+		TIMEOUT
+	);
 });


Reviews:
user: login: SBrandeis
body: Hi there - thank you for your conribution!

I have two minor comments - otherwise the PR looks good !

Ping @hanouticelina for a quick validation before merging
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T15:39:17Z
user: login: hanouticelina
body: looks good, thank you @nbarr07!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T16:08:50Z
user: login: Wauplin
body: Nice! :fire: 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T10:09:14Z

Review comments:
diff_hunk: @@ -0,0 +1,83 @@
+/**
+ * See the registered mapping of HF model ID => Nscale model ID here:
+ *
+ * https://huggingface.co/api/partners/nscale-cloud/models
+ *
+ * This is a publicly available mapping.
+ *
+ * If you want to try to run inference for a new model locally before it's registered on huggingface.co,
+ * you can add it to the dictionary "HARDCODED_MODEL_ID_MAPPING" in consts.ts, for dev purposes.
+ *
+ * - If you work at Nscale and want to update this mapping, please use the model mapping API we provide on huggingface.co
+ * - If you're a community member and want to add a new supported HF model to Nscale, please open an issue on the present repo
+ * and we will tag Nscale team members.
+ *
+ * Thanks!
+ */
+import { InferenceOutputError } from "../lib/InferenceOutputError";
+import type { BodyParams, UrlParams } from "../types";
+import { omit } from "../utils/omit";
+import {
+	BaseConversationalTask,
+	TaskProviderHelper,
+	type TextToImageTaskHelper,
+} from "./providerHelper";
+
+const NSCALE_API_BASE_URL = "https://inference.api.nscale.com";
+
+interface NscaleCloudBase64ImageGeneration {
+	data: Array<{
+		b64_json: string;
+	}>;
+}
+
+export class NscaleConversationalTask extends BaseConversationalTask {
+	constructor() {
+		super("nscale", NSCALE_API_BASE_URL);
+	}
+}
+
+export class NscaleTextToImageTask extends TaskProviderHelper implements TextToImageTaskHelper {
+	constructor() {
+		super("nscale", NSCALE_API_BASE_URL);
+	}
+
+	preparePayload(params: BodyParams): Record<string, unknown> {
+		return {
+			...omit(params.args, ["inputs", "parameters"]),
+			...(params.args.parameters as Record<string, unknown>),
+			response_format: "b64_json",
+			prompt: params.args.inputs,
+			model: params.model,
+		};
+	}
+
path: packages/inference/src/providers/nscale.ts
user: login: SBrandeis
body: No need for a cast here, IMO

`TextToImageInput` needs to be imported from the `@huggingface/tasks` sibling package

```suggestion
	preparePayload(params: BodyParams<TextToImageInput>): Record<string, unknown> {
		return {
			...omit(params.args, ["inputs", "parameters"]),
			...params.args.parameters,
			response_format: "b64_json",
			prompt: params.args.inputs,
			model: params.model,
		};
	}

```
created_at: 2025-04-14T15:33:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,83 @@
+/**
+ * See the registered mapping of HF model ID => Nscale model ID here:
+ *
+ * https://huggingface.co/api/partners/nscale-cloud/models
+ *
+ * This is a publicly available mapping.
+ *
+ * If you want to try to run inference for a new model locally before it's registered on huggingface.co,
+ * you can add it to the dictionary "HARDCODED_MODEL_ID_MAPPING" in consts.ts, for dev purposes.
+ *
+ * - If you work at Nscale and want to update this mapping, please use the model mapping API we provide on huggingface.co
+ * - If you're a community member and want to add a new supported HF model to Nscale, please open an issue on the present repo
+ * and we will tag Nscale team members.
+ *
+ * Thanks!
+ */
+import { InferenceOutputError } from "../lib/InferenceOutputError";
+import type { BodyParams, UrlParams } from "../types";
+import { omit } from "../utils/omit";
+import {
+	BaseConversationalTask,
+	TaskProviderHelper,
+	type TextToImageTaskHelper,
+} from "./providerHelper";
+
+const NSCALE_API_BASE_URL = "https://inference.api.nscale.com";
+
+interface NscaleCloudBase64ImageGeneration {
+	data: Array<{
+		b64_json: string;
+	}>;
+}
+
+export class NscaleConversationalTask extends BaseConversationalTask {
+	constructor() {
+		super("nscale", NSCALE_API_BASE_URL);
+	}
+}
+
+export class NscaleTextToImageTask extends TaskProviderHelper implements TextToImageTaskHelper {
+	constructor() {
+		super("nscale", NSCALE_API_BASE_URL);
+	}
+
+	preparePayload(params: BodyParams): Record<string, unknown> {
+		return {
+			...omit(params.args, ["inputs", "parameters"]),
+			...(params.args.parameters as Record<string, unknown>),
+			response_format: "b64_json",
+			prompt: params.args.inputs,
+			model: params.model,
+		};
+	}
+
+	makeRoute(params: UrlParams): string {
+		void params;
+		return "v1/images/generations";
+	}
+
path: packages/inference/src/providers/nscale.ts
user: login: SBrandeis
body: 
```suggestion
	makeRoute(): string {
		return "v1/images/generations";
	}

```
created_at: 2025-04-14T15:33:50Z
author_association: CONTRIBUTOR

PR comments:
user: login: Wauplin
created_at: 2025-03-28T09:17:01Z
body: Hey @nbarr07 , thanks for your PR :) Even though we are looking forward to integrate new providers, we'll have to hold on for a bit on this one. Integrating a new provider takes time and effort on our side to make things right so for now we've decided to focus on the existing ones. Will let you know when we are ready to move forward! :hugs: 
author_association: CONTRIBUTOR
user: login: julien-c
created_at: 2025-04-02T11:14:45Z
body: Hi @nbarr07 we are currently finishing a refactoring of Inference Providers integration code in #1315, this should be merged soon, but we will need to rewrite part of your implementation (should be even simpler to integrate), will ping again after it's been merged.
author_association: MEMBER
user: login: nbarr07
created_at: 2025-04-02T13:18:11Z
body: Hi @julien-c , thanks for the heads-up about the refactoring, if there's anything we can do on our side let me know!
author_association: CONTRIBUTOR
user: login: hanouticelina
created_at: 2025-04-08T09:31:35Z
body: Hi @nbarr07,
We've merged a refactoring for Inference Provider integration into main, which should make adding new providers much easier.
Could you merge main into your branch and update your PR accordingly? it should be relatively straightforward with the new structure:
1 - You have to update the `PROVIDERS` mapping here: [inference/src/lib/getProviderHelper.ts#L49](https://github.com/huggingface/huggingface.js/blob/9b3e2cf9e43c88fb03e6fda2aaf913385181c599/packages/inference/src/lib/getProviderHelper.ts#L49) and add nscale-cloud (let's ensure we respect the alphabetical order) :
```ts
import * as NScaleCloud from "../providers/nscale-cloud";
...
export const PROVIDERS: Record<InferenceProvider, Partial<Record<InferenceTask, TaskProviderHelper>>> = {
	...
	"nscale-cloud": {
		"text-to-image": new NScaleCloud.NScaleCloudTextToImageTask(),
		"conversational": new NScaleCloud.NScaleCloudConversationalTask(),
		"text-generation": new NScaleCloud.NScaleCloudTextGenerationTask(),
	},
	},
        ...
```
2 - Update `packages/inference/src/providers/nscale-cloud.ts` to implement the classes:
```ts
import { BaseConversationalTask, BaseTextGenerationTask } from "./providerHelper";

const NSCALE_API_BASE_URL = "https://inference.api.nscale.com";

export class NScaleCloudConversationalTask extends BaseConversationalTask {
	constructor() {
		super("nscale-cloud", NSCALE_API_BASE_URL);
	}
}

export class NScaleCloudTextGenerationTask extends BaseTextGenerationTask {
	constructor() {
		super("nscale-cloud", NSCALE_API_BASE_URL);
	}

	override makeRoute(): string {
		return "v1/chat/completions";
	}
}

export class NScaleCloudTextToImageTask extends TaskProviderHelper implements TextToImageTaskHelper {
	constructor() {
		super("nscale-cloud", NSCALE_API_BASE_URL);
	}

	makeRoute(): string {
		return "v1/images/generations";
	}

	preparePayload(params: BodyParams): Record<string, unknown> {
		// refer to Nebius implementation of preparePayload
	}
         
         getResponse(
		response: unknown,
		url?: string,
		headers?: HeadersInit,
		outputType?: "url" | "blob"
	): Promise<string | Blob>{
		// Refer to Nebius implementation of getResponse
	}
}
```
You can check Nebius implementation of text-to-image, it looks like it's the same payload and response format as NScale-cloud.
let us know if you need any help! you can find more details in the documentation : https://huggingface.co/docs/inference-providers/register-as-a-provider#2-js-client-integration.
author_association: CONTRIBUTOR
user: login: nbarr07
created_at: 2025-04-14T07:17:03Z
body: Hey @hanouticelina, 
thanks for the heads up, I've merged the latest changes into my branch and updated the PR. One question I have is does the provider name in inference need to march the organisation username on Huggingface or can they be different? 
Let me know if there's any adjustments needed.
author_association: CONTRIBUTOR
user: login: julien-c
created_at: 2025-04-14T09:44:20Z
body: it's better if they're the same, do you want us to check if we can move your org to `nscale` namespace?
author_association: MEMBER
user: login: nbarr07
created_at: 2025-04-14T12:04:17Z
body: yes that would be great, thanks!
author_association: CONTRIBUTOR
user: login: HuggingFaceDocBuilderDev
created_at: 2025-04-14T15:43:51Z
body: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/huggingface.js/pr_1260). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
author_association: NONE
user: login: nbarr07
created_at: 2025-04-15T07:02:27Z
body: @hanouticelina @SBrandeis thank you for the review and changes! 
I've just made a last change in the readme (the namespace had to be modified). Lmk when we can merge it, thanks!
author_association: CONTRIBUTOR
user: login: nbarr07
created_at: 2025-04-22T12:22:37Z
body: Hey @hanouticelina @SBrandeis , just to get an idea, when could we expect to merge the PR? 
author_association: CONTRIBUTOR
user: login: hanouticelina
created_at: 2025-04-24T08:02:59Z
body: let's merge this, thank you @nbarr07 again for the contribution!
author_association: CONTRIBUTOR