I currently read a doc for API call, but I have problem. Please simply reply with 'ok' after I sent doc below.

---


afghan26

Search Query
Search...
⌘+K
Dashboard
Explore
Pricing
Docs
Blog
Changelog

meta / meta-llama-3-70b-instruct 

A 70 billion parameter language model from Meta, fine tuned for chat completions

Warm
Public
3.7M runs
GitHub
License

Star
Run with an API
Playground
API
Examples
README
TABLE OF CONTENTS



Python
01
Get started
02
Learn more
Authentication
Setup
Run the model
Streaming
Streaming in the browser
Prediction lifecycle
Webhooks
Access a prediction
Cancel a prediction
Async Python methods
03
Schema
04
API reference
Authentication
Whenever you make an API request, you need to authenticate using a token. A token is like a password that uniquely identifies your account and grants you access.

The following examples all expect your Replicate access token to be available from the command line. Because tokens are secrets, they should not be in your code. They should instead be stored in environment variables. Replicate clients look for the REPLICATE_API_TOKEN environment variable and use it if available.

To set this up you can use:

export REPLICATE_API_TOKEN=r8_QKw**********************************

Visibility

Copy
Some application frameworks and tools also support a text file named .env which you can edit to include the same token:

REPLICATE_API_TOKEN=r8_QKw**********************************

Visibility

Copy
The Replicate API uses the Authorization HTTP header to authenticate requests. If you’re using a client library this is handled for you.

You can test that your access token is setup correctly by using our account.get endpoint:

What is cURL?
curl https://api.replicate.com/v1/account -H "Authorization: Bearer $REPLICATE_API_TOKEN"
# {"type":"user","username":"aron","name":"Aron Carroll","github_url":"https://github.com/aron"}

Copy
If it is working correctly you will see a JSON object returned containing some information about your account, otherwise ensure that your token is available:

echo "$REPLICATE_API_TOKEN"
# "r8_xyz"

Copy
../auth-token.js

Setup
First you’ll need to ensure you have a Python environment setup:

python -m venv .venv
source .venv/bin/activate

Copy
Then install the replicate Python library:

pip install replicate

Copy
In a main.py file, import replicate:

import replicate

Copy
This will use the REPLICATE_API_TOKEN API token you’ve set up in your environment for authorization.

Run the model
Use the replicate.run() method to run the model:

input = {
    "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
    "prompt_template": "

system
\n\nYou are a helpful assistant

user
\n\n{prompt}

assistant
\n\n"
}

output = replicate.run(
    "meta/meta-llama-3-70b-instruct",
    input=input
)
print("".join(output))
#=> "Let's break this problem down step by step.\n\nStep 1: S...

Copy
The run() function returns the output directly, which you can then use or pass as the input to another model. If you want to access the full prediction object (not just the output), use the replicate.models.predictions.create method instead. This will return a Prediction object that includes the prediction id, status, logs, etc.

Streaming
This model supports streaming. This allows you to receive output as the model is running:

input = {
    "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
    "prompt_template": "

system
\n\nYou are a helpful assistant

user
\n\n{prompt}

assistant
\n\n"
}

for event in replicate.stream(
    "meta/meta-llama-3-70b-instruct",
    input=input
):
    print(event, end="")
    #=> "Let"

Copy
Streaming in the browser
The Python library is intended to be run on the server. However once the prediction has been created its output can be streamed directly from the browser.

The streaming URL uses a standard format called Server Sent Events (or text/event-stream) built into all web browsers.

A common implementation is to use a web server to create the prediction using replicate.predictions.create , passing the stream property set to true . Then the urls.stream property of the response contains a url that can be returned to your frontend application:

input = {
    "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
    "prompt_template": "

system
\n\nYou are a helpful assistant

user
\n\n{prompt}

assistant
\n\n"
}

# POST /run_prediction
def handler(request):
    prediction = replicate.models.predictions.create(
        "meta/meta-llama-3-70b-instruct",
        input=input,
        stream=True,
    )

    return Response.json({
        'url': prediction.urls['stream']
    })
    # Returns {"url": "https://replicate-stream..."}
}

Copy
Make a request to the server to create the prediction and use the built-in EventSource object to read the returned url.

const response = await fetch("/run_prediction", { method: "POST" });
const { url } = await response.json();

const source = new EventSource(url);
source.addEventListener("output", (evt) => {
  console.log(evt.data) //=> "Let"
});
source.addEventListener("done", (evt) => {
  console.log("stream is complete");
});

Copy
Prediction lifecycle
Running predictions and trainings can often take significant time to complete, beyond what is reasonable for an HTTP request/response.

When you run a model on Replicate, the prediction is created with a “starting” state, then instantly returned. This will then move to "processing" and eventual one of “successful”, "failed" or "canceled".

Starting
Running
Succeeded
Failed
Canceled
You can explore the prediction lifecycle by using the prediction.reload() method update the prediction to it's latest state.

Show example
Webhooks
Webhooks provide real-time updates about your prediction. Specify an endpoint when you create a prediction, and Replicate will send HTTP POST requests to that URL when the prediction is created, updated, and finished.

It is possible to provide a URL to the predictions.create() function that will be requested by Replicate when the prediction status changes. This is an alternative to polling.

To receive webhooks you’ll need a web server. The following example uses AIOHTTP, a basic webserver built on top of Python’s asyncio library, but this pattern will apply to most frameworks.

Show example
Then create the prediction passing in the webhook URL and specify which events you want to receive out of "start" , "output" ”logs” and "completed".

input = {
    "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
    "prompt_template": "

system
\n\nYou are a helpful assistant

user
\n\n{prompt}

assistant
\n\n"
}

callback_url = "https://my.app/webhooks/replicate"
replicate.models.predictions.create(
  "meta/meta-llama-3-70b-instruct",
  input=input,
  webhook=callback_url,
  webhook_events_filter=["completed"]
)

# The server will now handle the event and log:
#=> Prediction(id='z3wbih3bs64of7lmykbk7tsdf4', ...)

Copy
The replicate.run() method is not used here. Because we're using webhooks, and we don’t need to poll for updates.

From a security perspective it is also possible to verify that the webhook came from Replicate, check out our documentation on verifying webhooks for more information.

Access a prediction
You may wish to access the prediction object. In these cases it’s easier to use the replicate.predictions.create(), replicate.models.predictions.create() or replicate.deployments.predictions.create() functions which will return the prediction object.

Though note that these functions will only return the created prediction, and it will not wait for that prediction to be completed before returning. Use replicate.predictions.get() to fetch the latest prediction.

import replicate

input = {
    "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
    "prompt_template": "

system
\n\nYou are a helpful assistant

user
\n\n{prompt}

assistant
\n\n"
}

prediction = replicate.models.predictions.create(
  "meta/meta-llama-3-70b-instruct",
  input=input
)
#=> Prediction(id='z3wbih3bs64of7lmykbk7tsdf4', ...)

Copy
Cancel a prediction
You may need to cancel a prediction. Perhaps the user has navigated away from the browser or canceled your application. To prevent unnecessary work and reduce runtime costs you can use prediction.cancel() method to call the predictions.cancel endpoint.

input = {
    "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
    "prompt_template": "

system
\n\nYou are a helpful assistant

user
\n\n{prompt}

assistant
\n\n"
}

prediction = replicate.models.predictions.create(
  "meta/meta-llama-3-70b-instruct",
  input=input
)

prediction.cancel()

Copy
Async Python methods
asyncio is a module built into Python's standard library for writing concurrent code using the async/await syntax.

Replicate's Python client has support for asyncio. Each of the methods has an async equivalent prefixed with async_<name>.

input = {
    "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
    "prompt_template": "

system
\n\nYou are a helpful assistant

user
\n\n{prompt}

assistant
\n\n"
}

prediction = replicate.models.predictions.create(
  "meta/meta-llama-3-70b-instruct",
  input=input
)

prediction = await replicate.models.predictions.async_create(
  "meta/meta-llama-3-70b-instruct",
  input=input
)

Copy
Replicate

About
Guides
Terms
Privacy
Status
GitHub
X
Discord
Support