bigcode

/

starcoder

like 2.71k

Hi, you can find an example here or see the docs.

In case it helps, I wrote one for stop-sequences, similar to OpenAI usage. Could probably be done faster/more cleverly, but for large LLMs where each forward is the bottleneck, it's saved plenty of time by allowing for early termination. Also wrote in some utilities for extracting the completion up to the stop-sequence and it's log-probabilities, as this was neede for my use-case.