Package: xxtest
LMQLA programming language for large language models.Documentation »Explore Examples·Playground IDE·Report BugLMQL is an open source programming language for large language models (LLMs) based on asuperset of Python. LMQL goes beyond traditional templating languages by providing full Python support, yet a lightweight programming interface.LMQL is designed to make working with language models like OpenAI, 🤗 Transformers more efficient and powerful through its advanced functionality, including multi-variable templates, conditional distributions, constraints, datatype constraints and control flow.Features:Python Syntax: Write your queries usingfamiliar Python syntax, fully integrated with your Python environment (classes, variable captures, etc.)Rich Control-Flow: LMQL offers full Python support, enabling powerfulcontrol flow and logicin your prompting logic.Advanced Decoding: Take advantage of advanced decoding techniques likebeam search, best_k, and more.Powerful Constraints Via Logit Masking: Applyconstraints to model output, e.g. to specify token length, character-level constraints, datatype and stopping phrases to get more control of model behavior.Optimizing Runtime:LMQL leverages speculative execution to enable faster inference, constraint short-circuiting, more efficient token use andtree-based caching.Sync and Async API: Execute hundreds of queries in parallel with LMQL'sasynchronous API, which enables cross-query batching.Multi-Model Support: Seamlessly use LMQL withOpenAI API, Azure OpenAI, and 🤗 Transformers models.Extensive Applications: Use LMQL to implement advanced applications likeschema-safe JSON decoding,algorithmic prompting,interactive chat interfaces, andinline tool use.Library Integration: Easily employ LMQL in your existing stack leveragingLangChainorLlamaIndex.Flexible Tooling: Enjoy an interactive development experience withLMQL's Interactive Playground IDE, andVisual Studio Code Extension.Output Streaming: Stream model output easily viaWebSocket, REST endpoint, or Server-Sent Event streaming.Explore LMQLA simple example program in LMQL looks like this:argmax"Greet LMQL:[GREETINGS]\n"if"Hi there"inGREETINGS:"Can you reformulate your greeting in the speech of victorian-era English: [VIC_GREETINGS]\n""Analyse what part of this response makes it typically victorian:\n"foriinrange(4):"-[THOUGHT]\n""To summarize:[SUMMARY]"from"openai/text-davinci-003"wherestops_at(GREETINGS,".")andnot"\n"inGREETINGSandstops_at(VIC_GREETINGS,".")andstops_at(THOUGHT,".")Program Output:The main body of an LMQL program reads like standard Python (with control-flow), where top-level strings are interpreted as model input with template variables like[GREETINGS].Theargmaxkeyword in the beginning specifies the decoding algorithm used to generate tokens, e.g.argmax,sampleor
even advanced branching decoders likebeam search andbest_k.Thefromandwhereclauses specify the model and constraints that are employed during decoding.Overall, this style of language model programming facilitates guidance of the model's reasoning process, and constraining
of intermediate outputs using anexpressive constraint language.Learn more about LMQL by exploring ourExample Showcaseor by running your own programs in ourbrowser-based Playground IDE.Getting StartedTo install the latest version of LMQL run the following command with Python ==3.10 installed.pip install lmqlLocal GPU Support:If you want to run models on a local GPU, make sure to install LMQL in an environment with a GPU-enabled installation of PyTorch >= 1.11 (cf.https://pytorch.org/get-started/locally/) and install viapip install lmql[hf].Running LMQL ProgramsAfter installation, you can launch the LMQL playground IDE with the following command:lmql playgroundUsing the LMQL playground requires an installation of Node.js. If you are in a conda-managed environment you can install node.js viaconda install nodejs=14.20 -c conda-forge. Otherwise, please see the official Node.js websitehttps://nodejs.org/en/download/for instructions how to install it on your system.This launches a browser-based playground IDE, including a showcase of many exemplary LMQL programs. If the IDE does not launch automatically, go tohttp://localhost:3000.Alternatively,lmql runcan be used to execute local.lmqlfiles. Note that when using local HuggingFace Transformers models in the Playground IDE or vialmql run, you have to first launch an instance of the LMQL Inference API for the corresponding model via the commandlmql serve-model.Configuring OpenAI API CredentialsIf you want to use OpenAI models, you have to configure your API credentials. To do so, create a fileapi.envin the active working directory, with the following contents.openai-org: <org identifier>
openai-secret: <api secret>For system-wide configuration, you can also create anapi.envfile at$HOME/.lmql/api.envor at the project root of your LMQL distribution (e.g.src/in a development copy).Installing the Latest Development VersionTo install the latest (bleeding-edge) version of LMQL, you can also run the following command:pip install git+https://github.com/eth-sri/lmqThis will install thelmqlpackage directly from themainbranch of this repository. We do not continously test themainversion, so it may be less stable than the latest PyPI release.Setting Up a Development EnvironmentTo setup acondaenvironment for local LMQL development with GPU support, run the following commands:# prepare conda environment
conda env create -f scripts/conda/requirements.yml -n lmql
conda activate lmql

# registers the `lmql` command in the current shell
source scripts/activate-dev.shOperating System: The GPU-enabled version of LMQL was tested to work on Ubuntu 22.04 with CUDA 12.0 and Windows 10 via WSL2 and CUDA 11.7. The no-GPU version (see below) was tested to work on Ubuntu 22.04 and macOS 13.2 Ventura or Windows 10 via WSL2.Development without GPUThis section outlines how to setup an LMQL development environment without local GPU support. Note that LMQL without local GPU support only supports the use of API-integrated models likeopenai/text-davinci-003. Please see the OpenAI API documentation (https://platform.openai.com/docs/models/gpt-3-5) to learn more about the set of available models.To setup acondaenvironment for LMQL with no GPU support, run the following commands:# prepare conda environment
conda env create -f scripts/conda/requirements-no-gpu.yml -n lmql-no-gpu
conda activate lmql-no-gpu

# registers the `lmql` command in the current shell
source scripts/activate-dev.sh