Package: simple-ai-server
SimpleAIA self-hosted alternative to the not-so-open AI API. It is focused on replicating the main endpoints for LLM:Text completion (/completions/)✔️ Nonstreamresponses✔️streamresponsesChat (/chat/completions/) [example]✔️ Nonstreamresponses✔️streamresponsesEdits (/edits/) [example]Embeddings (/embeddings/) [example]Not supported (yet):images,audio,files,fine-tunes,moderationsIt allows you to experiment with competing approaches quickly and easily.Why this project?Well first of all it's a fun little project, and perhaps a better use of my time than watching some random dog videos on Reddit or YouTube. I also believe it can be a great way to:experiment with new models and not be too dependent on a specific API provider,create benchmarks to decide which approach works best for you,handle some specific use cases where you cannot fully rely on an external service, without the need of re-writing everythingIf you find interesting use cases, feel free to share your experience.InstallationOn a machine with Python 3.9+:[Latest] From source:pipinstallgit+https://github.com/lhenault/simpleAIFrom Pypi:pipinstallsimple_ai_serverSetupStart by creating a configuration file to declare your models:simple_aiinitIt should createmodels.toml, where you declare your different models (see how below). Then start the server with:simple_aiserve[--host127.0.0.1][--port8080]You can then see the docs and try itthere.Integrating and declaring a modelModel integrationModels are queried throughgRPC, in order to separate the API itself from the model inference, and to support several languages beyond Python through this protocol.To expose for instance an embedding model in Python, you simply have to import a few things, and implements the.embed()method of yourEmbeddingModelclass:importloggingfromdataclassesimportdataclassfromsimple_ai.api.grpc.embedding.serverimportserve,LanguageModelServicer@dataclass(unsafe_hash=True)classEmbeddingModel:defembed(self,inputs:list=[],)->list:# TODO : implements the embed methodreturn[[]]if__name__=='__main__':model_servicer=LanguageModelServicer(model=EmbeddingModel())serve(address='[::]:50051',model_servicer=model_servicer)For a completion task, follow the same logic, but importfrom simple_ai.api.grpc.completion.serverinstead, and implements acompletemethod.Declaring a modelTo add a model, you first need to deploy a gRPC service (using the provided.protofile and / or the tools provided insrc/api/). Once your model is live, you only have to add it to themodels.tomlconfiguration file. For instance, let's say you've locally deployed allama.cppmodel available on port 50051, just add:[llama-7B-4b][llama-7B-4b.metadata]owned_by='Meta / ggerganov'permission=[]description='C++ implementation of LlaMA model, 7B parameters, 4-bit quantization'[llama-7B-4b.network]url='localhost:50051'type='gRPC'You can see see and try of the provided examples inexamples/directory (might require GPU).UsageThanks tothe Swagger UI, you can see and try the different endpointshere:Or you can directly use the API with the tool of your choice.curl-X'POST'\'http://127.0.0.1:8080/edits/'\-H'accept: application/json'\-H'Content-Type: application/json'\-d'{"model": "alpaca-lora-7B","instruction": "Make this message nicer and more formal","input": "This meeting was useless and should have been a bloody email","top_p": 1,"n": 1,"temperature": 1,"max_tokens": 256}'It's also compatible withOpenAI python client:importopenai# Put anything you want in `API key`openai.api_key='Free the models'# Point to your own urlopenai.api_base="http://127.0.0.1:8080"# Do your usual things, for instance a completion query:print(openai.Model.list())completion=openai.Completion.create(model="llama-7B",prompt="Hello everyone this is")Common issues and solutionsAdding a CORS middlewareIf you encounter CORS issues, it is suggested to not use thesimple_ai servecommand, but to rather use your own script to add your CORS configuration, using theFastAPI CORS middleware.For instance you can createmy_server.pywith:fromsimple_ai.serverimportappfromfastapi.middleware.corsimportCORSMiddlewaredefadd_cors(app):origins=["http://localhost","http://localhost:8080"]app.add_middleware(CORSMiddleware,allow_origins=origins,allow_credentials=True,allow_methods=["*"],allow_headers=["*"],)returnappdefserve_app(host="127.0.0.1",port=8080,**kwargs):app=add_cors(app)uvicorn.run(app=app,host=host,port=port)if__name__=="__main__":serve_app(host="127.0.0.1",port=8080)And run it aspython3 my_server.pyinstead.I needd/v1prefix in the endpointsSome projects have decided to include the/v1prefix as part of the endpoints, while OpenAI client includes it in itsapi_baseparameter. If you need to have it as part of the endpoints for your project, you can use a custom script instead ofsimple_ai serve:importuvicornfromsimple_ai.serverimportappasv1_appfromfastapiimportAPIRouter,FastAPIsai_app=FastAPI()sai_app.mount("/v1",v1_app)defserve_app(app=sai_app,host="0.0.0.0",port=8080):uvicorn.run(app=app,host=host,port=port)if__name__=="__main__":serve_app()ContributeThis is very much work in progress and far from being perfect, so let me know if you want to help. PR, issues, documentation, cool logo, all the usual candidates are welcome.Development EnvironmentIn order for the following steps to work it is required to have make and poetry installed on your system.To install the development environment run:make install-devThis will install all dev dependencies as well as configure your pre-commit helpers.