Package: aiproxy-python
🦉 AIProxy🦉AIProxyis a Python library that serves as a reverse proxy LLM APIs including ChatGPT and Claude2. It provides enhanced features like monitoring, logging, and filtering requests and responses. This library is especially useful for developers and administrators who need detailed oversight and control over the interaction with LLM APIs.✅ Streaming support: Logs every bit of request and response data with token count – never miss a beat! 💓✅ Custom monitoring: Tailor-made for logging any specific info you fancy. Make it your own! 🔍✅ Custom filtering: Flexibly blocks access based on specific info or sends back your own responses. Be in control! 🛡️✅ Multiple AI Services: Supports ChatGPT (OpenAI and Azure OpenAI Service), Claude2 on AWS Bedrock, and is extensible by yourself! 🤖✅ Express dashboard: We provide template forApache Supersetthat's ready to use right out of the box – get insights quickly and efficiently!  📊🚀 Quick startInstall.$pipinstallaiproxy-pythonRun.$python-maiproxy[--hosthost][--portport][--openai_api_keyOPENAI_API_KEY]Use.importopenaiclient=openai.Client(base_url="http://127.0.0.1:8000/",api_key="YOUR_API_KEY")resp=client.chat.completions.create(model="gpt-3.5-turbo",messages=[{"role":"user","content":"hello!"}])print(resp)Enjoy😊🦉🛠️ Custom entrypointTo customize🦉AIProxy, make your custom entrypoint and configure logger and filters here.fromcontextlibimportasynccontextmanagerimportloggingfromfastapiimportFastAPIfromaiproxyimportChatGPTProxy,AccessLogWorkerimportthreading# Setup loggerlogger=logging.getLogger()logger.setLevel(logging.INFO)log_format=logging.Formatter("%(asctime)s%(levelname)8s%(message)s")streamHandler=logging.StreamHandler()streamHandler.setFormatter(log_format)logger.addHandler(streamHandler)# Setup access log worker# worker = AccessLogWorker()worker=CustomAccessLogWorker(accesslog_cls=CustomAccessLog)# 🌟 Instantiate your custom access log worker# Setup proxy for ChatGPTproxy=ChatGPTProxy(api_key=YOUR_API_KEY,access_logger_queue=worker.queue_client)proxy.add_filter(CustomRequestFilter1())# 🌟 Set your custom filter(s)proxy.add_filter(CustomRequestFilter2())# 🌟 Set your custom filter(s)proxy.add_filter(CustomResponseFilter())# 🌟 Set your custom filter(s)# Setup server application@asynccontextmanagerasyncdeflifespan(app:FastAPI):# Start access log workerthreading.Thread(target=worker.run,daemon=True).start()yield# Stop access log workerworker.queue_client.put(None)app=FastAPI(lifespan=lifespan,docs_url=None,redoc_url=None,openapi_url=None)proxy.add_route(app,"/chat/completions")Run with uvicorn with some params if you need.$uvicornrun:app--host0.0.0.0--port8080To use Azure OpenAI, instantiateChatGPTProxywithAsyncAzureOpenAI.azure_client=openai.AsyncAzureOpenAI(api_key="YOUR_API_KEY",api_version="2023-10-01-preview",azure_endpoint="https://{DEPLOYMENT_ID}.openai.azure.com/")proxy=ChatGPTProxy(async_client=azure_client,access_logger_queue=worker.queue_client)To use Claude2 on AWS Bedrock, instantiateClaude2Proxy.fromaiproxy.claude2importClaude2Proxyclaude_proxy=Claude2Proxy(aws_access_key_id="YOUR_AWS_ACCESS_KEY_ID",aws_secret_access_key="YOUR_AWS_SECRET_ACCESS_KEY",region_name="your-bedrock-region",access_logger_queue=worker.queue_client)claude_proxy.add_route(app,"/model/anthropic.claude-v2")Client side. We test API with boto3.importboto3importjson# Make client with dummy credssession=boto3.Session(aws_access_key_id="dummy",aws_secret_access_key="dummy",)bedrock=session.client(service_name="bedrock-runtime",region_name="private",endpoint_url="http://127.0.0.1:8000")# Call APIresponse=bedrock.invoke_model(modelId="anthropic.claude-v2",body=json.dumps({"prompt":"Human: うなぎとあなごの違いは？\nAssistant: ","max_tokens_to_sample":100}))# Show responseprint(json.loads(response["body"].read()))🔍 MonitoringBy default, seeaccesslogtable inaiproxy.db. If you want to use other RDBMS like PostgreSQL, set SQLAlchemy-formatted connection string asconnection_strargument when instancingAccessLogWorker.And, you can customize log format as below:This is an example to addusercolumn to request log. In this case, the customized log are stored into table namedcustomaccesslog, the lower case of your custom access log class.fromdatetimeimportdatetimeimportjsonimporttracebackfromsqlalchemyimportColumn,Stringfromaiproxy.accesslogimportAccessLogBase,AccessLogWorkerclassCustomAccessLog(AccessLogBase):user=Column(String)classCustomGPTRequestItem(ChatGPTRequestItem):defto_accesslog(self,accesslog_cls:_AccessLogBase)->_AccessLogBase:accesslog=super().to_accesslog(accesslog_cls)# In this case, set value of "x-user-id" in request header to newly added colmun "user"accesslog.user=self.request_headers.get("x-user-id")returnaccesslog# Use your custom accesslogworker=CustomAccessLogWorker(accesslog_cls=CustomAccessLog)# Use your custom request itemproxy=ChatGPTProxy(access_logger_queue=worker.queue_client,request_item_class=CustomGPTRequestItem)NOTE: By defaultAccessLog, OpenAI API Key in the request headers is masked.🛡️ FilteringThe filter receives all requests and responses, allowing you to view and modify their content. For example:Detect and protect from misuse: From unknown apps, unauthorized users, etc.Trigger custom actions: Doing something triggered by a request.This is an example for custom request filter that protects the service from banned user. uezo will receive "you can't use this service" as the ChatGPT response.fromtypingimportUnionfromaiproxyimportRequestFilterBaseclassBannedUserFilter(RequestFilterBase):asyncdeffilter(self,request_id:str,request_json:dict,request_headers:dict)->Union[str,None]:banned_user=["uezo"]user=request_json.get("user")# Return string message to return response right after this filter ends (not to call ChatGPT)ifnotuser:return"user is required"elifuserinbanned_user:return"you can't use this service"# Enable this filterproxy.add_filter(BannedUserFilter())Try it.resp=client.chat.completions.create(model="gpt-3.5-turbo",messages=messages,user="uezo")print(resp)ChatCompletion(id='-',choices=[Choice(finish_reason='stop',index=0,message=ChatCompletionMessage(content="you can't use this service",role='assistant',function_call=None,tool_calls=None))],created=0,model='request_filter',object='chat.completion',system_fingerprint=None,usage=CompletionUsage(completion_tokens=0,prompt_tokens=0,total_tokens=0))Another example is the model overwriter that forces the user to use GPT-3.5-Turbo.classModelOverwriteFilter(RequestFilterBase):asyncdeffilter(self,request_id:str,request_json:dict,request_headers:dict)->Union[str,None]:request_model=request_json["model"]ifnotrequest_model.startswith("gpt-3.5"):print(f"Change model from{request_model}-> gpt-3.5-turbo")# Overwrite request_jsonrequest_json["model"]="gpt-3.5-turbo"Lastly,ReplayFilterthat retrieves content for a specific request_id from the histories. This is an exceptionally cool feature for developers to test AI-based applications.classReplayFilter(RequestFilterBase):asyncdeffilter(self,request_id:str,request_json:dict,request_headers:dict)->Union[str,None]:# Get request_id to replay from request headerrequest_id=request_headers.get("x-aiproxy-replay")ifnotrequest_id:returndb=worker.get_session()try:# Get and return the response content from historiesr=db.query(AccessLog).where(AccessLog.request_id==request_id,AccessLog.direction=="response").first()ifr:returnr.contentelse:return"Record not found for{request_id}"exceptExceptionasex:logger.error(f"Error at ReplayFilter:{str(ex)}\n{traceback.format_exc()}")return"Error at getting response for{request_id}"finally:db.close()request_idis included in HTTP response headers asx-aiproxy-request-id.NOTE:Responsefilter doesn't work whenstream=True.📊 DashboardWe provide an Apache Superset template as our express dashboard. Please follow the steps below to set up.Install Superset.$pipinstallapache-supersetGet dashboard.zip from release page and extract it to the same directory as aiproxy.db.https://github.com/uezo/aiproxy/releases/tag/v0.3.0Set required environment variables.$exportSUPERSET_CONFIG_PATH=$(pwd)/dashboard/superset_config.py
$exportFLASK_APP=supersetMake database.$supersetdbupgradeCreate admin user. Change username and password as you like.$supersetfabcreate-admin--usernameadmin--firstnameAIProxyAdmin--lastnameAIProxyAdmin--emailadmin@localhost--passwordadminInitialize Superset.$supersetinitImport 🦉AIProxy dashboard template. Execute this command in the same directory as aiproxy.db. If you execute from a different location, open the Database connections page in the Superset after completing these steps and modify the database connection string to the absolute path.$supersetimport-directorydashboard/resourcesStart Superset.$supersetrun-p8088Open and customize the dashboard to your liking, including the metrics you want to monitor and their conditions.👍http://localhost:8088📕 Superset official docs:https://superset.apache.org/docs/intro💡 TipsCORSConfigure CORS if you call API from web apps.https://fastapi.tiangolo.com/tutorial/cors/Retry🦉AIProxy does not retry when API returns 5xx error because the OpenAI official client library retries. Settmax_retriestoChatGPTProxyif you call 🦉AIProxy from general HTTP client library.proxy=ChatGPTProxy(api_key=args.openai_api_key,access_logger_queue=worker.queue_client,max_retries=2# OpenAI's default is 2)DatabaseYou can use other RDBMS that is supported by SQLAlchemy. You can use them by just changing connection string. (and, install client libraries required.)Example for PostgreSQL🐘$pipinstallpsycopg2-binary# connection_str = "sqlite:///aiproxy.db"connection_str=f"postgresql://{USER}:{PASSWORD}@{HOST}:{PORT}/{DATABASE}"worker=AccessLogWorker(connection_str=connection_str)Example for SQL Server or Azure SQL DatabaseThis is a temporary workaroud from AIProxy >= 0.3.6. SetAIPROXY_USE_NVARCHAR=1to use NVARCHAR internally.$exportAIPROXY_USE_NVARCHAR=1Install ODBC driver (version 18 in this example) andpyodbcthen set connection string as follows:# connection_str = "sqlite:///aiproxy.db"connection_str=f"mssql+pyodbc:///?odbc_connect=DRIVER={ODBCDriver18forSQLServer};SERVER=YOUR_SERVER;PORT=1433;DATABASE=YOUR_DB;UID=YOUR_UID;PWD=YOUR_PWD"worker=AccessLogWorker(connection_str=connection_str)RestrictionsFeaturesChatGPTChatGPTStreamClaude2Claude2StreamRequest filter✅✅✅Instant response is non-stream with status code 400Response filter✅N/A✅N/AAPI keysServer/ClientServer/ClientServerServer🛟 SupportFor support, questions, or contributions, please open an issue in the GitHub repository. Please contact me directly when you need an enterprise or business support😊.⚖️ License🦉AIProxyis released under theApache License v2.Made with ❤️ by Uezo, the representive of Unagiken.