Package: yodo1-toolkit
Yodo1 Python ToolkitYodo1 Python ToolkitInstallSSOSetupUser with FastAPIsqlalchemyUse with FastAPIUse without FastAPIDefine ModelDefine SchemaRabbit MQHow to use ConsumerConsume MQ with apm enabledHow to use SenderSend MQ with apm enabledSend MQ with FastAPI apm enabledProgress BarInstallpipinstallyodo1-toolkitSSOSetupCreate base instance onapp/base.pyfile.fromtypingimportDictfromfastapiimportDependsfromyodo1.ssoimportJWTHelper,JWTPayloadauth=JWTHelper()# Define helper class# This is to add custom operation after get user info, like setup APM context# https://www.elastic.co/guide/en/apm/agent/python/master/api.html#api-set-user-contextdefget_current_user_dict(payload:JWTPayload=Depends(auth.current_payload))->Dict:elasticapm.set_user_context(username=payload.name,email=payload.email,user_id=payload.sub)return{"sub":payload.sub,"email":payload.email,}Setup public key when api startupfromfastapiimportFastAPIfromapp.baseimportauthapp=FastAPI()@app.on_event("startup")asyncdefstartup_event()->None:# Setup public key via sso verser url, this is path to public_key file.auth.setup_with_sso_server("<public_key_url>")# Or just setup with piblic_key str# >>> public_key = "-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BA ..."# >>> auth.setup_keys(public_key=public_key)User with FastAPIfromapp.baseimportauth,get_current_user_dictfromyodo1.ssoimportJWTPayload...# use payload without custom model@router.get("/get_payload")asyncdefget_payload(payload:JWTPayload=Depends(auth.current_payload)):returnf"Hello,{payload.sub}"# use custom user model@router.get("/user")asyncdefget_payload(payload:Dict=Depends(get_current_user_dict)):returnf"Hello,{payload["sub"]}"sqlalchemyCreate base instances onapp/base.pyfile.fromsqlalchemyimportcreate_enginefromyodo1.sqlalchemyimportDBManagerengine=create_engine("<db_rui>",pool_size=0,pool_recycle=600,max_overflow=-1)db=DBManager(engine=engine)Use with FastAPIfromapp.baseimportdb...@router.post("/update")asyncdefhello_world(payload=Depends(auth.current_payload),session=Depends(db.get_session))->ItemModel:item:ItemModel=session.query(ItemModel).first()returnitem...Use without FastAPIfromapp.baseimportdbsession=db.SessionLocal()session.query(Model).all()session.commit()session.close()Define Modelfromsqlalchemyimport(INTEGER,Column,TEXT)fromyodo1.sqlalchemyimportBaseDBModelclassItemModel(BaseDBModel):__tablename__="item_list"__table_args__={"extend_existing":True}id=Column(INTEGER,primary_key=True,autoincrement=True,nullable=False)title=Column(TEXT,nullable=False,comment="notification title")Define Schemafromyodo1.pydanticimportBaseSchema,BaseDateSchemaclassOutputModelSchema(BaseSchema):id:inttitle:strclassOutputModelWithDateSchema(BaseDateSchema):id:inttitle:strRabbit MQHow to use Consumerimportloggingimportrandomimporttimeimportpikafromyodo1.rabbitmq.multi_threadimportMultiThreadConsumer,MQAction,CallbackResult# We can change pika log level to reduce logs.logging.getLogger("pika").setLevel(logging.INFO)logging.basicConfig(level="DEBUG")defdemo_callback(method_frame:pika.spec.Basic.Deliver,header_frame:pika.spec.BasicProperties,message_body:bytes)->CallbackResult:"""Demo callback function:param method_frame: method_frame from MQ Message:param header_frame: header_frame from MQ Message:param message_body: MQ Message body:return: whether should ack"""logging.info(f"Received message in Queue:{method_frame.routing_key}, delivery_tag:{method_frame.delivery_tag}")time.sleep(30)ifrandom.random()>0.5:# Failed to process, should nack with `requeue=False`returnCallbackResult(MQAction.ack)else:# Process success, should ackreturnCallbackResult(MQAction.nack)consumer=MultiThreadConsumer(uri="amqps://xxxx",verbose=True)consumer.setup_queue_consumer(queue_name="test.consumer.a.debug",exchange_name="target-exchange",handler_function=demo_callback)try:consumer.start_consuming()exceptKeyboardInterrupt:consumer.stop_consuming()consumer.close()Consume MQ with apm enabledimportelasticapmapm_client=elasticapm.Client(service_name="awesome-api",server_url="https://apm-host",secret_token="token",environment="test",service_version="2.0.0")consumer=MultiThreadConsumer(uri="amqps://xxxx",apm_client=apm_client,verbose=True)AsyncRabbitis Deprecated due to stability, will remove from version 0.3.0. Please useyodo1.rabbitmq.MultiThreadConsumerHow to use SenderRabbitHttpSenderis a thread safe sender with send MQ directly using the HTTP client.importjsonimportaio_pikafromyodo1.rabbitmqimportRabbitHttpSender# Recommend to share one sender client for each app worker# Init with URIuri="https://username:password@rabbit-host/virtualhost"rabbit_sender=RabbitHttpSender(uri=uri)# Make sure we have defined target queue and exchange relation on the startup@app.on_event("startup")asyncdefstartup_event()->None:# Register the exchange.# We need to define the relation in the coderabbit_sender.declare_exchange(exchange_name="only-queue")defdo_some_magic_and_publish_to_exchange():do_magic()rabbit_sender.publish(exchange_name="exchange-1",message_body={"magic":"done"})defdo_some_magic_and_publish_to_queue_withou_exchange():do_magic()# We can publish message directly to a queue using special exchange ""rabbit_sender.publish(exchange_name="",routing_key="target-queue-name",message_body={"magic":"done"})Send MQ with apm enabledimportelasticapmapm_client=elasticapm.Client(service_name="awesome-api-consumer",server_url="https://apm-host",secret_token="token",environment="test",service_version="2.0.0")# init sender with apm clienturi="https://username:password@rabbit-host/virtualhost"rabbit_sender=RabbitHttpSender(uri=uri)rabbit_sender.publish(event_name="alian-found",# Must have a event name when using apm clientexchange_name="",routing_key="target-queue-name",message_body={"magic":"done"})rabbit_sender.publish(event_name="alian-found",# Must have a event name when using apm clientexchange_name="exchange-1",message_body={"magic":"done"})Send MQ with FastAPI apm enabledfromelasticapm.contrib.starletteimportElasticAPM,make_apm_clientfromfastapiimportFastAPIapm_client=make_apm_client({"SERVICE_NAME":"demo-api","SECRET_TOKEN":"xxxx","SERVER_URL":"https://apm-host","ENVIRONMENT":"test","SERVICE_VERSION":"1.0.0",})app=FastAPI()app.add_middleware(ElasticAPM,client=apm_client)# init sender with apm clienturi="https://username:password@rabbit-host/virtualhost"rabbit_sender=RabbitHttpSender(uri=uri,apm_client=apm_client)rabbit_sender.publish(event_name="alian-found",# Must have a event name when using apm clientexchange_name="exchange-1",message_body={"magic":"done"})Progress BarA simple progress bar can display properly on k8s and Grafana.importloggingfromyodo1.progressimportProgressBarlogging.basicConfig(level="DEBUG")p=ProgressBar(total=100,desc="Hacking ...",step=5)foriinrange(100):p.update()This is the outputINFO:yodo1.progress: 12.0%  |>>>>>>                                            | 12/100 Hacking ...
INFO:yodo1.progress: 24.0%  |>>>>>>>>>>>>                                      | 24/100 Hacking ...
INFO:yodo1.progress: 36.0%  |>>>>>>>>>>>>>>>>>>                                | 36/100 Hacking ...
INFO:yodo1.progress: 48.0%  |>>>>>>>>>>>>>>>>>>>>>>>>                          | 48/100 Hacking ...
INFO:yodo1.progress: 60.0%  |>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>                    | 60/100 Hacking ...
INFO:yodo1.progress: 72.0%  |>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>              | 72/100 Hacking ...
INFO:yodo1.progress: 84.0%  |>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>        | 84/100 Hacking ...
INFO:yodo1.progress: 96.0%  |>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  | 96/100 Hacking ...
INFO:yodo1.progress: 100.0% |>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>| 100/100 Hacking ...Use it with ThreadPoolExecutor.progress=ProgressBar(total=1000,desc="Processing... ")withThreadPoolExecutor(max_workers=20)asexecutor:forindex,rowindf.iterrows():future=executor.submit(do_somethingid=row.id)# Update progress bar when the job donefuture.add_done_callback(lambdax:progress.update())