Package: dataligo
DataLigoThis library helps to read and write data from most of the data sources. It accelerate the ML and ETL process without worrying about the multiple data connectors.Installationpipinstall-UdataligoInstall from sourcesAlternatively, you can also clone the latest version from therepositoryand install it directly from the source code:pipinstall-e.Quick tour>>>fromdataligoimportLigo>>>fromtransformersimportpipeline>>>ligo=Ligo('./ligo_config.yaml')# Check the sample_ligo_config.yaml for reference>>>print(ligo.get_supported_data_sources_list())['s3','gcs','azureblob','bigquery','snowflake','redshift','starrocks','postgresql','mysql','oracle','mssql','mariadb','sqlite','elasticsearch','mongodb','dynamodb','redis']>>>mongodb=ligo.connect('mongodb')>>>df=mongodb.read_as_dataframe(database='reviewdb',collection='reviews',return_type='pandas')# Default return_type is pandas>>>df.head()_idReview064272bb06a14f52787e0a09egoodandinteresting164272bb06a14f52787e0a09fThisclassisveryhelpfultome.Currently,I...264272bb06a14f52787e0a0a0like!ProfandTAsarehelpfulandthediscussi...364272bb06a14f52787e0a0a1Easytofollowandincludesalotbasicandim...464272bb06a14f52787e0a0a2Reallyniceteacher!Icouldgotthepointeazl...>>>classifier=pipeline("sentiment-analysis")>>>reviews=df.Review.tolist()>>>results=classifier(reviews,truncation=True)>>>forresultinresults:>>>print(f"label:{result['label']}, with score:{round(result['score'],4)}")label:POSITIVE,withscore:0.9999label:POSITIVE,withscore:0.9997label:POSITIVE,withscore:0.9999label:POSITIVE,withscore:0.999label:POSITIVE,withscore:0.9967>>>df['predicted_label']=[result['label']forresultinresults]>>>df['predicted_score']=[round(result['score'],4)forresultinresults]# Write the results to the MongoDB>>>mongodb.write_dataframe(df,'reviewdb','review_sentiments')Example DataLigo PipelineETL PipelineML PipelineSupported ConnectorsData SourcesTypepandaspolarsdaskS3datalake[x] read[x] write[x] read[x] write[ ] read[ ] writeGCSdatalake[x] read[x] write[x] read[x] write[ ] read[ ] writeAzure Blob Storagedatalake[x] read[x] write[x] read[x] write[ ] read[ ] writeSnowflakedatawarehouse[x] read[x] write[x] read[x] write[ ] read[ ] writeBigQuerydatawarehouse[x] read[x] write[x] read[x] write[x] read[ ] writeStarRocksdatawarehouse[x] read[x] write[x] read[x] write[x] read[ ] writeRedshiftdatawarehouse[x] read[x] write[x] read[x] write[x] read[ ] writePostgreSQLdatabase[x] read[x] write[x] read[x] write[x] read[ ] writeMySQLdatabase[x] read[x] write[x] read[x] write[x] read[ ] writeMariaDBdatabase[x] read[x] write[x] read[x] write[x] read[ ] writeMsSQLdatabase[x] read[x] write[x] read[x] write[x] read[ ] writeOracledatabase[x] read[x] write[x] read[x] write[x] read[ ] writeSQLitedatabase[x] read[x] write[x] read[x] write[x] read[ ] writeMongoDBnosql[x] read[x] write[x] read[x] write[ ] read[ ] writeElasticSearchnosql[x] read[x] write[x] read[x] write[ ] read[ ] writeDynamoDBnosql[x] read[x] write[x] read[x] write[ ] read[ ] writeRedis(beta)nosql[x] read[ ] write[ ] read[ ] write[ ] read[ ] writeAcknowledgementSome functionalities of DataLigo are inspired by the following packages.ConnectorXDataLigo used Connectorx to read data from most of the RDBMS databases to utilize the performance benefits and inspired the return_type parameter from itdynamo-pandasDataLigo used dynamo-pandas to read and write data from DynamoDB