Saravanan666/pypi-org huggingface dataset:
Package: databases-extended
DatabasesDatabases gives you simple asyncio support for a range of databases.It allows you to make queries using the powerfulSQLAlchemy Coreexpression language, and provides support for PostgreSQL, MySQL, and SQLite.Databases is suitable for integrating against any async Web framework, such asStarlette,Sanic,Responder,Quart,aiohttp,Tornado, orFastAPI.Documentation:https://www.encode.io/databases/Community:https://discuss.encode.io/c/databasesRequirements: Python 3.6+Installation$pipinstalldatabasesYou can install the required database drivers with:$pipinstalldatabases[postgresql]$pipinstalldatabases[mysql]$pipinstalldatabases[sqlite]Driver support is provided using one ofasyncpg,aiomysql, oraiosqlite.QuickstartFor this example we'll create a very simple SQLite database to run some
queries against.$pipinstalldatabases[sqlite]$pipinstallipythonWe can now run a simple example from the console.Note that we want to useipythonhere, because it supports usingawaitexpressions directly from the console.# Create a database instance, and connect to it.fromdatabasesimportDatabasedatabase=Database('sqlite:///example.db')awaitdatabase.connect()# Create a table.query="""CREATE TABLE HighScores (id INTEGER PRIMARY KEY, name VARCHAR(100), score INTEGER)"""awaitdatabase.execute(query=query)# Insert some data.query="INSERT INTO HighScores(name, score) VALUES (:name, :score)"values=[{"name":"Daisy","score":92},{"name":"Neil","score":87},{"name":"Carol","score":43},]awaitdatabase.execute_many(query=query,values=values)# Run a database query.query="SELECT * FROM HighScores"rows=awaitdatabase.fetch_all(query=query)print('High Scores:',rows)Check out the documentation onmaking database queriesfor examples of how to start using databases together with SQLAlchemy core expressions.— ⭐️ —Databases isBSD licensedcode. Designed & built in Brighton, England.
Package: databases-extensions
databases-query-extensionsPagination, filter, search and sorting extensions for sqlalchey with encode/databasesInstallationpip install databases-query-extensionsExampleConstruct page paramspage_params=PaginationParams.construct(first=3,filter={"completed.is_true":""})apply pagination and get resultsquery=example.select()pagination=CursorPagination(database,query)paginated=awaitpagination.page(page_params)apply filters, search, sorting or pagination and get queryquery = example.select()
    cursor_column="created_date"
    query = query_builder(query, page_params, cursor_column)A complete query params example{"first":100,"after":<cursor>,"afterWith":<cursor>,"search":{"columns":["col1","col2"],"value":"new"},"filter":{"conversationType.eq":"Call""conversationStatus.eq":"NoAanswer"},"sort":"createdAt","order":"asc"}
Package: databases-watcher
Python Database Watcher LibraryIn a nutshellPython Database Watcher Libraryis a small library with a set of utilities to help you to monitor and watch the database changes.Install>pipinstalldatabases_watcherSupported databasesRedisSupported modes:Queue watch mode:redis://[[user]:[password]]@host:port/?db=[INT]&queue=[STRING]Pub/Sub mode:redis+pubsub://[[user]:[password]]@host:port/?db=[INT]&channel=[STRING]Changes watch mode(TODO):redis+watch://[[user]:[password]]@host:port/?db=[INT]&queue=[STRING]TODO: improve watch modeUsage examplesRedisQueue watch mode (Sync mode)fromdatabases_watcherimportconnect_databasedefmain():redis_watch=connect_database("redis://localhost:6501/?db=0&queue=default")redis_watch.send_message("hello!")redis_watch.send_json_message({"message":"hello!"})print(next(redis_watch.read_messages()))print(next(redis_watch.read_json_messages()))if__name__=='__main__':main()Queue watch mode (Async mode)importasynciofromdatabases_watcherimportconnect_database_asyncasyncdefmain():redis_watch=awaitconnect_database_async("redis://localhost:6501/?db=0&queue=default")awaitredis_watch.send_message("hello!")awaitredis_watch.send_json_message({"message":"hello!"})asyncformessageinredis_watch.read_messages():print(message)breakasyncformessageinredis_watch.read_json_messages():print(message)breakif__name__=='__main__':asyncio.run(main())Pub/Sub watch mode (Sync mode)importtimeimportthreadingfromdatabases_watcherimportconnect_databaseCONNECTION_STRING="redis+pubsub://localhost:6501/?db=0&channel=default"defbackground_read_pubsub():redis_watch=connect_database(CONNECTION_STRING)print(next(redis_watch.read_messages()))print(next(redis_watch.read_json_messages()))defmain():redis_watch=connect_database(CONNECTION_STRING)t=threading.Thread(target=background_read_pubsub)t.start()time.sleep(2)redis_watch.send_message("hello!")redis_watch.send_json_message({"message":"hello!"})t.join()if__name__=='__main__':main()Pub/Sub watch mode (Async mode)importasynciofromdatabases_watcherimportconnect_database_asyncCONNECTION_STRING="redis+pubsub://localhost:6501/?db=0&channel=default"asyncdefbackground_read_pubsub():redis_watch=awaitconnect_database_async(CONNECTION_STRING)asyncformessageinredis_watch.read_messages():print(message)breakasyncformessageinredis_watch.read_json_messages():print(message)breakasyncdefmain():redis_watch=awaitconnect_database_async(CONNECTION_STRING)asyncio.create_task(background_read_pubsub())awaitasyncio.sleep(2)awaitredis_watch.send_message("hello!")awaitredis_watch.send_json_message({"message":"hello!"})if__name__=='__main__':asyncio.run(main())LicenseDictionary Search is Open Source and available under theMIT.ContributionsContributions are very welcome. SeeCONTRIBUTING.mdor skim existing tickets to see where you could help out.
Package: database-testing-tools
database-testing-toolsDatabase Testing Tools comprises a suite of generalised tests to run on your database, and produce reports for a notebook. It is designed for pre-production checks for curated databases, but can also be useful for monitoring changes over time. It can be particularly useful when combined withpapermill, which can be used to parameterize, execute and store notebooks as part of your pipeline.Installationpipinstalldatabase_testing_toolsUsageThe package implements aTesterclass, from which you can invoke a number of built-in tests and optional plotting features.fromdatabase_testing_toolsimportTestertest=Tester()database="my_database_name"out=test.check_all_tables_return_data(db_name=database)Tests return two outputs: the result of the test; and some nice graphical representation of the test - either a markdown table, or a plot using theAltairplotting library.The class also collects the results of the tests for an overall report, which you can access with theTester.get_results_table()function.You can finda notebookwhich demonstrates the functions available in the package.Curated database assumptions - SCD2 vs SnapshottingThe tests in this package assume the database being tested is versioned in one of two ways:Slowly Changing Dimensions type 2 (SCD2): where the database includes records of inserts and updates to records. In curated databases a start time and end time will be added to each record. The start time is when the record is either created or last updated; the end time is either when the record was next updated, or a date arbitrarily far in the future to signify that it is the current version of the record. By filtering on these dates, it is possible to view the state of the database at a given point in time.Snapshots: when SCD2 isn't possible or is impractical, versions of the database are recorded with an associated snapshot date based on date of extraction.These methods require different methods to compare versions of the database over time, so there are some separate functions to handle them in their own way.Query enginesBy default, the package usespydbtoolsto run the queries in the package on a given Amazon Athena database. However, it is possible to connect to your own database by making an extension of the BaseQueryEngine class defined inutils.py.An engine forDuckDBis included, and is used for the demo and unit tests in this package. A basic use of the DuckDB engine looks like this:importduckdbfromdatabase_testing_tools.utilsimportDuckDbEnginefromdatabase_testing_toolsimportTesterconn=duckdb.connect()query_engine=DuckDbEngine(conn)tester=Tester(query_engine=query_engine)database="some_database"tester.check_all_tables_return_data(db_name=database)Alternative database engines should follow a similar pattern to the DuckDB engine - an initialiser that includes the database connection object, and an implementation of therun_queryfunction that runs a given SQL query and returns the result as a pandas dataframe. As the built-in tests usethe Trino SQL dialect, it is recommended to transpile the SQL withsqlglotto ensure the tests are executed correctly.
Package: databeakers
beakersbeakers is an experimental lightweight declarative ETL framework for PythonIt is still very much in flux with no correctness or stability guarantees.No contributions yet please, but feel free to poke around/ask questions.Featuresdeclarative ETL graph comprised of Python functions & Pydantic modelsdeveloper-friendly CLI for running processessync/async task executiondata checkpoints stored in local database for intermediate caching & resuming interrupted runsrobust error handling, including retriesGuiding PrinciplesLightweight- Writing a single python file should be enough to get started. It should be as easy to use as a script in that sense.Data-centric- Know what data is added at each step.Modern Python- Take full advantage of recent additions to Python, including type hints,asyncio, and libraries likepydanticandrich.Developer Experience- Focused on the developer experience: a nice CLI, helpful error messages.Anti-PrinciplesUnlike most tools in this space, this is not a complete "enterprise grade" ETL solution.It isn't a perfect analogy by any means but it could be saiddatabeakersis toluigiwhatflaskis toDjango.
If you are building your entire business around ETL, it makes sense to invest in the infrastructure & tooling to make that work.
Maybe structuring your code around beakers will make it easier to migrate to one of those tools than if you had written a bespoke script.
Plus, beakers is Python, so you can always start by running it from within a bigger framework.ConceptsLike most ETL tools, beakers is built around a directed acyclic graph (DAG).The nodes on this graph are known as "beakers", and the edges are often called "transforms".(Note: These names aren't final, suggestions welcome.)BeakersEach node in the graph is called a "beaker". A beaker is a container for some data.Each beaker has a name and a type.
The name is used to refer to the beaker elsewhere in the graph.
The type, represented by apydanticmodel, defines the structure of the data. By leveragingpydanticwe get a lot of nice features for free, like validation and serialization.TransformEdges in the graph represent dataflow between beakers. Each edge has a concept of a "source beaker" and a "destination beaker".These come in two main flavors:Transforms- A transform places new data in the destination beaker based on data already in the source beaker.
An example of this might be a transform that takes a list of URLs and downloads the HTML for each one, placing the results in a new beaker.Filter- A filter can be used to stop the flow of data from one beaker to another based on some criteria.SeedA concept somewhat unique to beakers is the "seed". A seed is a function that returns initial data for a beaker.This is useful for things like starting the graph with a list of URLs to scrape, or a list of images to process.A beaker can have any number of seeds, for example one might have a short list of URLs to use for testing, and another that reads from a database.
Package: databeaver
DataBeaver - Data Model OrchestrationDataBeaver is a tool that allows teams to easily realize, test, version, and share their data models.OverviewThe aim of data beaver is to create a deterministic process for realizing a given data model.
Data beaver can be used as easily a command line utility or integrated into your application via the DataBeaver class.Installing DataBeaverTo install Data Beaver you can run the command below.pipinstalldatabeaverUsing Data BeaverWhat Can I Do With Data BeaverBuild your data model - Traditionally creating your data was a labor intensive process. First you would write some sql
then you would run the manually one by one. With Data Beaver a simplebeaver buildcommand will generate all the models
in their correct order, respecting dependencies.(Not Yet) Visualize Model Dependencies - In a large model it can be tricky to keep track on what tables depend on what
other tables. Instead of trying to keep track of that yourself, let DataBeaver extract it directly from the sql and generate
and image file for you all with the simplebeaver visualizecommandUsing DataBeaver as a command line applicationStep 1 - Create a New ProjectBefore we start we need to create the basic directory structure needed and a configuration file that can be edited.Commandbeavercreate-project--name=<projectName>Execute a modelData Beaver can be used as either a command line application or as a module for a more direct integration.Module UsagefromdatabeaverimportDataModelmodel=DataModel()model.build()Major ReleasesVersionGoalStatus0.1.0Add the command line utility 'beaver' and 'DataBeaver' classIn Progress0.2.0Build a Model against in Postgres0.3.0Build database models in MySQLMinor ReleasesVersionPurpose0.0.37DataBeaver.build() now returns info on the models and files that were processedAdditional DocumentationClassesConfiguration
Package: data-bench
data-bench-pythonis a python package used to develop consumer and
producer agents found in theData Benchworkload. This module
does not install Data Bench.InstallInstall with pip:$pipinstalldata_benchOr install with setup.py:$gitclonehttps://github.com/Data-Bench/data-bench-python.git$cddata-bench-python&&python3setup.pyinstallUninstall with pip:$pipuninstalldata_bench
Package: data-bench-faker
data_bench_fakeris a provider for theFakerPython package
which creates stable, scalable data for theData Benchbenchmark.InstallInstall with pip:$pipinstalldata-bench-fakerOr install with setup.py:$gitclonehttps://github.com/erikoshaughnessy/data-bench-faker.git$cddata-bench-faker&&python3setup.pyinstallUninstall with pip:$pipuninstalldata-bench-fakerUsageAdd theDataProviderto yourFakerinstance:fromfakerimportFakerfromdata_bench_fakerimportDataProviderfake=Faker()fake.add_provider(DataProvider)customer=fake.customer()account=fake.customer_account(customer)company=fake.Company()# note fake.company() creates a company name.View the methods defined by DataProvider:$pydoc3data_bench_faker.DataProvider
Package: databend
Databend Python BindingThis crate intends to build a native python binding.InstallationpipinstalldatabendUsageBasic:fromdatabendimportSessionContextctx=SessionContext()df=ctx.sql("select number, number + 1, number::String as number_p_1 from numbers(8)")# convert to pyarrowdf.to_py_arrow()# convert to pandasdf.to_pandas()Register external table:supported functions:register_parquetregister_ndjsonregister_csvregister_tsvctx.register_parquet("pa","/home/sundy/dataset/hits_p/",pattern=".*.parquet")ctx.sql("select * from pa limit 10").collect()Tenant separation:ctx=SessionContext(tenant="a")DevelopmentSetup virtualenv:python-mvenv.venvActivate venv:source.venv/bin/activateInstallmaturin:pipinstall"maturin[patchelf]"Build bindings:maturindevelopRun tests:maturindevelop-EtestBuild API docs:maturindevelop-Edocs
pdocdatabendStorage configurationMeta Storage directory(Catalogs, Databases, Tables, Partitions, etc.):./.databend/_metaData Storage directory:./.databend/_dataCache Storage directory:./.databend/_cacheLogs directory:./.databend/logsMoreDatabend python api is inspired byarrow-datafusion-python, thanks for their great work.
Package: databend-driver
databend-driverBuildcdbindings/python
maturindevelopUsageBlockingfromdatabend_driverimportBlockingDatabendClientclient=BlockingDatabendClient('databend+http://root:root@localhost:8000/?sslmode=disable')conn=client.get_conn()conn.exec("""CREATE TABLE test (i64 Int64,u64 UInt64,f64 Float64,s   String,s2  String,d   Date,t   DateTime)""")rows=conn.query_iter("SELECT * FROM test")forrowinrows:print(row.values())Asyncioimportasynciofromdatabend_driverimportAsyncDatabendClientasyncdefmain():client=AsyncDatabendClient('databend+http://root:root@localhost:8000/?sslmode=disable')conn=awaitclient.get_conn()awaitconn.exec("""CREATE TABLE test (i64 Int64,u64 UInt64,f64 Float64,s   String,s2  String,d   Date,t   DateTime)""")rows=awaitconn.query_iter("SELECT * FROM test")asyncforrowinrows:print(row.values())asyncio.run(main())APIsAsyncDatabendClientclassAsyncDatabendClient:def__init__(self,dsn:str):...asyncdefget_conn(self)->AsyncDatabendConnection:...AsyncDatabendConnectionclassAsyncDatabendConnection:asyncdefinfo(self)->ConnectionInfo:...asyncdefversion(self)->str:...asyncdefexec(self,sql:str)->int:...asyncdefquery_row(self,sql:str)->Row:...asyncdefquery_iter(self,sql:str)->RowIterator:...asyncdefstream_load(self,sql:str,data:list[list[str]])->ServerStats:...BlockingDatabendClientclassBlockingDatabendClient:def__init__(self,dsn:str):...defget_conn(self)->BlockingDatabendConnection:...BlockingDatabendConnectionclassBlockingDatabendConnection:definfo(self)->ConnectionInfo:...defversion(self)->str:...defexec(self,sql:str)->int:...defquery_row(self,sql:str)->Row:...defquery_iter(self,sql:str)->RowIterator:...defstream_load(self,sql:str,data:list[list[str]])->ServerStats:...RowclassRow:defvalues(self)->tuple:...RowIteratorclassRowIterator:defschema(self)->Schema:...def__iter__(self)->RowIterator:...def__next__(self)->Row:...def__aiter__(self)->RowIterator:...asyncdef__anext__(self)->Row:...FieldclassField:@propertydefname(self)->str:...@propertydefdata_type(self)->str:...SchemaclassSchema:deffields(self)->list[Field]:...ServerStatsclassServerStats:@propertydeftotal_rows(self)->int:...@propertydeftotal_bytes(self)->int:...@propertydefread_rows(self)->int:...@propertydefread_bytes(self)->int:...@propertydefwrite_rows(self)->int:...@propertydefwrite_bytes(self)->int:...@propertydefrunning_time_ms(self)->float:...ConnectionInfoclassConnectionInfo:@propertydefhandler(self)->str:...@propertydefhost(self)->str:...@propertydefport(self)->int:...@propertydefuser(self)->str:...@propertydefdatabase(self)->str|None:...@propertydefwarehouse(self)->str|None:...Developmentcd tests
make upcdbindings/python
pipenvinstall--dev
pipenvrunmaturindevelop
pipenvrunbehavetests/*
Package: databend-py
# databend-pyDatabend Cloud Python Driver with native http interface support[![image](https://img.shields.io/pypi/v/databend-py.svg)](https://pypi.org/project/databend-py)[![image](https://coveralls.io/repos/github/databendcloud/databend-py/badge.svg?branch=master)](https://coveralls.io/github/databendcloud/databend-py?branch=master)[![image](https://img.shields.io/pypi/l/databend-py.svg)](https://pypi.org/project/databend-py)[![image](https://img.shields.io/pypi/pyversions/databend-py.svg)](https://pypi.org/project/databend-py)# Installationpip install databend-py# UsageUse the next code to check connection:>` python > >>> from databend_py import Client > >>> client = Client( >host='tenant--warehouse.ch.datafusecloud.com',>database="default",>user="user",>port="443",>password="password",settings={"copy_purge":True,"force":True})> >>>print(client.execute("SELECT1"))> `The [host]{.title-ref}, [user]{.title-ref}, [password]{.title-ref} info
will be found in databend cloud warehouse connect page as flows:Pure Client example:>` python > >>> from databend_py import Client > >>> > >>> client =Client.from_url('http://root@localhost:8000/db?secure=False&copy_purge=True')> >>> > >>>client.execute('SHOWTABLES') >[[],[('test',)]]> >>>client.execute("showtables",with_column_types=True)>[[('Tables_in_default','String')],[('test',)]]#[[(column_name,column_type)],[(data,)]]> >>>client.execute('DROPTABLE IF EXISTS test') > [] > >>>client.execute('CREATETABLE test (xInt32)')> [] > >>> client.execute( > ...     'INSERT INTO test (x) VALUES',[(1,)]> ... ) > 1 > >>>client.execute('INSERTINTO test (x) VALUES',[(200,)])> 1 > `More usages examples find [here](./examples).# FeaturesBasic SQL.TLS support.Query settings.Types support:
-   Float32/64
-   [U]Int8/16/32/64/128/256
-   Date/Date32/DateTime('timezone')/DateTime64('timezone')
-   String
-   Array(T)
-   Nullable(T)
-   Bool# CompatibilityIf databend version >= v0.9.0 or later, you need to use databend-py
version >= v0.3.0.# LicenseDatabend Python Driver is distributed under the [Apache
license]{.title-ref}.
Package: databend-sqlalchemy
databend-sqlalchemyDatabend dialect for SQLAlchemy.InstallationThe package is installable through PIP::pip install databend-sqlalchemyUsageThe DSN format is similar to that of regular Postgres::from sqlalchemy import create_engine, text
    from sqlalchemy.engine.base import Connection, Engine
    engine = create_engine(
        f"databend://{username}:{password}@{host_port_name}/{database_name}?sslmode=disable"
    )
    connection = engine.connect()
    result = connection.execute(text("SELECT 1"))
    assert len(result.fetchall()) == 1

    import connector
    cursor = connector.connect('databend://root:@localhost:8000?sslmode=disable').cursor()
    cursor.execute('SELECT * FROM test')
    # print(cursor.fetchone())
    # print(cursor.fetchall())
    for i in cursor.next():
        print(i)CompatibilityIf databend version >= v0.9.0 or later, you need to use databend-sqlalchemy version >= v0.1.0.The databend-sqlalchemy usedatabend-pyas internal driver when version < v0.4.0, but when version >= v0.4.0 it usedatabend driver python bindingas internal driver. The only difference between the two is that the connection parameters provided in the DSN are different. When using the corresponding version, you should refer to the connection parameters provided by the corresponding Driver.