Saravanan666/pypi-org huggingface dataset:
Package: clickhouse-connect-pure
ClickHouse ConnectA high performance core database driver for connecting ClickHouse to Python, Pandas, and SupersetPandas DataFramesNumpy ArraysPyArrow TablesSuperset ConnectorSQLAlchemy 1.3 and 1.4 (limited feature set)ClickHouse Connect currently uses the ClickHouse HTTP interface for maximum compatibility.Installationpip install clickhouse-connectClickHouse Connect requires Python 3.7 or higher.Superset ConnectivityClickHouse Connect is fully integrated with Apache Superset.  Previous versions of ClickHouse Connect utilized a
dynamically loaded Superset Engine Spec, but as of Superset v2.1.0 the engine spec was incorporated into the main
Apache Superset project and removed from clickhouse-connect in v0.6.0.  If you have issues connecting to earlier
versions of Superset, please use clickhouse-connect v0.5.25.When creating a Superset Data Source, either use the provided connection dialog, or a SqlAlchemy DSN in the formclickhousedb://{username}:{password}@{host}:{port}.SQLAlchemy ImplementationClickHouse Connect incorporates a minimal SQLAlchemy implementation (without any ORM features) for compatibility with
Superset.  It has only been tested against SQLAlchemy versions 1.3.x and 1.4.x, and is unlikely to work with more
complex SQLAlchemy applications.Complete DocumentationThe documentation for ClickHouse Connect has moved toClickHouse Docs
Package: clickhouse-ddl
ClickHouse DDL ParserDescriptionClickhouse metadata (DDL) grabberDependenciesPython 3clickhouse-driverInstallationpipinstallclickhouse_ddlUsageExample script in this project:Copyexport.json.examplefile toexport.jsonSet your Clickhouse database connection paramsRunexport.pyCode:#!/usr/local/bin/python# -*- coding: utf-8 -*-importosfromclickhouse_ddl.ConfigimportConfigfromclickhouse_ddl.ClickHouseDDLimportClickHouseDDLif__name__=='__main__':Config.Parse({"threads":8,"path_git":"git","new_line":"\n","connect":{"host":"localhost","port":9000,"database":"default","username":"user_name","password":"pass_word"}})# Export to folderpath_result=os.path.join(os.path.dirname(os.path.abspath(__file__)),"result")ddl=ClickHouseDDL(path_result)LinksGitHubPyPI
Package: clickhouse-django-logger
clickhouse-django-loggerОписаниеЭтот проект предоставляет возможность логировать ошибки Django в базу данных Clickhouse.
Логи записываются через отложенные задачи в celery, в отдельной очереди, чтобы не нагружать другие воркеры.УстановкаУстановите через pip:pipinstallclickhouse-django-logger# Установка стабильной версииpipinstall-Ugit+https://github.com/single-service/clickhouse_django_logger.git@master# Установка версии разработкиДобавьте clickhouse_django_logger в INSTALLED_APPS:INSTALLED_APPS=INSTALLED_APPS+("clickhouse_django_logger",)Установите переменные среды для Clickhouse logger в settings.py:# Обязательные переменныеDJANGO_CLICKHOUSE_LOGGER_HOST="clickhouse"DJANGO_CLICKHOUSE_LOGGER_PORT=8123DJANGO_CLICKHOUSE_LOGGER_USER="default"DJANGO_CLICKHOUSE_LOGGER_PASSWORD="default"# Необязательные переменные# Для создания отдельной очереди для celery (логи будут писаться только в очереди указанной тут)# если переменная не указана, то будет использована стандартная очередьUSE_LOGGER_QUEUE_NAME="logger"# Для сортировки логов например по контурам (local, development, release)DJANGO_CLICKHOUSE_SERVER_NAME="release"# по умолчанию developmentДобавьте Clickhouse logger в конфигурацию логгера в settings.py:LOGGING={"version":1,"disable_existing_loggers":False,"formatters":{'verbose':{'format':'%(levelname)s %(asctime)s %(module)s %(message)s'},'simple':{'format':'%(levelname)s %(message)s'},},"handlers":{'console':{'level':'DEBUG','class':'logging.StreamHandler','formatter':'simple'},'error':{'level':'INFO','class':'logging.handlers.WatchedFileHandler','filename':'logs/error.log','formatter':'verbose'},'mail_admins':{'level':'ERROR','class':'django.utils.log.AdminEmailHandler',},'info':{'level':'INFO','class':'logging.handlers.WatchedFileHandler','filename':'logs/info.log','formatter':'verbose'},"django_clickhouse_logger":{"level":"INFO","class":"clickhouse_django_logger.handlers.LoggerHandler",},},"loggers":{'django.request':{'handlers':['mail_admins','django_clickhouse_logger','console'],'level':'ERROR','propagate':False,},'django.template':{'handlers':['django_clickhouse_logger','mail_admins'],'level':'ERROR','propagate':False,},'info':{'handlers':['django_clickhouse_logger'],'level':'INFO',},'celery':{'handlers':['console','django_clickhouse_logger'],'level':'ERROR','propagate':True},},}КонтактыЕсли у вас есть вопросы или предложения, не стесняйтесь связаться со мной по электронной почте:singleservice2022@gmail.com
Package: clickhouse-driver
ClickHouse Python DriverClickHouse Python Driver with native (TCP) interface support.Asynchronous wrapper is available here:https://github.com/mymarilyn/aiochFeaturesExternal data for query processing.Query settings.Compression support.TLS support.Types support:Float32/64[U]Int8/16/32/64/128/256Date/Date32/DateTime(‘timezone’)/DateTime64(‘timezone’)String/FixedString(N)Enum8/16Array(T)Nullable(T)BoolUUIDDecimalIPv4/IPv6LowCardinality(T)SimpleAggregateFunction(F, T)Tuple(T1, T2, …)NestedMap(key, value)Query progress information.Block by block results streaming.Reading query profile info.Receiving server logs.Multiple hosts support.Python DB API 2.0 specification support.Optional NumPy arrays support.DocumentationDocumentation is available athttps://clickhouse-driver.readthedocs.io.UsageThere are two ways to communicate with server:using pure Client;using DB API.Pure Client example:>>>fromclickhouse_driverimportClient>>>>>>client=Client('localhost')>>>>>>client.execute('SHOW TABLES')[('test',)]>>>client.execute('DROP TABLE IF EXISTS test')[]>>>client.execute('CREATE TABLE test (x Int32) ENGINE = Memory')[]>>>client.execute(...'INSERT INTO test (x) VALUES',...[{'x':100}]...)1>>>client.execute('INSERT INTO test (x) VALUES',[[200]])1>>>client.execute(...'INSERT INTO test (x) '...'SELECT * FROM system.numbers LIMIT%(limit)s',...{'limit':3}...)[]>>>client.execute('SELECT sum(x) FROM test')[(303,)]DB API example:>>>fromclickhouse_driverimportconnect>>>>>>conn=connect('clickhouse://localhost')>>>cursor=conn.cursor()>>>>>>cursor.execute('SHOW TABLES')>>>cursor.fetchall()[('test',)]>>>cursor.execute('DROP TABLE IF EXISTS test')>>>cursor.fetchall()[]>>>cursor.execute('CREATE TABLE test (x Int32) ENGINE = Memory')>>>cursor.fetchall()[]>>>cursor.executemany(...'INSERT INTO test (x) VALUES',...[{'x':100}]...)>>>cursor.rowcount1>>>cursor.executemany('INSERT INTO test (x) VALUES',[[200]])>>>cursor.rowcount1>>>cursor.execute(...'INSERT INTO test (x) '...'SELECT * FROM system.numbers LIMIT%(limit)s',...{'limit':3}...)>>>cursor.rowcount0>>>cursor.execute('SELECT sum(x) FROM test')>>>cursor.fetchall()[(303,)]LicenseClickHouse Python Driver is distributed under theMIT license.
Package: clickhouse-driver-decorators
clickhouse-driver-decoratorsBunch of decorators to decorateclickhouse-driver'sexecutemethodAvailable decorators:add_column_names-- converts each row to dict or namedtuple with column names as keys;apply_callback-- applies a given callback to each row of result;convert_string_to_datetime-- converts string datetime to python datetime object in a needed timezone for a given column. String datetime have to be in UTC;convert_timestamp_to_datetime-- converts timestamp to python datetime object in a needed timezone for a given column.transform_to_pandas_dataframe-- creates pandas frame based on result from dbExamplesFor a closer look, check out thetestsfromtypingimportGeneratorfrompandasimportDataFramefromclickhouse_driverimportClientfromclickhouse_driver_decoratorsimport*defreplace_empty_with_none(row:tuple)->Generator:yieldtuple([Noneifv==''elsevforvinrow])classA:def__init__(self,client:Client):self.__client=clientdefget_data(self)->Generator[]:returnself.__decorated_execute("""selecttoUnixTimestamp(now()) as now_ts,formatDateTime(now(), '%F%T', 'UTC') as now_str,'' as empty""")@add_column_names()@apply_callback(on_row_callable=replace_empty_with_none)@convert_timestamp_to_datetime(columns_to_convert={'now_ts':'Europe/Moscow'})@convert_string_to_datetime(date_format='%Y-%m-%d%H:%M:%S',columns_to_convert={'now_str':'Europe/Moscow'})def__decorated_execute(self,query:str):returnself.__client.execute(query,with_column_types=True)classB:def__init__(self,client:Client):self.__client=clientdefget_pandas_df(self)->DataFrame:returnself.__decorated_execute("select 'val1' as col1, 'val2' as col2")@transform_to_pandas_dataframe()def__decorated_execute(self,query:str):returnself.__client.execute(query,with_column_types=True)
Package: clickhouse-driver-fork-0-2-4
ClickHouse Python DriverClickHouse Python Driver with native (TCP) interface support.Asynchronous wrapper is available here:https://github.com/mymarilyn/aiochFeaturesExternal data for query processing.Query settings.Compression support.TLS support.Types support:Float32/64[U]Int8/16/32/64/128/256Date/Date32/DateTime(‘timezone’)/DateTime64(‘timezone’)String/FixedString(N)Enum8/16Array(T)Nullable(T)BoolUUIDDecimalIPv4/IPv6LowCardinality(T)SimpleAggregateFunction(F, T)Tuple(T1, T2, …)NestedMap(key, value)Query progress information.Block by block results streaming.Reading query profile info.Receiving server logs.Multiple hosts support.Python DB API 2.0 specification support.Optional NumPy arrays support.DocumentationDocumentation is available athttps://clickhouse-driver.readthedocs.io.UsageThere are two ways to communicate with server:using pure Client;using DB API.Pure Client example:>>>fromclickhouse_driverimportClient>>>>>>client=Client('localhost')>>>>>>client.execute('SHOW TABLES')[('test',)]>>>client.execute('DROP TABLE IF EXISTS test')[]>>>client.execute('CREATE TABLE test (x Int32) ENGINE = Memory')[]>>>client.execute(...'INSERT INTO test (x) VALUES',...[{'x':100}]...)1>>>client.execute('INSERT INTO test (x) VALUES',[[200]])1>>>client.execute(...'INSERT INTO test (x) '...'SELECT * FROM system.numbers LIMIT%(limit)s',...{'limit':3}...)[]>>>client.execute('SELECT sum(x) FROM test')[(303,)]DB API example:>>>fromclickhouse_driverimportconnect>>>>>>conn=connect('clickhouse://localhost')>>>cursor=conn.cursor()>>>>>>cursor.execute('SHOW TABLES')>>>cursor.fetchall()[('test',)]>>>cursor.execute('DROP TABLE IF EXISTS test')>>>cursor.fetchall()[]>>>cursor.execute('CREATE TABLE test (x Int32) ENGINE = Memory')>>>cursor.fetchall()[]>>>cursor.executemany(...'INSERT INTO test (x) VALUES',...[{'x':100}]...)>>>cursor.rowcount1>>>cursor.executemany('INSERT INTO test (x) VALUES',[[200]])>>>cursor.rowcount1>>>cursor.execute(...'INSERT INTO test (x) '...'SELECT * FROM system.numbers LIMIT%(limit)s',...{'limit':3}...)>>>cursor.rowcount0>>>cursor.execute('SELECT sum(x) FROM test')>>>cursor.fetchall()[(303,)]LicenseClickHouse Python Driver is distributed under theMIT license.
Package: clickhouse-http-client
clickhouse-http-clientclickhouse http client.InstallUse source:$ git clone https://github.com/tomoncle/clickhouse-http-client.git
$ cd clickhouse-http-client
$ sudo python setup.py installUse pip:$ pip install clickhouse-http-clientUsageinit:from clickhouse_http_client import ClickHouse
ck = ClickHouse(password="123456")query:ck.text("select * from table_name")
ck.json("select * from table_name")delete:ck.delete("table_name", "id IS NOT NULL")save:ck.insert("table_name", {"name":"tomoncle", "age":"27"})update:ck.update("table_name", "status=1", {"name":"tomoncle", "age":"27"})
Package: clickhouse-migrate
ClickHouse MigratePython library for creating and applying migrations in ClickHouse database.InstallationInstallation via PyPi:pipinstallclickhouse-migrateUsageTo use this tool it is required to create and provide a directory in which the migration files will be created,
stored and from which they will be applied.clickhouse-migratetool supports clustered ClickHouse setup and therefore it requires ALL clickhouse servers that are clustered as a database parameter.Environment variablesRequired parameters can be provided as environment variables# Comma separated ClickHouse connection strings from one cluster# Can be as a single connection string in case of standalone clickhouse instanceCLICKHOUSE_MIGRATE_DATABASES="clickhouse+native://<user>:<pass>@<host1>:<port>, ... ,clickhouse+native://<user>:<pass>@<host2>:<port>"CLICKHOUSE_MIGRATE_DIRECTORY=<pathtodirectorywithmigrationsfiles>Create migration fileOne can create a new migration via callingclickhouse-migrate create_migrationcommand.Command parameters:-n/--name <name of a new migration file>— this is a required parameter-dir/--migration_dir— optional parameter for providing path to directory with migration files, default value./migrations,
can be replaced byCLICKHOUSE_MIGRATE_DIRECTORYenvironment variableExample usage:clickhouse-migratecreate_migration-ntest_migration-dirmigrationsAfter calling the above-mentioned command a blank migration fileYYYY-MM-DD-HH-mm-ss_<migration_name>.pywill be created in themigrationsdirectory which was defined in command line arguments.The content of the created file will look like thisfromclickhouse_migrateimportStepmigrations=[Step(sql="")]One should useStepdataclass to create migrations. Migrations would be applied in the same order that they are stated in themigrationslist variable.
All of listed changes in themigrationlist variable are by default applied to all databases listed in the*.iniconfig file, since usually changes are required for all replicated instances.If an SQL query stated in theStepdataclass should not be executed for all DB replicas one may useStep(sql="<sql query>", replicated=False)statement.This rule does not apply in case there is only one connection string in*.inifile.Apply migrationsOne can apply migrations created viaclickhouse-migrate create_migrationcommand by callingclickhouse-migrate migrate.
This command will check for already applied migrations and will only apply new ones.Command parameters:-db/--databases— optional parameter for providing database connection strings,
can be replaced byCLICKHOUSE_MIGRATE_DATABASESenvironment variable-dir/--migration_dir— optional parameter for providing path to directory with migration files,
can be replaced byCLICKHOUSE_MIGRATE_DIRECTORYenvironment variableExample usage:clickhouse-migratemigrate-dir/home/my_project/migrations-dbclickhouse+native://default:@host1:9000/db-dbclickhouse+native://default:@host2:9000/dbAfter calling this command all changes from migration files will be applied step-by-step. Changes are stored inclickhouse_migratetable.
Package: clickhouse-migrations
Clickhouse MigrationsPython library for creating and applying migrations in ClickHouse database.Development and Maintenance of large-scale db systems many times requires constant changes to the actual DB system.
Holding off the scripts to migrate these will be painful.Features:Supports multi statements - more than one query per migration file.Allow running migrations out-of-boxSimple file migrations format: {VERSION}_{name}.sqlSupports Cluster deployments, makes sure that migrations state is consistent on all cluster nodesKnown alternativesThis package originally forked fromclickhouse-migrator.PackageDifferencesclickhouse-migratorDoesn't support multistatement in a single file , to heavy because of pandas, looks like abandoneddjango-clickhouseNeed djangoclickhouse-migrateDoesn't support multistatementInstallationYou can install from pypi usingpip install clickhouse-migrations.UsageIn command lineclickhouse-migrations--db-hostlocalhost\--db-userdefault\--db-passwordsecret\--db-nametest\--migrations-dir./migrationsIn codefromclickhouse_migrations.clickhouse_clusterimportClickhouseClustercluster=ClickhouseCluster(db_host,db_user,db_password)cluster.migrate(db_name,migrations_home,cluster_name=None,create_db_if_no_exists=True,multi_statement=True)ParameterDescriptionDefaultdb_hostClickhouse database hostnamelocalhostdb_portClickhouse database port9000db_userClickhouse userdefaultdb_passwordClichouse passworddefaultdb_nameClickhouse database nameNonemigrations_homePath to list of migration files<project_root>cluster_nameName of Clickhouse topology cluster from <remote_servers>Nonecreate_db_if_no_existsIf thedb_nameis not present, enabling this will create the dbTruemulti_statementAllow multiple statements in migration filesTrueNotesThe Clickhouse driver does not natively support executing multipe statements in a single query.
To allow for multiple statements in a single migration, you can use the multi_statement param.
There are two important caveats:This mode splits the migration text into separately-executed statements by a semi-colon ;. Thus cannot be used when a statement in the migration contains a string with a semi-colon.The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
Package: clickhouse-migrator
Clickhouse MigratorClickhouseis known for its scale to store and fetch large datasets.Development and Maintenance of large-scale db systems many times requires constant changes to the actual DB system.
Holding off the scripts to migrate these will be painful.We found there is nothing existing earlier and developed one inspired by,Flyway,AlembicThis is a python library, which you can execute as a pre-hook using sys python.
Or as a migration framework before deployment/server-startup in your application as required.Publishing to pypipython -m buildpython -m twine upload --verbose --repository pypi dist/*InstallationYou can install from pypi usingpip install clickhouse-migrator.Usage# <= v1.0.4frommigration_lib.migrateimportmigratemigrate(db_name,migrations_home,db_host,db_user,db_password,create_db_if_no_exists)# > v1.0.4fromclickhouse_migrate.migrateimportmigratemigrate(db_name,migrations_home,db_host,db_user,db_password,create_db_if_no_exists)ParameterDescriptionDefaultdb_nameClickhouse database nameNonemigrations_homePath to list of migration files<project_root>db_hostClickhouse database hostnamelocalhostdb_password*********create_db_if_no_existsIf thedb_nameis not present, enabling this will create the dbTruedb_portDatabase port incase your server runs on a non default portNone (defaults to 9000)queue_execCommand Pipelining (wait for system mutations if any to complete) for every commandTrueFolder and Migration file patternsThe filenames are pretty similar to howflywaykeeps it.Your first version filename should be prefixed withV1__(double underscore)
These migrations are executed one by one, failures in between will stop and not further version files will be executed.Multi statement and single statement migrationsIf your migration is a single statement, you can create a file in the migration folder using the .sql extension and push your migration statement in there.If you want to execute more than one statement in your migration, you can use a json file using the array syntax. Note that when using a json file, contents should be a valid json array as show. Ensure to keep migrations logical. Its not a good practise to push all migrations to one json file and neither is it wise to in all cases have them each statement in one file.["CREATE TABLE pytest.sample1(id UInt32, name String) ENGINE MergeTree PARTITION BY tuple() ORDER BY tuple()","CREATE TABLE pytest.sample2(id UInt32, name String) ENGINE MergeTree PARTITION BY tuple() ORDER BY tuple()","CREATE TABLE pytest.sample3(id UInt32, name String) ENGINE MergeTree PARTITION BY tuple() ORDER BY tuple()"]
Package: clickhouse-plantuml
PlantUML generator for ClickHouse tablesThis is a very early version of diagrams generator. It parsessystem.tablestable and producesPlantUMLdiagrams source. Here's example of produced diagram:@startuml
' This diagram is generated with https://github.com/Felixoid/clickhouse-plantuml
!define Table(x) class x << (T,mistyrose) >>
!define View(x) class x << (V,lightblue) >>
!define MaterializedView(x) class x << (m,orange) >>
!define Distributed(x) class x << (D,violet) >>

hide empty methods
hide stereotypes
skinparam classarrowcolor gray

Distributed(graphite.data) {
  ENGINE=**Distributed**
  ..engine config..
  cluster: graphite_data
  database: graphite
  table: data_lr
  sharding_key: cityHash64(Path)
  ==columns==
  Path: String
  Value: Float64
  Time: UInt32
  Date: Date
  Timestamp: UInt32
}

Table(graphite.data_lr) {
  ENGINE=**ReplicatedGraphiteMergeTree**
  ..engine config..
  rollup_config: graphite_rollup
  ..replication..
  zoo_path: /clickhouse/tables/graphite.data_lr/{shard}
  replica: {replica}
  ==columns==
  Path: String <size:15><&signal></size>
  Value: Float64
  Time: UInt32 <size:15><&signal></size>
  Date: Date <size:15><&list-rich></size>
  Timestamp: UInt32
  ..<size:15><&list-rich></size>partition key..
  toYYYYMMDD(toStartOfInterval(Date, toIntervalDay(3)))
  ..<size:15><&signal></size>sorting key..
  Path, Time
}

Table(graphite.index) {
  ENGINE=**ReplicatedReplacingMergeTree**
  ..engine config..
  version: Version
  ..replication..
  zoo_path: /clickhouse/tables/graphite.index/1
  replica: {replica}
  ==columns==
  Date: Date <size:15><&list-rich></size> <size:15><&signal></size>
  Level: UInt32 <size:15><&signal></size>
  Path: String <size:15><&signal></size>
  Version: UInt32
  ..<size:15><&list-rich></size>partition key..
  toYYYYMM(Date)
  ..<size:15><&signal></size>sorting key..
  Level, Path, Date
}

Table(graphite.tagged) {
  ENGINE=**ReplicatedReplacingMergeTree**
  ..engine config..
  version: Version
  ..replication..
  zoo_path: /clickhouse/tables/graphite.tagged/1
  replica: {replica}
  ==columns==
  Date: Date <size:15><&list-rich></size> <size:15><&signal></size>
  Tag1: String <size:15><&signal></size>
  Path: String <size:15><&signal></size>
  Tags: Array(String)
  Version: UInt32
  ..<size:15><&list-rich></size>partition key..
  toYYYYMM(Date)
  ..<size:15><&signal></size>sorting key..
  Tag1, Path, Date
}

graphite.data_lr -|> graphite.data
@endumlAnd how it looks after running PlantUML:Usagepythonsetup.pyinstall
clickhouse-plantuml
Package: clickhouse-query
ClickHouse QueryClickhouse Query is a Python library for building complex ClickHouse queries by a fluent API and a set of functions.Installationpipinstallclickhouse-queryUsageimportclickhouse_queryaschq=(ch.QuerySet().from_("my_table").prewhere(date__year__gte="2024").select(count=ch.Count()).limit(10))sql,sql_params=ch.get_sql(q)# sql = 'SELECT count() AS count FROM click PREWHERE (greaterOrEquals(toYear(date), %(__U_1)s)) LIMIT %(__U_2)f'# sql_params = {'__U_1': 2024, '__U_2': 10}data=q.execute()# data = [{'count': 100}]Custom FunctionsIf there is not a function that you need, you can easily create it by extending theFunctionclass. like this:classMyFunction(ch.Function):function="my_function"Custom Aggregation FunctionsIf there is not a aggregation function that you need, you can easily create it by extending theAggregationFunctionclass. like this:classMyAggregationFunction(ch.AggregationFunction):function="my_aggregation_function"LicenseThis project is licensed under the MIT License - see theLICENSEfile for details