Package: vulkn
VULKИVULKИ (Vulkn) is a specialized, environmentally friendly, eco-system for manipulating and managing large volumes of data at petabyte scale.Vulkn is underpinned by the high-performance Open Source OLAP database, Yandex ClickHouse. Rather than re-invent the wheel (or mask the amazing technology that is ClickHouse), Vulkn focuses on the areas of DataOps, simplification and automation. Users can easily extend Vulkn, and the SQL layer, with their own SQL clauses and functions via Python as well as use Python bindings to work with ClickHouse in much the same way they would other modern data systems, specifically those based on functional programming.Short-term RoadmapFebruary/March 2020Simple SQL and Vulkn ORM UDF supportFunction cleanup/introspectionSupport all types/functions for current stable/testingSupport for user/group/quota/workload management (in ClickHouse core)HTTP interface (rather than fork/exec client)(done)Proof Of Concept phase endsInitial implementation of Vulkn serverDocumentationDocumentation is available athttp://docs.vulkndata.io. Note this is currently incomplete and content will be updated/provided over the coming weeks.Initial ReleaseThe initial release (19.0.0) is a Proof Of Concept. Expect APIs to change in the coming releases as Vulkn reaches stability. The following features are available:SQL rewriting engine with library supportWorkspaces (very proof of concept)Python data type bindingsVector types - ArrayVector, ColumnVector, MergeVectorSQL extensions - CHUNK BY, VECTORIZE BYSQL vector functions (basic SQL:2003 Window Functions)InstallationVulkn has only been tested with recent versions of Ubuntu and Python 3.7. You will need to have a
working Python 3.7.x environment with pip.Ensure you have installed ClickHouse (both server and client):sudoaptinstallclickhouse-serverclickhouse-clientclickhouse-commonYou should also ensure your systems console encoding is set to a valid non-latin/ASCII encoding such as en_US.UTF-8 (or your preferred/local extended character set)sudolocalectlset-localeLANG=en_US.UTF-8Installation with pipInstall Vulkn via pip.sudoadd-apt-repositoryppa:deadsnakes/ppa
sudoaptupdate
sudoaptinstall-ypython3.7python3.7-devpython3-pip
sudopython3.7-mpipinstallvulknInstallation from source (for developers)Install Vulkn viagit clone.gitclonehttps://github.com/VulknData/vulkn.gitcdvulknInstall required packages. Note that it may make sense to do this within a virtual environment.pipinstall-rrequirements.txtYou can start the Vulkn CLI using the following. This creates a temporary Vulkn Workspace.cdscriptssourceenv.sh
./v--local# Or use ./v to connect to your running local ClickHouse instanceGetting StartedLet's try out the Python API with a 25 million row time-series table. In this example we'll create a table from a group of ArrayVectors with 10,000 unique series, a random DateTime dimension over the course of a day and two metrics (temperature, bytes). For the metrics we'll simulate a double peak day by generating concatenated normal distribution curves. These operations should execute in several seconds on any modern computer with 4-8GB of RAM.If you installed Vulkn via pip usevulkn --localto start a local session (or follow Step 3. from the source installation option above).Note that if you are connecting to a remote (or local ClickHouse instance not managed by Vulkn) you may need to manually create the "vulkn" database and ensure the connecting user can create tables under this database.ironman@hulk~/$vulkn--local2020.01.0111:52:14.072561[1]{}<Trace>Pipe:Pipecapacityis1.00MiB2020.01.0111:52:14.072619[1]{}<Information>:StartingClickHouse19.19.1.2024withrevision544302020.01.0111:52:14.072651[1]{}<Information>Application:startingup2020.01.0111:52:14.078485[1]{}<Trace>Application:Willmlockalltopreventexecutablememoryfrombeingpagedout.Itmaytakeafewseconds.2020.01.0111:52:14.103994[1]{}<Trace>Application:Thememorymapofclickhouseexecutablehasbeenmlock'ed2020.01.01 11:52:14.104171 [ 1 ] {} <Debug> Application: Set max number of file descriptors to 1048576 (was 1024).2020.01.01 11:52:14.104180 [ 1 ] {} <Debug> Application: Initializing DateLUT.2020.01.01 11:52:14.104182 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'.2020.01.01 11:52:14.105815 [ 1 ] {} <Debug> ConfigReloader: Loading config '/tmp/jason-1f1b3dd6-703c-40d8-9573-f0f07fcda44a/etc/users.xml'2020.01.01 11:52:14.106233 [ 1 ] {} <Information> Application: Loading metadata from /tmp/jason-1f1b3dd6-703c-40d8-9573-f0f07fcda44a/clickhouse/2020.01.01 11:52:14.106817 [ 1 ] {} <Information> DatabaseOrdinary (default): Total 0 tables and 0 dictionaries.2020.01.01 11:52:14.106827 [ 1 ] {} <Information> DatabaseOrdinary (default): Starting up tables.2020.01.01 11:52:14.106876 [ 1 ] {} <Debug> Application: Loaded metadata.2020.01.01 11:52:14.106887 [ 1 ] {} <Information> BackgroundSchedulePool: Create BackgroundSchedulePool with 16 threads2020.01.01 11:52:14.109096 [ 1 ] {} <Information> Application: Listening for http://[::1]:81242020.01.01 11:52:14.109151 [ 1 ] {} <Information> Application: Listening for connections with native protocol (tcp): [::1]:90012020.01.01 11:52:14.109202 [ 1 ] {} <Information> Application: Listening for http://127.0.0.1:81242020.01.01 11:52:14.109235 [ 1 ] {} <Information> Application: Listening for connections with native protocol (tcp): 127.0.0.1:90012020.01.01 11:52:14.109584 [ 1 ] {} <Information> Application: Available RAM: 62.60 GiB; physical cores: 8; logical cores: 16.2020.01.01 11:52:14.109597 [ 1 ] {} <Information> Application: Ready for connections.Добро пожаловать to VULKИ version 19.0.1!██╗   ██╗██╗   ██╗██╗     ██╗  ██╗███╗   ██╗██║   ██║██║   ██║██║     ██║ ██╔╝████╗  ██║██║   ██║██║   ██║██║     █████╔╝ ██╔██╗ ██║╚██╗ ██╔╝██║   ██║██║     ██╔═██╗ ██║╚██╗██║╚████╔╝ ╚██████╔╝███████╗██║  ██╗██║ ╚████║╚═══╝   ╚═════╝ ╚══════╝╚═╝  ╚═╝╚═╝  ╚═══╝The developer friendly real-time analytics engine powered by ClickHouse.VULKИ entrypoint initialized as 'v'.

>>># Create four ArrayVectors simulating a time-seriesseries_key=(ArrayVector.range(1,10000).cast('String').map('concat',String('device-')).take(25000000).cache().alias('series_key'))timestamp=(ArrayVector.rand(DateTime('2019-01-01 00:00:00'),DateTime('2019-01-01 23:59:59'),25000000).cache().alias('timestamp'))temperature=(ArrayVector.norm(80,10,12500000).join(ArrayVector.norm(95,5,12500000)).cache().alias('temperature'))bytes=ArrayVector.rand(1,8192,25000000).cache().alias('bytes')# Combine the ArrayVectors to create a table, returning a DataTabledata=v.table.fromVector('default.mydata',(series_key,timestamp,temperature,bytes),engines.Memory(),replace=True)data.count().show()# Use SQL to aggregations and other queriesv.q("""SELECTseries_key,max(timestamp),avg(temperature),min(bytes)FROM default.mydataGROUP BY series_key""").show()# Use Vulkn time-series SQL extensions - calculate the moving average using the current row and two behind/ahead (5 value window)v.q("""SELECTseries_key,timestamp,bytes,vectorWindowAgg(avg, bytes, -2, 2) AS moving_avgFROM default.mydataWHERE series_key = 'device-1'VECTORIZE BY (series_key, timestamp)""").limit(20).show()# You can also use the DataTable interface to build up complex queriesv.table('default.mydata').select(funcs.agg.max(col('bytes')).alias('max_bytes')).s# Results can be returned as a 'pretty print' table (.s, .show()), list of dictionaries (a record) (.r, .to_records()) or a Pandas DataFrame (.p, .to_pandas())v.table('default.mydata').all().limit(4).r[{'series_key':'device-1','timestamp':'2019-01-01 08:42:29','temperature':94.10629119151686,'bytes':6206},{'series_key':'device-2','timestamp':'2019-01-01 08:32:24','temperature':75.25278421080404,'bytes':5070},{'series_key':'device-3','timestamp':'2019-01-01 09:47:07','temperature':83.09766288572209,'bytes':3617},{'series_key':'device-4','timestamp':'2019-01-01 17:37:49','temperature':77.4298465579733,'bytes':3283}]# Data sets are also discoverable via the 'data' dictionary[tfortinv.data.keys()ift=='default.mydata']mydata=v.data['default.mydata']mydata# <vulkn.datatable.BaseTableDataTable object at 0x7f587af97908># Inspect the DataTable propertiesmydata.desc().srownametypedefault_typedefault_expressioncommentcodec_expressionttl_expression-----------------------------------------------------------------------------------------------------1series_keyStringnannannannannan2timestampDateTimenannannannannan3temperatureFloat64nannannannannan4bytesUInt64nannannannannan(4rows)# Create Pandas DataFrames from a Vulkn querymydata.all().limit(10).pseries_keytimestamptemperaturebytes0device-12019-01-0119:47:2492.57327477591device-22019-01-0110:10:3497.79587563722device-32019-01-0112:43:2092.76974776923device-42019-01-0122:57:4298.03709054984device-52019-01-0104:47:1386.35244236645device-62019-01-0109:09:1786.67986149976device-72019-01-0112:01:0599.48172761897device-82019-01-0117:03:0790.48829549918device-92019-01-0105:28:4392.5322156539device-102019-01-0110:47:3898.021234192mydata.all().limit(10).p.dtypesseries_keyobjecttimestampobjecttemperaturefloat64bytesint64dtype:objectLet's explore the python API.# Native Vulkn functions/ClickHouse SQL in the Python APIprint((UInt8(8).alias('v')!=3).alias('k').cast('String').startswith('a').intDiv(3))# Using and re-using aliases and variabless=String('abc,foo').split(',')col1=s[1].alias('col1')clone_col1=String(name='a')# Tying it all together with lambdasfunc=funcs.arrayMap(lambdax=String(n='x'),y=String(n='y'):x.length()>3,s,col1,clone_col1)# Preview the rendered function.print(func)# arrayMap((x, y) -> greater(length("x"),3), splitByString(',','abc,foo'), arrayElement(splitByString(',','abc,foo'),1) AS `col1`, "a")# You can also create libraries of re-usable functions using the Python API.# Note - Native Python functions and the ability to register functions with the SQL engine are coming in a future release.defextract_key(column,alias):key_id=String(n=column).lower().split('-')[2]key=key_id.cast('UInt64').alias(alias)returnkeydefapply_cutoff(arg1,arg2,arg3):returnfuncs.if_(col(arg1)>arg2,arg3,arg2)# Python datatablesdf=(v.table('default.mydata').select(extract_key('series_key','key'),apply_cutoff('bytes',100,100).alias('newcolumn')).orderBy('key'))df.show_sql()"SELECT cast(splitByString('-', lower(series_key))[2], 'UInt64') AS key, if(bytes > 100, 100, 100) AS newcolumn FROM default.mydata ORDER BY key ASC;"# Load external dataexternal_data=(v.read.format('csv').options(header=True,overwrite=True,infer_schema=True,engine=engines.MergeTree()).load('file:///data/my-example-data.csv',database='default',table='example'))external_data.count().show()LimitationsThe following limitations apply to the 19.0.0 Proof Of Concept release and will be addressed in future versions:Security, authentication and authorization is currently not available. We will be implementing LDAP authentication and fine-grained access control.All operations are synchronous and single-threaded however operations on the underlying ClickHouse datastore will utilise multiple cores if available. Future versions will be asynchronous.Single node. Session management is focussed on a single node only at this stage. You can use Vulkn with a multi-node ClickHouse cluster however sessions and tables will be limited to the originating/master node only. Future versions will be completely distributed allowing for grid computing.Sub-selects are required for certain operations with SQL extensions.Stream processing is not supported. Vulkn will look to leverage ClickHouse's impressive ingestion performance and real-time reporting rather than provide a dedicated streaming solution.Most operations are not constrained by memory. In the future Vulkn will offer more options to serialize to disk.Vector matrices are not yet supported however standard table functions are fine.Data shuffling is currently offloaded to ClickHouse. There are known performance issues with data shuffle operations and distributed joins within ClickHouse that will be addressed by the core team soon.Did we mention this is a POC? Things are brittle and will break. The code is very ugly in places and we intend on major refactoring under the hood ASAP.OverviewSQL EngineDataOps meets SQL. The SQL engine intercepts and rewrites SQL statements, facilitating clever data orchestration and extension with only a handful of lines of code. The initial SQL engine is very simple at this stage. This release includes a simple library interface for registering SQL extensions and vector functions to support theVECTORIZE BYextension.WorkspacesA workspace consists of a stand-alone, local instance of the ClickHouse server. When a workspace is instantiated a local instance of ClickHouse is started under the current user. This is coupled to the Python API, along with either temporary or permanent storage. Users can nominate for either temporary or permanent workspaces and manage their workspaces via the vulkn CLI. Permanent workspaces can be thought of as private ClickHouse databases, complete with SQL interfaces, persistent data and managed configuration. Multiple users can launch multiple workspaces on a single machine if desired.Workspaces will be extended in the future to offer both local and remote storage with the ability to pull in datasets from remote databases, object stores and filesystems for analysis within the local instance. Given the performance profile of ClickHouse this could offset Cloud costs - organisations will find it faster, cheaper and greener to switch off Hadoop/Spark clusters and simply run offline analytics on a single laptop that pull from a main store when required.Python BindingsPython bindings for most functions and objects are available as well as a simple DataTable interface.Data TypesThe available data types are:StringDateDateTimeFloat32, Float64Int8, Int16, Int32, Int64UInt8, UInt16, UInt32, UInt64ArrayFunctional styles/notation is preferred and types behave like Python types in most cases:>>>mystr=String('a quick brown box').split(' ').length()[2]>>>print(mystr)length(splitByString(' ','a quick brown box'))>>>one_element=String('a quick brown box').split(' ')[2]>>>print(one_element)arraySlice(splitByString(' ','a quick brown box'),2,1)Array slices are also supported:>>>mystr_with_commas=String('a quick brown box').split(' ')[1:3].join(',')>>>print(mystr_with_commas)arrayStringConcat(',',arraySlice(splitByString(' ','a quick brown box'),1,3))>>>v.select(mystr_with_commas).srowarrayStringConcat(arraySlice(splitByString(\'\',\'a quick brown box\'), 1, 3),\',\')---------------------------------------------------------------------------------------------1a,quick,brown(1row)You can also mix and match Python types with Vulkn types:>>>mystr=String('a quick brown box').split(' ').length()>3>>>print(mystr)greater(length(splitByString(' ','a quick brown box')),3)Type information is propagated through the function interfaces so String.length() returns a UInt type.Note that the function interface will be redesigned in future releases however functionality / APIs will remain mostly unchanged.DataTable interfaceDataTables mimic similar concepts found within Pandas, Apache Spark, Apache Flink and other such systems. They provide an object/functional interface to authoring re-usable transformations and code.Mixing DataTables with sub-selects, JOINS and raw SQL (v.q()) are all supported.Example:v=Vulkn()df=(v.table('metrics').select('column','another column',String(n='myid')).where('column > 3').orderBy('another column'))df2=df.select('count()')Results can be returned as Pandas DataFrames, dictionaries, JSON or CSV. Basic type mappings with Python/Pandas is supported however nested and array types are returned as objects.Support for nested and array types for Python and Pandas outputs will be available in a future release as will transparent external data joins / loading with Python objects and Pandas DataFrames.Specialized Vector typesVulkn provides two vector types - ArrayVector and ColumnVector. ArrayVector types offer moderate performance and are suitable for up to 100 million elements. ColumnVectors can be unlimited (within reason) although some operations will be limited by available memory. ArrayVectors can be converted to ColumnVectors and vice-versa.Vectors are useful for generating sample datasets or incremental vector calculations from a large table. Both types support most of the same operations and are interchangeable.Note that vector types are not intended/available for Vulkn queries and supplement existing relational processing. The Array type and general purpose data type functions can be used to manipulate tables.Vector creation methods:range(start, end)- generate a vector of Integers/UIntegers starting at start and ending at end.rand(start, end, length)- generate a random vector of Integers/UIntegers with values between $start and $end of length $length.norm(mean, stddev, count)- generate a vector of random distribution (bell curve) of count elements with centered around mean of width stddev.table(Vulkn table/query)- create a vector from a Vulkn table or query result.toColumnVector- creates a ColumnVector object from an ArrayVector.toArrayVector- creates an ArrayVector object from a ColumnVector.Operations:sort- sorts the vector by keyreverse/sort_reverse- sort the vector by key descendingshuffle- randomly sort the vectorcast(type)- cast the vector to the specified typecut(length)- divide the vector in rows of equal elementstake- either reduce or extend the vector by the specified limit. Note that in some cases limit is more efficient than take for reducing the size of a vector.limit(limit)- reduce the length of the vector by the specified limitflatten- flatten a vector containing arraysprev- shift the entire vector backward one elementnext- shift the entire vector forward one elementdelta- delta the vector with the preceding valuejoin(other_vector)- join/concatenate two vectorsmove- shift the vector positions elements left or rightwavg- calculate the weighted average of one vector by anothermap- apply a function to all elements in the vectormaplag- apply a function pairwise in a lagging fashion - f(current, previous)maplead- apply a function pairwise in a leading fashion - f(current, next)Table creation methodsv.table.fromVector(name, (vectors,), storage.Type)Multiple vectors can be formed into a standard table called name using engines -engines.<Type>Engine Typesengines.Memory(settings)engines.Log(settings)engines.StripeLog(settings)engines.TinyLog(settings)engines.MergeTree(partitionBy, PrimaryKey, OrderBy, settings)SQL ExtensionsCHUNK BYStatus: alphaExample:SELECT<cols>FROMtableCHUNKBY(key,2)Notes: N/ACHUNK BYdynamically generatesUNION ALLstatements with a modulosWHEREclause based on the chunk key. This is a simple and effective work around for corner cases where ClickHouse is unable to utilise all CPU cores in some queries.In the example above the statement will be rewritten to:SELECT*FROM(SELECT<cols>FROMtableWHEREcityHash64(key)%2=0UNIONALLSELECT<cols>FROMtableWHEREcityHash64(key)%2=1)This statement will most likely be deprecated as the execution mechanisms within ClickHouse address the various non-performant cases.VECTORIZE BYStatus: alphaExample:SELECTkey,sort_key,other_col1,other_col2,metric1,metric2,vectorFunction(metric1)ASvector_metric1vectorFunction(metric2)ASvector_metric2FROMtableVECTORIZEBY(key,other_col1,other_col2...other_coln,sort_key)Notes:GROUP BY,LIMIT BY,LIMIT,ORDER BYandSETTINGSare not supported inVECTORIZE BYstatements (use sub-selects if required)VECTORIZE BYis a specialized time-series focused extension that creates column vectors for the specific metric for each key sorted by the sort_key (usually time). Additional columns that are not part of the key, sort_key or metrics must be specified in theVECTORIZE BYclause (other_col..). Vector columns are automatically determined using the leading double-underscore syntax. Non-vector columns will most likely fail to evaluate correctly in vector* functions within aVECTORIZE BYstatement. This syntax may change in the future as the API matures.When usingVECTORIZE BYeach row has visibility across the entire set of rows for that given key allowing the user to calculate complex metrics that are dependant on part of or the entire dataset for that key. Most SQL:2003 window functions have been implemented using this mechanism in addition to specialized time-series functions.For example - to compare the lagging delta of a value of a particular metric against the average for that series you can use the following. Note how thevectorAgg(avg)function calculates an average value that can be compared across rows and against other vector function derived values. Moving averages / sliding windows are also supported using thevectorWindowAggfunction.SELECTid,time,volume,avg_volume,abs(delta_volume)ASdelta_volume,abs(round(abs(delta_volume)/avg_volume,2))ASavg_distanceFROM(SELECTid,time,volume,vectorAgg(avg,__volume)ASavg_volume,vectorDelta(__volume)ASdelta_volumeFROMstocksVECTORIZEBY(id,time))idtimevolumeavg_volumedelta_volumeavg_distance5222432019-01-0110:02:3421981926.9005222432019-01-0110:11:557061926.914920.775222432019-01-0110:27:4934231926.927171.415222432019-01-0110:40:3213741926.920491.065222432019-01-0110:42:1817421926.93680.195222432019-01-0111:17:4639161926.921741.135222432019-01-0111:27:4614771926.924391.275222432019-01-0111:33:1812391926.92380.125222432019-01-0112:17:5928641926.916250.845222432019-01-0112:32:543301926.925341.3241091292019-01-0110:24:3238502323.735201.5141091292019-01-0110:50:3513322323.725181.0841091292019-01-0111:00:5928972323.715650.6741091292019-01-0111:16:26162323.728811.2441091292019-01-0111:18:558732323.78570.3741091292019-01-0111:21:5835532323.726801.1541091292019-01-0111:34:4416992323.718540.841091292019-01-0111:55:2232272323.715280.6641091292019-01-0112:02:3523492323.78780.3841091292019-01-0112:14:1334412323.710920.4720rowsinset.Elapsed:1.298sec.Processed50.00millionrows,1.39GB(38.52millionrows/s.,1.07GB/s.)Under the hoodVECTORIZE BYconverts values to arrays grouped by the key and non-key columns, sorted by the sort_key. Vector operations are then executed on the underlying arrays. Because these operations can touch the entire keys dataset per row the performance ofVECTORIZE BYmay seem very different to a non-VECTORIZE-BYquery. At face value, Non-VECTORIZE-BYqueries may be able to scan 1 billion cells/s for a given column, whereas a similarVECTORIZEquery will only scan 40 million rows/s. In reality theVECTORIZE BYquery may actually touch the equivalent of 500 million cells/s however the complexity of the operations involved hide this fact.As a footnote statements generated byVECTORIZE BYresult in very complex ARRAY JOIN statements. Fortunately these statements can be persisted as views within ClickHouse.This statement will most likely be deprecated when window functions are available within ClickHouse core.Future versions of Vulkn will include a server component that will expose all extensions and custom functions through a normal ClickHouse ODBC endpoint.SQL vector functionsThe following standard vector functions are part of the core library.As vector functions return vectors (even scalar average functions return vectors) you can chain functions together to calculate complex derivatives and other metrics. For instance, assuming a 15minute timeseries counter, the following calculates the max temperature across a rolling hour every 15mins and then calculates the average of the max for the rolling hour. We've also calculated cumulative sum of the rolling average (2 periods behind, 2 ahead) of the delta of the temperature.SELECTdevice,location,timestamp,temperature,vectorWindowAgg(max,temperature,-3)ASmax_rolling,vectorAgg(avg,vectorWindowAgg(max,temperature,-3))ASavg_max_rolling,vectorAccumulate(vectorWindowAgg(avg,vectorDelta(temperature),-2,2))ASdelta_temp_window_avg_accumulateFROMdevice_seriesWHEREdevice='mydevice-123'ANDtimestampBETWEEN'2019-01-01'AND'2019-02-01'VECTORIZEBY(device,location,timestamp)Note that the formula for the column max_rolling must be repeated within the nested vectorAgg function. This is due to limitations with the current Vulkn parser. Fortunately, whilst it may seem that we're 'recalculating' the max_rolling value, the underlying ClickHouse engine tracks the relational algebra and ensures the formula is only calculated once.Supported Vector functionsRudimentary SQL:2003 support usingVECTORIZE BY (key ..., sort_key).ROW_NUM - vectorRowNumber(vector_column)- for each (key, sort_key) row emit a number starting from 1FIRST - vectorFirst(vector_column)- emit the first row in the (key, sort_key)LAST - vectorLast(vector_column)- emit the last row in the (key, sort_key)LAG - vectorLag(vector_column (, offset=1, default='NULL'))- emit the column value offset rows behind the current rowLEAD - vectorLead(vector_column (, offset=1, default='NULL'))- emit the column value offset rows ahead of the current rowDENSE_RANK - vectorDenseRank(vector_column)- as per ROW_NUM, calculate the rank of each value relative to the other excluding tiesCUME_DIST - vectorCumeDist(vector_column)- calculate the cumulative distance to a maximum of 1 of the first row to the lastPERCENT_RANK - vectorPercentRank(vector_column)- as above except use percentages starting from 0NTH_VALUE - vectorNth(vector_column, nth)- return every nth value in the setMisc. standard aggregates (min, max, avg..) usingvectorAgg(aggregate_function, vector_column)Others:vectorWindow(vector_column, (, window_backwards=0, window_forwards=0))- generate sub-vectors of abs(window_forwards)+abs(window_backwards) length (returns a vector of vectors)vectorWindowAgg(aggregate_function, vector_column, window_backwards=0, window_forwards=0)- calculate the aggregate_function on sub-vectors and returns a single value for each sub-vector.vectorCut(vector_column, length)- divide the column vector into vectors of length sub-vectorsvectorDelta(vector_column)- delta the current value of column with the previous valuevectorAccumulate(vector_column)- maintain a running sum for the specific columnFinance:vectorElliotWaveOscillator(vector_column)- Calculate the Elliot Wave Oscillator of the vectorvectorDollarValue(price_vector, volume_vector)- Calculate the Dollar ValueFAQIsn't this just an ORM?No. The SQL / object interface is only one minor / enabling feature. By having a tightly integrated SQL / object / type interface that is free from the usual constraints of an ORM we can implement mechanisms that blur the lines between ORMs, BigData frameworks (like Spark), time-series systems, APL systems, DBMS management utilities, API generation, ML, AI and DataOps orchestration/automation. Vulkn also enables DataOps extensions, that leverage many of the ideas found within DevOps, to allow users to shape the query language itself. The POC release only targets SELECT statements where the AST can be transformed into single statement however we will be extending this to encompass a wider range of possibilities.Does Vulkn compete with Hadoop, Spark, Flink?Spark, Hadoop etc.. have a large, mature and established eco-system with a specific trajectory and path. ClickHouse is capable of ingesting and querying massive volumes of data in real-time (at velocities that exceed the aforementioned systems) however it has a very rigid API that does not allow for custom code without re-compiling the framework (unlike Spark or Hadoop). Custom code has advantages with performance being the limitation. If, at some point, users see benefit in Vulkn or ClickHouse over other tools, we suspect it will be to compliment their respective strengths.Is Vulkn a time-series database?Not specifically although ClickHouse is capable of besting the fastest Open Source and commercial time-series databases in most cases and can match them on a $-for-$ basis for compute. ClickHouse supports in-memory dictionaries/symbol tables, key/value, both in-memory and disk backed tables with memory buffers, distributed execution and ingestion, federated queries, some window/vector operations (more with Vulkn), streaming materialized views and ASOF-joins.Does Vulkn support native Python functions?We have been able to validate 1 million rows/s per core (Core i7) for trivial/untuned Python serialization of a 5 column width table of various data types. 1 million rows/s per core is most likely acceptable for those cases where serialization out to Python is a hard requirement although such throughput is generally too slow for any real work. Preferably users should use the Python API to author Vulkn native functions which support billions of rows/s per core.ClickHouse -> Python serialization has been validated by others (within the clickhouse-driver project) at over 25-50 million elements per second per core depending on data type/width. We anticipate using this library over our rudimentary tests in the future.Once we have the serialization performance nailed down we will be adding support for distributed Pandas/Dask, and by extension, pure-Python functions.Do you support AI and ML interfacesRudimentary ML is supported within ClickHouse via basic regression and classification methods and CatBoost integration with GPU support. Rather than create an eco-system of ML and AI within ClickHouse, Vulkn will focus on DataOps orchestration of ML and AI coupled with ClickHouse.Apache Spark .. or other project .. benchmark is much faster than Vulkn or ClickHouse - what gives?There are plenty of instances where other technologies will outperform ClickHouse and, coupled with their respective feature set, these may be better solutions for your specific problem. That said, beware of 'snake oil' benchmarks that focus on pre-computed or indexed fields, min/max counts or datasets that are prepared and presented in-memory only. Considering the whole of a problem, from ingestion, through analysis and to ensuring data consistency at rest, ClickHouse, coupled with Vulkn, should prove to be compelling.Will you provide other language bindings?Most likely no, however existing ClickHouse drivers and ORMs in other languages should work with Vulkn SQL without a problem.When will the road-map be published?Soon.I like where this is going and I want feature X!Great! Create a feature request on GitHub and we'll review.Can I contribute?Absolutely! Please seeCONTRIBUTINGfor more info.Do you provide commercial/Enterprise support?Not at this stage. Feel free to ping us @admin@vulkndata.io.Can I use Vulkn in my non-GPL FLOSS or commercial project?Yes! You can use most popular OSI approved licenses for your project. We have used the GPL exclusion mechanism to specifically allow aggregate works providing the aggregate work is released under one of the listed OSI approved licenses and Vulkn itself remains unmodified.The allowable OSI licenses are:Apache License 2.0 (Apache-2.0)3-clause BSD license (BSD-3-Clause)2-clause BSD license (BSD-2-Clause)GNU General Public License (GPL)GNU Lesser General Public License (LGPL)MIT license (MIT)Mozilla Public License 2.0 (MPL-2.0)Common Development and Distribution License 1.0 (CDDL-1.0)Eclipse Public License 2.0 (EPL-2.0)The GPLv3 enables you to use Vulkn internally within your organization or as part of a web service with no requirement to share your application code or licence it under the GPLv3.If you modify Vulkn you will have to either contribute your changes or adhere to the requirements of the GPLv3.If you are wanting to package and distribute the covered (unmodified) work as part of a larger work under a commercial license thats fine too providing you comply using interfaces compatible with the GPL ("at arms length" - pipes, sockets and command-line arguments) or seek an exception from the Licensor.If you want to modify the work, include it in a proprietary aggregate/license and distribute it, you will need to reach out to us. It is not our intention to expose part of or your entire proprietary/commercial application to an OSI licence. We merely want to protect our efforts thus far as it applies to reciprocal adaptions, modification and minimal appropriate investment to enable sustainable development of the software by the authors.