Package: shoots
Shoots is dataframe storage server. Currently is supports pandas, but most likely I will add support for polars in the future.Shoots is entirely written in Python and is designed for Python users.Shoots comes in 2 parts, a server and a client library.Shoots is very early software, but is in a usable state. It is stored ongithub. Issues and contributions welcome.shoots_serverThe server tries to be a fairly faithful Apache Flight Server, meaning that you should be able to use the Apache Arrow Flight client libraries directly. It is entirely built upon the upstreamApache Arrow project.Under the hood, the server receives and serves pandas dataframes, storing thenm on disk inApache Parquetformat. However, shoots is designed so that, as a user, you don't need to know about the underlying storage formats and libraries.shoots_clientThe client pieces wrap theApache FlightClientto offer an interface for pandas developers, abstracting away the Apache Arrow and Flight concepts.usagestarting up the serverterminalRunning the server is a simple matter of running the python module, depending on your system:pythonshoots_server.pyorpython3shoots_server.pystart up optionsShootsServer supports the follow CLI arguments:--port: Port number to run the Flight server on. Defaults to 8081.--bucket_dir: Path to the bucket directory. Defaults to ./buckets.--host: Host IP address for where the server will run. Defaults to localhost.For example, to run on localhost, but for a different port and bucket directory:python3shoots_server.py--port=8082--bucket_dir="/foo/bar"To enable TLS on the server, provide an SSL certificate and key.--cert_file: Path to file for cert file for TLS. Defaults to None.--key_file: Path to file for key file for TLS. Defaults to None.To enable JWT-based authentication, provide a secret string:--secret: A secret string use to generate a JWT and authorize clients with that JWT. TLS must be enabled.These options can also be set via environment variables.SHOOTS_PORTSHOOTS_BUCKET_DIRSHOOTS_HOSTSHOOTS_CERT_FILESHOOTS_KEY_FILESHOOTS_SECRETpythonYou can also start up the server in Python. It is best to start it on a thread or you won't be able to cleanly shut it down.frompyarrow.flightimportLocationimportthreadinglocation=Location.for_grpc_tcp("localhost",8081)server=ShootsServer(location,bucket_dir="/foo/bar")#bucket_dir is optionalserver_thread=threading.Thread(target=server.run)server_thread.start()Running SecurelyTo run the server with TLS enabled start the server with both the a certificate and key.  This is accomplished by passing the strings for the certificate and key as a tuple.Add TLSwithopen(cert_file,'r')ascert_file_content:cert_data=cert_file_content.read()withopen(key_file,'r')askey_file_content:key_data=key_file_content.read()server=ShootsServer(location,bucket_dir=self.bucket_dir,certs=(cert_data,key_data))server.start()Note below that if you are usig a self-signed certificate, you should create the certificate and key with a root certificate that can be shared with the client, so that the client can verify that the it is the actually expected server that is responding.Using a JWTThe server can require a JWT from the client to authenticate that the client is legit. This requires TLS to be enabled so that the jwt is not passed around in clear text. To enable JWT authantication, supply a secret to the server, generate a JWT, and then the client can use that token to authenticate with the server.server=ShootsServer(self.location,bucket_dir=self.bucket_dir,certs=(cert_data,key_data),secret="some_secret_to_generte a token")token=server.generate_admin_jwt()# give the token to the clientserver.start()Note that a token will be generated and printed to standard out at start up as well.See below for how to create a client that uses the token.shutting down the serverfrom the shoots clientShoots supports ashutdownaction. You can call it from the shoots client:fromshoots_clientimportShootsClientshoots=ShootsClient("localhost",8081)shoots.shutdown()from python codeThe Shoots shutdown function handles the threading, so you can simply call it.server.shutdown()creating a client objectconnect to a server running in insecure modeShootsClient requires a host name and port number:shoots=ShootsClient("localhost",8081)connect to a server with TLS enabledYou can enable tls by setting use TLS to True.shoots=ShootsClient("localhost",8081,True)If the server is using a self signed certicate for TLS, you need to provide the root certificate to the client. You do this by passing in the certificate a string.root_cert=""withopen(path_to_root_cert)asroot_cert_file:root_cert=root_cert_file.read()shoots=ShootsClient("localhost",8081,True,root_cert)connect to a server that requires a JWTIf the server requires token authentication, then assuming you have the token, use it when you create the client object.client=ShootsClient("localhost",port,True,root_cert,token=token)client.ping()storing a dataframeUse the client library to create an instance of the client, andput()a dataframe. Assuming you are running locally:fromshoots_clientimportShootsClientfromshoots_clientimportPutModeimportpandasaspdshoots=ShootsClient("localhost",8081)df=pd.read_csv('sensor_data.csv')shoots.put("sensor_data",dataframe=df,mode=PutMode.REPLACE)retrieving a dataframeYou can simply get a dataframe back by using its name:df0=shoots.get("sensor_data")print(df0)You can also submit a sql query to bring back a subset of the data:sql='select "Sensor_1" from sensor_data where "Sensor_2" < .2'df1=shoots.get("sensor_data",sql=sql)print(df1)Shoots useApache DataFusionfor executing SQL. TheDataFusion dialectis well document.listing dataframesYou can retrieve a list of dataframes and their schemas, using thelist()method.results=shoots.list()print("dataframes stored:")forrinresults:print(r["name"])dataframesstored:
sensor_datadeleting dataframesYou can delete a dataframe using thedelete()method:shoots.delete("sensor_data")resampling dataframesYou can resample (aka "downsample") dataframes on the server by sending either a command for a time series dataframe, or just send SQL for any arbitrary dataframe.resample with SQLself.client.resample(source="my_source_dataframe",target="my_resampled_dataframe",sql="SELECT * FROM my_source_dataframe LIMIT 10",mode=PutMode.APPEND)resample time seriesself.client.resample(source="my_source_dataframe",target="my_resampled_dataframe",rule="10s",time_col="timestamp",aggregation_func="mean",mode=PutMode.APPEND)bucketsYou can organize your dataframes in buckets. This is essentially a directory where your dataframes are stored.creating bucketsBuckets are implicitly created as needed if you use the "bucket" parameter input():shoots.put("sensor_data",dataframe=df,mode=PutMode.REPLACE,bucket="my-bucket")df1=shoots.get("sensor_data",bucket="my-bucket")print(df1)listing bucketsYou can use thebuckets()method to list available buckets:print("buckets:")print(shoots.buckets())buckets:['my-bucket','foo']deleting bucketsYou can delete buckets with thedelete_bucket()method. You can force a deletion of all the dataframes contained in a bucket by usingBucketDeleteMode.DELETE_CONTENTS, otherwise you need to delete all of the dataframes first.:print("buckets before deletion:")print(shoots.buckets())shoots.delete_bucket("my-bucket",mode=BucketDeleteMode.DELETE_CONTENTS)print("buckets after deletion:")print(shoots.buckets())bucketsbeforedeletion:['my-bucket','foo']bucketsafterdeletion:['foo']Running TestsThere are tests for checking running in insecure mode (no TLS or JWt), TLS only (no JWT), or JWT(also with TLS). The tests are designed to run without extra dependencies.To run the tests, navigate to the project directory, add the project directory to your python path, and run the tests (from the project directory):$exportPYTHONPATH="/path/to/shoots:$PYTHONPATH"$python3tests/run_tests.pyThis will run all 3 test casees in parallel.To run a single test, you can drop into the tests directory and run the test directly:$cdteststests$python3-munittesttls_test.TLSTestThere is an additional test for testing large data sets which is, by necessity, slow. As such it is not included in the ```run_tests.py`` program. To run this test:tests$python3-munittestlarge_datasets_test.LargeDatasetsTestLicenseThis edition of the code is licensed under the MIT license.RoadmapI intend to work on the following in the coming weeks, in no particular order:add a runtime option for the root bucket directory, use it for testingpip packagingpattern matching forlist()downsampling via sql on the servercombining dataframes on the servercompressing and cleaning dataframes on the serverauthenticationUI with SQL tree view browser and editor