Package: dremio-simple-query
dremio_simple_queryThe purpose of this library is to easily query a Dremio source using Arrow Flight for analytics.LEARN MORE ABOUT DREMIOUse Dremio to Help:Govern your dataJoin your data across sources (Iceberg, Delta, S3, JSON, CSV, RDBMS, and more)Accelerate your queries across data sourcesReduce your Data Warehouse WorkloadsWith this library your analysts can more easily get their data from Dremio and easily get to work running local analytics with Arrow, Pandas, Polars and DuckDB. This library can grab large datasets performantly thanks to using Apache Arrow Flight.Getting Your URI and TokenProtocolEndpointResultDremio Cloud (NA)grpc+tls://data.dremio.cloud:443grpc+tls://data.dremio.cloud:443Dremio Cloud (EU)grpc+tls://data.eu.dremio.cloud:443grpc+tls://data.eu.dremio.cloud:443Dremio Software (SSL)grpc+tls://<ip-address>:32010grpc+tls://<ip-address>:32010Dremio Software (NoSSL)grpc://<ip-address>:32010grpc://<ip-address>:32010Getting your tokenFor Dremio Cloud can get token from interface or REST APIFor Dremio Software can get token from Rest APIThe get_token function is included to help get the token from the Dremio Rest API.fromdremio_simple_query.connectimportget_token,DremioConnection## URL to Login Endpointlogin_endpoint="http://localhost:9047/apiv2/login"## Payload for Loginpayload={"userName":username,"password":password}## Get token from APItoken=get_token(uri=login_endpoint,payload=payload)## URL Dremio Software Flight Endpointarrow_endpoint="grpc://localhost:32010"## Establish Clientdremio=DremioConnection(token,arrow_endpoint)Setting up your connectionfromdremio_simple_query.connectimportDremioConnectionfromosimportgetenvfromdotenvimportload_dotenvload_dotenv()## Dremio Person Tokentoken=getenv("TOKEN")## Arrow Endpoint (See Dremio Documentation)uri=getenv("ARROW_ENDPOINT")## Create Dremio Arrow Connectiondremio=DremioConnection(token,uri)Query (Get Arrow Back)If you want to get Arrow Data back you can run a query like so.stream=dremio.toArrow("SELECT * FROM arctic.table1;")The.toArrowmethod returns aFlightStreamReaderobject which can be converted into typical Arrow objects.Arrow Tablearrow_table=stream.read_all()Arrow RecordBatchReaderbatch_reader=stream.to_reader()toPandas (Get Pandas Dataframe Back)df=dremio.toPandas("SELECT * FROM arctic.table1;")toPolars (Get Polars Dataframe Back)df=dremio.toPolars("SELECT * FROM arctic.table1;")Querying with DuckDBUsing the DuckDB Relation APIUsing the.toDuckDBmethod the query results will be returned as a DuckDB relation.duck_rel=dremio.toDuckDB("SELECT * FROM arctic.table1")result=duck_rel.query("table1","SELECT * from table1").fetchall()result2=duck_rel.filterprint(result)Querying Arrow Objects with DuckDBfromdremio_simple_query.connectimportDremioConnectionfromosimportgetenvfromdotenvimportload_dotenvimportduckdb## DuckDB Connectioncon=duckdb.connection()load_dotenv()## Dremio Person Tokentoken=getenv("TOKEN")## Arrow Endpoint (See Dremio Documentation)uri=getenv("ARROW_ENDPOINT")## Create Dremio Arrow Connectiondremio=DremioConnection(token,uri)## Get Data from Dremiostream=dremio.toArrow("SELECT * FROM arctic.table1;")## Turn into Arrow Tablemy_table=stream.read_all()## Query with Duckdbresults=con.execute("SELECT * FROM my_table;").fetchall()print(results)