Package: opteryx
Query your data, where it lives.A unified SQL interface to unlock insights across your diverse data sources, from blobs stores to databases -effortless cross-platform data analytics.Documentation•Install•Examples•Get InvolvedWhat is Opteryx?Opteryx champions the SQL-on-everything approach, streamlining cross-platform data analytics by federating SQL queries across diverse data sources, including database systems like Postgres and datalake file formats like Parquet. The goal is to enhance your data analytics process by offering a unified way to access data from across your organization.Opteryx is a Python library that combines elements of in-process database engines like SQLite and DuckDB with federative features found in systems like Presto and Trino. The result is a versatile tool for querying data across multiple data sources in a seamless fashion.Opteryx offers the following features:SQL queries on data files generated by other processes, such as logsA command-line tool for filtering, transforming, and combining filesIntegration with familiar tools like pandas and PolarsEmbeddable as a low-cost engine, enabling portability and allowing for hundreds of analysts to leverage ad hoc databases with easeUnified and federated access to data on disk, in the cloud, and in on-premises databases, not only through the same interface but in the same queryWhy Use Opteryx?Familiar InterfaceOpteryx supports key parts of thePython DBAPIandSQL92 standardstandards which many analysts and engineers will already know how to use.Consistent SyntaxOpteryx creates a common SQL-layer over multiple data platforms, allowing backend systems to be upgraded, migrated or consolidated without changing any Opteryx code.Where possible, errors and warnings returned by Opteryx help the user to understand how to fix their statement to reduce time-to-success for even novice SQL users.Bring your own DataOpteryx supports multiple query engines, dataframe APIs and storage formats. You can mix-and-match sources in a single query. Opteryx can evenJOINdatasets stored in different formats and different platforms in the same query, such as Parquet and MySQL.Opteryx allows you to query your data directly in the systems where they are stored, eliminating the need to duplicate data into a common store for analytics. This saves you the cost and effort of maintaining duplicates.Opteryx can push parts of your query to the source query engine, allowing queries to run at the speed of the backend, rather than your local computer.And if there's not a connector in the box for your data platform; bespoke connectors can be added.Consumption-Based Billing FriendlyOpteryx is well-suited for deployments to environments which are pay-as-you-use, like Google Cloud Run. Great for situations where you have low-volume usage, or multiple environments, where the costs of many traditional database deployment can quickly add up.Python EcosystemOpteryx is Open Source Python, it quickly and easily integrates into Python code, including Jupyter Notebooks, so you can start querying your data within a few minutes. Opteryx integrates with many of your favorite Python data tools, you can use Opteryx to run SQL against pandas and Polars DataFrames, and even execute aJOINon an in-memory DataFrame and a remote SQL dataset.Time TravelDesigned for data analytics in environments where decisions need to be replayable, Opteryx allows you to query data as at a point in time in the past to replay decision algorithms against facts as they were known in the past. You can even self-join tables historic data, great for finding deltas in datasets over time.(data must be structured to enable temporal queries)FastBenchmarks on M2 Pro Mac running an ad hocGROUP BYover a 6 million row parquet file via the CLI in ~1/4th of a second from a cold start (no caching and predefined schema).(different systems will have different performance characteristics)Instant ElasticityDesigned to run in Knative and similar environments like Google Cloud Run, Opteryx can scale down to zero, and scale up to respond to thousands of concurrent queries within seconds.InstallInstalling from PyPI is recommended.pipinstallopteryxTo build Opteryx from source, refer to thecontribution guides.Opteryx installs with a small set of libraries it needs for core functionality, such asNumpy,PyArrow, andorjson. Some features require additional libraries to be installed, you are notified of these libraries as they are required.ExamplesFilter a Dataset on the Command LineExecute a Simple Query in PythonExecute SQL on a pandas DataFrameQuery Data on Local DiskQuery Data on GCSQuery Data in SQLiteTry Opteryx now using ourinteractive labsonBinder.Filter a Dataset on the Command LineIn this example, we are running Opteryx from the command line to filter one of the internal example datasets and display the results on the console.python-mopteryx"SELECT * FROM \$astronauts WHERE 'Apollo 11' IN UNNEST(missions);"this example is complete and should run as-isExecute a Simple Query in PythonIn this example, we are showing the basic usage of the Python API by executing a simple query that makes no references to any datasets.# Import the Opteryx SQL query engine library.importopteryx# Execute a SQL query to evaluate the expression 4 * 7.# The result is stored in the 'result' variable.result=opteryx.query("SELECT 4 * 7;")# Display the first row(s) of the result to verify the query executed correctly.result.head()ID4 * 7128this example is complete and should run as-isExecute SQL on a pandas DataFrameIn this example, we are running a SQL statement on a pandas DataFrame and returning the result as a new pandas DataFrame.# Required importsimportopteryximportpandas# Read data from the exoplanets.csv file hosted on Google Cloud Storage# The resulting DataFrame is stored in the variable `pandas_df`.pandas_df=pandas.read_csv("https://storage.googleapis.com/opteryx/exoplanets/exoplanets.csv")# Register the pandas DataFrame with Opteryx under the alias "exoplanets"# This makes the DataFrame available for SQL-like queries.opteryx.register_df("exoplanets",pandas_df)# Perform an SQL query to group the data by `koi_disposition` and count the number# of occurrences of each distinct `koi_disposition`.# The result is stored in `aggregated_df`.aggregated_df=opteryx.query("SELECT koi_disposition, COUNT(*) FROM exoplanets GROUP BY koi_disposition;").pandas()# Display the aggregated DataFrame to get a preview of the result.aggregated_df.head()koi_disposition  COUNT(*)
0       CONFIRMED      2293
1  FALSE POSITIVE      5023
2       CANDIDATE      2248this example is complete and should run as-isQuery Data on Local DiskIn this example, we are querying and filtering a file directly. This example will not run as written because the file being queried does not exist.# Import the Opteryx query engine.importopteryx# Execute a SQL query to select the first 5 rows from the 'space_missions.parquet' table.# The result will be stored in the 'result' variable.result=opteryx.query("SELECT * FROM 'space_missions.parquet' LIMIT 5;")# Display the result.# This is useful for quick inspection of the data.result.head()IDCompanyLocationPriceLaunched_atRocketRocket_StatusMissionMission_Status0RVSN USSRSite 1/5, Baikonur Cosmodrome,null1957-10-04 19:28:00Sputnik 8K71PSRetiredSputnik-1Success1RVSN USSRSite 1/5, Baikonur Cosmodrome,null1957-11-03 02:30:00Sputnik 8K71PSRetiredSputnik-2Success2US NavyLC-18A, Cape Canaveral AFS, Flnull1957-12-06 16:44:00VanguardRetiredVanguard TV3Failure3AMBALC-26A, Cape Canaveral AFS, Flnull1958-02-01 03:48:00Juno IRetiredExplorer 1Success4US NavyLC-18A, Cape Canaveral AFS, Flnull1958-02-05 07:33:00VanguardRetiredVanguard TV3BUFailurethis example requires a data file,space_missions.parquet.Query Data in SQLiteIn this example, we are querying a SQLite database via Opteryx. This example will not run as written because the file being queried does not exist.# Import the Opteryx query engine and the SqlConnector from its connectors module.importopteryxfromopteryx.connectorsimportSqlConnector# Register a new data store with the prefix "sql", specifying the SQL Connector to handle it.# This allows queries with the 'sql' prefix to be routed to the appropriate SQL database.opteryx.register_store(prefix="sql",# Prefix for distinguishing this particular storeconnector=SqlConnector,# Specify the connector to handle queries for this storeremove_prefix=True,# Remove the prefix from the table name when querying SQLiteconnection="sqlite:///database.db"# SQLAlchemy connection string for the SQLite database)# Execute a SQL query to select specified columns from the 'planets' table in the SQL store,# limiting the output to 5 rows. The result is stored in the 'result' variable.result=opteryx.query("SELECT name, mass, diameter, density FROM sql.planets LIMIT 5;")# Display the result.# This is useful for quickly verifying that the query executed correctly.result.head()IDnamemassdiameterdensity1Mercury0.33487954272Venus4.871210452433Earth5.971275655144Mars0.642679239335Jupiter1898.01429841326this example requires a data file,database.db.Query Data on GCSIn this example, we are to querying a dataset on GCS in a public bucket called 'opteryx'.# Import the Opteryx query engine and the GcpCloudStorageConnector from its connectors module.importopteryxfromopteryx.connectorsimportGcpCloudStorageConnector# Register a new data store named 'opteryx', specifying the GcpCloudStorageConnector to handle it.# This allows queries for this particular store to be routed to the appropriate GCP Cloud Storage bucket.opteryx.register_store("opteryx",# Name of the store to registerGcpCloudStorageConnector# Connector to handle queries for this store)# Execute a SQL query to select all columns from the 'space_missions' table located in the 'opteryx' store,# and limit the output to 5 rows. The result is stored in the 'result' variable.result=opteryx.query("SELECT * FROM opteryx.space_missions LIMIT 5;")# Display the result.# This is useful for quickly verifying that the query executed correctly.result.head()IDCompanyLocationPriceLaunched_atRocketRocket_StatusMissionMission_Status0RVSN USSRSite 1/5, Baikonur Cosmodrome,null1957-10-04 19:28:00Sputnik 8K71PSRetiredSputnik-1Success1RVSN USSRSite 1/5, Baikonur Cosmodrome,null1957-11-03 02:30:00Sputnik 8K71PSRetiredSputnik-2Success2US NavyLC-18A, Cape Canaveral AFS, Flnull1957-12-06 16:44:00VanguardRetiredVanguard TV3Failure3AMBALC-26A, Cape Canaveral AFS, Flnull1958-02-01 03:48:00Juno IRetiredExplorer 1Success4US NavyLC-18A, Cape Canaveral AFS, Flnull1958-02-05 07:33:00VanguardRetiredVanguard TV3BUFailurethis example is complete and should run as-isFurther ExamplesFor further examples, check out theinteractive labsonBinder.CommunityGet InvolvedStar this repoContribute— join us in building Opteryx, through writing code, or inspiring others to use it.Let us knowyour ideas, how you are using Opteryx, or report abug or feature request.See thecontributor documentationfor Opteryx. It's easy to get started, and we're really friendly if you need any help!If you're interested in contributing to the code now, check outGitHub issues. Feel free to ask questions or open a draft PR.SecuritySee the projectSecurity Policyfor information about reporting vulnerabilities.LicenseOpteryx is licensed underApache 2.0except where specific modules note otherwise.StatusOpteryx is in beta. Beta means different things to different people, to us, being beta means:Core functionality has good regression test coverage to help ensure stabilitySome edge cases may have undetected bugsPerformance tuning is incompleteChanges are focused on feature completion, bugs, performance, reducing debt, and securityCode structure and APIs are not stable and may changeRelated ProjectsorsoDataFrame librarymabelStreaming data APIsmesosMySQL connector for Opteryx