Package: dbl-discoverx
DiscoverXMulti-table operations over the lakehouse.Run a single command to execute operations across many tables.Operations examplesOperations are applied concurrently across multiple tablesMaintenanceVACUUM all tables(example notebook)OPTIMIZE with z-order on tables having specified columnsDetect tables having too many small files (example notebook)Visualise quantity of data written per table per periodGovernancePII detection with Presidio (example notebook)Text Analysis with MosaicML and Databricks MLflow (example notebook)Text Analysis with OpenAI GPT (example notebook)GDPR right of access: extract user data from all tables at onceGDPR right of erasure: delete user data from all tables at onceSearch in any columnUpdate Owner of Data Objects (example notebook)Semantic classificationSemantic classification of columns by semantic class: email, phone number, IP address, etc.Select data based on semantic classesDelete data based on semantic classesCustomArbitrary SQL template execution across multiple tablesCreate Mlflow gateway routes for MosaicML and OpenAI (example notebook)Scan using User Specified Data Source Formats (example notebook)Getting startedInstall DiscoverX, in Databricks notebook type%pip install dbl-discoverxGet startedfrom discoverx import DX
dx = DX(locale="US")You can now run operations across multiple tables.As an illustration, consider the scenario where you need to retrieve a single row from various tables within a catalog that begins with "dev_" and includes the term "sample" in their names. To achieve this, the following code block utilizes the dx.from_tables function, which applies an SQL query to extract JSON-formatted data:dx.from_tables("dev_*.*.*sample*")\
  .with_sql("SELECT to_json(struct(*)) AS row FROM {full_table_name} LIMIT 1")\
  .apply()Available functionalityThe availabledxfunctions arefrom_tables("<catalog>.<schema>.<table>")selects tables based on the specified pattern (use*as a wildcard). Returns aDataExplorerobject with methodshaving_columnsrestricts the selection to tables that have the specified columnswith_concurrencydefines how many queries are executed concurrently (10 by defailt)with_sqlapplies a SQL template to all tables. After this command you can apply anaction. See in-depth documentationhere.unpivot_string_columnsreturns a melted (unpivoted) dataframe with all string columns from the selected tables. After this command you can apply anactionscan(experimental) scans the lakehouse with regex expressions defined by the rules and to power the semantic classification.introgives an introduction to the libraryscanscans the lakehouse with regex expressions defined by the rules and to power the semantic classification.Documentationdisplay_rulesshows the rules available for semantic classificationsearchsearches the lakehouse content for by leveraging the semantic classes identified with scan (eg. email, ip address, etc.).Documentationselect_by_classselects data from the lakehouse content by semantic class.Documentationdelete_by_classdeletes from the lakehouse by semantic class.Documentationfrom_tables ActionsAfter awith_sqlorunpivot_string_columnscommand, you can apply the following actions:explainexplains the queries that would be executeddisplayexecutes the queries and shows the first 1000 rows of the result in a unioned dataframeapplyreturns a unioned dataframe with the result from the queriesRequirementsADatabricks workspaceUnity CatalogProject SupportPlease note that all projects in the /databrickslabs github account are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs).  They are provided AS-IS and we do not make any guarantees of any kind.  Please do not submit a support ticket relating to any issues arising from the use of these projects.Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo.  They will be reviewed as time permits, but there are no formal SLAs for support.