Package: pingpong-datahub
Metadata IngestionThis module hosts an extensible Python-based metadata ingestion system for DataHub.
This supports sending data to DataHub using Kafka or through the REST API.
It can be used through our CLI tool, with an orchestrator like Airflow, or as a library.Getting StartedPrerequisitesBefore running any metadata ingestion job, you should make sure that DataHub backend services are all running. If you are trying this out locally, the easiest way to do that is throughquickstart Docker images.Install from PyPIThe folks over atAcryl Datamaintain a PyPI package for DataHub metadata ingestion.# Requires Python 3.6+python3-mpipinstall--upgradepipwheelsetuptools
python3-mpipinstall--upgradeacryl-datahub
datahubversion# If you see "command not found", try running this instead: python3 -m datahub versionIf you run into an error, try checking thecommon setup issues.Installing PluginsWe use a plugin architecture so that you can install only the dependencies you actually need. Click the plugin name to learn more about the specific source recipe and any FAQs!Sources:Plugin NameInstall CommandProvidesfileincluded by defaultFile source and sinkathenapip install 'acryl-datahub[athena]'AWS Athena sourcebigquerypip install 'acryl-datahub[bigquery]'BigQuery sourcebigquery-usagepip install 'acryl-datahub[bigquery-usage]'BigQuery usage statistics sourcedatahub-business-glossaryno additional dependenciesBusiness Glossary File sourcedbtno additional dependenciesdbt sourcedruidpip install 'acryl-datahub[druid]'Druid Sourcefeastpip install 'acryl-datahub[feast]'Feast sourcegluepip install 'acryl-datahub[glue]'AWS Glue sourcehivepip install 'acryl-datahub[hive]'Hive sourcekafkapip install 'acryl-datahub[kafka]'Kafka sourcekafka-connectpip install 'acryl-datahub[kafka-connect]'Kafka connect sourceldappip install 'acryl-datahub[ldap]'(extra requirements)LDAP sourcelookerpip install 'acryl-datahub[looker]'Looker sourcelookmlpip install 'acryl-datahub[lookml]'LookML source, requires Python 3.7+metabasepip install 'acryl-datahub[metabase]Metabase sourcemodepip install 'acryl-datahub[mode]'Mode Analytics sourcemongodbpip install 'acryl-datahub[mongodb]'MongoDB sourcemssqlpip install 'acryl-datahub[mssql]'SQL Server sourcemysqlpip install 'acryl-datahub[mysql]'MySQL sourcemariadbpip install 'acryl-datahub[mariadb]'MariaDB sourceopenapipip install 'acryl-datahub[openapi]'OpenApi Sourceoraclepip install 'acryl-datahub[oracle]'Oracle sourcepostgrespip install 'acryl-datahub[postgres]'Postgres sourceredashpip install 'acryl-datahub[redash]'Redash sourceredshiftpip install 'acryl-datahub[redshift]'Redshift sourcesagemakerpip install 'acryl-datahub[sagemaker]'AWS SageMaker sourcesnowflakepip install 'acryl-datahub[snowflake]'Snowflake sourcesnowflake-usagepip install 'acryl-datahub[snowflake-usage]'Snowflake usage statistics sourcesql-profilespip install 'acryl-datahub[sql-profiles]'Data profiles for SQL-based systemssqlalchemypip install 'acryl-datahub[sqlalchemy]'Generic SQLAlchemy sourcesupersetpip install 'acryl-datahub[superset]'Superset sourcetableaupip install 'acryl-datahub[tableau]'Tableau sourcetrinopip install 'acryl-datahub[trino]Trino sourcestarburst-trino-usagepip install 'acryl-datahub[starburst-trino-usage]'Starburst Trino usage statistics sourcenifipip install 'acryl-datahub[nifi]Nifi sourceSinksPlugin NameInstall CommandProvidesfileincluded by defaultFile source and sinkconsoleincluded by defaultConsole sinkdatahub-restpip install 'acryl-datahub[datahub-rest]'DataHub sink over REST APIdatahub-kafkapip install 'acryl-datahub[datahub-kafka]'DataHub sink over KafkaThese plugins can be mixed and matched as desired. For example:pipinstall'acryl-datahub[bigquery,datahub-rest]'You can check the active plugins:datahubcheckpluginsBasic Usagepipinstall'acryl-datahub[datahub-rest]'# install the required plugindatahubingest-c./examples/recipes/example_to_datahub_rest.ymlThe--dry-runoption of theingestcommand performs all of the ingestion steps, except writing to the sink. This is useful to ensure that the
ingestion recipe is producing the desired workunits before ingesting them into datahub.# Dry rundatahubingest-c./examples/recipes/example_to_datahub_rest.yml--dry-run# Short-formdatahubingest-c./examples/recipes/example_to_datahub_rest.yml-nThe--previewoption of theingestcommand performs all of the ingestion steps, but limits the processing to only the first 10 workunits produced by the source.
This option helps with quick end-to-end smoke testing of the ingestion recipe.# Previewdatahubingest-c./examples/recipes/example_to_datahub_rest.yml--preview# Preview with dry-rundatahubingest-c./examples/recipes/example_to_datahub_rest.yml-n--previewInstall using DockerIf you don't want to install locally, you can alternatively run metadata ingestion within a Docker container.
We have prebuilt images available onDocker hub. All plugins will be installed and enabled automatically.Limitation: the datahub_docker.sh convenience script assumes that the recipe and any input/output files are accessible in the current working directory or its subdirectories. Files outside the current working directory will not be found, and you'll need to invoke the Docker image directly.# Assumes the DataHub repo is cloned locally../metadata-ingestion/scripts/datahub_docker.shingest-c./examples/recipes/example_to_datahub_rest.ymlInstall from sourceIf you'd like to install from source, see thedeveloper guide.RecipesA recipe is a configuration file that tells our ingestion scripts where to pull data from (source) and where to put it (sink).
Here's a simple example that pulls metadata from MSSQL and puts it into datahub.# A sample recipe that pulls metadata from MSSQL and puts it into DataHub# using the Rest API.source:type:mssqlconfig:username:sapassword:${MSSQL_PASSWORD}database:DemoDatatransformers:-type:"fully-qualified-class-name-of-transformer"config:some_property:"some.value"sink:type:"datahub-rest"config:server:"http://localhost:8080"Running a recipe is quite easy.datahubingest-c./examples/recipes/mssql_to_datahub.ymlA number of recipes are included in theexamples/recipesdirectory. For full info and context on each source and sink, see the pages described in thetable of plugins.Handling sensitive information in recipesWe automatically expand environment variables in the config (e.g.${MSSQL_PASSWORD}),
similar to variable substitution in GNU bash or in docker-compose files. For details, seehttps://docs.docker.com/compose/compose-file/compose-file-v2/#variable-substitution. This environment variable substitution should be used to mask sensitive information in recipe files. As long as you can get env variables securely to the ingestion process there would not be any need to store sensitive information in recipes.TransformationsIf you'd like to modify data before it reaches the ingestion sinks – for instance, adding additional owners or tags – you can use a transformer to write your own module and integrate it with DataHub.Check out thetransformers guidefor more info!Using as a libraryIn some cases, you might want to construct Metadata events directly and use programmatic ways to emit that metadata to DataHub. In this case, take a look at thePython emitterand theJava emitterlibraries which can be called from your own code.Programmatic PipelineIn some cases, you might want to configure and run a pipeline entirely from within your custom python script. Here is an example of how to do it.programmatic_pipeline.py- a basic mysql to REST programmatic pipeline.DevelopingSee the guides ondeveloping,adding a sourceandusing transformers.