Package: datajunction
DataJunction Python ClientThis is a short introduction into the Python version of the DataJunction (DJ) client.
For a full comprehensive intro into the DJ functionality please check outdatajunction.io.InstallationTo install:pip install datajunctionIntroWe have three top level client classes that help you choose the right path for your DataJunction actions.DJClientfor basic read only access to metrics, dimensions, SQL and data.DJBuilderfor those who would like to modify their DJ data model, build new nodes and/or modify the existing ones.DJAdminfor the administrators of the system to define the connections to your data catalog and engines.DJ Client : Basic AccessHere you can see how to access and use the most common DataJunction features.ExamplesTo initialize the client:fromdatajunctionimportDJClientdj=DJClient("http://localhost:8000")NOTEIf you are running in our demo docker environment please change the above URL to "http://dj:8000".You are now connected to your DJ service and you can start looking around. Let's see what namespaces we have in the system:dj.list_namespaces()['default']Next let's see what metrics and dimensions exist in thedefaultnamespace:dj.list_metrics(namespace="default")['default.num_repair_orders','default.avg_repair_price','default.total_repair_cost','default.avg_length_of_employment','default.total_repair_order_discounts','default.avg_repair_order_discounts','default.avg_time_to_dispatch']dj.list_dimensions(namespace="default")['default.date_dim','default.repair_order','default.contractor','default.hard_hat','default.local_hard_hats','default.us_state','default.dispatcher','default.municipality_dim']Now let's pick two metrics and see what dimensions they have in common:dj.common_dimensions(metrics=["default.num_repair_orders","default.total_repair_order_discounts"],name_only=True)['default.dispatcher.company_name','default.dispatcher.dispatcher_id','default.dispatcher.phone','default.hard_hat.address','default.hard_hat.birth_date','default.hard_hat.city',...And finally let's ask DJ to show us some data for these metrics and some dimensions:dj.data(metrics=["default.num_repair_orders","default.total_repair_order_discounts"],dimensions=["default.hard_hat.city"])|default_DOT_num_repair_orders|default_DOT_total_repair_order_discounts|city||-----------------------------|----------------------------------------|-----------||4|5475.110138|JerseyCity||3|11483.300049|Billerica||5|6725.170074|Southgate|...ReferenceList of all available DJ client methods:DJClient:listlist_namespaces( prefix: Optional[str])list_dimensions( namespace: Optional[str])list_metrics( namespace: Optional[str])list_cubes( namespace: Optional[str])list_sources( namespace: Optional[str])list_transforms( namespace: Optional[str])list_nodes( namespace: Optional[str], type_: Optional[NodeType])list_catalogs()list_engines()findcommon_dimensions( metrics: List[str], name_only: bool = False)common_metrics( dimensions: List[str], name_only: bool = False)executesql( metrics: List[str],
dimensions: Optional[List[str]],
filters: Optional[List[str]],
engine_name: Optional[str],
engine_version: Optional[str])data( metrics: List[str],
dimensions: Optional[List[str]],
filters: Optional[List[str]],
engine_name: Optional[str],
engine_version: Optional[str],
async_: bool = True)DJ Builder : Data ModellingIn this section we'll show you few examples to modify the DJ data model and its nodes.Start HereTo initialize the DJ builder:fromdatajunctionimportDJBuilderdjbuilder=DJBuilder("http://localhost:8000")NOTEIf you are running in our demo docker container please change the above URL to "http://dj:8000".NamespacesTo access a namespace or check if it exists you can use the same simple call:djbuilder.namespace("default")Namespace(dj_client=...,namespace='default')djbuilder.namespace("foo")[DJClientException]:Namespace`foo`doesnotexist.To create a namespace:djbuilder.create_namespace("foo")Namespace(dj_client=...,namespace='foo')To delete (or restore) a namespace:djbuilder.delete_namespace("foo")djbuilder.restore_namespace("foo")NOTE:Thecascadeparameter in both of above methods allows for cascading
effect applied to all underlying nodes and namespaces. Use it with caution!TagsYou can read existing tags as well as create new ones.djbuilder.tag(name="deprecated",description="This node has been deprecated.",tag_type="standard",tag_metadata={"contact":"Foo Bar"})Tag(dj_client=...,name='deprecated',description='This node has been deprecated.',tag_type='standard',tag_metadata={"contact":"Foo Bar"})djbuilder.tag("official")[DJClientException]:Tag`official`doesnotexist.To create a tag:djbuilder.create_tag(name="deprecated",description="This node has been deprecated.",tag_type="standard",tag_metadata={"contact":"Foo Bar"})Tag(dj_client=...,name="deprecated",description="This node has been deprecated.",tag_type="standard",tag_metadata={"contact":"Foo Bar"})To add a tag to a node:repair_orders=djbuilder.source("default.repair_orders")repair_orders.tags.append(djbuilder.tag("deprecated"))repair_orders.save()NodesTo learn whatNodemeans in the context of DJ, please check outthis datajuntion.io page.To list all (or some) nodes in the system you can use thelist_<node-type>()methods described
in theDJ Client : Basic Accesssection or you can use the namespace based method:All nodes for a given namespace can be found with:djbuilder.namespace("default").nodes()Specific node types can be retrieved with:djbuilder.namespace("default").sources()djbuilder.namespace("default").dimensions()djbuilder.namespace("default").metrics()djbuilder.namespace("default").transforms()djbuilder.namespace("default").cubes()To create a source node:repair_orders=djbuilder.create_source(name="repair_orders",display_name="Repair Orders",description="Repair orders",catalog="dj",schema_="roads",table="repair_orders",)Nodes can also be created in draft mode:repair_orders=djbuilder.create_source(...,mode=NodeMode.DRAFT)To create a dimension node:repair_order=djbuilder.create_dimension(name="default.repair_order_dim",query="""SELECTrepair_order_id,municipality_id,hard_hat_id,dispatcher_idFROM default.repair_orders""",description="Repair order dimension",primary_key=["repair_order_id"],)To create a transform node:large_revenue_payments_only=djbuilder.create_transform(name="default.large_revenue_payments_only",query="""SELECTpayment_id,payment_amount,customer_id,account_typeFROM default.revenueWHERE payment_amount > 1000000""",description="Only large revenue payments",)To create a metric:num_repair_orders=djbuilder.create_metric(name="default.num_repair_orders",query="""SELECTcount(repair_order_id)FROM repair_orders""",description="Number of repair orders",)ReferenceList of all available DJ builder methods:DJBuilder:namespacesnamespace( namespace: str)create_namespace( namespace: str)delete_namespace(self, namespace: str, cascade: bool = False)restore_namespace(self, namespace: str, cascade: bool = False)nodesdelete_node(self, node_name: str)restore_node(self, node_name: str)nodes: sourcesource(self, node_name: str)create_source( ..., mode: Optional[NodeMode] = NodeMode.PUBLISHED)register_table( catalog: str, schema: str, table: str)nodes: transformtransform(self, node_name: str)create_transform( ..., mode: Optional[NodeMode] = NodeMode.PUBLISHED)nodes: dimensiondimension(self, node_name: str)create_dimension( ..., mode: Optional[NodeMode] = NodeMode.PUBLISHED)nodes: metricmetric(self, node_name: str)create_metric( ..., mode: Optional[NodeMode] = NodeMode.PUBLISHED)nodes: cubecube(self, node_name: str)create_cube( ..., mode: Optional[NodeMode] = NodeMode.PUBLISHED)DJ System AdministrationIn this section we'll describe how to manage your catalog and engines.Start HereTo initialize the DJ admin:fromdatajunctionimportDJAdmindjadmin=DJAdmin("http://localhost:8000")NOTEIf you are running in our demo docker container please change the above URL to "http://dj:8000".ExamplesTo list available catalogs:djadmin.list_catalogs()['warehouse']To list available engines:djadmin.list_engines()[{'name':'duckdb','version':'0.7.1'}]To create a catalog:djadmin.add_catalog(name="my-new-catalog")To create a new engine:djadmin.add_engine(name="Spark",version="3.2.1",uri="http:/foo",dialect="spark")To linke an engine to a catalog:djadmin.link_engine_to_catalog(engine="Spark",version="3.2.1",catalog="my-new-catalog")ReferenceList of all available DJ builder methods:DJAdmin:Catalogslist_catalogs()  # in DJClientget_catalog( name: str)add_catalog( name: str)Engineslist_engines()  # in DJClientget_engine( name: str)add_engine( name: str,version: str, uri: Optional[str], dialect: Optional[str])Togetherlink_engine_to_catalog( engine_name: str, engine_version: str, catalog: str)