Package: mlplatformutils
mlplatformutilsmlplatformutils package for observability and ML Pipeline ProcessingThis framework supports Azure Machine Learning training Pipeline supporting across computes such as Azure Synapse Spark, Virtual Machines Clusters, Azure Kubernetes Cluster, Azure Databricks. It supports reading/writing data from Azure Data Lake Gen2 in parquet and DELTA format, Azure Data Explorer (Kusto), Azure Sql DB instnces. The framework suports Python and Spark scalably. Writes with Spark with capabilties such a dynamic partitition overwrites, repartitioning are fully supported. In operating data reads and writes from such sources, The framework integrates built-in lineage framework providing column level lineage across the systems on a scalable Graph leveraging Azure Cosmos Gremlin Graph DB service. This enables a robust upstream dependency tracking and proactive alerting & eventing.  All operations are suported over Service Principal (Client Id, Client Secrets) for applications and processing. The package also provides creating and managing computes, PIP dependecies for Azure Machine Learning Workspace and the training definitions.Descriptionapp_insights_logger- ContainstelemetryloggerClass with Functions to Manage and Log Telemetry into Azure Application InsightstrackEventtrackTracetrackExceptionlogEventgather_event_detailslineagegraph- ContainsLineageGraphClass with functions to manage Graph on Azure Cosmos DB enabled with Gremlinadd_vertexget_verticesis_vertexupdate_vertexinsert_edgesdrop_vertexdrop_edgequery_graphupdate_lineage_graphconnect_lineage_graphplatformutils- Contains platform utility functions to check, install depedencies, check Azure ML Computeis_package_installedinstall_pipget_environmentset_environmentassert_amlcomputeread_setup_inisparkutils- Contains functions to read data from sources such as (Azure Data Lake Gen2, Azure Data Explorer (Kusto), Azure Sql Server) and write (Azure Data Lake Gen2)while ensuring integrated Lineage Graph Logging.read_from_adls_gen2write_to_adls_gen2read_from_kustosynapseread_from_kustoread_from_azsqlread_sstream_from_adls_gen1sparkcoreutils- Contains functions to read data from sources such as (Azure Data Lake Gen2, Azure Data Explorer (Kusto), Azure Sql Server, Azure Data Lake Gen1 Structured Streams) and write (Azure Data Lake Gen2)withoutintegrated Lineage Graph Logging.read_from_adls_gen2write_to_adls_gen2read_from_kustoread_from_azsqlread_sstream_from_adls_gen1pandasutils- Contains functions to read data from Azure Data Lake Gen2 (from Delta Format or Parquet Format) into Pandas Dataframe without Spark while ensuring integrated Lineage Graph Logging.read_from_delta_as_pandasread_parquet_file_from_adlsgen2_as_pandasread_parquet_directory_from_adlsgen2_as_pandaswrite_pandas_as_parquet_file_to_adlsgen2pandascoreutils- Contains functions to read data from Azure Data Lake Gen2 (from Delta Format or Parquet Format) into Pandas Dataframe without Sparkwithoutintegrated Lineage Graph Logging.read_from_delta_as_pandasread_parquet_file_from_adlsgen2_as_pandasread_parquet_directory_from_adlsgen2_as_pandaswrite_pandas_as_parquet_file_to_adlsgen2freshnessutils- Contains functions to add freshness details into Azure Cosmos (NoSQL) document db. This helps with the details on the freshness metrics on evaluating the SLA, and downstream processing. It captures and provides details on model, training dataset freshness for the most recent and historical processing.add_freshnessupsert_freshnessquery_freshnessExamplesfrom mlplatformutils.core.platformutils import is_package_installedprint(is_package_installed("pandas"))from mlplatformutils.core.app_insights_logger import telemetryloggerfrom mlplatformutils.core.lineagegraph import LineageGraphfrom mlplatformutils.core.sparkutils import write_to_adls_gen2, read_from_adls_gen2, read_sstream_from_adls_gen1from mlplatformutils.core.pandasutils import write_pandas_as_parquet_file_to_adlsgen2, read_parquet_directory_from_adlsgen2_as_pandasfrom mlplatformutils.core.sparkcoreutils import write_to_adls_gen2, read_from_adls_gen2, read_sstream_from_adls_gen1from mlplatformutils.core.pandascoreutils import write_pandas_as_parquet_file_to_adlsgen2, read_parquet_directory_from_adlsgen2_as_pandasfrom mlplatformutils.core.freshnessutils import add_freshness, upsert_freshness, query_freshnessimport mlplatformutils.core.version as vrprint(vr.__version__)NotesWhen Running this Lineage Package from Jupyter Nootebook, the below 3 Lines Help overcome JupyterNotebookRuntimeError: Cannot run the event loop while another loop is runningimport asyncioimport nest_asyncionest_asyncio.apply()Structure.|-- LICENSE.txt|-- README.rst|-- setup.cfg|-- setup.py|-- src|   |-- mlplatformutils|   |   |-- __init__.py|   |   |-- core|   |   |-- |-- __init__.py|   |   |-- |-- sparkcoreutils.py|   |   |-- |-- sparkutils.py|   |   |-- |-- platformutils.py|   |   |-- |-- pandascoreutils.py|   |   |-- |-- pandasutils.py|   |   |-- |-- lineagegraph.py|   |   |-- |-- freshnessutils.py|   |   |-- |-- app_insights_logger.py|-- tests|   |-- __init__.py|   |-- core|   |-- |--__init__.py|   |-- |-- test_sparkcoreutils.py|   |-- |-- test_sparkutils.py|   |-- |-- test_platformutils.py|   |-- |-- test_pandascoreutils.py|   |-- |-- test_pandasutils.py|   |-- |-- test_lineagegraph.py|   |-- |-- test_freshnessutils.py|   |-- |-- test_app_insights_logger.pyInstructionsinstall twine - twine is a utility package that is used for publishing Python packages on PyPIpython -m pip install twineBuild Package - create the source distribution of the packagepython setup.py sdistUpload Package to PyPI**python -m twine upload dist/***