Package: azure-monitor-ingestion
Azure Monitor Ingestion client library for PythonThe Azure Monitor Ingestion client library is used to send custom logs toAzure Monitorusing theLogs Ingestion API.This library allows you to send data from virtually any source to supported built-in tables or to custom tables that you create in Log Analytics workspace. You can even extend the schema of built-in tables with custom columns.Resources:Source codePackage (PyPI)Package (Conda)API reference documentationService documentationSamplesChange logGetting startedPrerequisitesPython 3.7 or laterAnAzure subscriptionAnAzure Log Analytics workspaceAData Collection EndpointAData Collection RuleInstall the packageInstall the Azure Monitor Ingestion client library for Python withpip:pipinstallazure-monitor-ingestionCreate the clientAn authenticated client is required to upload Logs to Azure Monitor. The library includes both synchronous and asynchronous forms of the clients. To authenticate, create an instance of a token credential. Use that instance when creating aLogsIngestionClient. The following examples useDefaultAzureCredentialfrom theazure-identitypackage.Synchronous clientsConsider the following example, which creates synchronous clients for uploading logs:importosfromazure.identityimportDefaultAzureCredentialfromazure.monitor.ingestionimportLogsIngestionClientendpoint=os.environ['DATA_COLLECTION_ENDPOINT']credential=DefaultAzureCredential()logs_client=LogsIngestionClient(endpoint,credential)Asynchronous clientsThe asynchronous forms of the client APIs are found in the.aio-suffixed namespace. For example:importosfromazure.identity.aioimportDefaultAzureCredentialfromazure.monitor.ingestion.aioimportLogsIngestionClientendpoint=os.environ['DATA_COLLECTION_ENDPOINT']credential=DefaultAzureCredential()logs_client=LogsIngestionClient(endpoint,credential)Configure clients for non-public Azure cloudsBy default,LogsIngestionClientis configured to connect to the public Azure cloud. To connect to non-public Azure clouds, some additional configuration is required. The appropriate scope for authentication must be provided using thecredential_scopeskeyword argument. The following example shows how to configure the client to connect to Azure US Government:logs_client=LogsIngestionClient(endpoint,credential_scopes=["https://monitor.azure.us//.default"])Key conceptsData Collection EndpointData Collection Endpoints (DCEs) allow you to uniquely configure ingestion settings for Azure Monitor.This articleprovides an overview of data collection endpoints including their contents and structure and how you can create and work with them.Data Collection RuleData collection rules (DCR) define data collected by Azure Monitor and specify how and where that data should be sent or stored. The REST API call must specify a DCR to use. A single DCE can support multiple DCRs, so you can specify a different DCR for different sources and target tables.The DCR must understand the structure of the input data and the structure of the target table. If the two don't match, it can use a transformation to convert the source data to match the target table. You may also use the transform to filter source data and perform any other calculations or conversions.For more information, seeData collection rules in Azure Monitor, and seethis articlefor details about a DCR's structure. For information on how to retrieve a DCR ID, seethis tutorial.Log Analytics workspace tablesCustom logs can send data to any custom table that you create and to certain built-in tables in your Log Analytics workspace. The target table must exist before you can send data to it. The following built-in tables are currently supported:CommonSecurityLogSecurityEventsSyslogWindowsEventsLogs retrievalThe logs that were uploaded using this library can be queried using theAzure Monitor Queryclient library.ExamplesUpload custom logsUpload with custom error handlingUpload custom logsThis example shows uploading logs to Azure Monitor.importosfromazure.core.exceptionsimportHttpResponseErrorfromazure.identityimportDefaultAzureCredentialfromazure.monitor.ingestionimportLogsIngestionClientendpoint=os.environ['DATA_COLLECTION_ENDPOINT']credential=DefaultAzureCredential()client=LogsIngestionClient(endpoint=endpoint,credential=credential,logging_enable=True)rule_id=os.environ['LOGS_DCR_RULE_ID']body=[{"Time":"2021-12-08T23:51:14.1104269Z","Computer":"Computer1","AdditionalContext":"context-2"},{"Time":"2021-12-08T23:51:14.1104269Z","Computer":"Computer2","AdditionalContext":"context"}]try:client.upload(rule_id=rule_id,stream_name=os.environ['LOGS_DCR_STREAM_NAME'],logs=body)exceptHttpResponseErrorase:print(f"Upload failed:{e}")Upload with custom error handlingTo upload logs with custom error handling, you can pass a callback function to theon_errorparameter of theuploadmethod. The callback function is called for each error that occurs during the upload and should expect one argument that corresponds to anLogsUploadErrorobject. This object contains the error encountered and the list of logs that failed to upload.# Example 1: Collect all logs that failed to upload.failed_logs=[]defon_error(error):print("Log chunk failed to upload with error: ",error.error)failed_logs.extend(error.failed_logs)# Example 2: Ignore all errors.defon_error_pass(error):passclient.upload(rule_id=rule_id,stream_name=os.environ['LOGS_DCR_STREAM_NAME'],logs=body,on_error=on_error)TroubleshootingFor details on diagnosing various failure scenarios, see ourtroubleshooting guide.Next stepsTo learn more about Azure Monitor, see theAzure Monitor service documentation.SamplesThe following code samples show common scenarios with the Azure Monitor Ingestion client library.Logs Ingestion samplesUpload a list of logs(async sample)Upload a list of logs with custom error handling(async sample)Upload the contents of a file(async sample)Upload data in a pandas DataFrame(async sample)ContributingThis project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visitcla.microsoft.com.When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repositories using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct. For more information, see theCode of Conduct FAQor contactopencode@microsoft.comwith any additional questions or comments.Release History1.0.3 (2023-11-07)Other ChangesAdd type validation for thelogsparameter in theuploadmethod. (#32591)1.0.2 (2023-06-15)Bugs FixedFixed issue preventing custom authentication policies or credential scopes to be passed to the client. (#30739)1.0.1 (2023-04-11)Bugs FixedFixed an issue where log entry sizes were miscalculated when chunking. (#29584)1.0.0 (2023-02-16)Features AddedAdded newon_errorparameter to theuploadmethod to allow users to handle errors in their own way.AnLogsUploadErrorclass was added to encapsulate information about the error. An instance of this class is passed to theon_errorcallback.Added IO support for upload. Now IO streams can be passed in using thelogsparameter. (#28373)Breaking ChangesRemoved support for max_concurrencyOther ChangesRemovedmsrestdependency.Added requirement forisodate>=0.6.0(isodatewas required bymsrest).Added requirement fortyping-extensions>=4.0.1.1.0.0b1 (2022-07-15)FeaturesVersion (1.0.0b1) is the first preview of our efforts to create a user-friendly and Pythonic client library for Azure Monitor Ingestion.
For more information about this, and preview releases of other Azure SDK libraries, please visithttps://azure.github.io/azure-sdk/releases/latest/python.html.Added~azure.monitor.ingestion.LogsIngestionClientto send logs to Azure Monitor along with~azure.monitor.ingestion.aio.LogsIngestionClient.