Package: azure-storage-logging
azure-storage-loggingprovides functionality to send output from
the standard Python logging APIs to Microsoft Azure Storage.Dependenciesazure-storage 0.33 or newerInstallationInstall the package via pip:pip install azure-storage-loggingUsageThe moduleazure_storage_logging.handlersin the package contains
the following logging handler classes. Each of them uses a different
type of Microsoft Azure Storage to send its output to. They all are subclasses
of the standard Python logging handler classes, so you can make use of them
in the standard ways of Python logging configuration.In addition tothe standard formats for logging,
the special format%(hostname)sis also available in your message formatter
for the handlers. The format is introduced for ease of identifying the source
of log messages which come from many computers and go to the same storage.TableStorageHandlerTheTableStorageHandlerclass is a subclass oflogging.Handlerclass,
and it sends log messages to Azure table storage and store them
as entities in the specified table.The handler puts a formatted log message from applications in themessageproperty of a table entity along with some system-defined properties
(PartitionKey,RowKey, andTimestamp) like this:PartitionKeyRowKeyTimestampmessageXXXXXXXXXXXXXXYYYY-MM-DD …log messageXXXXXXXXXXXXXXYYYY-MM-DD …log messageXXXXXXXXXXXXXXYYYY-MM-DD …log messageclassazure_storage_logging.handlers.TableStorageHandler(account_name=None, account_key=None, protocol=’https’, table=’logs’, batch_size=0, extra_properties=None, partition_key_formatter=None, row_key_formatter=None, is_emulated=False)Returns a new instance of theTableStorageHandlerclass.
The instance is initialized with the name and the key of your
Azure Storage account and some optional parameters.Thetablespecifies the name of the table that stores log messages.
A new table will be created if it doesn’t exist. The table name must
conform to the naming convention for Azure Storage table, seethe naming convention for tablesfor more details.Theprotocolspecifies the protocol to transfer data between
Azure Storage and your application,httpandhttpsare supported.You can specify thebatch_sizein an integer if you want to use
batch transaction when creating new log entities. If thebatch_sizeis greater than 1, all new log entities will be transferred to the
table at a time when the number of new log messages reaches thebatch_size. Otherwise, a new log entity will be transferred to
the table every time a logging is performed. Thebatch_sizemust be
up to 100 (maximum number of entities in a batch transaction for
Azure Storage table).Theextra_propertiesaccepts a sequence ofthe formats for logging.
The handler-specific one%(hostname)sis also acceptable.
The handler assigns an entity property for every format specified inextra_properties. Here is an example of using extra properties:import logging
from azure_storage_logging.handlers import TableStorageHandler

# configure the handler and add it to the logger
logger = logging.getLogger('example')
handler = TableStorageHandler(account_name='mystorageaccountname',
                              account_key='mystorageaccountkey',
                              extra_properties=('%(hostname)s',
                                                '%(levelname)s'))
logger.addHandler(handler)

# output log messages
logger.info('info message')
logger.warning('warning message')
logger.error('error message')And it will create the log entities, that have the extra properties
in addition to the regular propertymessage, into the table like this:PartitionKeyRowKeyTimestamphostnamelevelnamemessageXXXXXXXXXXXXXXYYYY-MM-DD …myhostINFOinfo messageXXXXXXXXXXXXXXYYYY-MM-DD …myhostWARNINGwarn messageXXXXXXXXXXXXXXYYYY-MM-DD …myhostERRORerror messageYou can specify an instance of your customlogging.Formattersfor thepartition_key_formatteror therow_key_formatterif you want to implement your own keys for the table.
The default formatters will be used for partition keys and row keys
if no custom formatter for them is given to the handler.
The default values for partition keys are provided by the format%(asctime)sand the date format%Y%m%d%H%M(provides a unique
value per minute). The default values for row keys are provided by the
format%(asctime)s%(msecs)03d-%(hostname)s-%(process)d-%(rowno)02dand the date format%Y%m%d%H%M%S.Note that the format%(rowno)dis a handler-specific one only
available for row keys. It would be formatted to a sequential and
unique number in a batch that starts from 0. The format is introduced
to avoid collision of row keys generated in a batch, and it would
always be formatted to 0 if you don’t use batch transaction for logging
to the table.setPartitionKeyFormatter(fmt)Sets the handler’s formatter for partition keys tofmt.setRowKeyFormatter(fmt)Sets the handler’s formatter for row keys tofmt.QueueStorageHandlerTheQueueStorageHandlerclass is a subclass oflogging.Handlerclass,
and it pushes log messages to specified Azure storage queue.You can pop log messages from the queue in other applications
using Azure Storage client libraries.classazure_storage_logging.handlers.QueueStorageHandler(account_name=None, account_key=None, protocol=’https’, queue=’logs’, message_ttl=None, visibility_timeout=None, base64_encoding=False, is_emulated=False)Returns a new instance of theQueueStorageHandlerclass.
The instance is initialized with the name and the key of your
Azure Storage account and some optional parameters.Thequeuespecifies the name of the queue that log messages are added.
A new queue will be created if it doesn’t exist. The queue name must
conform to the naming convention for Azure Storage queue, seethe naming convention for queuesfor more details.Theprotocolspecifies the protocol to transfer data between
Azure Storage and your application,httpandhttpsare supported.Themessage_ttlspecifies the time-to-live interval for the message,
in seconds. The maximum time-to-live allowed is 7 days. If this
parameter is omitted, the default time-to-live is 7 days.Thevisibility_timeoutspecifies the visibility timeout value,
in seconds, relative to server time. If not specified, the default
value is 0 (makes the message visible immediately). The new value
must be larger than or equal to 0, and cannot be larger than 7 days.
Thevisibility_timeoutcannot be set to a value later than themessage_ttl, and should be set to a value smaller than themessage_ttl.Thebase64_encodingspecifies the necessity for encoding
log text in Base64. If you set this toTrue, Unicode log text
in a message is encoded in utf-8 first and then encoded in Base64.
Some of Azure Storage client libraries or tools assume that
text messages in Azure Storage queue are encoded in Base64,
so you can set this toTrueto receive log messages correctly
with those libraries or tools.BlobStorageRotatingFileHandlerTheBlobStorageRotatingFileHandlerclass is a subclass oflogging.handlers.RotatingFileHandlerclass. It performs
log file rotation and stores the outdated one in Azure blob storage
container when the current file reaches a certain size.classazure_storage_logging.handlers.BlobStorageRotatingFileHandler(filename, mode=’a’, maxBytes=0, encoding=None, delay=False, account_name=None, account_key=None, protocol=’https’, container=’logs’, zip_compression=False, max_connections=1, max_retries=5, retry_wait=1.0, is_emulated=False)Returns a new instance of theBlobStorageRotatingFileHandlerclass. The instance is initialized with the name and the key of your
Azure Storage account and some optional parameters.SeeRotatingFileHandlerfor its basic usage. The handler keeps the latest log file into the
local file system. Meanwhile, the handler sends the outdated log file
to the blob container immediately and then removes it from the local
file system.Thecontainerspecifies the name of the blob container that stores
outdated log files. A new container will be created if it doesn’t exist.
The container name must conform to the naming convention for
Azure Storage blob container, seethe naming convention for blob containersfor more details.Theprotocolspecifies the protocol to transfer data between
Azure Storage and your application,httpandhttpsare supported.Thezip_compressionspecifies the necessity for compressing
every outdated log file in zip format before putting it in
the container.Themax_connectionsspecifies a maximum number of parallel
connections to use when the blob size exceeds 64MB.
Set to 1 to upload the blob chunks sequentially.
Set to 2 or more to upload the blob chunks in parallel,
and this uses more system resources but will upload faster.Themax_retriesspecifies a number of times to retry
upload of blob chunk if an error occurs.Theretry_waitspecifies sleep time in secs between retries.The only two formatters%(hostname)sand%(process)dare
acceptable as a part of thefilenameor thecontainer. You can save
log files in a blob container dedicated to each host or process by
naming containers with these formatters, and also can store log files
from multiple hosts or processes in a blob container by naming log files
with them.Be careful to use the%(process)dformatter in thefilenamebecause inconsistent PIDs assigned to your application every time it
gets started are included as a part of the name of log files to search
for rotation. You should use the formatter in thefilenameonly when
the log file is generated by a long-running application process.Note that the hander class doesn’t take thebackupCountparameter,
unlike RotatingFileHandler does. The number of outdated log files
that the handler stores in the container is unlimited, and the files
are saved with the extension that indicates the time in UTC when
they are replaced with a new one. If you want to keep the amount of
outdated log files in the container in a certain number, you will
need to do that using Azure management portal or other tools.BlobStorageTimedRotatingFileHandlerTheBlobStorageTimedRotatingFileHandlerclass is a subclass oflogging.handlers.TimedRotatingFileHandlerclass. It performs
log file rotation and stores the outdated one to Azure blob storage
container at certain timed intervals.classazure_storage_logging.handlers.BlobStorageTimedRotatingFileHandler(filename, when=’h’, interval=1, encoding=None, delay=False, utc=False, account_name=None, account_key=None, protocol=’https’, container=’logs’, zip_compression=False, max_connections=1, max_retries=5, retry_wait=1.0, is_emulated=False)Returns a new instance of theBlobStorageTimedRotatingFileHandlerclass. The instance is initialized with the name and the key of your
Azure Storage account and some optional parameters.SeeTimedRotatingFileHandlerfor its basic usage. The handler keeps the latest log file into the
local file system. Meanwhile, the handler sends the outdated log file
to the blob container immediately and then removes it from the local
file system.Thecontainerspecifies the name of the blob container that stores
outdated log files. A new container will be created if it doesn’t exist.
The container name must conform to the naming convention for
Azure Storage blob container, seethe naming convention for blob containersfor more details.Theprotocolspecifies the protocol to transfer data between
Azure Storage and your application,httpandhttpsare supported.Thezip_compressionspecifies the necessity for compressing
every outdated log file in zip format before putting it in
the container.Themax_connectionsspecifies a maximum number of parallel
connections to use when the blob size exceeds 64MB.
Set to 1 to upload the blob chunks sequentially.
Set to 2 or more to upload the blob chunks in parallel,
and this uses more system resources but will upload faster.Themax_retriesspecifies a number of times to retry
upload of blob chunk if an error occurs.Theretry_waitspecifies sleep time in secs between retries.The only two formatters%(hostname)sand%(process)dare
acceptable as a part of thefilenameor thecontainer. You can save
log files in a blob container dedicated to each host or process by
naming containers with these formatters, and also can store log files
from multiple hosts or processes in a blob container by naming log files
with them.Be careful to use the%(process)dformatter in thefilenamebecause inconsistent PIDs assigned to your application every time it
gets started are included as a part of the name of log files to search
for rotation. You should use the formatter in thefilenameonly when
the log file is generated by a long-running application process.Note that the hander class doesn’t take thebackupCountparameter,
unlike TimedRotatingFileHandler does. The number of outdated log files
that the handler stores in the container is unlimited.
If you want to keep the amount of outdated log files in the container
in a certain number, you will need to do that using Azure
management portal or other tools.ExampleHere is an example of the configurations and the logging that uses
three different types of storage from the logger:LOGGING = {
    'version': 1,
    'formatters': {
        'simple': {
            'format': '%(asctime)s %(message)s',
        },
        'verbose': {
            'format': '%(asctime)s %(levelname)s %(hostname)s %(process)d %(message)s',
        },
        # this is the same as the default, so you can skip configuring it
        'partition_key': {
            'format': '%(asctime)s',
            'datefmt': '%Y%m%d%H%M',
        },
        # this is the same as the default, so you can skip configuring it
        'row_key': {
            'format': '%(asctime)s%(msecs)03d-%(hostname)s-%(process)d-%(rowno)02d',
            'datefmt': '%Y%m%d%H%M%S',
        },
    },
    'handlers': {
        'file': {
            'account_name': 'mystorageaccountname',
            'account_key': 'mystorageaccountkey',
            'protocol': 'https',
            'level': 'DEBUG',
            'class': 'azure_storage_logging.handlers.BlobStorageTimedRotatingFileHandler',
            'formatter': 'verbose',
            'filename': 'example.log',
            'when': 'D',
            'interval': 1,
            'container': 'logs-%(hostname)s',
            'zip_compression': False,
        },
        'queue': {
            'account_name': 'mystorageaccountname',
            'account_key': 'mystorageaccountkey',
            'protocol': 'https',
            'queue': 'logs',
            'level': 'CRITICAL',
            'class': 'azure_storage_logging.handlers.QueueStorageHandler',
            'formatter': 'verbose',
        },
        'table': {
            'account_name': 'mystorageaccountname',
            'account_key': 'mystorageaccountkey',
            'protocol': 'https',
            'table': 'logs',
            'level': 'INFO',
            'class': 'azure_storage_logging.handlers.TableStorageHandler',
            'formatter': 'simple',
            'batch_size': 20,
            'extra_properties': ['%(hostname)s', '%(levelname)s'],
            'partition_key_formatter': 'cfg://formatters.partition_key',
            'row_key_formatter': 'cfg://formatters.row_key',
        },
    },
    'loggers': {
        'example': {
            'handlers': ['file', 'queue', 'table'],
            'level': 'DEBUG',
        },
    }
}

import logging
from logging.config import dictConfig

dictConfig(LOGGING)
logger = logging.getLogger('example')
logger.debug('debug message')
logger.info('info message')
logger.warning('warning message')
logger.error('error message')
logger.critical('critical message')NoticeSetis_emulatedtoTrueat initialization of the logging handlers
if you want to use this package with Azure storage emulator.LicenseApache License 2.0CreditsMichiya Takahashi