Differential logging of computing processes

Techniques and solutions are provided to facilitate logging of computing processes. A computing process generates multiple log entries. A first portion of the multiple log entries are written to a first log. A second portion of the multiple log entries are written to the first log and to a second log, where the second portion is less than the first portion, such as being a proper subset of the first portion. Log entries can be determined to be written to the second log by scanning all or a portion of a log, by evaluating log entries as they are generated, or through computer code that causes log entries to be written to the second log. Typically, the second log is maintained for a longer period of time than the first log.

FIELD

The present disclosure generally relates to logging associated with computing processes. Particular implementations provide that a first portion of log entries are written to a first log and a second portion of log entries are written to a second log, where in particular examples the second portion is a proper subset of the first portion.

BACKGROUND

Many computing processes involve the production of logs. Logs can be used for various purposes, and the information needed for a particular log can depend on particular purposes for which the log may be used. Some uses may be more immediate than others. Particularly for uses where log data should be maintained for long period of time, maintaining log data may not be feasible. In such cases, logs may be discarded, or logs may be maintained, but at a cost of storage resources and potentially computing resources needed to process the logs. Accordingly, room for improvement exists.

SUMMARY

Techniques and solutions are provided to facilitate logging of computing processes. A computing process generates multiple log entries. A first portion of the multiple log entries are written to a first log. A second portion of the multiple log entries are written to the first log and to a second log, where the second portion is less than the first portion, such as being a proper subset of the first portion. Log entries can be determined to be written to the second log by scanning all or a portion of a log, by evaluating log entries as they are generated, or through computer code that causes log entries to be written to the second log. Typically, the second log is maintained for a longer period of time than the first log.

In one aspect, the present disclosure provides a method for differential logging. A computer-implemented process is executed, such as a process associated with database backup or recovery. During execution of the computer-implemented process, a first log entry is generated. It is determined that the first log entry is to be stored in a first log. The first log entry is caused to be stored in the first log.

During execution of the computer-implemented process, a second log entry is generated. It is determined that the second log entry is to be stored in the first log. The second log entry is caused to be stored in the first log. It is determined that the second log entry is to be stored in a second log. The second log entry is caused to be stored in the second log. The first log entry is not stored in the second log.

Although the terms “first” and “second” have been used with respect to the method, it should be appreciated that these terms are used to convey that the log entries are different, not that one log entry is generated before the other. For example, in some cases the second log entry is generated and stored in the first and second logs before the first log entry is generated.

The present disclosure also includes computing systems and tangible, non-transitory computer readable storage media configured to carry out, or including instructions for carrying out, an above-described method (or operations). As described herein, a variety of other features and advantages can be incorporated into the technologies as desired.

DETAILED DESCRIPTION

Many computing processes involve the production of logs. Logs can be used for various purposes, and the information needed for a particular log can depend on particular purposes for which the log may be used. Some uses may be more immediate than others. Particularly for uses where log data should be maintained for long period of time, maintaining log data may not be feasible. In such cases, logs may be discarded, or logs may be maintained, but at a cost of storage resources and potentially computing resources needed to process the logs. Accordingly, room for improvement exists.

Different processes or use scenarios can use different information that may be captured by a logging process. Often, a larger amount of log information is useful when the log is reviewed at periods close in time to when the activity occurred than for when the log is reviewed at later points in time. As an example, backup processes, such as for a database, can be carried out, either in response to a manual command or according to a scheduled or triggered process. For a relatively short period after logs are generated, more detailed information contained in the log can be useful. For example, if a backup operation fails, log information may be reviewed to determine individual logged backup operations to determine a cause of backup failure or to determine how to proceed from a backup failure (for example, resuming backup operations from a point prior to failure). Or, even if a backup operation completes, it may be useful to review detailed log information to confirm that a backup process operated as expected, including confirming that all information to be included in a backup job was in fact processed.

As time goes by, it can be increasingly less likely that detailed log information will be needed. However, at least some portions of a log can still be relevant at time periods further removed from the logged process. As an example, auditors may require access to logs for a longer period of time, such as to determine that backup operations occurred at a particular frequency and to determine whether backup operations completed successfully or not. So, while log entries indicating that a backup job started and whether the backup completed, completed with errors, or did not complete may be useful, more granular details of the backup operation captured in the log may not be relevant.

Typically, the above issues with logging have been addressed in two ways. One way of addressing the issue is to simply discard the logs after a period of time, even though some logged information might still be desired at later date. Or, the logs can be kept, even though the logs may require a large amount of storage space. In addition to taking up storage space, unnecessary log entries can take extra computing resources to load and process, including in searching a log for particular entries that might be of interest.

The present disclosure provides for differential logging. That is, an overall logging process can produce multiple versions of a log (collection of log entries). At least one version of the log can have a comparatively large amount of information and can be stored for a different time period than a log having a comparatively smaller amount of information. While typically larger logs are maintained for a shorter period of time than smaller logs, the retention periods can be reversed, or can be the same, without departing from at least some aspects of the present disclosure.

Differential logging can be carried out in a variety of ways. In one example, logs from an existing logging process are analyzed, such as to filter a log for entries that should be maintained in long term storage. These entries can be extracted and stored. In another example, log entries can be filtered on the fly. For instance, when a log entry is to be written to a first log, it can be analyzed to determine whether it satisfies criteria to be written to a second log.

In a further example, code that implements a process, such as process that generates log entries as part of a logging process or a dedicated logging process, can push log entries to selected logs, which can eliminate the need to scan log entries on the fly or to scan a generated log for entries to be included in another log.

In a specific implementation, described logging techniques can be used with database backup procedures. An existing database backup procedure can generate a backup log that contains a variety of entries, such as an entry indicating that a backup job started, an entry indicating that a backup job completed successfully, an entry indicating that a backup job completed with errors, an entry that a backup job failed, and entries for individual operations in a backup process (e.g., individual files or other data that was processed for backup). In a first example, all or a portion of the produced log is scanned to determine log entries to be included in a further log, such as a log with only a subset of the entries of the original log. In a second example, as the log entries are generated during backup, they are analyzed to determine whether they should be written to a secondary log in addition to being written to a primary log. In a third example, code that implements the backup process affirmatively generates appropriate log entries. For example, when code that starts a backup operation is executed, the code can cause a log entry to be written to both a primary log and a secondary log. Other operations, such backup information for particular files, may be associated with code that only writes information to a primary log.

Disclosed technologies can also be used with other database processes, including database process that add, delete, or modify database instances. For example, log entries can be written when the size of a database changes or when resources allocated to a database are modified. Similarly, at least certain configuration changes to a database can be logged.

Disclosed technologies can find use in a variety of computing environments. For example, logging techniques can be used by individual entities for their computing systems, or can be used in a hyperscalar or other type of centralized computing system where logging operations may be carried out on behalf of many entities. Disclosed techniques, including those using a logging service, can be beneficial in “private cloud” environments, where it can be beneficial to have all log activity occur within a single entity's computing systems, as opposed to transmitting logging information to computing systems operated by third parties. Logging can be carried out for processes that are automated or for processes that are manually triggered. For example, backup operations can be manually triggered, can be triggered according to a schedule, or can be triggered by various events.

Particularly for infrastructure or other service providers that provide computing resources to multiple entities, the maintenance of computing environments can be complex and time consuming. For example, it can be complex and time consuming to monitor the state of a database application, such as a database application running in a hyperscalar environment. Attempts have been made to automate the monitoring and maintenance of computing environments. In particular, software programs, such as KUBERNETES (the Linux Foundation Non-Profit Corporation), have been developed to automate deployment, scaling, and maintenance of containerized software applications (e.g., applications running within DOCKER (Docker, Inc.) containers).

Automation programs can include features that implement an “operator pattern” (or “operator”). An operator pattern provides a software agent that periodically polls the state of a computing environment. If a current state of the computing environment does not conform to a specified state, the automation program can take steps to bring the computing environment back to the specified state. Thus, maintenance can be implemented in a declarative manner (telling the operator what state to maintain), rather than using an imperative approach (where specific commands would be provided in order to alter a computing environment to maintain a state). For example, if a computing environment implements a database application, and a new database is added, an operator may be responsible for marshalling computing resources for the database and configuring the database for use. Operators can also take actions according to a schedule in response to events, including conducting backup or recovery operations.

Certain of the following Examples describe how disclosed techniques can be implemented in the specific embodiment of a computing environment using an automation platform, such as KUBERNETES, using containerized applications, such as applications running in DOCKER containers. However, disclosed technologies can be use in other environments that use an operator pattern. Or, disclosed technologies can be used in environments that do not use an operator pattern or which are not automatically managed.

Example 2) —Example Computing Environment with Differential Logging

FIG.1illustrates an example computing environment100in which disclosed technologies can be implemented. The computing environment100can include a computing system104that executes one or more applications or processes108. Examples of applications and processes108include database applications or processes carried out by an operating system. The applications can themselves have processes, which can be included in the applications or processes108, but the processes need not be part of a computer program that is typically thought of as an “application.” For example, an operating system, framework, or “middleware” can be an interface between computing hardware and other applications, including applications with which end users interact. As another distinction, application processes may be associated with applications that are used for comparatively specific tasks, whereas other processes (non-application processes) may be associated with software programs that are used for more general purposes (e.g., managing memory, hardware, user input and output devices, etc.). Applications and their processes can be the same as, or analogous to, non-application processes in some cases. For example, an operating system may perform backup operations, as may a database application.

The applications or processes108can including logging functionality112. Certain actions performed by the applications or processes108can result in log entries that are stored in one or more logs116. For example, when a particular instruction or sets of instructions is executed by the applications or processes108, code can insert a log entry in one or more of the logs116. In at least some cases, the logs116are stored externally to the applications or processes108, such as being stored in a file, such as a text file. In other cases, the logs can be configured to be more specifically used by the applications or processes108, rather than for use by end users or other software applications. Logs116are typically persisted for at least some period of time, such as being stored on disk or another type of non-volatile storage. However, at least some logs, or log entries, can be used for a more limited duration, and can optionally be stored in volatile storage. For example, some log information can be maintained only until it is determined by a relevant application or process108that the log information is no longer needed (for example, because a particular function, routine, or other subprocess executed successfully), and then at least that relevant log information can be deleted from volatile or non-volatile storage.

For the purposes of the present disclosure, at least some of the information in a log116is intended to be stored beyond the completion of a subprocess that resulted in a log entry. Even more particularly, the present disclosure relates to log entries for which differential storage is desired—some types of log entries are to be stored for a longer period of time than other types of log entries.

Although in some cases logging functionality112is included within a particular application or process108, in other cases logging can be carried out by an “external” application or process120, where “external” refers to the application or process that generates log entries being different than the application or process108that executed the logged activity. For example, an external application or process120can monitor settings or repositories and log any changes that may have been made by an application or process108that interacts with such settings or repositories.

Log entries can be processed in a variety of ways such that all or a portion of the log entries are maintained in storage for a first duration and a portion of log entries are maintained for a second duration. As mentioned, typically a smaller portion of log entries are maintained in longer term storage compared with log entries stored in shorter term storage, although this need not be the case in all implementations of the disclosed technologies. Log processing techniques will be discussed in further Examples, but generally include post-processing of a log to determine log entries that should be stored for a different period of time than the processed log (“differential storage”), processing log entries as they are generated and storing at least a portion of the log entries in differential storage, or incorporating into a process commands that write a first portion of log entries to a first store and a second portion of the log entries to a second store, where in some, but not all, cases the second portion is a proper subset of the first portion, but in any case the second portion container fewer log entries than the first portion.

Logs can be maintained in one or more locations. For instance,FIG.1illustrates the computing system104as having long term log retention132and short term log retention134. One or more logs (where a “log” can refer to an entire log/greater number of log entries or selected/a fewer number of log entries) can be stored at other locations instead of, or in addition to, being stored in the computing system104.FIG.1illustrates that the computing system104can be in communication with a logging service138, where the logging service includes one or both of log term log retention142and short term log retention144. Logs can be stored in other storage locations, in addition to or in place of one or both of the computing system104or the logging service138, where storage148is shown as having short term log retention150, storage154is shown as having long term log retention156, and storage160is shown as having long term log retention162and short term log retention164.

The computing system104can include components to help generate or store differential logs. For example, the computing system104can include a log scanner168. The log scanner168can use a set of filters, or selection criteria,170to analyze logs116, or individual log entries as they are generated, where log entries satisfying filter/selection criteria can be stored in the long term retention132and/or the short term log retention134, or transmitted to the logging service138to be stored in the long term log retention142or in the short term log retention144. In some cases, some log data, such as short term log data is stored on the computing system104, while other log data, such as long term log data, is stored elsewhere, such as by the logging service138or storage148,154,160.

The computing system104can include a log agent174that communicates with the logging service138. For example, the log agent174can send entire logs or selected log entries to the logging service138, where relevant log entries can be stored in the long term log retention142or the short term log retention144. When logs or log entries are not filtered or classified (e.g., for storage type) by the computing system104, the logging service138may include a log scanner178and filters180that can be configured analogously to the log scanner168and the filter/selection criteria170.

In at least some implementations, the computing environment100includes a plurality of computing systems, such as including computing system104and one or more additional computing systems184having log agents188. The additional computing systems184can be configured in one of the manners described for the computing system104, although a given computing system184can be configured differently than a particular implementation of the computing system104. The additional computing systems184include a log agent188for sending log information (e.g., logs or log entries) to the logging service138. The use of computing systems184with log agents188can be useful when, for example, logging is carried out for multiple computing systems in the computing environment100, such as when the computing environment represents a cloud-based/hyperscalar computing environment. In this implementation, the computing system104can include application/processes108, logs116, etc. for multiple users of the cloud/hyperscalar environment, where users may be associated with discrete computing systems104,184or where multiple users may use a single computing system.

Example 3) —Example Computing Environment with Automated Management of Containerized Applications

FIG.2illustrates an example computing environment200in which disclosed technologies can be implemented. The computing environment200can include a plurality of computing clusters210, shown as clusters210a-210c. Cluster210ais illustrated showing details of components that can be included in the clusters210. The cluster210acan include a plurality of computing systems214. At least a portion of the computing systems214can include one or more virtual machines218.

A computing system214, or a virtual machine218running within a computing system, can host one or more pods222, where a given pod can in turn host one or more applications230running inside a respective container226. At least a portion of the pods222can include resources234, such as having all or a portion of a storage volume assigned to the pod. One or more pods222can be combined into a service224, where the pods in a service can be located on the same computing system214or on different computing systems.

A cluster210can be managed by a control plane238. Among other things, the control plane238can prove an API (not shown inFIG.2) for interacting with the cluster210and its constituent components. The API can include methods for accessing custom resources defined for the cluster210, and can implement an operator pattern for controlling such resources. In a particular example, the operator pattern can manage backup or restore processes for a database system, or can manage other database processes.

The control plane238is further shown inFIG.3as including an API server242and an operator246.

Example 4) —Example Processing of Logs or Log Entries for Differential Logging

FIG.4provides a flowchart of a method400for differential logging. The method400can represent a technique for “on the fly” log scanning that scans log entries as they are generated as part of a process or a technique for scanning a collection of log entries, such as all or a portion of a generated log. At410, a log entry is received from a process or is read during log scanning. It is determined at420whether the entry should be added to an alternate log. If not, the method400returns to410(provided the logged process continues or log entries remain to be processed). If so, the log entry is written to an alternate log (e.g., a collection of log entries) at430, after which the method400returns to410(provided the logged process continues or log entries remain to be processed). Writing the log entry at420can include sending the log entry to a logging service, such as the logging service138ofFIG.1.

Example 5) —Example Differential Logging During Process Execution

FIG.5provides a flowchart of a method500for differential logging. The method500can represent a technique for logging where a software-implemented process contains instructions that indicate whether a particular step in the process should result in a log entry being written to a first log (or being sent to a logging service), being written to a second log (or being sent to a logging service), or being written to both the first and second logs (which can be accomplished by sending a log entry to the logging service with directions to store the log entry in different logs maintained by the logging service or by sending the log entries to different logging services or different processes of a single logging service).

At510a step (e.g., an instruction or set of instructions embodied in software instructions generated from software code) of a process is executed. It is determined at520whether the process step is a logged event for a first log. If so, a log entry is written to a first log (or sent to a log service, which can be implemented as described in Example 2) at530, where the method500then proceeds to540. If the step is determined at520not to be a logged event, in a first embodiment, the process500returns to510where a next step in the process is executed. In a second embodiment, the process500proceeds to540.

At540it is determined whether the step should be written to an alternate log, such as a log that is maintained for a shorter time than a log to which a log entry was written at530. If so, a log entry is written to the alternate log (or is sent to a logging service) at550. If the step is determined at540not to be written to the alternate log, the method500returns to510(provided the process has additional steps).

Example 6) —Example Computer-Implemented Process with Differential Logging Operations

FIG.6illustrates an example computer-implemented process600that includes a plurality of process steps608(shown as steps608a-608d), where each process step includes at least one action612(shown as actions612a-612d) and at least a portion of the process steps include logging commands616(shown as logging commands616a-616d).

A process action612can represent one or more lines of software code, where the one or more lines implement the particular process step608. Typically, the logging commands616are placed in the code such that they are executed when the corresponding code for the process action612executes, or depending on the execution results of the corresponding code (e.g., whether a process completes successfully or not). It should be appreciated that multiple logging commands616can be included for a given process action612, and at least some of these logging commands can be conditional (e.g., write to log X if true, write to log Y if false).

As a particular example, the process600can represent a database backup process. A particular process step608can represent initiation of the backup process, and the corresponding process action612can be an action that initiates the backup process. In this case, a logging command616associated with the process step608can cause a log entry to be written to one or more logs indicating that the backup process was initiated. Note that in a similar manner as some process steps608need not have logging commands616, some process steps need not have process actions612. That is, a given process step608can simply be a log writing step. In addition, logging commands616can be carried out in a different order than that illustrated for the process600. For example, a logging command616can write a log entry indicating that a process step608for a database backup job was initiated before any process actions612are executed that carry out the backup job.

In the process600, two logs (collections of log entries) can be written to. One log can be stored for a comparatively shorter time and another log can be stored for a comparatively longer time. In a process step608a, a process action612ais executed and a logging command616aresults in a log entry being written to the first log. A process step608binvolves the execution of a process action612band a logging command616bresults in a log entry being written to a second log. Process step608cincludes the execution of a process action612cand logging commands616c,616dthat result in log entries in the first log and in the second log. The final illustrated process step608dinvolves the execution of a process action616dbut does not involve any logging commands.

Example 7) —Example Differential Logging in Database Backup and Recovery Processes

Disclosed techniques can find use in a variety of scenarios. However, a particular use case involves logging associated with database backup and recovery operations. Databases can maintain critical information, and some entities which maintain databases may be subject to governmental or contractual obligations with respect to data. Database service providers, such as those who provide database hardware, software, or management services, including those who provide database services in a hyperscalar environment (e.g., “cloud based” services) can be subject to government or contractual obligations.

Part of maintaining a database, including as part of providing databases services to customers, can include creating backup copies of a database at regular intervals, upon triggering events, or when manually requested. Backup data may be encrypted, and periodically keys used to encrypt the data may be changed to help maintain data security. Backup data can be deleted, such as when the backup data becomes stale or is superseded by a more recent backup or if the database represented by the backup was deprovisioned/removed. Backup data can also be restored to a database, such as if the database experiences a hardware failure or a software issue that requires a restart of the database system.

One or more of these database backup/recovery operations can be associated with logging activity, which can benefit from disclosed technologies. The amount of information in logs associated with database backup and recovery operations can be voluminous. The volume is compounded when considering the number of backup and recovery operations that may occur (for example, if backups are made on a daily basis) and the number of databases that may be involved, particularly as even a single entity may maintain multiple databases and database service providers may provide one or multiple databases to multiple clients.

Logging information can include detailed operations about backup or recovery processes, including detailed logs of how backup jobs were processed, such as individual tables or files that were processed. This information may not be commonly used, but can be useful, such as if a backup or recovery job fails. Because the detailed logged information is most likely to be useful at times close to when the job occurred, such as to troubleshoot a failure, detailed logs may be kept for a comparatively short period, such as days or weeks. After that, the cost (in terms of computing resources) of maintaining the data can outweigh the likelihood that the data may later be of use. In addition, maintaining data backups can raise security or confidentiality concerns (for example, data protection and privacy laws or regulations may limit how long data can be maintained, particularly in the absence of an authorized reason for maintaining the data).

However, as mentioned above, there can be cases where a portion of logged data may be useful for longer time periods. For instance, while the minute details of how backup job was carried out may not be relevant after a certain period of time, it still may be useful, such as for auditing purposes, to have records showing that a backup job was initiated, whether the job completed successfully or not, and information to describe at least certain parameters of the backup job (e.g., identifying when a backup job was initiated, an identifier of a database system being backed up, an identifier of a backup job that can be referenced to determine parameters of the backup, such as data to be included in the backup). Prior to the disclosed technologies, detailed backup logs were typically not kept for extended durations, meaning that backup job information would be unavailable after a comparatively short period of time. Even if some backup logs were maintained for longer periods of time, they typically were the same logs that were initially generated, meaning that the logs were voluminous and contained information that was less likely to be useful after a comparatively short period of time passed, resulting in both wasted resources in maintaining the logs and in locating information of interest in the logs if it turned out the logs were reviewed at a later date.

FIG.7illustrates a computing environment700where a backup operator710, such as described in Example 3, is used to manage at least certain backup and recovery operations for a database715. One task performed by the backup operator710is a backup data job720, such as a recurring backup job or a backup job triggered by particular criteria being satisfied or when manually requested. The data backup job720reads at least a portion of data in the database715(e.g., a particular selection of tables and views stored by the database, and optionally information used in executing database operations, such as data dictionaries, statistics, query plans, or indexes) and stores a copy of the data. The backup data can be stored in the database715or in another location (not shown). For example, backup data may be stored in a remote system that may have higher storage capacity than the database715, or if it otherwise desired not to occupy storage of the database715with backup data. As has previously been described, generally, logs associated with a data backup job720can include details about when a job was started, job parameters, individual job steps (e.g., individual backup operations for particular data processed by the backup job), and an indication of whether the job completed successfully.

Another task that can be performed by the backup operator710is a backup deprovisioning job725. A backup deprovisioning job725removes data from one or more prior data backup jobs720. For example, data backups may be removed when a database is deprovisioned (deleted), including when a customer of a database service provider terminates their relationship with the provider, when the database is deleted (such as because it is no longer being used or because all or a portion of the database is being transitioned to another database, which can be a database having a different storage model or schema than the original database), or when the database backup is superseded by another database backup or the database backup becomes stale. Although shown as communicating with the database715, in some cases a deprovisioning job725communicates with another computing system, such as a system remote from the database system that is used to store backup information. Log entries created during a backup deprovisioning job725can include an identifier of when the deprovisioning job was initiated, parameters for the deprovisioning job (e.g., how database backups to be removed are to be identified), individual deprovisioning operations (e.g., commands to delete particular data in the backup job), or an indication of whether a backup deprovisioning job completed successfully.

The backup operator710can institute a backup recovery job730, where backup data, such as from a backup job720, is restored to the database715. Log entries created during the backup recovery job730can include an identifier of when the job was initiated, an identifier of the database715, an identifier of a particular backup to be restored, log entries for individual restoration operations (e.g., processing of particular files, tables, etc. in backup data), or an indication of whether the recovery job completed successfully.

An encryption key change job735can also be performed by the backup operator710. In some cases, newly received keys are used to encrypt newly created backups. In other cases, newly received keys are used to encrypt newly created backups but are also used to reencrypt data from prior backup jobs720. Encryption keys can be used for additional purposes, such as to encrypt logs associated with backup or recovery operations. Logs for encryption key change jobs735can include entries for when a job was initiated, the target for a job (e.g., a particular database system715, particular backup data), operations performed during the job (e.g., keys removed, keys added, backup data that was re-encrypted), or indications of whether the job, or particular job components (e.g., backup data re-encryption), completed successfully.

FIG.8illustrates a computing environment800in which disclosed technologies can be used to facilitate the various backup operations shown in the computing environment700ofFIG.7. The computing environment800shows a backup data job810, a backup deprovision job815, a backup recovery job820, and a key change job825interacting with a log service830. The backup data job810, the backup deprovision job815, the backup recovery job820, and the key change job825can be analogous to the correspondingly titled jobs720-735ofFIG.7.

The log service830can be a service that processes and manages logs or log entries, including maintaining the logs in a storage835and retrieving logs or log entries upon request. In a particular example, including when disclosed technologies are used with an application management system such as DOCKER and KUBERNETES, the log service830can be the LOKI log aggregation system (Grafana Labs, New York, NY). LOKI can apply labels to logs or log entries, which can be used to develop indexes that can be used to search the logs. The logs themselves can be maintained in a compressed state. Log compression results in additional computing resource (storage) savings beyond those provided by differential logging itself—storing only selected log entries.

Log entries can be identified and retrieved from the storage835in response to queries from a variety of sources. In some cases, a user, such as a user of an entity whose data is reflected in the logs, can request manual reports840or automatic reports845can periodically be generated. In other cases, an auditor850can request that particular log data be retrieved from storage, such as to determine if backup jobs were conducted as required, and as a starting point to determine whether any failed backup jobs were adequately investigated and resolved.

Example 8) —Example Differential Logs and Log Storage Techniques

FIGS.9A-9Z and10A-10Pillustrate example entries in a log900. It can be seen that the log entries are detailed and numerous. While these log entries can be useful for some purposes, such as troubleshooting a failed process, they contain much more detail than is needed for other purposes, such as auditing. In some cases, the only entries fromFIGS.9A-9Z and10A-10Pthat are relevant to longer term uses, such as auditing, are:{“log”:“DB_H00/backup.log: 2021-11-01T10:22:37+00:00 P0002344 17cdb05b99b INFO BACKUP SAVE DATA started [1635762157979]\n”,“stream”:“stdout”, “time”:“2021-11-01T10:22:38.768102517Z”}{“log”:“DB_H00/backup.log: 2021-11-01T10:22:53+00:00 P0002344 17cdb05b99b INFO BACKUP SAVE DATA finished successfully\n”,“stream”:“stdout”,“time”:“2021-11-01T10:22:53.483382664Z”}

These log entries indicate when a backup process initiated, when it finished, and the manner in which the backup process completed (in this case that it completed successfully). It can thus be seen that differential logging can greatly reduce the amount of log information that is stored, such as when only a subset of information may be relevant to uses which may arise comparatively longer after the generation of the logs. As discussed in earlier examples, in some cases the two log entries above can be identified by scanning a complete or partial log, such as for particular keywords. Assuming such terms are unique, or can be combined with other terms or textual tokens to uniquely identify entries, string-based searching for “started” and “finished” may be used to identify the above entries. Or, rather than scanning a complete or partial log, log entries can be scanned as they are generated and entries satisfying search criteria can be stored in a primary log and an alternate log. However, text-based scanning can be both time and computing resource intensive, and can be difficult to implement in a way that catches all relevant log entries and excludes irrelevant log entries. Thus, particularly when a comparatively small number of events are to be logged, it can be advantageous to include in code implementing a particular process commands to write log entries to all relevant logs.

FIG.11illustrates additional examples of log entries1110that may be useful for long term retention for a backup process, such as for purposes of auditing to confirm whether backup operations were conducted and results of such backup operations. For a log entry1110a,FIG.11illustrates labels1120that were associated with the log entry1110a, where the labels can be used to index log entries, including so that log entry1110acan be retrieved in response to relevant search criteria (e.g., search criteria that includes one or more of the labels1120). In some cases, labels1120are not sufficient to retrieve particular log entries of interest, but can retrieve a smaller subset of log entries that can then be further searched, such as using text-based searching.

Note that the log labels1120in this case are not part of the log entry1110aitself. Instead, the log labels1120can be supplied in metadata provided with the log entries1120. In other cases, all or a portion of the log labels1120can be extracted from the log entries. Further, information extracted from a log entry1110can be applied to other log entries that do not include such information, in some cases. In the example shown, the “event” label1120acould be extracted from a log entry initiating the event, and subsequent relevant log entries could be labelled with the extracted event name.

FIG.12illustrates a method1200for differential logging according to an embodiment of the present disclosure. The method1200can be implemented in various disclosed computing environments, including the computing environment100ofFIG.1, the computing environment200ofFIG.2, the computing environment700ofFIG.7, or the computing environment800ofFIG.8.

At1210, a computer-implemented process is executed, such as a process associated with database backup or recovery. During execution of the computer-implemented process, a first log entry is generated at1220. At1230, it is determined that the first log entry is to be stored in a first log. The first log entry is caused to be stored in the first log at1240.

At1250, during execution of the computer-implemented process, a second log entry is generated. At1260, it is determined that the second log entry is to be stored in the first log. The second log entry is caused to be stored in the first log at1270. At1280, it is determined that the second log entry is to be stored in a second log. The second log entry is caused to be stored in the second log at1290. The first log entry is not stored in the second log.

Although the terms “first” and “second” have been used with respect to the method1200, it should be appreciated that these terms are used to convey that the log entries are different, not that one log entry is generated before the other. For example, in some cases the second log entry is generated and stored in the first and second logs before the first log entry is generated.

FIG.13depicts a generalized example of a suitable computing system1300in which the described innovations may be implemented. The computing system1300is not intended to suggest any limitation as to scope of use or functionality of the present disclosure, as the innovations may be implemented in diverse general-purpose or special-purpose computing systems.

With reference toFIG.13, the computing system1300includes one or more processing units1310,1315and memory1320,1325. InFIG.13, this basic configuration1330is included within a dashed line. The processing units1310,1315execute computer-executable instructions, such as for implementing components of the environment100ofFIG.1, the computer environment200ofFIG.2, the computer environment700ofFIG.7, or the computing environment800ofFIG.8, including as described in Examples 1-9. A processing unit can be a general-purpose central processing unit (CPU), processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. For example,FIG.13shows a central processing unit1310as well as a graphics processing unit or co-processing unit1315. The tangible memory1320,1325may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two, accessible by the processing unit(s)1310,1315. The memory1320,1325stores software1380implementing one or more innovations described herein, in the form of computer-executable instructions suitable for execution by the processing unit(s)1310,1315.

A computing system1300may have additional features. For example, the computing system1300includes storage1340, one or more input devices1350, one or more output devices1360, and one or more communication connections1370. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system1300. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing system1300, and coordinates activities of the components of the computing system1300.

The tangible storage1340may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium which can be used to store information in a non-transitory way, and which can be accessed within the computing system1300. The storage1340stores instructions for the software1380implementing one or more innovations described herein.

The input device(s)1350may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing system1300. The output device(s)1360may be a display, printer, speaker, CD-writer, or another device that provides output from the computing system1300.

In various examples described herein, a module (e.g., component or engine) can be “coded” to perform certain operations or provide certain functionality, indicating that computer-executable instructions for the module can be executed to perform such operations, cause such operations to be performed, or to otherwise provide such functionality. Although functionality described with respect to a software component, module, or engine can be carried out as a discrete software unit (e.g., program, function, class method), it need not be implemented as a discrete unit. That is, the functionality can be incorporated into a larger or more general purpose program, such as one or more lines of code in a larger or general purpose program.

Example 11) —Cloud Computing Environment

FIG.14depicts an example cloud computing environment1400in which the described technologies can be implemented. The cloud computing environment1400comprises cloud computing services1410. The cloud computing services1410can comprise various types of cloud computing resources, such as computer servers, data storage repositories, networking resources, etc. The cloud computing services1410can be centrally located (e.g., provided by a data center of a business or organization) or distributed (e.g., provided by various computing resources located at different locations, such as different data centers and/or located in different cities or countries).

The cloud computing services1410are utilized by various types of computing devices (e.g., client computing devices), such as computing devices1420,1422, and1424. For example, the computing devices (e.g.,1420,1422, and1424) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smart phones), or other types of computing devices. For example, the computing devices (e.g.,1420,1422, and1424) can utilize the cloud computing services1410to perform computing operators (e.g., data processing, data storage, and the like).

For clarity, only certain selected aspects of the software-based implementations are described. Other details that are well known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any specific computer language or program. For instance, the disclosed technology can be implemented by software written in C, C++, C#, Java, Perl, JavaScript, Python, Ruby, ABAP, SQL, XCode, GO, Adobe Flash, or any other suitable programming language, or, in some examples, markup languages such as html or XML, or combinations of suitable programming languages and markup languages. Likewise, the disclosed technology is not limited to any particular computer or type of hardware. Certain details of suitable computers and hardware are well known and need not be set forth in detail in this disclosure.