Patent Description:
The present disclosure provides a new and innovative system, methods and apparatus for generation of SLO specifications using annotations. In an example, first source code associated with a first computer-implemented service may be received. In various cases, a first annotation in the first source code may be received. The first annotation may include first metadata defining a name of an SLO specification. In various examples, a second annotation in the first source code may be received. The second annotation may include second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service. In some examples, the first computer-implemented service may be executed using the first source code. In various examples, the SLO specification may be generated based on the first annotation and the second annotation. In various cases, the SLO specification may include the name and a definition of the SLO of the first aspect of the first computer-implemented service.

In another example, a system may comprise at least one processor and non-transitory computer-readable memory. The non-transitory computer-readable memory may store instructions that, when executed by the at least one processor are configured to receive first source code defining a first computer-implemented service. The non-transitory computer-readable memory may store instructions that, when executed by the at least one processor are configured to receive a first annotation in the first source code, the first annotation comprising first metadata defining a name of an SLO specification. The instructions, when executed by the at least one processor may be further configured to receive a second annotation in the first source code, the second annotation comprising second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service. In some examples, the instructions, when executed by the at least one processor may be further configured to execute the first computer-implemented service using the first source code. In various examples, the instructions, when executed by the at least one processor may be further configured to generate the SLO specification based on the first annotation and the second annotation. The SLO specification may include the name and a definition of the SLO of the first aspect of the first computer-implemented service.

In yet another example, another system may be described. The system may comprise an integrated development environment. In some examples, the system may comprise a compiler. In still other examples, the system may comprise an annotation processor. In some cases, the integrated development environment may be effective to receive first source code associated with a first computer-implemented service. In various examples, the integrated development environment may be further effective to receive a first annotation in the first source code, the first annotation comprising first metadata defining a name of an SLO specification. In some examples, the integrated development environment may be further effective to receive a second annotation in the first source code, the second annotation comprising second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service. In various examples, the compiler may be effective to compile the first source code to generate executable code for the first computer-implemented service. In some other examples, the annotation processor may be effective to receive the first annotation from the first source code. In some examples, the annotation processor may be further effective to receive the second annotation from the first source code. In still other examples, the annotation processor may be effective to generate the SLO specification based on the first annotation and the second annotation. The SLO specification may include the name and a definition of the SLO of the first aspect of the first computer-implemented service.

According to an aspect of the present invention there is provided a method according to any of accompanying Claims <NUM> to <NUM>.

According to a further aspect of the present disclosure there is provided a system according to either of accompanying Claims <NUM> or <NUM>.

Additional features and advantages of the disclosed method and apparatus are described in, and will be apparent from, the following Detailed Description and the Figures.

OpenSLO is an open-source service level objective (SLO) language that declaratively defines reliability and performance targets using a YAML specification. SLOs include reliability targets for various aspects of computer-implemented services that allow organizations to determine how to create, operate, and run cloud-based services and/or applications. SLOs are important metrics that are used to set the operational expectations of computer-implemented services. An SLO specification is a standardized machine-readable definition format for defining the SLOs for a given computer-implemented service. An example of a standardized SLO specification is the OpenSLO specification promulgated by Nobl9 among others. Since OpenSLO specifications and other SLO specifications define SLOs in a standardized, vendor-agnostic way (excluding platform-specific details), OpenSLO specifications can be easily ingested by other systems and used to programmatically monitor the performance of the computer-implemented services to which the SLOs pertain. The various techniques described herein may be used in conjunction with any SLO specifications. However, for brevity, the OpenSLO specification will hereinafter be referred to for illustrative purposes.

However, there is currently no way to automatically generate OpenSLO specifications at the code level of the underlying computer-implemented service to which the SLOs pertain. As such, OpenSLO documents are manually generated by developers. Generation of the OpenSLO documents can be a tedious and time-intensive operation. Additionally, such documents may become outdated over time as changes are made to the underlying services. Developers often need to access these OpenSLO documents to ensure that their code performs well per the SLOs defined by the OpenSLO document. Developers are typically required to maintain their applications over time. However, as the code of the application changes, the manually-generated OpenSLO documents may become stale (e.g., if corresponding updates are not made to the OpenSLO documents). In some examples, OpenSLO documents may be programmatically ingested by software testing components to monitor the performance of a service. However, if stale OpenSLO documents are used, the testing may result in the wrong tests being performed wasting CPU compute time, bandwidth, memory, and/or power. The lack of availability to programmatically generate OpenSLO documents at the source code level of a computer-implemented service is a technical problem that can lead to a disconnect between the service and the reliability and performance targets for that service.

Described herein are technical solutions (e.g., systems and techniques) that may be used to automatically generate OpenSLO specifications (e.g., OpenSLO documents) using annotations in the source code. For example, Java annotations can be used to annotate a Java service interface with annotation values that logically map to the equivalent fields defined in the OpenSLO specification. Programmatic generation of OpenSLO specifications ensures that automatic testing components used to test performance of the underlying service are performing the correct tests, rather than outdated tests based on stale SLOs (which can waste compute resources). Including SLOs in the code using annotations not only automates the generation of OpenSLO specifications, but also keeps the objectives/requirements specified by the SLOs within the source code itself. This can force developers to consider their code in terms of operational requirements specified by the SLOs (and the annotations) and can be used to generate automated testing scripts that reference the OpenSLO specification. Annotations (e.g., Java annotations) can be consumed by annotation processors to generate the corresponding OpenSLO YAML descriptor (e.g., in the appropriate field of the OpenSLO specification). Additionally, annotations can be consumed by annotation processors to generate markup based documentation (e.g., HTML, PDF, etc.). Additionally, annotations can be consumed by annotation processors to generate scripts that may test the corresponding computer-implemented service according to the SLOs defined in the programmatically-generated OpenSLO specification.

Although many of the examples described herein use Java annotations to generate OpenSLO specifications, other annotations may be used depending on the programming language of the source code. For example, C#, Ruby, VB. NET, and other programming languages also support annotation integration and may be used to programmatically generate OpenSLO specifications.

<FIG> is a block diagram of a system <NUM> comprising an OpenSLO generator <NUM> configured in communication with an integrated development environment (IDE) <NUM>, according to various examples of the present disclosure. The OpenSLO generator <NUM> may be implemented using software, hardware, and/or some combination thereof. In the example OpenSLO generator <NUM> depicted in <FIG>, the OpenSLO generator <NUM> may include one or more physical host(s), including physical host 110A. Physical host 110A may in turn include one or more physical processor(s) (e.g., CPU 112A) communicatively coupled to one or more memory device(s) (e.g., MDs 114A-B) and one or more input/output device(s) (e.g., I/O 116A). As used herein, physical processor or processors 112A refer to devices capable of executing instructions encoding arithmetic, logical, and/or I/O operations. In one illustrative example, a processor may follow Von Neumann architectural model and may include an arithmetic logic unit (ALU), a control unit, and a plurality of registers. In an example, a processor may be a single core processor which is typically capable of executing one instruction at a time (or process a single pipeline of instructions), or a multi-core processor which may simultaneously execute multiple instructions and/or threads. In another example, a processor may be implemented as a single integrated circuit, two or more integrated circuits, or may be a component of a multi-chip module (e.g., in which individual microprocessor dies are included in a single integrated circuit package and hence share a single socket). A processor may also be referred to as a central processing unit ("CPU").

As discussed herein, memory devices 114A-B refer to volatile or non-volatile memory devices, such as RAM, ROM, EEPROM, or any other device capable of storing data. In an example, memory devices 114A may be persistent storage devices such as hard drive disks ("HDD"), solid state drives ("SSD"), and/or persistent memory (e.g., Non-Volatile Dual In-line Memory Module ("NVDIMM")). Memory devices 114A-B may additionally include replication of data to prevent against data loss due to a failure in any one device. This replication may be implemented through, for example, a redundant array of independent disks ("RAID") setup. RAID arrays may be designed to increase performance, to provide live data backup, or a combination of both. As discussed herein, I/O device(s) 116A refer to devices capable of providing an interface between one or more processor pins and an external device, the operation of which is based on the processor inputting and/or outputting binary data. CPU(s) 112A may be interconnected using a variety of techniques, ranging from a point-to-point processor interconnect, to a system area network, such as an Ethernet-based network. Local connections within physical hosts 110A, including the connections between processors 112A and memory devices 114A-B and between processors 112A and I/O device 116A may be provided by one or more local buses of suitable architecture, for example, peripheral component interconnect (PCI).

In an example, physical host 110A may run one or more isolated guests, for example, VM <NUM>, which may in turn host additional virtual environments (e.g., VMs and/or containers). In an example, a container (e.g., storage container <NUM>, service containers 150A-B) may be an isolated guest using any form of operating system level virtualization, for example, Red Hat® OpenShift®, Docker® containers, chroot, Linux®-VServer, FreeBSD® Jails, HP-UX® Containers (SRP), VMware ThinApp®, etc. Storage container <NUM> and/or service containers 150A-B may run directly on a host operating system (e.g., host OS <NUM>) or run within another layer of virtualization, for example, in a virtual machine (e.g., VM <NUM>). In an example, containers that perform a unified function may be grouped together in a container cluster that may be deployed together (e.g., in a Kubernetes® pod). In an example, a given service may require the deployment of multiple VMs, containers and/or pods in multiple physical locations. In an example, VM <NUM> may be a VM executing on physical host 110A.

OpenSLO generator <NUM> may run one or more VMs (e.g., VMs <NUM>), by executing a software layer (e.g., hypervisor <NUM>) above the hardware and below the VM <NUM>, as schematically shown in <FIG>. In an example, the hypervisor <NUM> may be a component of respective host operating system <NUM> executed on physical host 110A, for example, implemented as a kernel based virtual machine function of host operating system <NUM>. In another example, the hypervisor <NUM> may be provided by an application running on host operating system 118A. In an example, hypervisor <NUM> may run directly on physical host 110A without an operating system beneath hypervisor <NUM>. Hypervisor <NUM> may virtualize the physical layer, including processors, memory, and I/O devices, and present this virtualization to VM <NUM> as devices, including virtual central processing unit ("VCPU") 190A, virtual memory devices ("VMD") 192A, virtual input/output ("VI/O") device 194A, and/or guest memory 195A. In an example, another virtual guest (e.g., a VM or container) may execute directly on host OSs <NUM> without an intervening layer of virtualization.

In an example, a VM <NUM> may be a virtual machine and may execute a guest operating system 196A which may utilize the underlying VCPU 190A, VMD 192A, and VI/O 194A. Processor virtualization may be implemented by the hypervisor <NUM> scheduling time slots on physical CPUs 112A such that from the guest operating system's perspective those time slots are scheduled on a virtual processor 190A. VM <NUM> may run on any type of dependent, independent, compatible, and/or incompatible applications on the underlying hardware and host operating system <NUM>. The hypervisor <NUM> may manage memory for the host operating system <NUM> as well as memory allocated to the VM <NUM> and guest operating system 196A such as guest memory 195A provided to guest OS 196A. In an example, storage container <NUM> and/or service containers 150A, 150B are similarly implemented.

In an example, in addition to distributed storage provided by storage container <NUM>, a storage controller may additionally manage storage in dedicated storage nodes (e.g., NAS, SAN, etc.). In an example, a storage controller may deploy storage in large logical units with preconfigured performance characteristics (e.g., storage nodes 170A). In an example, access to a given storage node (e.g., storage node 170A) may be controlled on an account and/or tenant level. In an example, a service container (e.g., service containers 150A-B) may require persistent storage for application data, and may request persistent storage with a persistent storage claim to an orchestrator (not shown in <FIG>). In the example, a storage controller may allocate storage to service containers 150A-B through a storage node (e.g., storage nodes 170A) in the form of a persistent storage volume. In an example, a persistent storage volume for service containers 150A-B may be allocated a portion of the storage capacity and throughput capacity of a given storage node (e.g., storage nodes 170A). In various examples, the storage container <NUM> and/or service containers 150A-B may deploy compute resources (e.g., storage, cache, etc.) that are part of a compute service that is distributed across multiple clusters (not shown in <FIG>).

The various virtualized computing systems (e.g., service containers 150A, 150B, VM <NUM>) may be examples of computing environments that may deploy one or more of the techniques described herein for programmatic generation of an OpenSLO document <NUM> using java annotations <NUM> of source code <NUM>. In various examples, the service containers 150A, 150B, and/or VM <NUM> may be used to implement an annotation processor. An annotation processor may be a tool build in Java that is used for scanning and/or processing of annotations at compile time. An annotation processor for a certain annotation takes java code (or compiled byte code) as input and generates files as output. In the various embodiments described herein, annotation processors take source code (e.g., java code) as input and generate OpenSLO documents <NUM> that include the various SLOs <NUM> defined in the java annotations <NUM>. In various examples, the OpenSLO document <NUM> may be in YAML. Additionally, OpenSLO "documents" and "specifications" may generally refer to any OpenSLO data that is generated according to the OpenSLO standard using source code annotations.

The foregoing example is merely one possible implementation of an OpenSLO generator <NUM>. The actual deployment of the various services and/or systems of the OpenSLO generator <NUM> are implementation-specific details and may be modified as desired in accordance with the present disclosure. The OpenSLO generator <NUM> may be deployed across any number of physical computing devices and/or virtualized computing environments, depending on the desired implementation.

IDE <NUM> may be a software development application that includes a source code editor, build automation tools, and/or debugging tools. In some examples, IDE <NUM> may also include one or more compilers and/or interpreters. SLOs <NUM> may be performance objectives for various aspects of the computer-implemented service defined by source code <NUM>. In various examples, the SLOs <NUM> may be associated with an SLA. Java annotations <NUM> may be provided to represent the SLOs <NUM> in the source code. In turn, the annotated source code may be provided to OpenSLO generator <NUM> (e.g., an annotation processor) that may generate the OpenSLO document <NUM> upon compilation of the source code <NUM>.

Below is an example of an OpenSLO document <NUM> and a corresponding Java definition:
<IMG>
<IMG>.

The corresponding Java definition (e.g., of the source code) may be:
<IMG>
<IMG>.

For example, the Java annotation at line <NUM> of the Java source code (e.g., @Metadata(name="my-amazing-slo", displayName="My amazing SLO") may define a name of the OpenSLO document <NUM>. Accordingly, at line <NUM> of the OpenSLO document <NUM> above, the displayName field has the value "My amazing SLO" as defined in the Java annotation.

Additionally, the Java annotations at lines <NUM> and <NUM> provide the SLO which indicates that a value of <NUM> should be exceeded <NUM>% of the time (e.g., @Objective(displayName"Weiner Shirt-zel Front Page", op">", target "<NUM>", value="<NUM>"). Accordingly, the SLO defined using this Java annotation is included at lines <NUM>-<NUM> of the OpenSLO document <NUM> above. Declarative programming is used at lines <NUM>-<NUM> of the Java definition to provide actions (e.g., GET, and Produces(MediaType. TEXT_PLAIN)) and a uniform resource locator (URL) (e.g., Path("myPath") for the computer-implemented service that is defined by the source code (e.g., myAwesomeService(){. Other Java annotation may be provided to generate other SLOs in the OpenSLO document <NUM>. Additionally, an annotation parser (e.g., OpenSLO generator <NUM>) may use Java annotation to determine a path at which to publish the OpenSLO document <NUM> and/or may define one or more scripts that may be used to evaluate the service (as described in further detail below). The OpenSLO generator <NUM> includes logic to parse source code <NUM> including Java annotations <NUM> to populate the relevant fields of the OpenSLO document <NUM>.

<FIG> is flowchart illustrating an example process <NUM> for generating OpenSLO specifications using annotations, according to an example of the present disclosure. Although the example process <NUM> is described with reference to the flowchart illustrated in <FIG>, it will be appreciated that many other methods of performing the acts associated with the process <NUM> may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, blocks may be repeated, and some of the blocks described may be optional. The process <NUM> may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software, or a combination of both. In some examples, the actions described in the blocks of the process <NUM> may represent a series of instructions comprising computer-readable machine code executable by one or more processing units of one or more computing devices. In various examples, the computer-readable machine codes may be comprised of instructions selected from a native instruction set of and/or an operating system (or systems) of the one or more computing devices.

The process <NUM> includes receiving first source code associated with a first computer-implemented service (block <NUM>). For example, first source code defining the first computer-implemented service may be entered into an IDE by a developer. The first computer-implemented service may be associated with an SLA and one or more SLOs.

The process <NUM> includes receiving a first annotation in the first source code. The first annotation includes first metadata defining a name of an OpenSLO specification (block <NUM>). For example, the developer may use a Java annotation to provide a name of an OpenSLO specification (e.g., an OpenSLO document) for the first computer-implemented service.

The process <NUM> includes receiving a second annotation in the first source code. The second annotation comprises second metadata defining an SLO (block <NUM>). For example, there may be an SLO associated with a latency requirement for a particular aspect of the first computer-implemented service. The second annotation may define the SLO and may be used to programmatically populate the relevant fields of the OpenSLO specification during compile time.

In an example, the process <NUM> includes executing the first computer-implemented service using the first source code (block <NUM>). The first source code is compiled in order to generate machine language that can be executed by a computing device. The first computer-implemented service is defined by the first source code.

In an example, the process <NUM> includes generating the OpenSLO specification based on the first annotation and the second annotation (block <NUM>). In various examples, during compilation of the first source code, the first source code may be parsed to identify the annotations by an annotation processor. The annotation processor programmatically generates the OpenSLO specification with the name specified by the first annotation and with a definition of the SLO related to the first aspect of the first computer-implemented service.

<FIG> is flowchart illustrating an example process <NUM> for generating OpenSLO specifications using an annotation processor according to an example of the present disclosure. Although the example process <NUM> is described with reference to the flowchart illustrated in <FIG>, it will be appreciated that many other methods of performing the acts associated with the process <NUM> may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, blocks may be repeated, and some of the blocks described may be optional. The process <NUM> may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software, or a combination of both. In some examples, the actions described in the blocks of the process <NUM> may represent a series of instructions comprising computer-readable machine code executable by one or more processing units of one or more computing devices. In various examples, the computer-readable machine codes may be comprised of instructions selected from a native instruction set of and/or an operating system (or systems) of the one or more computing devices.

In various examples, the process <NUM> includes receiving, by an annotation processor, first source code of a first computer-implemented service (block <NUM>). The first source code defining the first computer-implemented service which may be entered into an IDE by a developer. The first computer-implemented service may be associated with an SLA and one or more SLOs. The annotation processor receives the first source code during compilation of the first source code.

The process <NUM> may include parsing, by the annotation processor, the first source code to identify a first annotation in the first source code, the first annotation comprising first metadata (block <NUM>). In various examples, the first annotation may pertain to a field of an OpenSLO specification that may be programmatically generated by the annotation processor during compilation of the first source code. For example, the first annotation may be a Java annotation that may be used to define an SLO to be included in the OpenSLO specification.

The process <NUM> may include parsing, by the annotation processor, the first source code to identify a second annotation in the first source code, the second annotation comprising second metadata defining an SLO (block <NUM>). For example, there may be an SLO associated with a throughput requirement for a particular aspect of the first computer-implemented service. The second annotation may define the SLO and may be used by the annotation processor to programmatically populate the relevant fields of the OpenSLO specification during compile time.

The process <NUM> includes generating, by the annotation processor, the OpenSLO specification based on the first annotation and the second annotation (block <NUM>). As previously described, the annotations of the first source code may define the OpenSLO specification to be generated by the annotation processor during compilation of the first source code. Accordingly, various annotations (e.g., Java annotations) included in the first source code may define and/or provide values or other data to be used to populate the OpenSLO specification template to generate the relevant OpenSLO specification for the first computer-implemented service.

<FIG> illustrates a flow diagram of an example generation of OpenSLO specifications using annotations, according to various aspects of the present disclosure. Although the examples below are described with reference to the flow diagram illustrated in <FIG>, many other methods of performing the acts associated with <FIG> may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, and some of the blocks described are optional. The methods may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software, or a combination of both. In illustrated example <NUM>, an IDE <NUM> receives first source code for a first computer-implemented service (e.g., a cloud-based application) (block <NUM>). The IDE <NUM> receives a first Java annotation in the first source code (block <NUM>). The first Java annotation includes metadata that indicates the name of an SLO specification. In various examples, declaration of the name of the SLO specification may be used to generate the SLO specification with the specified name (i.e., the SLO specification may not yet exist) during compilation of the first source code.

The IDE <NUM> receives a second Java annotation in the first source code (block <NUM>). The second Java annotation includes metadata indicating a first SLO (e.g., an SLO associated with an SLA for the first computer-implemented service). IDE <NUM> may receive an instruction to compile the first source code (block <NUM>). The compilation may be performed by a compiler of the IDE <NUM> or may be performed by a separate compiler.

Open SLO generator <NUM> (e.g., an annotation processor) may parse the first source code (block <NUM>) and may identify the first Java annotation (block <NUM>) and the second Java annotation (block <NUM>). The annotation processor generates the OpenSLO specification (block <NUM>) that includes the name specified by the first Java annotation and that includes the SLO defined by the second Java annotation. The OpenSLO generator <NUM> may use the metadata included in the Java annotations to generate the corresponding OpenSLO YAML descriptor. The OpenSLO generator <NUM> may populate the fields of the corresponding OpenSLO YAML descriptors using the metadata included in the Java annotations (block <NUM>). In some further examples, the OpenSLO generator <NUM> may use Java annotations to generate markup based documentation (e.g., HTML, pdf, etc.) describing the first computer-implemented service and/or the OpenSLO specification. Further, the OpenSLO generator <NUM> may publish the OpenSLO specification at a path specified in the first source code (e.g., using a third Java annotation) (block <NUM>). In various examples, the OpenSLO specification may be published by a remote system <NUM>. Accordingly, the remote system <NUM> (e.g., a web server) may store the OpenSLO specification in a data store (block <NUM>).

<FIG> illustrates a flow diagram of an example of testing a computer-implemented service using an OpenSLO specification generated using annotations, according to various aspects of the present disclosure. Although the examples below are described with reference to the flow diagram illustrated in <FIG>, many other methods of performing the acts associated with <FIG> may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, and some of the blocks described are optional. The methods may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software, or a combination of both.

In the illustrated example <NUM>, an OpenSLO generator <NUM> (e.g., an annotation processor and/or compiler) generates an OpenSLO specification for a first computer-implemented service using Java annotations included in the source code for the first computer-implemented service (block <NUM>). The OpenSLO specification may be provided to a testing component <NUM> (block <NUM>). In various examples, Java annotations included in the source code may be used to generate testing scripts that can be executed (e.g., by the testing component). The testing scripts may reference various YAMI, descriptors of the OpenSLO specification for the first computer-implemented service. Accordingly, in some examples, both the testing scripts and the OpenSLO specification may be programmatically generated from the first source code by an annotation processor using annotations included in the first source code. Additionally, the testing scripts may reference the various YAML descriptors of the OpenSLO specification to determine whether or not various SLOs defined in the OpenSLO specification are being met when the first computer-implemented service is subjected to some compute load (also specified by the testing scripts).

The testing component <NUM> may receive the OpenSLO specification (block <NUM>) and may execute the testing script(s) (block <NUM>). The testing component <NUM> may send instructions to execute the first computer-implemented service (block <NUM>) to a system under test <NUM>. In various examples, the system under test <NUM> may represent a desired deployment (e.g., using one or more clusters, nodes, VMs, Docker containers, etc.) of the first computer-implemented service. The system under test <NUM> may execute the first computer-implemented service (block <NUM>) according to configuration data provided by testing component <NUM> (not shown in <FIG>). In various examples, the configuration data may be generated using the testing scripts and may be specified using Java annotations in the source code.

The testing component <NUM> may send instructions to the system under test <NUM> to generate a first load on the first computer-implemented service (block <NUM>). For example, the testing component <NUM> may specify an amount of traffic sent to a first service of the first computer-implemented service. The system under test <NUM> may process the first load (block <NUM>) and may determine the relevant results according to instructions provided by the testing component <NUM> (as part of the testing script(s)) (block <NUM>). For example, the average latency and/or throughput during a specified time period may be provided as result data. The testing component <NUM> may receive the result data (block <NUM>) and may programmatically evaluate the result data using the OpenSLO specification (block <NUM>). In various examples, the result data may be evaluated by the testing scripts. The testing component <NUM> may generate performance data (block <NUM>). The performance data may indicate whether the first computer-implemented service is meeting one or more SLOs defined by the OpenSLO specification. The Java annotations in the first source code may be used to generate the testing scripts based on a combination of SLO-specific Java annotations and testing annotations (e.g., JAX-RS or similar) and can be used to programmatically evaluate the first computer-implemented service.

<FIG> is block diagram of a system <NUM> effective to generate an OpenSLO specification <NUM> using annotations (e.g., first annotation <NUM> and/or second annotation <NUM>), according to an example of the present disclosure.

System <NUM> comprises at least one processor <NUM> and non-transitory computer-readable memory <NUM>. The memory <NUM> stores instructions <NUM>. The instructions <NUM> areexecuted by the at least one processor <NUM> to perform various techniques described herein related to generation of the OpenSLO specification <NUM>.

The at least one processor <NUM> receives the first source code <NUM>. The first source code <NUM> defines the functionality of a first computer-implemented service <NUM>. The at least one processor <NUM> receives a first annotation <NUM> in the first source code <NUM>. The first annotation <NUM> comprises first metadata <NUM> that defines a name of an OpenSLO specification <NUM>. The at least one processor <NUM> receives a second annotation <NUM> in the first source code <NUM>. The second annotation <NUM> comprises second metadata <NUM> that defines an SLO <NUM> of a first aspect of the first computer-implemented service <NUM>.

The at least one processor <NUM> may execute the first computer-implemented service <NUM> using the first source code <NUM>. In some further examples, the at least one processor <NUM> generates the OpenSLO specification <NUM> using the first annotation <NUM> and the second annotation <NUM>. The OpenSLO specification <NUM> comprises the name of the OpenSLO specification <NUM>' and a definition of the SLO <NUM> of the first aspect of the first computer-implemented service <NUM>.

<FIG> is block diagram <NUM> of an annotation processor <NUM> in communication with an integrated development environment <NUM> according to an example of the present disclosure. The integrated development environment <NUM> receives first source code <NUM> that is associated with a first computer-implemented service. The integrated development environment <NUM> receives a first annotation <NUM> in the first source code <NUM>. The first annotation <NUM> includes first metadata <NUM> defining a name of an OpenSLO specification <NUM>.

The integrated development environment <NUM> receives a second annotation <NUM> in the first source code <NUM>. The second annotation <NUM> includes second metadata <NUM> defining an SLO <NUM> of a first aspect of the first computer-implemented service. The compiler <NUM> receive the first source code <NUM> and compiles the first source code <NUM> to generate executable code for the first computer-implemented service <NUM>.

Annotation processor <NUM> receive the first annotation <NUM> from the first source code <NUM> and the second annotation <NUM> from the first source code <NUM>. Annotation process <NUM> generate the OpenSLO specification <NUM> based on the first annotation <NUM> and the second annotation <NUM>. The OpenSLO specification <NUM> include the name of the OpenSLO specification <NUM>' and the definition of the SLO <NUM> of the first aspect of the first computer-implemented service.

It will be appreciated that all of the disclosed methods and procedures described herein can be implemented using one or more computer programs or components. These components may be provided as a series of computer instructions on any conventional computer readable medium or machine readable medium, including volatile or non-volatile memory, such as RAM, ROM, flash memory, magnetic or optical disks, optical memory, or other storage media. The instructions may be provided as software or firmware, and/or may be implemented in whole or in part in hardware components such as ASICs, FPGAs, DSPs or any other similar devices. The instructions may be executed by one or more processors, which when executing the series of computer instructions, performs or facilitates the performance of all or part of the disclosed methods and procedures.

Claim 1:
A method of generation of service-level objective, SLO, specifications comprising:
receiving first source code (<NUM>) associated with a first computer-implemented service;
the first source code (<NUM>) comprising first and second annotations;
compiling the first source code (<NUM>) to generate executable code for the first computer-implemented service;
executing the first computer-implemented service using the executable code (<NUM>); and
characterised in that:
the first annotation comprises first metadata defining a name of an SLO specification and the second annotation comprises second metadata defining a service-level objective, SLO, of a first aspect of the first computer-implemented service;
generating the SLO specification based on the first annotation and the second annotation, the SLO specification including the name and a definition of the SLO of the first aspect of the first computer-implemented service; and
wherein the SLO specification is generated using the first metadata and the second metadata, using an annotation processor during compilation of the first source code (<NUM>).