Patent Publication Number: US-2023153073-A1

Title: Generation of service-level objective specifications using java annotation

Description:
BACKGROUND 
     Providers of compute services such as software developers and web service developers often enter into service-level agreements (SLAs) with customers. An SLA is a commitment to providing certain agreed-upon aspects of the service. For example, an SLA may define certain levels of availability, quality of a service, and/or may define various roles and responsibilities for the service. An SLA typically includes service-level objectives (SLOs). While an SLA is the overall agreement specifying the service to be provided, how the service is supported, responsibilities, etc., SLOs define specific measurable characteristics of the SLA. For example, an SLO may comprise one or more quality of service (QoS) measurements that may define minimum levels of throughput, quality, availability, etc. SLO specifications may be machine-readable documents that may define the SLOs of a service and which may be ingested by other software/systems in order to verify that SLOs are being met and/or to monitor performance. 
     SUMMARY 
     The present disclosure provides a new and innovative system, methods and apparatus for generation of SLO specifications using annotations. In an example, first source code associated with a first computer-implemented service may be received. In various cases, a first annotation in the first source code may be received. The first annotation may include first metadata defining a name of an SLO specification. In various examples, a second annotation in the first source code may be received. The second annotation may include second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service. In some examples, the first computer-implemented service may be executed using the first source code. In various examples, the SLO specification may be generated based on the first annotation and the second annotation. In various cases, the SLO specification may include the name and a definition of the SLO of the first aspect of the first computer-implemented service. 
     In another example, a system may comprise at least one processor and non-transitory computer-readable memory. The non-transitory computer-readable memory may store instructions that, when executed by the at least one processor are configured to receive first source code defining a first computer-implemented service. The non-transitory computer-readable memory may store instructions that, when executed by the at least one processor are configured to receive a first annotation in the first source code, the first annotation comprising first metadata defining a name of an SLO specification. The instructions, when executed by the at least one processor may be further configured to receive a second annotation in the first source code, the second annotation comprising second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service. In some examples, the instructions, when executed by the at least one processor may be further configured to execute the first computer-implemented service using the first source code. In various examples, the instructions, when executed by the at least one processor may be further configured to generate the SLO specification based on the first annotation and the second annotation. The SLO specification may include the name and a definition of the SLO of the first aspect of the first computer-implemented service. 
     In yet another example, another system may be described. The system may comprise an integrated development environment. In some examples, the system may comprise a compiler. In still other examples, the system may comprise an annotation processor. In some cases, the integrated development environment may be effective to receive first source code associated with a first computer-implemented service. In various examples, the integrated development environment may be further effective to receive a first annotation in the first source code, the first annotation comprising first metadata defining a name of an SLO specification. In some examples, the integrated development environment may be further effective to receive a second annotation in the first source code, the second annotation comprising second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service. In various examples, the compiler may be effective to compile the first source code to generate executable code for the first computer-implemented service. In some other examples, the annotation processor may be effective to receive the first annotation from the first source code. In some examples, the annotation processor may be further effective to receive the second annotation from the first source code. In still other examples, the annotation processor may be effective to generate the SLO specification based on the first annotation and the second annotation. The SLO specification may include the name and a definition of the SLO of the first aspect of the first computer-implemented service. 
     Additional features and advantages of the disclosed method and apparatus are described in, and will be apparent from, the following Detailed Description and the Figures. 
    
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
         FIG.  1    is a block diagram of an OpenSLO generator configured in communication with an integrated development environment, according to various examples of the present disclosure. 
         FIG.  2    is flowchart illustrating an example process for generating OpenSLO specifications using annotations, according to an example of the present disclosure. 
         FIG.  3    is flowchart illustrating an example process for generating OpenSLO specifications using an annotation processor according to an example of the present disclosure. 
         FIG.  4    illustrates a flow diagram of an example generation of OpenSLO specifications using annotations, according to various aspects of the present disclosure. 
         FIG.  5    illustrates a flow diagram of an example of testing a computer-implemented service using an OpenSLO specification generated using annotations, according to various aspects of the present disclosure. 
         FIG.  6    is block diagram of a system effective to generate an OpenSLO specification using annotations, according to an example of the present disclosure. 
         FIG.  7    is block diagram of an annotation processor in communication with an integrated development environment according to an example of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS 
     OpenSLO is an open-source service level objective (SLO) language that declaratively defines reliability and performance targets using a YAML specification. SLOs include reliability targets for various aspects of computer-implemented services that allow organizations to determine how to create, operate, and run cloud-based services and/or applications. SLOs are important metrics that are used to set the operational expectations of computer-implemented services. An SLO specification is a standardized machine-readable definition format for defining the SLOs for a given computer-implemented service. An example of a standardized SLO specification is the OpenSLO specification promulgated by Nob19 among others. Since OpenSLO specifications and other SLO specifications define SLOs in a standardized, vendor-agnostic way (excluding platform-specific details), OpenSLO specifications can be easily ingested by other systems and used to programmatically monitor the performance of the computer-implemented services to which the SLOs pertain. The various techniques described herein may be used in conjunction with any SLO specifications. However, for brevity, the OpenSLO specification will hereinafter be referred to for illustrative purposes. 
     However, there is currently no way to automatically generate OpenSLO specifications at the code level of the underlying computer-implemented service to which the SLOs pertain. As such, OpenSLO documents are manually generated by developers. Generation of the OpenSLO documents can be a tedious and time-intensive operation. Additionally, such documents may become outdated over time as changes are made to the underlying services. Developers often need to access these OpenSLO documents to ensure that their code performs well per the SLOs defined by the OpenSLO document. Developers are typically required to maintain their applications over time. However, as the code of the application changes, the manually-generated OpenSLO documents may become stale (e.g., if corresponding updates are not made to the OpenSLO documents). In some examples, OpenSLO documents may be programmatically ingested by software testing components to monitor the performance of a service. However, if stale OpenSLO documents are used, the testing may result in the wrong tests being performed wasting CPU compute time, bandwidth, memory, and/or power. The lack of availability to programmatically generate OpenSLO documents at the source code level of a computer-implemented service is a technical problem that can lead to a disconnect between the service and the reliability and performance targets for that service. 
     Described herein are technical solutions (e.g., systems and techniques) that may be used to automatically generate OpenSLO specifications (e.g., OpenSLO documents) using annotations in the source code. For example, Java annotations can be used to annotate a Java service interface with annotation values that logically map to the equivalent fields defined in the OpenSLO specification. Programmatic generation of OpenSLO specifications ensures that automatic testing components used to test performance of the underlying service are performing the correct tests, rather than outdated tests based on stale SLOs (which can waste compute resources). Including SLOs in the code using annotations not only automates the generation of OpenSLO specifications, but also keeps the objectives/requirements specified by the SLOs within the source code itself. This can force developers to consider their code in terms of operational requirements specified by the SLOs (and the annotations) and can be used to generate automated testing scripts that reference the OpenSLO specification. Annotations (e.g., Java annotations) can be consumed by annotation processors to generate the corresponding OpenSLO YAML descriptor (e.g., in the appropriate field of the OpenSLO specification). Additionally, annotations can be consumed by annotation processors to generate markup based documentation (e.g., HTML, PDF, etc.). Additionally, annotations can be consumed by annotation processors to generate scripts that may test the corresponding computer-implemented service according to the SLOs defined in the programmatically-generated OpenSLO specification. 
     Although many of the examples described herein use Java annotations to generate OpenSLO specifications, other annotations may be used depending on the programming language of the source code. For example, C#, Ruby, VB.NET, and other programming languages also support annotation integration and may be used to programmatically generate OpenSLO specifications. 
       FIG.  1    is a block diagram of a system  100  comprising an OpenSLO generator  122  configured in communication with an integrated development environment (IDE)  124 , according to various examples of the present disclosure. The OpenSLO generator  122  may be implemented using software, hardware, and/or some combination thereof. In the example OpenSLO generator  122  depicted in  FIG.  1   , the OpenSLO generator  122  may include one or more physical host(s), including physical host  110 A. Physical host  110 A may in turn include one or more physical processor(s) (e.g., CPU  112 A) communicatively coupled to one or more memory device(s) (e.g., MDs  114 A-B) and one or more input/output device(s) (e.g., I/O  116 A). As used herein, physical processor or processors  112 A refer to devices capable of executing instructions encoding arithmetic, logical, and/or I/O operations. In one illustrative example, a processor may follow Von Neumann architectural model and may include an arithmetic logic unit (ALU), a control unit, and a plurality of registers. In an example, a processor may be a single core processor which is typically capable of executing one instruction at a time (or process a single pipeline of instructions), or a multi-core processor which may simultaneously execute multiple instructions and/or threads. In another example, a processor may be implemented as a single integrated circuit, two or more integrated circuits, or may be a component of a multi-chip module (e.g., in which individual microprocessor dies are included in a single integrated circuit package and hence share a single socket). A processor may also be referred to as a central processing unit (“CPU”). 
     As discussed herein, memory devices  114 A-B refer to volatile or non-volatile memory devices, such as RAM, ROM, EEPROM, or any other device capable of storing data. In an example, memory devices  114 A may be persistent storage devices such as hard drive disks (“HDD”), solid state drives (“SSD”), and/or persistent memory (e.g., Non-Volatile Dual In-line Memory Module (“NVDIMM”)). Memory devices  114 A-B may additionally include replication of data to prevent against data loss due to a failure in any one device. This replication may be implemented through, for example, a redundant array of independent disks (“RAID”) setup. RAID arrays may be designed to increase performance, to provide live data backup, or a combination of both. As discussed herein, I/O device(s)  116 A refer to devices capable of providing an interface between one or more processor pins and an external device, the operation of which is based on the processor inputting and/or outputting binary data. CPU(s)  112 A may be interconnected using a variety of techniques, ranging from a point-to-point processor interconnect, to a system area network, such as an Ethernet-based network. Local connections within physical hosts  110 A, including the connections between processors  112 A and memory devices  114 A-B and between processors  112 A and I/O device  116 A may be provided by one or more local buses of suitable architecture, for example, peripheral component interconnect (PCI). 
     In an example, physical host  110 A may run one or more isolated guests, for example, VM  155 , which may in turn host additional virtual environments (e.g., VMs and/or containers). In an example, a container (e.g., storage container  160 , service containers  150 A-B) may be an isolated guest using any form of operating system level virtualization, for example, Red Hat® OpenShift®, Docker® containers, chroot, Linux®-VServer, FreeB SD® Jails, HP-UX® Containers (SRP), VMware ThinApp®, etc. Storage container  160  and/or service containers  150 A-B may run directly on a host operating system (e.g., host OS  118 ) or run within another layer of virtualization, for example, in a virtual machine (e.g., VM  155 ). In an example, containers that perform a unified function may be grouped together in a container cluster that may be deployed together (e.g., in a Kubernetes® pod). In an example, a given service may require the deployment of multiple VMs, containers and/or pods in multiple physical locations. In an example, VM  155  may be a VM executing on physical host  110 A. 
     OpenSLO generator  122  may run one or more VMs (e.g., VMs  122 ), by executing a software layer (e.g., hypervisor  120 ) above the hardware and below the VM  155 , as schematically shown in  FIG.  1   . In an example, the hypervisor  120  may be a component of respective host operating system  118  executed on physical host  110 A, for example, implemented as a kernel based virtual machine function of host operating system  118 . In another example, the hypervisor  120  may be provided by an application running on host operating system  118 A. In an example, hypervisor  120  may run directly on physical host  110 A without an operating system beneath hypervisor  120 . Hypervisor  120  may virtualize the physical layer, including processors, memory, and I/O devices, and present this virtualization to VM  155  as devices, including virtual central processing unit (“VCPU”)  190 A, virtual memory devices (“VIVID”)  192 A, virtual input/output (“VI/O”) device  194 A, and/or guest memory  195 A. In an example, another virtual guest (e.g., a VM or container) may execute directly on host OSs  118  without an intervening layer of virtualization. 
     In an example, a VM  155  may be a virtual machine and may execute a guest operating system  196 A which may utilize the underlying VCPU  190 A, VIVID  192 A, and VI/O  194 A. Processor virtualization may be implemented by the hypervisor  120  scheduling time slots on physical CPUs  112 A such that from the guest operating system&#39;s perspective those time slots are scheduled on a virtual processor  190 A. VM  155  may run on any type of dependent, independent, compatible, and/or incompatible applications on the underlying hardware and host operating system  118 . The hypervisor  120  may manage memory for the host operating system  118  as well as memory allocated to the VM  155  and guest operating system  196 A such as guest memory  195 A provided to guest OS  196 A. In an example, storage container  160  and/or service containers  150 A,  150 B are similarly implemented. 
     In an example, in addition to distributed storage provided by storage container  160 , a storage controller may additionally manage storage in dedicated storage nodes (e.g., NAS, SAN, etc.). In an example, a storage controller may deploy storage in large logical units with preconfigured performance characteristics (e.g., storage nodes  170 A). In an example, access to a given storage node (e.g., storage node  170 A) may be controlled on an account and/or tenant level. In an example, a service container (e.g., service containers  150 A-B) may require persistent storage for application data, and may request persistent storage with a persistent storage claim to an orchestrator (not shown in  FIG.  1   ). In the example, a storage controller may allocate storage to service containers  150 A-B through a storage node (e.g., storage nodes  170 A) in the form of a persistent storage volume. In an example, a persistent storage volume for service containers  150 A-B may be allocated a portion of the storage capacity and throughput capacity of a given storage node (e.g., storage nodes  170 A). In various examples, the storage container  160  and/or service containers  150 A-B may deploy compute resources (e.g., storage, cache, etc.) that are part of a compute service that is distributed across multiple clusters (not shown in  FIG.  1   ). 
     The various virtualized computing systems (e.g., service containers  150 A,  150 B, VM  155 ) may be examples of computing environments that may deploy one or more of the techniques described herein for programmatic generation of an OpenSLO document  151  using java annotations  128  of source code  130 . In various examples, the service containers  150 A,  150 B, and/or VM  155  may be used to implement an annotation processor. An annotation processor may be a tool build in Java that is used for scanning and/or processing of annotations at compile time. An annotation processor for a certain annotation takes java code (or compiled byte code) as input and generates files as output. In the various embodiments described herein, annotation processors take source code (e.g., java code) as input and generate OpenSLO documents  151  that include the various SLOs  126  defined in the java annotations  128 . In various examples, the OpenSLO document  151  may be in YAML. Additionally, OpenSLO “documents” and “specifications” may generally refer to any OpenSLO data that is generated according to the OpenSLO standard using source code annotations. 
     The foregoing example is merely one possible implementation of an OpenSLO generator  122 . The actual deployment of the various services and/or systems of the OpenSLO generator  122  are implementation-specific details and may be modified as desired in accordance with the present disclosure. The OpenSLO generator  122  may be deployed across any number of physical computing devices and/or virtualized computing environments, depending on the desired implementation. 
     IDE  124  may be a software development application that includes a source code editor, build automation tools, and/or debugging tools. In some examples, IDE  124  may also include one or more compilers and/or interpreters. SLOs  126  may be performance objectives for various aspects of the computer-implemented service defined by source code  130 . In various examples, the SLOs  126  may be associated with an SLA. Java annotations  128  may be provided to represent the SLOs  126  in the source code. In turn, the annotated source code may be provided to OpenSLO generator  122  (e.g., an annotation processor) that may generate the OpenSLO document  151  upon compilation of the source code  130 . 
     Below is an example of an OpenSLO document  151  and a corresponding Java definition: 
     
       
         
           
               
               
               
             
               
                   
                   
               
             
            
               
                   
                  1 
                  apiVersion: openslo/v1alpha 
               
               
                   
                  2 
                  kind: SLO 
               
               
                   
                  3 
                  metadata: 
               
               
                   
                  4 
                   displayName: My amazing SLO 
               
               
                   
                  5 
                   name: my-amazing-slo 
               
               
                   
                  6 
                  spec: 
               
               
                   
                  7 
                   budgetingMethod: Occurrences 
               
               
                   
                  8 
                   description: Latency Alert 
               
               
                   
                  9 
                   indicator: 
               
               
                   
                 10 
                    thresholdMetric: 
               
               
                   
                 11 
                     source: Prometheus 
               
               
                   
                 12 
                     queryType: promql 
               
               
                   
                 13 
                     query: 
               
            
           
           
               
            
               
                 latency_west_c7{code=″ALL″,instance=″localhost:3000″,job=″prometheus″,service=″globacoun 
               
               
                 t″} 
               
            
           
           
               
               
               
            
               
                   
                 14 
                   objectives: 
               
               
                   
                 15 
                   - displayName: Weiner Shirt-zel Front Page 
               
               
                   
                 16 
                    op: gt 
               
               
                   
                 17 
                    target: 0.99 
               
               
                   
                 18 
                    value: 2000 
               
               
                   
                 19 
                   service: my-awesome-service 
               
               
                   
                 20 
                   timeWindows: 
               
               
                   
                 21 
                   - calendar: 
               
               
                   
                 22 
                     startTime: ″2021-05-01 00:00:00″ 
               
               
                   
                 23 
                     timeZone: UTC 
               
               
                   
                 24 
                    count: 1 
               
               
                   
                 25 
                    isRolling: false 
               
               
                   
                 26 
                    unit: Month 
               
            
           
           
               
               
            
               
                   
                 The corresponding Java definition (e.g., of the source code) may be: 
               
            
           
           
               
               
               
            
               
                   
                 10 
                 public class SampleService { 
               
               
                   
                 11 
               
               
                   
                 12 
                  @Metadata(name=”my-amazing-slo”, displayName=”My amazing SLO”) 
               
               
                   
                 13 
                  @BudgetingMethod(Budgeting.OCCURRENCES) 
               
               
                   
                 14 
                  @Description(“Latency Alert”) 
               
               
                   
                 15 
                  @Indicator( @ThresholdMetric( source=”prometheus”, 
               
            
           
           
               
            
               
                 queryType=”promql”, 
               
               
                 query=”latency_west_c7{code=’ALL’,instance=’localhost:3000’,job=’prometheus&#39;,service=’glo 
               
               
                 bacount’}” )) 
               
            
           
           
               
               
               
            
               
                   
                 16 
                  @Objectives({ 
               
               
                   
                 17 
                   @Objective(displayName=”Weiner Shirt-zel Front Page”, op=”&gt;”, 
               
            
           
           
               
            
               
                 target=”0.99”, value=”2000”) 
               
            
           
           
               
               
               
            
               
                   
                 18 
                  , b=3) 
               
               
                   
                 19 
                 }) 
               
               
                   
                 20 
                 @TimeWindows({ 
               
               
                   
                 21 
                  @TimeWindow( 
               
               
                   
                 22 
                   @Calendar(startTime=″2021-05-01 00:00:00″, timeZone=”UTC”), 
               
               
                   
                 23 
                   count=1, 
               
               
                   
                 24 
                   isRolling=false, 
               
               
                   
                 25 
                   unit=Units.MONTH 
               
               
                   
                 26 
                 ) 
               
               
                   
                 27 
                 }) 
               
               
                   
                 28 
                 @GET 
               
               
                   
                 29 
                 @Produces(MediaType.TEXT_PLAIN) 
               
               
                   
                 30 
                 @Path(“myPath”) 
               
               
                   
                 31 
                 public String myAwesomeService( ){ ...} 
               
               
                   
                 32 
               
               
                   
                 33 
                 } 
               
               
                   
                   
               
            
           
         
       
     
     For example, the Java annotation at line  12  of the Java source code (e.g., @Metadata(name=“my-amazing-slo”, displayName=“My amazing SLO”) may define a name of the OpenSLO document  151 . Accordingly, at line  4  of the OpenSLO document  151  above, the displayName field has the value “My amazing SLO” as defined in the Java annotation. 
     Additionally, the Java annotations at lines  16  and  17  provide the SLO which indicates that a value of 2000 should be exceeded 99% of the time (e.g., @Objective(displayName=“Weiner Shirt-zel Front Page”, op=“&gt;”, target=“0.99”, value=“2000”). Accordingly, the SLO defined using this Java annotation is included at lines  15 - 18  of the OpenSLO document  151  above. Declarative programming is used at lines  28 - 30  of the Java definition to provide actions (e.g., GET, and Produces(MediaType.TEXT PLAIN)) and a uniform resource locator (URL) (e.g., Path(“myPath”) for the computer-implemented service that is defined by the source code (e.g., myAwesomeService( ){ . . . }). Other Java annotation may be provided to generate other SLOs in the OpenSLO document  151 . Additionally, an annotation parser (e.g., OpenSLO generator  122 ) may use Java annotation to determine a path at which to publish the OpenSLO document  151  and/or may define one or more scripts that may be used to evaluate the service (as described in further detail below). The OpenSLO generator  122  includes logic to parse source code  130  including Java annotations  128  to populate the relevant fields of the OpenSLO document  151 . 
       FIG.  2    is flowchart illustrating an example process  200  for generating OpenSLO specifications using annotations, according to an example of the present disclosure. Although the example process  200  is described with reference to the flowchart illustrated in  FIG.  2   , it will be appreciated that many other methods of performing the acts associated with the process  200  may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, blocks may be repeated, and some of the blocks described may be optional. The process  200  may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software, or a combination of both. In some examples, the actions described in the blocks of the process  200  may represent a series of instructions comprising computer-readable machine code executable by one or more processing units of one or more computing devices. In various examples, the computer-readable machine codes may be comprised of instructions selected from a native instruction set of and/or an operating system (or systems) of the one or more computing devices. 
     The example process  200  includes receiving first source code associated with a first computer-implemented service (block  210 ). For example, first source code defining the first computer-implemented service may be entered into an IDE by a developer. The first computer-implemented service may be associated with an SLA and one or more SLOs. 
     In an example, the process  200  may include receiving a first annotation in the first source code. The first annotation may include first metadata defining a name of an OpenSLO specification (block  215 ). For example, the developer may use a Java annotation to provide a name of an OpenSLO specification (e.g., an OpenSLO document) for the first computer-implemented service. 
     The process  200  may include receiving a second annotation in the first source code. The second annotation may comprise second metadata defining an SLO (block  220 ). For example, there may be an SLO associated with a latency requirement for a particular aspect of the first computer-implemented service. The second annotation may define the SLO and may be used to programmatically populate the relevant fields of the OpenSLO specification during compile time. 
     In an example, the process  200  may include executing the first computer-implemented service using the first source code (block  225 ). In various examples, the first source code is compiled in order to generate machine language that can be executed by a computing device. The first computer-implemented service may be defined by the first source code. 
     In an example, the process  200  may include generating the OpenSLO specification based on the first annotation and the second annotation (block  230 ). In various examples, during compilation of the first source code, the first source code may be parsed to identify the annotations by an annotation processor. The annotation processor may programmatically generate the OpenSLO specification with the name specified by the first annotation and with a definition of the SLO related to the first aspect of the first computer-implemented service. 
       FIG.  3    is flowchart illustrating an example process  300  for generating OpenSLO specifications using an annotation processor according to an example of the present disclosure. Although the example process  300  is described with reference to the flowchart illustrated in  FIG.  3   , it will be appreciated that many other methods of performing the acts associated with the process  300  may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, blocks may be repeated, and some of the blocks described may be optional. The process  300  may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software, or a combination of both. In some examples, the actions described in the blocks of the process  300  may represent a series of instructions comprising computer-readable machine code executable by one or more processing units of one or more computing devices. In various examples, the computer-readable machine codes may be comprised of instructions selected from a native instruction set of and/or an operating system (or systems) of the one or more computing devices. 
     In various examples, the process  300  may include receiving, by an annotation processor, first source code of a first computer-implemented service (block  310 ). For example, first source code defining the first computer-implemented service may be entered into an IDE by a developer. The first computer-implemented service may be associated with an SLA and one or more SLOs. The annotation processor may receive the first source code during compilation of the first source code. 
     The process  300  may include parsing, by the annotation processor, the first source code to identify a first annotation in the first source code, the first annotation comprising first metadata (block  315 ). In various examples, the first annotation may pertain to a field of an OpenSLO specification that may be programmatically generated by the annotation processor during compilation of the first source code. For example, the first annotation may be a Java annotation that may be used to define an SLO to be included in the OpenSLO specification. 
     The process  300  may include parsing, by the annotation processor, the first source code to identify a second annotation in the first source code, the second annotation comprising second metadata defining an SLO (block  320 ). For example, there may be an SLO associated with a throughput requirement for a particular aspect of the first computer-implemented service. The second annotation may define the SLO and may be used by the annotation processor to programmatically populate the relevant fields of the OpenSLO specification during compile time. 
     In some examples, process  300  may include generating, by the annotation processor, the OpenSLO specification based on the first annotation and the second annotation (block  325 ). As previously described, the annotations of the first source code may define the OpenSLO specification to be generated by the annotation processor during compilation of the first source code. Accordingly, various annotations (e.g., Java annotations) included in the first source code may define and/or provide values or other data to be used to populate the OpenSLO specification template to generate the relevant OpenSLO specification for the first computer-implemented service. 
       FIG.  4    illustrates a flow diagram of an example generation of OpenSLO specifications using annotations, according to various aspects of the present disclosure. Although the examples below are described with reference to the flow diagram illustrated in  FIG.  4   , many other methods of performing the acts associated with  FIG.  4    may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, and some of the blocks described are optional. The methods may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software, or a combination of both. In illustrated example  400 , an IDE  424  may receive first source code for a first computer-implemented service (e.g., a cloud-based application) (block  410 ). The IDE  424  may receive a first Java annotation in the first source code (block  412 ). The first Java annotation may include metadata that indicates the name of an SLO specification. In various examples, declaration of the name of the SLO specification may be used to generate the SLO specification with the specified name (i.e., the SLO specification may not yet exist) during compilation of the first source code. 
     The IDE  424  may receive a second Java annotation in the first source code (block  414 ). The second Java annotation may include metadata indicating a first SLO (e.g., an SLO associated with an SLA for the first computer-implemented service). IDE  424  may receive an instruction to compile the first source code (block  416 ). The compilation may be performed by a compiler of the IDE  424  or may be performed by a separate compiler. 
     Open SLO generator  422  (e.g., an annotation processor) may parse the first source code (block  418 ) and may identify the first Java annotation (block  420 ) and the second Java annotation (block  428 ). The annotation processor may generate the OpenSLO specification (block  430 ) that includes the name specified by the first Java annotation and that includes the SLO defined by the second Java annotation. The OpenSLO generator  422  may use the metadata included in the Java annotations to generate the corresponding OpenSLO YAML descriptor. The OpenSLO generator  422  may populate the fields of the corresponding OpenSLO YAML descriptors using the metadata included in the Java annotations (block  432 ). In some further examples, the OpenSLO generator  422  may use Java annotations to generate markup based documentation (e.g., HTML, pdf, etc.) describing the first computer-implemented service and/or the OpenSLO specification. Further, the OpenSLO generator  422  may publish the OpenSLO specification at a path specified in the first source code (e.g., using a third Java annotation) (block  434 ). In various examples, the OpenSLO specification may be published by a remote system  426 . Accordingly, the remote system  426  (e.g., a web server) may store the OpenSLO specification in a data store (block  436 ). 
       FIG.  5    illustrates a flow diagram of an example of testing a computer-implemented service using an OpenSLO specification generated using annotations, according to various aspects of the present disclosure. Although the examples below are described with reference to the flow diagram illustrated in  FIG.  5   , many other methods of performing the acts associated with  FIG.  5    may be used. For example, the order of some of the blocks may be changed, certain blocks may be combined with other blocks, and some of the blocks described are optional. The methods may be performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software, or a combination of both. 
     In the illustrated example  500 , an OpenSLO generator  522  (e.g., an annotation processor and/or compiler) may generate an OpenSLO specification for a first computer-implemented service using Java annotations included in the source code for the first computer-implemented service (block  510 ). The OpenSLO specification may be provided to a testing component  524  (block  512 ). In various examples, Java annotations included in the source code may be used to generate testing scripts that can be executed (e.g., by the testing component). The testing scripts may reference various YAML descriptors of the OpenSLO specification for the first computer-implemented service. Accordingly, in some examples, both the testing scripts and the OpenSLO specification may be programmatically generated from the first source code by an annotation processor using annotations included in the first source code. Additionally, the testing scripts may reference the various YAML descriptors of the OpenSLO specification to determine whether or not various SLOs defined in the OpenSLO specification are being met when the first computer-implemented service is subjected to some compute load (also specified by the testing scripts). 
     The testing component  524  may receive the OpenSLO specification (block  514 ) and may execute the testing script(s) (block  516 ). The testing component  524  may send instructions to execute the first computer-implemented service (block  518 ) to a system under test  526 . In various examples, the system under test  526  may represent a desired deployment (e.g., using one or more clusters, nodes, VMs, Docker containers, etc.) of the first computer-implemented service. The system under test  526  may execute the first computer-implemented service (block  520 ) according to configuration data provided by testing component  524  (not shown in  FIG.  5   ). In various examples, the configuration data may be generated using the testing scripts and may be specified using Java annotations in the source code. 
     The testing component  524  may send instructions to the system under test  526  to generate a first load on the first computer-implemented service (block  528 ). For example, the testing component  524  may specify an amount of traffic sent to a first service of the first computer-implemented service. The system under test  526  may process the first load (block  530 ) and may determine the relevant results according to instructions provided by the testing component  524  (as part of the testing script(s)) (block  532 ). For example, the average latency and/or throughput during a specified time period may be provided as result data. The testing component  524  may receive the result data (block  534 ) and may programmatically evaluate the result data using the OpenSLO specification (block  536 ). In various examples, the result data may be evaluated by the testing scripts. The testing component  524  may generate performance data (block  538 ). The performance data may indicate whether the first computer-implemented service is meeting one or more SLOs defined by the OpenSLO specification. The Java annotations in the first source code may be used to generate the testing scripts based on a combination of SLO-specific Java annotations and testing annotations (e.g., JAX-RS or similar) and can be used to programmatically evaluate the first computer-implemented service. 
       FIG.  6    is block diagram of a system  600  effective to generate an OpenSLO specification  626  using annotations (e.g., first annotation  610  and/or second annotation  616 ), according to an example of the present disclosure. 
     System  600  may comprise at least one processor  604  and non-transitory computer-readable memory  603 . The memory  603  may store instructions  606 . The instructions  606  may be executed by the at least one processor  604  to perform various techniques described herein related to generation of the OpenSLO specification  626 . 
     The at least one processor  604  may receive the first source code  602 . The first source code  602  may define the functionality of a first computer-implemented service  622 . The at least one processor  604  may receive a first annotation  610  in the first source code  602 . The first annotation  610  may comprise first metadata  612  that defines a name of an OpenSLO specification  614 . The at least one processor  604  may receive a second annotation  616  in the first source code  602 . The second annotation  616  may comprise second metadata  618  that defines an SLO  620  of a first aspect of the first computer-implemented service  624 . 
     In some examples, the at least one processor  604  may execute the first computer-implemented service  622  using the first source code  602 . In some further examples, the at least one processor  604  may generate the OpenSLO specification  626  using the first annotation  610  and the second annotation  616 . The OpenSLO specification  626  may comprise the name of the OpenSLO specification  614 ′ and a definition of the SLO  628  of the first aspect of the first computer-implemented service  622 . 
       FIG.  7    is block diagram  700  of an annotation processor  766  in communication with an integrated development environment  750  according to an example of the present disclosure. The integrated development environment  750  may receive first source code  702  that is associated with a first computer-implemented service. The integrated development environment  750  may receive a first annotation  710  in the first source code  702 . The first annotation  710  may include first metadata  712  defining a name of an OpenSLO specification  714 . 
     The integrated development environment  750  may receive a second annotation  716  in the first source code  702 . The second annotation  716  may include second metadata  718  defining an SLO  720  of a first aspect of the first computer-implemented service. The compiler  762  may receive the first source code  702  and may compile the first source code  702  to generate executable code for the first computer-implemented service  764 . 
     Annotation processor  766  may receive the first annotation  710  from the first source code  702  and the second annotation  716  from the first source code  702 . Annotation process  766  may generate the OpenSLO specification  726  based on the first annotation  710  and the second annotation  716 . The OpenSLO specification  726  may include the name of the OpenSLO specification  714 ′ and the definition of the SLO  728  of the first aspect of the first computer-implemented service. 
     It will be appreciated that all of the disclosed methods and procedures described herein can be implemented using one or more computer programs or components. These components may be provided as a series of computer instructions on any conventional computer readable medium or machine readable medium, including volatile or non-volatile memory, such as RAM, ROM, flash memory, magnetic or optical disks, optical memory, or other storage media. The instructions may be provided as software or firmware, and/or may be implemented in whole or in part in hardware components such as ASICs, FPGAs, DSPs or any other similar devices. The instructions may be executed by one or more processors, which when executing the series of computer instructions, performs or facilitates the performance of all or part of the disclosed methods and procedures. 
     Aspects of the subject matter described herein may be useful alone or in combination with one or more other aspects described herein. In a 1st example aspect of the present disclosure, a method comprises receiving first source code associated with a first computer-implemented service; receiving a first annotation in the first source code, the first annotation comprising first metadata defining a name of an OpenSLO specification; receiving a second annotation in the first source code, the second annotation comprising second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service; executing the first computer-implemented service using the first source code; and generating the OpenSLO specification based on the first annotation and the second annotation, the OpenSLO specification including the name and a definition of the SLO of the first aspect of the first computer-implemented service. 
     In accordance with a 2nd example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 1st aspect), wherein the first annotation and the second annotation are Java annotations. 
     In accordance with a 3rd example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 1st aspect), compiling the first source code, wherein the OpenSLO specification is generated using the first metadata and the second metadata during compilation of the first source code. 
     In accordance with a 4th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 3rd aspect), wherein an annotation processor generates the OpenSLO specification during the compilation of the first source code. 
     In accordance with a 5th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 1st aspect), further comprising generating a third annotation in the first source code, the third annotation comprising third metadata defining a uniform resource locator (URL) at which the OpenSLO specification is published. 
     In accordance with a 6th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 1st aspect), generating, using at least one script, a first load for the first computer-implemented service; receiving, by the at least one script, the OpenSLO specification as an input; determining first metric data describing performance of the first aspect of the first computer-implemented service; and comparing the first metric data to the SLO of the first aspect of the first computer-implemented service. 
     In accordance with a 7th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 6th aspect), determining, using the at least one script, that the first metric data does not meet the SLO; and generating first output data indicating that the first metric data does not meet the SLO. 
     In accordance with an 8th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 1st aspect), generating, using the first metadata, a first YAML descriptor in the OpenSLO specification. 
     In accordance with a 9th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 1st aspect), wherein the OpenSLO specification comprises a standardized definition for defining service-level objectives (SLOs). 
     In accordance with a 10th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 1st aspect), wherein the second metadata logically maps to a field of the OpenSLO specification that is associated with the first aspect of the first computer-implemented service. 
     Aspects of the subject matter described herein may be useful alone or in combination with one or more other aspects described herein. In a 11th example aspect of the present disclosure, a system comprises at least one processor; and non-transitory computer-readable memory storing instructions that, when executed by the at least one processor are configured to: receive first source code defining a first computer-implemented service; receive a first annotation in the first source code, the first annotation comprising first metadata defining a name of an OpenSLO specification; receive a second annotation in the first source code, the second annotation comprising second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service; execute the first computer-implemented service using the first source code; and generate the OpenSLO specification based on the first annotation and the second annotation, the OpenSLO specification including the name and a definition of the SLO of the first aspect of the first computer-implemented service. 
     In accordance with a 12th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 11th aspect), wherein the first annotation and the second annotation are Java annotations. 
     In a 13th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 11th aspect), compiling the first source code, wherein the OpenSLO specification is generated using the first metadata and the second metadata during compilation of the first source code. 
     In a 14th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 13th aspect), wherein an annotation processor generates the OpenSLO specification during the compilation of the first source code. 
     In a 15th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 11th aspect), the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor are further configured to generate a third annotation in the first source code, the third annotation comprising third metadata defining a uniform resource locator (URL) at which the OpenSLO specification is published. 
     In a 16th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 11th aspect), the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor are further configured to: generate, using at least one script, a first load for the first computer-implemented service; receive, by the at least one script, the OpenSLO specification as an input; determine first metric data describing performance of the first aspect of the first computer-implemented service; and compare the first metric data to the SLO of the first aspect of the first computer-implemented service. 
     In accordance with a 17th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 16th aspect), the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor are further configured to: determine, using the at least one script, that the first metric data does not meet the SLO; and generate first output data indicating that the first metric data does not meet the SLO. 
     In accordance with a 18th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 11th aspect), the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor are further configured to generate, using the first metadata, a first YAML descriptor in the OpenSLO specification. 
     In accordance with a 19th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 11th aspect), wherein the OpenSLO specification comprises a standardized definition for defining service-level objectives (SLOs). 
     In accordance with a 20th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 11th aspect), wherein the second metadata logically maps to a field of the OpenSLO specification that is associated with the first aspect of the first computer-implemented service. 
     Aspects of the subject matter described herein may be useful alone or in combination with one or more other aspects described herein. In a 21st example aspect of the present disclosure, a system comprises an integrated development environment; a compiler; and an annotation processor; the integrated development environment effective to: receive first source code associated with a first computer-implemented service; receive a first annotation in the first source code, the first annotation comprising first metadata defining a name of an OpenSLO specification; and receive a second annotation in the first source code, the second annotation comprising second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service; the compiler effective to compile the first source code to generate executable code for the first computer-implemented service; and the annotation processor effective to: receive the first annotation from the first source code; receive the second annotation from the first source code; and generate the OpenSLO specification based on the first annotation and the second annotation, the OpenSLO specification including the name and a definition of the SLO of the first aspect of the first computer-implemented service. 
     In accordance with a 22nd example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 21st aspect), wherein the first annotation and the second annotation are Java annotations. 
     In accordance with a 23rd example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 21st aspect), wherein the OpenSLO specification is generated using the first metadata and the second metadata by the annotation processor during compilation of the first source code. 
     In accordance with a 24th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 21st aspect), the integrated development environment further effective to receive a third annotation in the first source code, the third annotation comprising third metadata defining a uniform resource locator (URL) at which the OpenSLO specification is published. 
     In accordance with a 25th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 21st aspect), further comprising a second computer-implemented service, wherein the second computer-implemented service is effective to: generate, using at least one script, a first load for the first computer-implemented service; receive, by the at least one script, the OpenSLO specification as an input; determine first metric data describing performance of the first aspect of the first computer-implemented service; and compare the first metric data to the SLO of the first aspect of the first computer-implemented service. 
     In accordance with a 26th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 25th aspect), wherein the second computer-implemented service is further effective to: determine, using the at least one script, that the first metric data does not meet the SLO; and generate first output data indicating that the first metric data does not meet the SLO. 
     In accordance with a 27th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 21st aspect), wherein the annotation processor effective to generate, using the first metadata, a first YAML descriptor in the OpenSLO specification. 
     In accordance with a 28th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 21st aspect), wherein the OpenSLO specification comprises a standardized definition for defining service-level objectives (SLOs). 
     In accordance with a 29th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 21st aspect), wherein the second metadata logically maps to a field of the OpenSLO specification that is associated with the first aspect of the first computer-implemented service. 
     Aspects of the subject matter described herein may be useful alone or in combination with one or more other aspects described herein. In a 30th example aspect of the present disclosure, a method comprising: receiving, by an annotation processor, first source code of a first computer-implemented service; parsing, by the annotation processor, the first source code to identify a first annotation in the first source code, the first annotation comprising first metadata defining a name of an OpenSLO specification; parsing, by the annotation processor, the first source code to identify a second annotation in the first source code, the second annotation comprising second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service; generating, by the annotation processor, the OpenSLO specification based on the first annotation and the second annotation, the OpenSLO specification including the name and a definition of the SLO of the first aspect of the first computer-implemented service. 
     In accordance with a 31st example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 30th aspect), wherein the first annotation and the second annotation are Java annotations. 
     In accordance with a 32nd example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 30th aspect), further comprising compiling the first source code, wherein the OpenSLO specification is generated using the first metadata and the second metadata during compilation of the first source code. 
     In accordance with a 33rd example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 32nd aspect), wherein the annotation processor generates the OpenSLO specification during the compilation of the first source code. 
     Aspects of the subject matter described herein may be useful alone or in combination with one or more other aspects described herein. In a 34th example aspect of the present disclosure, a system comprising: a means for generating first source code of a first computer-implemented service; a means for generating a first annotation in the first source code, the first annotation comprising first metadata defining a name of an OpenSLO specification; a means for generating a second annotation in the first source code, the second annotation comprising second metadata defining a service-level objective (SLO) of a first aspect of the first computer-implemented service; a means for executing the first computer-implemented service using the first source code; and a means for generating the OpenSLO specification based on the first annotation and the second annotation, the OpenSLO specification including the name and a definition of the SLO of the first aspect of the first computer-implemented service. 
     In accordance with a 35th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 34th aspect), further comprising a means for compiling the first source code, wherein the OpenSLO specification is generated using the first metadata and the second metadata during compilation of the first source code. 
     In accordance with a 36th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 35th aspect), wherein an annotation processor generates the OpenSLO specification during the compilation of the first source code. 
     In accordance with a 37th example aspect of the present disclosure, which may be used in combination with any one or more of other aspects described herein (e.g., the 34th aspect) further comprising a means for generating a third annotation in the first source code, the third annotation comprising third metadata defining a uniform resource locator (URL) at which the OpenSLO specification is published. 
     To the extent that any of these aspects are mutually exclusive, it should be understood that such mutual exclusivity shall not limit in any way the combination of such aspects with any other aspect whether or not such aspect is explicitly recited. Any of these aspects may be claimed, without limitation, as a system, method, apparatus, device, medium, etc. 
     It should be understood that various changes and modifications to the example embodiments described herein will be apparent to those skilled in the art. Such changes and modifications can be made without departing from the spirit and scope of the present subject matter and without diminishing its intended advantages. It is therefore intended that such changes and modifications be covered by the appended claims.