Patent Publication Number: US-9836328-B2

Title: System and method for improving memory usage in virtual machines at a cost of increasing CPU usage

Description:
This Application is a Continuation Application of U.S. patent application Ser. No. 13/782,661, filed on Mar. 1, 2013, which is a Continuation Application of U.S. patent application Ser. No. 13/738,814, filed on Jan. 10, 2013. 
    
    
     BACKGROUND OF THE INVENTION 
     Field of the Invention 
     The present invention relates generally to memory optimization and improving the efficiency of real memory use by applications and computing platforms such as physical and virtual machines. More specifically, in a system having a plurality of virtual machines, information is periodically collected on memory and CPU usage of each virtual machine and a memory optimizer uses this information to re-allocate memory among the virtual machines, as based on at least one memory optimization mechanism which can reduce memory usage of a virtual machine at a cost of increasing CPU usage. 
     Background of the Invention 
     It is often important to run multiple virtual machines (VMs) concurrently, particularly in cloud computing environments, where multiple applications can be run concurrently. However, poor memory usage can make a system almost unusable, as, for example, when too much paging between memory and disk can slow the system down. 
     SUMMARY OF THE INVENTION 
     In view of the foregoing and other exemplary problems, drawbacks, and disadvantages of conventional methods and systems, an exemplary feature of the present invention is to provide a method and structure to dynamically allocate memory between concurrently-running virtual machines. 
     In a first exemplary aspect of the present invention, described herein is an apparatus, including: at least one processor upon which can be executed a virtual memory optimizer for optimizing a memory usage among a plurality of concurrently-running virtual machines; and a memory that stores a set of computer readable instructions for implementing and executing the virtual memory optimizer, the memory optimizer performing a monitoring of a usage of memory by each virtual machine of the plurality of concurrently-running virtual machines and applying at least one memory optimization mechanism that reallocates memory among the concurrently-running virtual machines based on reducing a memory usage of a virtual machine as a tradeoff of increasing a central processing unit (CPU) usage to achieve the reduced memory usage. 
     In a second exemplary aspect of the present invention, also described herein is a system comprising a plurality of concurrently-running virtual machines (VMs), the system comprising at least one computer including: at least one processor upon which can be executed a virtual memory optimizer for optimizing a memory usage among the plurality of concurrently-running VMs; and a memory that stores a set of computer readable instructions for implementing and executing the virtual memory optimizer, the virtual memory optimizer providing at least one memory optimization mechanism which can reduce a memory usage of a virtual machine at a cost of increasing a central processing unit (CPU) usage. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The foregoing and other exemplary purposes, aspects and advantages will be better understood from the following detailed description of an exemplary embodiment of the invention with reference to the drawings, in which: 
         FIG. 1  exemplarily depicts a system  100  in accordance with the invention for optimizing memory usage among multiple virtual machines  101 ; 
         FIG. 2  is a description in flowchart format  200  of an exemplary method in accordance with the invention for optimizing memory usage among multiple virtual machines; 
         FIG. 3  is a description in flowchart format  300  of an exemplary method in accordance with the invention for handling excess CPU usage; 
         FIG. 4  exemplarily shows a computer system  400  for implementing the present invention; 
         FIG. 5  exemplarily shows hardware aspects  500  of a computer system implementing the present invention; and 
         FIG. 6  exemplarily shows portable memory devices  600 ,  602  that could be used for storing machine code that implements the present invention. 
     
    
    
     DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS OF THE INVENTION 
       FIG. 1  depicts an exemplary system having features of the present invention. A memory optimizer  102  optimizes memory use across multiple virtual machines  101 . The memory optimizer could be implemented in software and can execute on one or multiple processor nodes. The virtual machines  101  may also execute on one or more processor nodes. 
     Virtual machines may be implemented in multiple ways including but not limited to IBM&#39;s Dynamic Logical Partitioning (DLPAR) (described in various publications, including an internal IBM web page identified in the cited reference section on the front page of the issued patent version of this disclosure. This publication explains how DLPAR provides the ability to logically attach and detach a managed system&#39;s resources to and from a logical partition&#39;s operating system without rebooting. The contents of this publication are incorporated herein by reference), VMware virtualization platforms such as ESX, Xen, KVM, Java virtual machines, etc. Information on commonly used virtualization platforms is available in a number of publications, including a generic description in a wikipedia article entitled “Comparison_of_platform_virtual_machines”. The contents of this publication are incorporated herein by reference. 
     This publication explains how platform virtual machines are software packages that emulate a whole physical computer machine, often providing multiple virtual machines on one physical platform. This publication also provides a comparison of basic information about a relatively large number of platform virtual machine (VM) packages. 
     The memory optimizer  102  of the present invention could run on one or more nodes on which the virtual machines  101  execute. Alternatively, the memory optimizer  102  could run on one or more nodes which are distinct from the one or more nodes on which the virtual machines  101  execute. Alternatively, the memory optimizer  102  could run one or more nodes on which the virtual machines  101  execute as well as one or more nodes which are distinct from the one or more nodes on which the virtual machines  101  execute. 
     The piece of software, firmware, or hardware that creates and runs virtual machines is often referred to as a hypervisor. All or a fraction of the memory optimizer  102  could be part of a hypervisor. However, this is not necessary. The memory optimizer  102  can also execute independently from a hypervisor. 
     In the context of explaining the present invention, there is a pool of memory that can be allocated to multiple virtual machines. In other words, some memory m 1  could be allocated to either virtual machine vm 1  or virtual machine vm 2 . If vm 1  has a lot of free memory while vm 2  has little free memory, it is preferable to allocate m 1  to vm 2 . By contrast, if vm 2  has a lot of free memory while vm 1  has little free memory, it is preferable to allocate m 1  to vm 1 . 
     The memory optimizer  102  determines how memory should be allocated to different virtual machines. If vm 1  has a lot of free memory while vm 2  is running low on free memory, memory optimizer  102  might allocate some of vm 1 &#39;s free memory to vm 2 . 
     Once a virtual machine  101  starts running low on memory, its performance can degrade considerably. In some cases, insufficient memory can cause applications to not execute properly at all. It is therefore desirable to maintain proper amounts of memory for all virtual machines  101 . If sufficient memory exists throughout the system, the memory optimizer  102  can allocate memory across the virtual machines  101  to provide enough memory for each virtual machine  101 . If there is not enough memory in the system to prevent a virtual machine  101  from running low on memory, other actions need to be taken. 
     There are various special techniques that can be deployed to reduce the memory consumed by a virtual machine. One such technique is memory compression. Another technique is memory deduplication, in which duplicate memory pages are identified. Memory deduplication stores a single copy of duplicate memory pages. A third technique is delta encoding. For two pages p 1  and p 2  which are similar but not identical, it is not necessary to store entire versions of both p 1  and p 2 . Instead, p 1  could be stored along with a delta which encodes just the differences between p 1  and p 2 . The delta would typically be considerably smaller than p 2 . 
     The use of these and other techniques for virtual machines is described in D. Gupta et al, “Difference Engine: Harnessing Memory Redundancy in Virtual Machines”, Proceedings of OSDI 2008, the contents of which is incorporated herein by reference. There are other techniques that can be applied to reduce memory usage as well. For example, a virtual machine  101  might be a Java virtual machine (abbreviated JVM) running a Java workload. An overview of JVMs is described in various publications, including a Wikipedia article “Java_virtual_machine”, the content of which is incorporated herein by reference. Java allocates memory from an area known as the heap. In order to free memory from the heap, garbage collection is needed. 
     The concepts of “heap” and “garbage collection” are further explained in various publications, such as the Wikipedia article entitled “programming_language”, the Wikipedia article entitled “memory management”, and the Wikipedia article entitled “garbage collection”, the contents of these articles being hereby incorporated herein by reference. In summary, memory management is the process of managing computer memory, including mechanisms to dynamically allocate portions of memory to programs upon request and freeing it for reuse when no longer needed. Memory requests are satisfied by allocating portions from a large pool of memory referred to as “the heap”, and, at any given time, some parts of the heap are in use while some are “free” (i.e., unused) and thus available for future allocations. 
     Garbage collection is a form of automatic memory management that attempts to reclaim “garbage”, as referring to memory occupied by objects that are no longer in use by a program. Garbage collection is often portrayed as the opposite of manual memory management, a mechanism in which the programmer specifies which objects to deallocate and return to the memory system. Many computer languages, such as Java, C#, and several scripting languages, require garbage collection either as part of the language specification or effectively for practical implementation, while other languages, such as C, C++, were designed for use with manual memory management but have garbage collected implementations available. Still others, such as Ada, Modula-3, and C++/CLI allow both garbage collection and manual memory management to co-exist in the same application by using separate heaps, and others, such as D, are garbage collected but allow the user to manually delete objects and disable garbage collection when speed is required. 
     The entity that performs garbage collection is known as the garbage collector. Garbage collectors are typically implemented in software, but they can also be implemented using both software and hardware. For example, a system might have special hardware support to aid garbage collection. 
     Returning now to an exemplary embodiment of the present invention using JVM, if a smaller maximum heap size is used, the JVM consumes less memory. However, the garbage collector needs to run more frequently, which uses up additional CPU cycles. Thus, there is once again a memory/CPU trade-off. Larger maximum heap sizes use up more memory for the heap but save CPU cycles because less frequent garbage collections are needed. Smaller maximum heap sizes use up less memory but use more CPU cycles because more frequent garbage collections are needed. 
     Note that this technique of modifying Java heap space is applicable to other languages with automatic memory management and garbage collection, such as Lisp, Smalltalk, C#, many scripting languages, etc. The present invention is applicable to languages which use garbage collection in general. The term “heap” refers to the memory area managed using garbage collection and is not specific to the Java programming language. However, for ease of exposition, we exemplarily discuss concepts of the present invention in terms of Java and Java virtual machines. One skilled in the art could easily apply this invention to other languages with garbage collection. 
     Other techniques besides those mentioned above can be used for optimizing memory usage within the spirit and scope of this invention. 
     As recognized by the present inventors, a key problem with these techniques is that they consume CPU overhead. Thus, while they improve memory usage, they hurt CPU performance. The present inventors have recognized that what is needed is a selective way to apply these techniques. That is what the present invention provides, 
     Thus, in the present invention, memory optimizer  102  shown in  FIG. 1  takes a global view of all virtual machines  101  that potentially can be allocated memory under its control, as follows. When free memory throughout the system is low and a particular virtual machine  101  needs more memory, the memory optimizer identifies one or more virtual machines  101  which has spare CPU capacity to apply an optimization to free up memory. That way, a virtual machine  101  running on hardware with little spare CPU capacity will not have to devote additional CPU cycles in order to free up memory. 
       FIG. 2  depicts an exemplary method for managing memory across multiple virtual machines  101 . In Step  201 , the memory optimizer  102  is continuously monitoring the system. It collects information such as memory usage of virtual machines, CPU usage of virtual machines, etc. For such information collection, memory optimizer  102  can use standard tools (e.g. vmstat) in operating systems such as Linux, other variants of Unix, Windows, etc. to collect this information. Other methods and tools can also be used to monitor memory and CPU usage within the spirit and scope of this invention. 
     In step  202 , memory optimizer  102  determines that a particular virtual machine VMx (e.g., VM 1 ) needs more memory. This can be done in several ways. 
     For example, memory optimizer  102  might determine that the amount of free memory available to VM 1  is decreasing and getting close to 0 (memory which is available to a virtual machine  101  but is not being used is “free memory”); more specifically, the amount of free memory may fall below a threshold. Alternatively, the memory optimizer  102  might determine that VM 1  has no free memory and is paging. Alternatively, the memory optimizer  102  might have predictions of future memory needs for VM 1  based on empirical data of VM 1 &#39;s memory usage in the past. Even though VM 1  currently has some free memory, the memory optimizer  102  might predict that VM 1  is likely to run out of memory in the near future unless it is given more memory. Other methods for determining that VM 1  needs more memory are possible within the spirit and scope of the invention. 
     The memory optimizer  102  has a global view of how much memory each virtual machine  101  has available to it and how much memory a virtual machine  101  is using. If, in step  203 , the memory optimizer  102  determines that sufficient free memory exists from other virtual machines  101  to satisfy the memory needs of VM 1 , the memory optimizer  102 , in step  204 , allocates additional memory to VM 1  from one or more other virtual machines  101  with free memory. 
     If the memory optimizer  102  determines in step  203  that sufficient free memory from other virtual machines  101  does not exist to fully satisfy the memory needs of VM 1 , the memory optimizer in step  205  tries to identify one or more virtual machines  101  which can free up memory to give to VM 1  by applying an optimization. Such optimizations include but are not limited to the aforementioned memory compression, memory deduplication, delta encoding, and reducing heap space for Java applications and/or applications in other programming languages with automatic memory management and garbage collection. 
     The memory optimizer  102  uses knowledge of specific optimizations to estimate both the CPU overhead of applying an optimization and the amount of memory which would be freed by applying the optimization. For example, information on memory used by specific applications can be analyzed to determine both the amount of memory saved and the CPU overhead incurred for applying optimizations such as memory compression, memory deduplication, and delta encoding. 
     A specific example of a possible mechanism the memory optimizer  102  could use to estimate both the CPU overhead of applying an optimization and the amount of memory which could be freed by applying the optimization is IBM&#39;s amepat tool (Active Memory™ Expansion Planning and Advisory Tool), described in various publications including internal IBM publication located at the URL address identified in the references section. The content of this publication, incorporated herein by reference, describes amepat. Amepat provides information on memory saved and CPU overhead incurred by memory compaction. 
     Another example of how the overhead of applying an optimization and the amount of memory which could be freed by applying the optimization could be estimated is the following. For programming languages using garbage collection, such as Java, empirical data can be collected on overhead incurred by garbage collection as a function of heap size. If memory usage of individual applications is profiled, these estimates of garbage collection overheads can be more accurate. From this data, the memory optimizer  102  will be able to estimate the effect of heap size on garbage collection overhead. If a virtual machine has sufficient excess CPU capacity, the memory optimizer  102  can reduce the Java heap size appropriately to free up memory. 
     Based on estimates of CPU overhead incurred for an optimization and the memory expected to be freed, the memory optimizer  102  determines which optimization (s) should be applied to which virtual machines  101  to free memory. The CPU load on the virtual machines  101  is an important part of the decision. If a virtual machine  101  has high CPU utilization or is predicted to have high CPU utilization in the near future, that virtual machine  101  is not a good candidate to which to apply a CPU-intensive memory optimization. On the other hand, if a virtual machine  101  is consuming few CPU cycles and is not predicted to have a significant increase in CPU consumption in the near future, that virtual machine might be a better candidate for applying the memory optimization, provided the virtual machine is predicted to release a significant amount of memory as a result of applying the optimization. 
     Memory usage by a virtual machine  101  is also used to determine whether it is a good candidate for giving up memory. If a virtual machine  101  is not using up much memory, then it is not a good candidate for giving up memory by applying an optimization since it has little memory to give. If, on the other hand, a virtual machine  101  is using up a lot of memory which could be freed by applying an optimization, then it is a good candidate for giving up memory by applying an optimization. 
     There are several different criteria which can be applied to determine which optimizations should be applied to which virtual machine, including, but not limited, to the following:
         If a virtual machine&#39;s predicted CPU utilization after applying the optimization is below a threshold value (e.g. 0.5), apply the optimization.   If a virtual machine&#39;s predicted CPU utilization after applying the optimization is below a threshold value and the amount of memory predicted to be freed by the optimization exceeds a threshold (e.g. 100 Megabytes), apply the optimization.   If a virtual machine&#39;s predicted CPU utilization after applying the optimization is below a threshold value and the amount of memory predicted to be freed by the optimization exceeds a threshold proportion of the memory needed by VM 1 , (e.g. 0.05), apply the optimization.   Each virtual machine has a target CPU utilization. Give priority to applying optimizations to virtual machines which have the highest value of target CPU utilization minus actual CPU utilization.   Each virtual machine has a target memory usage. Give priority to applying optimizations to virtual machines which have the lowest values of target memory utilization minus actual memory utilization.       

     It should be mentioned that each of the thresholds mentioned above could be different for different virtual machines  101 . Alternatively, one or more thresholds could be the same for one or more virtual machines. 
     The memory optimizer  102  may apply one or more of the criteria above in determining which optimizations to apply to which virtual machines  101 . For example, the memory optimizer  102  could give priority to applying optimizations to virtual machines which have the highest value of target CPU utilization minus actual CPU utilization while also giving priority to applying optimizations to virtual machines which have the lowest values of target memory utilization minus actual memory utilization. 
     It may be possible to apply multiple different optimizations to free up memory. In this case, memory optimizer  102  makes intelligent choices of which optimization (s) to apply, and to what degree. Thus, in an exemplary embodiment of the present invention, memory optimizer  102  can make the choices of which optimization (s) to apply (or increase use of) using the following guidelines:
         Give priority to optimizations which consume relatively few CPU cycles.   Give priority to optimizations which result in relatively more memory savings.   Consider both the memory saved by an optimization and its CPU overhead in determining which optimization (s) to apply (or increase use of). An optimization is a good candidate if it is both relatively inexpensive in CPU cost and saves a relatively high amount of memory compared with other optimizations.       

     Memory optimizer  102  can also use other guidelines to determine which optimization (s) to apply or to increase use of. 
     In step  206 , one or more optimizations are applied (and/or the application of one or more optimizations currently being used is increased). The memory freed by the one or more optimizations is given to virtual machine vm 1 . 
     The memory optimizer  102  also has the ability to reduce (or eliminate use of) the amount of an optimization applied to a virtual machine  101  if the virtual machine  101  is consuming too many CPU cycles. This is illustrated in  FIG. 3 . 
     In step  201  of  FIG. 3 , the memory optimizer monitors the system. This is the same step as step  201  in  FIG. 2 . In step  301 , memory optimizer  102  detects a virtual machine VMx (e.g., VM 2 ) which is consuming too many CPU cycles. For example, VM 2  might have a CPU utilization exceeding a predetermined threshold, such as 90%. CPU utilization statistics may be determined from one or more measurements. In the case of multiple CPU utilization measurements, an aggregate figure, such as an average CPU utilization, may be determined. In some cases, step  301  might determine that even though VM 2  is not currently consuming too many CPU cycles, it is likely to do so in the near future based on an analysis of past behavior, and action must be taken now to prevent CPU overload from occurring. The memory optimizer  102  then determines what optimizations, if any, are being applied by VM 2 . If it detects optimizations being applied to VM 2 , it reduces (or eliminates) the level of one or more optimizations to reduce CPU usage in step  302 . As mentioned above, optimizations might include, but are not limited to, memory compression, memory deduplication, delta encoding, reducing heap space for Java applications and/or applications in other programming languages with automatic memory management and garbage collection. For example, the optimizer could increase the size of a Java heap in step  302  to reduce CPU cycles consumed by garbage collection. 
     There are multiple methods by which memory optimizer  102  could reduce (or eliminate) the amount of one or more optimizations applied to VM 2 . These include but are not limited to the following:
         VM 2  has a threshold CPU utilization (e.g. 0.5) which is being exceeded. Reduce (or eliminate) optimizations to bring the CPU utilization for VM 2  within the threshold CPU utilization.   Multiple optimizations are being applied to VM 2 . Give priority to reducing (or eliminating) optimizations which consume significant CPU cycles.   Multiple optimizations are being applied to VM 2 . Give priority to reducing (or eliminating) optimizations which result in less memory saved.   Multiple optimizations are being applied to VM 2 . Consider both the memory saved by an optimization and its CPU overhead in determining which optimization (s) to reduce. An optimization is a good candidate to reduce (or eliminate) if it is both relatively expensive in CPU cost and saves relatively little compared with other optimizations.       

     Exemplary Hardware Implementation 
     From the exemplary embodiments described above, it is clear that the present invention is directed to controlling memory allocation for virtual machines and involves control concepts that would typically involve software. However, as is well known in the art, software implementation inherently involves underlying hardware.  FIG. 4  exemplarily demonstrates a computer system  400  upon which a plurality of virtual machines operate using a hypervisor  401  that allocates and controls machine assets for a plurality of LPARs (logical partitions) upon which are respectively running VMs  402 . This system  400  might involve a single computer or could be implemented across a network of interconnected computers. As pointed out about, exemplary embodiments of the present invention involve DLPAR, in which computer resources can be dynamically reallocated without rebooting. 
       FIG. 5  illustrates a typical hardware configuration of an information handling/computer system in accordance with the invention and which preferably has at least one processor or central processing unit (CPU)  511 . In the context of the present invention, this computer would typically be a server on a network. 
     The CPUs  511  are interconnected via a system bus  512  to a random access memory (RAM)  514 , read-only memory (ROM)  516 , input/output (I/O) adapter  518  (for connecting peripheral devices such as disk units  521  and tape drives  540  to the bus  512 ), user interface adapter  522  (for connecting a keyboard  524 , mouse  526 , speaker  528 , microphone  532 , and/or other user interface device to the bus  512 ), a communication adapter  534  for connecting an information handling system to a data processing network, the Internet, an Intranet, a personal area network (PAN), etc., and a display adapter  536  for connecting the bus  512  to a display device  538  and/or printer  539  (e.g., a digital printer or the like). 
     In addition to the hardware/software environment described above, a different aspect of the invention includes a computer-implemented method for performing the above method. As an example, this method may be implemented in the particular environment discussed above. 
     Such a method may be implemented, for example, by operating a computer, as embodied by a digital data processing apparatus, to execute a sequence of machine-readable instructions. These instructions may reside in various types of non-transitory signal-bearing storage media. 
     Thus, this aspect of the present invention is directed to a programmed product, comprising non-transitory signal-bearing storage media tangibly embodying a program of machine-readable instructions executable by a digital data processor incorporating the CPU  511  and hardware above, to perform the method of the invention. 
     This signal-bearing storage media may include, for example, a RAM device  514  contained within the CPU  511 , as represented by the fast-access storage, for example, and used for programs being currently executed, or a ROM device  516  storing program instructions not currently being executed. Alternatively, the instructions may be contained in another signal-bearing storage media, such as a magnetic data storage diskette  600  ( FIG. 6 ), directly or indirectly accessible by the CPU  511 . 
     Whether contained in the diskette  600 , the computer/CPU  511 , or elsewhere, the instructions may be stored on a variety of machine-readable data storage media, such as DASD storage (e.g., a conventional “hard drive” or a RAID array), magnetic tape, electronic read-only memory (e.g., ROM, EPROM, or EEPROM), an optical storage device (e.g. CD-ROM, WORM, DVD, digital optical tape, etc.), paper “punch” cards, or other suitable signal-bearing storage media including memory devices in transmission hardware, communication links, and wireless, and including different formats such as digital and analog. In an illustrative embodiment of the invention, the machine-readable instructions may comprise software object code. 
     As is readily apparent from the above description, the present invention discusses a new method for dynamically controlling memory allocation for a plurality of concurrently-operating VM&#39;s, as based on applying one or more memory optimization mechanisms for reducing memory usage of a VM taking into account a cost of increasing CPU usage. The method also permits CPU usage to be controlled, as related to memory usage. 
     Although the present invention has been described in various exemplary embodiments, it should be apparent that variations of this exemplary embodiment are possible and considered as included in the present invention. 
     Therefore, it is noted that, Applicants&#39; intent is to encompass equivalents of all claim elements, even if amended later during prosecution.