Patent Publication Number: US-10324907-B2

Title: Genomic application data storage

Description:
This application is a continuation of co-pending U.S. patent application Ser. No. 13/804,717, entitled GENOMIC APPLICATION DATA STORAGE filed Mar. 14, 2013 which is incorporated herein by reference for all purposes. 
    
    
     BACKGROUND OF THE INVENTION 
     Genome data often comprises very large datasets and so processing of genome data (e.g., by researchers) cannot be done easily and/or in a reasonable amount of time using just any processing system. To manage such large datasets, distributed systems which can handle very large datasets are often used (e.g., Hadoop systems).  FIG. 1A  is a diagram showing an example of a university (or a company) in which researchers build their own systems for processing genome data and do not share systems with other researchers, even though those other researchers may work for the same university or company. This is an inefficient use of resources, since such systems will be sitting idle most of the time. New systems which can process and/or store very large datasets, such as genome data, would be desirable. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings. 
         FIG. 1A  is a diagram showing an example of a university (or a company) in which researchers build their own systems for processing genome data and do not share systems with other researchers, even though those other researchers may work for the same university or company. 
         FIG. 1B  is a flowchart illustrating an embodiment of a process for providing a processing service. 
         FIG. 2  is a diagram showing an embodiment of a user interface for setting up an account. 
         FIG. 3  is a diagram showing an embodiment of a user interface for specifying a new processing run. 
         FIG. 4  is a diagram showing an embodiment of a user interface which presents a proposed processing run. 
         FIG. 5  is a diagram showing an embodiment of a user interface in which a previous processing run is viewed. 
         FIG. 6  is a diagram showing an embodiment of a user interface which shows account information. 
         FIG. 7  is a diagram showing an embodiment of a decoupled processing and storage associated with a Hadoop system. 
         FIG. 8A  is a diagram showing an embodiment of a processing system which provides processing services to one or more users. 
         FIG. 8B  is a diagram showing a second embodiment of a processing system which provides processing services to one or more users. 
         FIG. 9  is a diagram showing an embodiment of a user interface in which sharing is enabled when a new processing run is specified. 
         FIG. 10  is a diagram showing an embodiment of a user interface in which sharing is enabled after processing has been performed. 
     
    
    
     DETAILED DESCRIPTION 
     The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions. 
     A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured. 
       FIG. 1B  is a flowchart illustrating an embodiment of a process for providing a processing service. In one example, a company or university builds a processing system which performs the process of  FIG. 1B  and makes the system available to its employees. As will be described in further detail below, in some embodiments, requests from different users (e.g., from different primary investigators or labs at a university or different employees at a company) are able to be serviced simultaneously without sharing information with other users unless permitted. In some embodiments, users from different companies or entities are serviced (e.g., the provided processing service is a third-party service and users who are willing to pay for the service are permitted to use the service). 
     At  100 , a request to perform a process on a set of data is received. For example, the set of data may include genome data and a researcher wants to perform one or more processes on the data, such as genome typing or generating statistical information about the genome data (e.g., determining correlations or distributions). 
     At  102 , a set of resources, including processing and storage, is obtained to run an appropriate Hadoop system to process the request received at  100 . Processing resources for a new distributed Hadoop system are allocated and the Hadoop software is provisioned via automated build processes (one example Hadoop software distribution is Pivotal HD). In one example, there is a pool of processing and storage resources associated with the newly provisioned Hadoop system created to deal with the request at  100 . Some portion or all of the pool of allocated resources will be used to service the request received at  100 , in some cases a portion of the pool may be kept in reserve (e.g., so that if a request from another user is received, there are processing resources still available). Automation and orchestration of resource allocation, storage allocation and Hadoop system provisioning are key components of the platform. In one example, automation is provided by the vCloud Automation Center product from VMware (known as vCAC). In the example above, processing resources are virtual and VMware vCloud Director is orchestrated by VCAC to allocate a virtual processing resource at  102 . Provisioning of storage resources on the Isilon scale out NAS platform (for example) is carried out by VCAC in parallel with EMC Integrated Storage Management (ISM) making calls to the Isilon API to automate storage provisioning tasks. Provisioning of virtual Hadoop compute nodes is done (at least in this example) through a combination of VMware Serengeti and Pivotal HD; these Hadoop compute nodes will be held as templates on the infrastructure and deployed based on the request made at  100 . Customization of the Hadoop nodes to meet the specific demands of the request at  100  may be done through custom scripts; this includes the linking of the Hadoop compute nodes to the Pivotal HD Hadoop File System stored directly on the Isilon scale out NAS array. 
     As is described above, in some embodiments, a processing resource allocated at  102  is a virtual processing resource. A virtual processing resource may be attractive because an application or toolkit which runs on a virtual processor is decoupled from the specific implementation of the underlying processor below the virtualization. This enables a variety of underlying processors to be employed and if a switch of underlying processors is desired, such a change is transparent to the application or toolkit because of the virtualization. 
     At  104 , the set of data is processed using the obtained set of resources. For example, the exemplary genome data may be processed using the Broad Institute&#39;s genome analysis toolkit where the toolkit runs on the processing and storage resources allocated at  102 . In some embodiments, processing at  104  requires resources additional resources, such as ports (sometimes referred to as I/Os). In such embodiments, those resources are allocated prior to processing the set of data at  104 . 
     At  106 , one or more storage resources associated with the distributed system are allocated, where a total amount of storage associated with the distributed system is independent of a total amount of processing associated with the distributed system. Similar to the example associated with step  102  described above, in some embodiments there is a pool of storage and some portion of the pool is allocated at  106  to service the request. In some embodiments, a storage resource allocated at  106  is virtual (e.g., and VMware vCenter Server and/or VMware vCloud is used to allocate a virtual storage resource at  106 ). 
     The processing results are stored in the allocated storage resources at  108 . For example, the storage may be network attached storage (NAS), such as EMC Isilon storage. In one example, processing at  104  includes processing genome data using a genome analysis toolkit and generating results in the form of a variant call format (VCF) file. A VCF file may contain unstructured data and in some embodiment the VCF file is transformed into relational data capable of being stored on and/or accessed from a relational database (e.g., the data is transformed to have column and/or row formatting in order to “fit” in a relational database). In one example, a transformation from unstructured data to relational data is performed using Greenplum HD. 
     In some embodiments, transformed relational data (e.g., transformed using Greenplum HD) is stored at  108  in a massively parallel processing (MPP) database which is designed or optimized to analyze very large datasets (e.g., on the order of terabytes or petabytes). One example of a MPP database is Greenplum Database. Greenplum Database (or other alternatives) may be attractive because such database applications offer a suite of database analysis tools designed for very large datasets. With the data is saved in relational form in Greenplum Database (or some other alternative), subsequent analysis is faster than having to transform the data into relational form each time before analysis, or trying to analyze the data in unstructured form. Another advantage to storing processing results in Greenplum Database (or some other alternative) is that such applications work seamlessly with other applications or tools which have additional or different features. For example, Greenplum Chorus, which enables sharing of data with collaborators or other select users, works seamlessly with Greenplum Database. 
     In some embodiments, a user associated with a request is charged by the system. For example, a company or university may make a processing system available to its employees or researchers but charges a user depending upon the processing resources allocated at  102  and/or the storage resources allocated at  106 . In some embodiments, a university gives each PI or lab a certain amount of credit and after the credit is used up the lab must pay for service. In some embodiments, a processing system is a third-party service available to any user who is willing to pay for the service. In such embodiments,  FIG. 1B  includes determining a cost (e.g., based on the amount of processing allocated at  102  and/or the amount of storage allocated at  106 ) and recording the cost. The cost may be saved in the account of the user who submitted the request at  100  so that the proper party is charged for service. 
     One feature of the process shown in  FIG. 1B  is that the total amount of storage associated with the distributed system is independent of the total amount of processing associated with the distributed system. For example, processing and storage on the distributed system are decoupled (an example of this is described in further detail below). For such systems, when the total amount of storage (as an example) is increased, it does not necessitate a corresponding increase in the total amount of processing. To a company or university which builds a system which performs the process of  FIG. 1B , this is an attractive feature because additional processing resources may not be required. For example, the total amount of processing in a system may be sufficient to service the level of requests coming in from all users, so it is not necessary to increase the total amount of processing. Distributed systems with decoupled storage and processing do not require the purchase and installation of additional processing which would be underutilized. 
     In contrast, some other systems have coupled storage and processing. For such systems, scaling up the total amount of storage (as an example) would also require a corresponding increase in the total amount of processing. This is undesirable because even before the increase in processing, the total amount of processing may be more than sufficient. For example, in some distributed systems where processing and storage are coupled and storage is the limiting factor, only about 5% of the total amount of processing is utilized, which is a significant underutilization. 
     Another feature of the process shown in  FIG. 1B  is that any amount of processing may be allocated at  102  and any amount of storage may be allocated at  106 . Using genome data as an example, processing and/or storage requirements may vary greatly depending upon the processes being performed and the data being processed. For example, performing genome typing on one set of genome data may have very different processing and/or storage requirements than a correlation analysis performed on another set of genome data. Being able to allocate an amount of storage independent of the amount of processing is desirable because a request can be serviced without allocating excess processing or storage resources. This is a desirable feature in a multi-user system. 
     In contrast, some other distributed systems were built by individual users for their personal use and were not intended to be shared with other users. As such, those other systems do not necessarily allocate storage resources and/or allocate processing resources. One difference between such other systems and the process shown in  FIG. 1B  is that those other systems may not be able support multiple users while keeping data confidential. For example, processing results stored at  108  in allocated storage resources are only available to the user who requested the service whereas processing results in other systems may be visible to any user. This may be undesirable, even if all users work for the same company or university. For example, some large research universities hire junior faculty in the same area of research (i.e., deliberately pitting colleagues at the same university against each other) with the understanding that the researcher who produces the best work will be offered tenure. In such a competitive environment, it is undesirable for one researcher to be able to see another researcher&#39;s work. 
     Another benefit of allocating processing at  102  and storage at  106  is evident in systems where the services provided by  FIG. 1B  have costs associated with them. For example, a university may make processing services available to its researchers but charges a researcher based on the amount of processing allocated at  102  and/or the amount of storage allocated at  106 . Being able to allocate varying amounts of storage resources at  102  and/or processing resources at  106  enables a system (if desired) to allocate resources in a manner that satisfies some cost constraint. For example, a less established researcher with less grant money may want to spend less money on processing services than a more established researcher with more grant money. 
     The following figures show a variety of user interfaces associated with various steps of  FIG. 1B . These figures are merely exemplary and are not intended to be limiting. For example, although certain user interface controls are shown (e.g., pull down menus, radio buttons, etc.), other user interface controls may be used. Also, the exemplary user interfaces are associated with a university which makes a processing system available for use by its researchers and therefore some information about potential users is known a priori (e.g., schools, departments, mailing addresses, and/or names of potential users). Some other scenarios (e.g., where a third-party processing system is made available to any user from any university or company) may require other information to be obtained and/or presented (e.g., a mailing address to send invoices to). Also, although genome data and processing of such data is used in the examples below, this is not intended to be limiting and the systems and techniques described herein may be used with any large dataset. For example, weather or climate models often comprise large datasets. 
       FIG. 2  is a diagram showing an embodiment of a user interface for setting up an account. In the example shown, a processing system is provided by a research university for use by its researchers. Researchers set up accounts via user interface  200 . Once an account is set up, a researcher (or someone in their lab) is able to access the services of the processing system and be billed accordingly. 
     Using pull down menu  202 , the user&#39;s school is identified as the school of medicine. In this example, once pull down menu  202  is specified, pull down menu  204  is populated with the departments and/or institutes for the specified school. For example, the school of medicine may have one set of departments/institutes and whereas the school of engineering has another set, and pull down menu  204  is populated accordingly depending upon the school specified in pull down menu  202 . 
     The user&#39;s department is identified as the genetics department in pull down menu  204 . The lab or principal investigator (PI) with which the user is associated with is specified in fillable field  206  to be the Mary Johnson lab. As used herein, the terms lab and PI are used interchangeably since for the purposes of explanation it is assumed a lab includes only one principal investigator and that it is permissible to share confidential information amongst people in the same lab. In fillable field  208 , the username is set to Mary.Johnson.Lab and the password is specified via fillable field  210 . 
     User interface  200  offers the user the option of specifying a budget in fillable field  210  and the period of time (e.g., annually, semi-annually, or quarterly) over which the specified budget applies. In some embodiments, a specified budget is a soft budget, which is tracked for the convenience of the user but is not necessarily used to disable services if the budget is exceeded. In some embodiments, budget field  212  is pre-populated depending upon the PI. For example, a school or department may allocate or credit a certain amount of money to each PI, and budget field  212  indicates the amount of money credited to the PI specified (e.g., via user interface controls  202 ,  204 , and/or  206 ). In some embodiments, a budget is a hard limit, and access to the processing system is denied if the budget is zero or negative, or if the cost of a processing run would exceed the available budget. In various embodiments, services must be pre-paid (e.g., sending in money for credit on the system before the services are used) or services may be paid for after the fact. 
     User interface  200  also permits a user to control whether permission is first obtained before a processing run is performed. If permission is required, the contact person (where the contact name is specified in fillable field  218  and the contact email address is specified in fillable field  220 ) is contacted for approval before a processing run is performed. 
       FIG. 3  is a diagram showing an embodiment of a user interface for specifying a new processing run. In the example shown, a user wants to perform a genome related process and selects tab  302  (i.e., start a new process) in user interface  300  in order to set up the desired processing run. The user&#39;s data (e.g., collected by the researcher or user) is specified in field  304  (e.g., after going through a file system hierarchy) and a desired toolkit (i.e., genome analysis toolkit) is specified in pull down menu  306 . The type of process (in this example, genome typing) is specified in pull down menu  308 . 
     Pull down menu  310  is an option in which the user can specify a desired level of performance. The user in this example has selected high performance, but other performance options (e.g., moderate performance, budget/low performance, etc.) may be selected. Costs scale with performance in this example, so better performance will cost more money. 
     In this example, proprietary data may be specified in optional pull down menu  312 . In this genomics related example, proprietary data may comprise genome sequences owned by a third party to whom royalties or payments are made if the proprietary data is used in the processing. In this particular example, pull down menu  312  is set to don&#39;t know/suggest. In some embodiments, when this option is set, the data specified in field  304 , the toolkit specified in pull down menu  306  and/or the type of process specified in pull down menu  308  is used to determine relevant proprietary data (if any). For example, the data specified in field  304  may relate to certain locations in a genome or certain markers and appropriate proprietary data (e.g., which covers those locations or markers) is selected by pre-processing the data specified in  304 . 
     Optional cost limit fillable field  314  is used to specify (if desired) a cap on the processing run being specified. If specified, the processing run which is quoted will try to stay within the specified cost limit. 
     Optional notes field  316  is used to record a user&#39;s notes. In this particular example, the user&#39;s notes indicate that the data (e.g., specified in field  304 ) comprises control samples. A note may be saved with and/or associated with its corresponding processing run, for example so that even if more than one person is working with the data, a person who did not initiate the processing run knows what the processing run relates to and/or has sufficient context to interpret or make sense of the processing results. 
     Once the user has set the fields in tab  302  to the desired values, the user presses quote button  318  and receives a proposed processing run. The following figure shows an example of a returned proposed processing run. 
       FIG. 4  is a diagram showing an embodiment of a user interface which presents a proposed processing run. Based on the information specified in  FIG. 3 , user interface  400  proposes the shown processing run. In field  402 , the proposed processing run includes two processors at a cost of $165. Although this particular example only varies the number of proposed processors, in some embodiments, different processing resources have different performance capabilities and field  402  in some embodiments includes both quantity and quality of processors (e.g., 1 high end processor and 1 mid-range processor). 
     The toolkit for the proposed run is shown in field  404  and the proprietary data is shown in field  406 . In this particular example, proprietary data field  312  in  FIG. 3  was set to don&#39;t know/suggest and the processing system (e.g., based on its analysis) is proposing to use the Genetix Inc. database at a cost of $25. The cost estimate (e.g., based on the number of processors and the proprietary data) is shown in field  408  and the runtime estimate is shown in field  410 . In some embodiments, a cost estimate also takes into consideration other allocated resources, such as allocated ports (e.g., some processing runs may be I/O intensive whereas other are not) and/or allocated storage. 
     If the proposed processing run is acceptable, the user presses submit button  414 . Depending upon the approval settings specified via radio buttons  216  in  FIG. 2 , the processing run is either initiated or is sent to the contact (e.g., specified in fields  218  and  220  in  FIG. 2 ) for approval. If the user wants to modify the proposed processing run (e.g., to reduce cost or to modify the type of process to perform), go back button  412  is pressed. 
       FIG. 5  is a diagram showing an embodiment of a user interface in which a previous processing run is viewed. In the example shown, tab  502  (i.e., previous results) is selected in user interface  500 . In some embodiments, all previous processing runs are saved automatically. In some embodiments, only selected processing runs are saved. In some embodiments, there is a cost associated with saving processing results and a quote is presented (e.g., similar to  FIG. 4 ) before a processing run is saved (not shown). 
     In tab  502 , two previous processing runs are shown in frames  504  and  506 . For each processing run presented, the data (e.g., C:\\Desktop\Jan2013Data or C:\\Desktop\Jan2013Data), the toolkit, type of process, proprietary data (if any), run date and time, and notes (if any) are displayed. To view a selected processing run, the corresponding view button (e.g., button  508  or  510 ) is selected. 
       FIG. 6  is a diagram showing an embodiment of a user interface which shows account information. In the example shown, tab  602  (i.e., account information) is selected in user interface  600 . Account activity section  610  shows credits, debits, and remaining budget for various transactions associated with the account. Tab  602  continues the example of  FIG. 2  where a user specified a budget of $10,000 in field  212 . The remaining budget column in account activity section  610  begins with a budget of $10,000 and deducts the costs of processing runs from that $10,000 budget. For example, at  612  the Feb. 6, 2013 processing run cost $205 and the remaining budget is updated to $10,000−$205=$9,795. At  614 , the Feb. 15, 2013 payment of $205 does not cause the remaining budget to change (at least in this example). The remaining budget is updated again at  616  for the Mar. 1, 2013 processing run: $9,795−$190=$9,605. 
     Account settings are shown in section  620  and at least some of the information shown in that section is set via user interface  200  in  FIG. 2 . For example, the school, department/institute, and lab/principal investigator displayed in fields  622 ,  624 , and  626 , respectively are set via user interface  200  in  FIG. 2 . The budget information (displayed in field  628  and radio buttons  630 ) and permission information (displayed in radio buttons  632  and fields  634  and  636 ) are also specified via user interface  200  in  FIG. 2 . If desired, account settings shown in section  620  may be changed using the user interface shown. 
       FIG. 7  is a diagram showing an embodiment of a decoupled processing and storage associated with a Hadoop system. This example shows one embodiment of a system which has a total amount of storage which is independent of a total amount of processing associated. In this example, Hadoop processing  700  is implemented using Greenplum HD and Hadoop storage  702  is implemented using Isilon. 
     In this example, Hadoop processing  700  and Hadoop storage  702  are virtualized and so a virtual infrastructure manager, such as Serengeti, has the ability to separately scale out the total amount of virtual processing or the total amount of virtual storage. For example, if users are increasing their utilization of a processing system (e.g., because more new users are signing up or because each user is submitting more processing requests) but the total amount of storage is sufficient, Hadoop processing  700  can be scaled out using Serengeti without being forced to scale out Hadoop storage  702 . Similarly, if more storage is desired but the total amount of processing resources is sufficient, Hadoop storage  702  may be scaled out without having to scale out Hadoop processing  700  unnecessarily. 
     One advantage to using Isilon (or some other alternative with similar capabilities) is that Hadoop storage  702  can be scaled out to very large sizes. For applications with very large datasets (e.g., genome data), a single file system in Hadoop storage  702  can be scaled out to 16 petabytes. 
     Another advantage to using Isilon (or some similar alternative) is that Isilon supports multiple protocols, such as network file system (NFS), common Internet file system (CIFS), and Hadoop Distributed File System (HDFS) on a single storage platform. This reduces extract, transfer, and load (ETL) experienced by the system. In contrast, some other systems which do not use Isilon (or some similar alternative) have to extract, transfer, and load the data onto another storage platform if a desired protocol is not supported. 
       FIG. 8A  is a diagram showing an embodiment of a processing system which provides processing services to one or more users. In the example shown,  FIG. 8A  is one embodiment of a system which performs the process shown in  FIG. 1B . Virtual datacenter manager  800  provisions and manages virtual datacenters and in one example is implemented using VMware vCloud Director. Virtual datacenter manager  800  enables multiple users (such as the researchers Mary Johnson and John Lee) to be serviced by the same system while keeping information associated with each user secure. For example, instance  810   a  is associated with the Mary Johnson lab or account and instance  810   b  is associated with the John Lee lab or account. 
     Each of instances  810   a  and  810   b  includes users and policies ( 812   a  and  812   b ), a virtual data center ( 814   a  and  814   b ), and a catalog ( 816   a  and  816   b ). Users and policies  812   a  and  812   b  include (at least) user and policy information associated with the Mary Johnson lab and John Lee lab, respectively. For example, it may have policies about whether permission is required to perform a processing run, the amount of a remaining budget, etc. In some embodiments, users and policies  812   a  and  812   b  are identical and include information for all users. 
     Virtual data centers  814   a  and  814   b  are secure and unique data centers for the dedicated use of the associated user (i.e., Mary Johnson and John Lee, respectively). A user associated with instance  810   a  cannot (for example) access the results of processing run on virtual datacenter  814   b  and vice versa. Any of a variety of tools or applications may be run on virtual datacenter  814   a  or  814   b . For example, genome analysis toolkit  850 , Greenplum HD  852 , and/or Serengeti  854  may be run on virtual datacenter  814   a  or  814   b  as desired. 
     Catalogs  816   a  and  816   b  are catalogs of services that are available to the users or instances. 
     Virtual datacenters  814   a  and  814   b  are implemented using either gold resources  822   a , silver resources  822   b , bronze resources  822   c  or a combination of resources types from the provider virtual data center  820 . Provider virtual data center  820  assembles groups of resources  822   a - 822   c  and allocates groups of resources for virtual datacenter  814   a  or  814   b  as appropriate for that particular user. For example, the quantity and/or quality of resources in gold resources  822   a  is better than silver resources  822   b , which in turn is better than bronze resources  822   c . For example, gold resources  822   a  may have more processing resources and/or storage resources than silver resources  822   b . Accordingly, costs (in embodiments where costs are charged to the user) vary accordingly. In other words, gold resources  822   a  cost more than silver resources  822   b  and silver resources  822   b  costs more than bronze resources  822   c . To meet the cost constraint of a given user (e.g., the Mary Johnson Lab may have a bigger budget to work with than the John Lee Lab), a group of resources is allocated based (at least in some embodiments) on a cost cap. Provider virtual data center  820  provides resource abstraction services and vCloud Director performs resource allocation steps outlined in  102  and  106  in  FIG. 1B . 
     Each of the groups of resources  822   a - 822   c  includes some Hadoop processing resources ( 832   a ), Hadoop storage resources ( 832   b ), and Hadoop port resources ( 832   c ) from virtual infrastructure manager  830 . In this example, virtual infrastructure manager  830  is implemented using VMware vCenter Server. If desired, the total amount of Hadoop processing  832   a , Hadoop storage  832   b , and/or Hadoop ports  832   c  may be scaled out (e.g., independently) using (for example) Serengeti. 
     In some embodiments, processing system  800  is built on or using converged infrastructure, such as Vblock. Converged Infrastructure provides best of breed Compute (Cisco UCS), Storage (EMC VNX and VMAX), Network infrastructure (Cisco Nexus) and virtualization software (VMware vCloud Suite) in a factory built, pre-validate and fully integrated infrastructure stack. In today&#39;s cloud environments where scaling up and/or out the underlying resources quickly is key to success, vBlock allows service providers in this space such as university or research facilities to quickly add resource to their service offering. vBlock customers benefit from optimized performance, joint product roadmap across all components, upgrade paths across the entire infrastructure stack and a single support organization. 
       FIG. 8B  is a diagram showing a second embodiment of a processing system which provides processing services to one or more users. In the example shown, there are four systems associated with four different university faculty members. Each faculty member has a system which includes a Hadoop compute cluster with various numbers of Hadoop compute only nodes: university faculty member A ( 880 ) has 6 Hadoop compute only nodes, university faculty member B ( 882 ) has 4 Hadoop compute only nodes, university faculty member C ( 884 ) has 4 Hadoop compute only nodes, and university faculty member D ( 886 ) has 4 Hadoop compute only nodes. Some faculty members also have systems which include a genome toolkit cluster while others do not. For example, university faculty members A and B ( 880  and  882 ) each have a genome tool kit cluster (with 4 genome analysis toolkits each), whereas university faculty members C and D ( 884  and  886 ) do not have a genome toolkit cluster. 
     A benefit to the embodiments of the processing system described herein is that selected information may be shared with collaborators without exposing confidential information outside of the scope of the collaboration. For example, within a given lab there are typically multiple research projects being worked on at the same time. One set of graduate students and/or postdoctoral candidates may be working on one research project while another set is working on another project. If the first group is collaborating with another lab, that group will want to share project information with the other lab. However, it would not be desirable to share research that the second set of graduate students and/or postdoctoral candidates is working on. The following figures show some example user interfaces in which sharing is enabled, but without the risk of exposing confidential information. 
       FIG. 9  is a diagram showing an embodiment of a user interface in which sharing is enabled when a new processing run is specified. In the example shown, tab  902  in user interface  900  is similar to tab  302  in user interface  300  in  FIG. 3 , except tab  902  has some additional user interface controls. In tab  902 , a user specifies via radio buttons  904  whether the processing run is to be shared. If so, the collaborator username (e.g., Mark.Jones.Lab) is obtained via fillable field  906 . 
     In this particular example, the collaborator username specified in fillable field  906  is associated with the processing system which provides user interface  900 , making it a quick and unique way of identifying a collaborator. In some embodiments, additional information associated with a collaborator is determined based on the username specified in fillable field  906  and is displayed (e.g., in real time) in tab  902  (not shown). For example, there may be two researchers named Mark Jones and displaying other information (e.g., company, university, school, department, e-mail address, etc.) associated with the specified collaborator username may be useful to ensure that the proper collaborator is selected by the user. 
       FIG. 10  is a diagram showing an embodiment of a user interface in which sharing is enabled after processing has been performed. In the example shown, tab  1002  in user interface  1000  is similar to tab  502  in user interface  500  in  FIG. 5 , except tab  1002  has some additional user interface controls. If a user wants to share data from processing run  1004  with a collaborator, share button  1008  is pressed which causes a window to be displayed in which the collaborator is identified (not shown). Similarly, if a user wants to share data from processing run  1006 , share button  1010  is pressed. Any number of processing runs may be shared with any number of collaborators and a collaborator may be specified in any number of ways. 
     The user interfaces shown in  FIGS. 9 and 10  are not necessarily mutually exclusive. In some embodiments, even if a user does not enable sharing when a processing run is being specified (e.g., as in  FIG. 9 ), the user still has the option of sharing data from that processing run after processing has completed (e.g., as in  FIG. 10 ). 
     In the backend, sharing may be performed using any appropriate application or tool. In one example, Greenplum Chorus is used to share data between collaborators by offering users the ability share selected data with specified collaborators. This enables a lab or principal investigator to share data related to a collaboration with a collaborator, without exposing data unrelated to the collaboration which would be undesirable because of the risk of being “scooped.” A lab or principal investigator may be working on multiple projects simultaneously, and a collaborator on one project may be a competitor on another project. Referring back to  FIGS. 7 and 8  using Greenplum Chorus with the exemplary systems shown therein may be attractive because Greenplum Chorus is designed to operate seamlessly with the system components shown. For other systems, there may be some other sharing application or tool which works better with those embodiments. 
     Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.