Patent Publication Number: US-9846635-B2

Title: Making production data available for testing in a non-production environment

Description:
FIELD OF THE TECHNOLOGY DISCLOSED 
     A metric &amp; event infrastructure platform serves an intelligent, integrated suite of applications that monitor and provide critical data for systems and applications. 
     BACKGROUND 
     The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also correspond to implementations of the claimed inventions. 
     In today&#39;s world, we are dealing with huge data volumes, popularly referred to as “Big Data”. Web applications that serve and manage millions of Internet users, such as FACEBOOK™, INSTAGRAM™, TWITTER™, banking websites, or even online retail shops, such as AMAZON.COM™ or EBAY™ are faced with the challenge of ingesting high volumes of data as fast as possible so that the end users can be provided with a real-time experience. 
     Another major contributor to Big Data is a concept and paradigm called “Internet of Things” (IoT). IoT is about a pervasive presence in the environment of a variety of things/objects that through wireless and wired connections are able to interact with each other and cooperate with other things/objects to create new applications/services. These applications/services are in areas likes smart cities (regions), smart car and mobility, smart home and assisted living, smart industries, public safety, energy and environmental protection, agriculture and tourism. Global data centers host thousands of enterprise companies, offering performance and security that enable organizations to serve and manage millions of Internet users of the IoT. 
     Currently, there is a need to collect and visualize metrics for disparate internet-connected devices. For example, global data centers that process the big data of an IoT ecosystem need to be monitored for reliability by their site reliability engineers. It has become imperative to increase the ability to customize the views needed by different groups of users for monitoring operational status of computing devices and systems. Production support teams for global data centers monitor the infrastructure of global data centers 24 hours a day, seven days a week—monitoring metrics for thousands of hardware systems, including servers and IoT devices. In turn, thousands of visualizations need to be designed and generated to display those metrics, with different teams across an enterprise developing and deploying visualizations. 
     Production support teams often use a real-time graphing system for monitoring and graphing the performance of computer systems, to monitor metrics and graph system performance. Historically, those who develop new dashboard visualizations have faced the challenges of developing and testing the new dashboard before deploying it in a production development. 
     When a designer exports a newly developed dashboard to a non-production server for testing, no production data is available in the development environment. That is, the developer is “flying blind” in the sense that they miss the production metrics—they do not know whether the dashboard will function as desired when loaded into production and displaying production data. Also, when a new dashboard is submitted to the production server environment, the new dashboard overwrites the existing dashboard on the server. Errors—often caused by the inability to test the design with production data prior to deployment—can result in down time during which no dashboard is available to production support teams who need to monitor the metrics, so no system covered by the affected dashboard(s) can be monitored. 
     Therefore, an opportunity arises to provide systems and methods for developing and testing dashboards with production data in non-production environments. Efficient monitoring of global data centers, increased system reliability and uptime percentages, and improved user experience may result. 
     The disclosed technology relates to making production data available for testing in a non-production environment. 
     SUMMARY 
     A simplified summary is provided herein to help enable a basic or general understanding of various aspects of exemplary, non-limiting implementations that follow in the more detailed description and the accompanying drawings. This summary is not intended, however, as an extensive or exhaustive overview. Instead, the sole purpose of this summary is to present some concepts related to some exemplary non-limiting implementations in a simplified form as a prelude to the more detailed description of the various implementations that follow. 
     The disclosed technology includes a development server with application programming interfaces (APIs) interposed between a development app and a production server, providing development access to production data before deployment into the production environment. 
     A disclosed development server includes a processor coupled in communication with a persistent development code store that holds development code, the processor coupled in communication with a production server. An app server running on the processor serves a development app that is a modified version of a production app, in response to a request from a developer device; and exposes development APIs that mirror production APIs of the production server, and relay of messages between the development APIs and the production APIs. The development server includes a code manager running on the processor that accesses the development code in the persistent development code store, in response to a request from a development app. The development server also includes a deployment app that includes a user interface for selection among development code definitions of dashboards. The development server additionally replicates access to production server APIs relied on to retrieve and process time series data from a data store, the replication including access to development server APIs that mirror relative addressing of production APIs. Users can select among dashboards defined using JSON, and can select to deploy the selected dashboards to the production server. 
     A disclosed method of making production data available for development testing includes providing a development server code with app server code that, when run on a processor, serves a development app code that is a modified version of a production app, in response to a request from a developer device; and exposes development APIs that mirror relative addressing of production APIs and relay selected messages between the development app and the production server. The development server code also includes a code manager that accesses the development code in a persistent development code store, in response to a request from the developer app or browser. The method includes receiving at least one data assembly and rendering request from a development server that includes or references a definition of data assembly and rendering requirements; processing each data assembly and rendering request against a data store; and returning results of the processing via the development server. 
     Other aspects and advantages of the technology disclosed can be seen on review of the drawings, the detailed description and the claims, which follow. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The included drawings are for illustrative purposes and serve only to provide examples of possible structures and process operations for one or more implementations of this disclosure. These drawings in no way limit any changes in form and detail that may be made by one skilled in the art without departing from the spirit and scope of this disclosure. A more complete understanding of the subject matter may be derived by referring to the detailed description and claims when considered in conjunction with the following figures, wherein like reference numbers refer to similar elements throughout the figures. 
         FIG. 1  illustrates an example environment usable for making production data available for testing in a non-production environment. 
         FIG. 2  illustrates a global data center usable in the example environment shown in  FIG. 1 , according to one implementation of the technology disclosed. 
         FIG. 3  shows an example for developing dashboards when production data is not available for design testing prior to the release of a dashboard to a production environment. 
         FIG. 4  shows an example implementation of a process for developing dashboards when production data is available for design testing prior to the release to a production environment. 
         FIG. 5  shows example data infographics displayed in response to a request from a developer device. 
         FIG. 6  shows an example publish dialog for deploying selected development code to the production server. 
         FIG. 7  shows an example published data infographic—a deployed selected development code that represents a dashboard. 
         FIG. 8  shows a snippet of a JSON file that represents the dashboard shown in  FIG. 7 . 
         FIG. 9  shows an example user interface for selection of deployment of a development code definition to the production server. 
         FIG. 10  shows an example workflow for a method of making production data available from a production server for development testing. 
         FIG. 11  is a block diagram of an exemplary multi-tenant system suitable for integration with the environment of  FIG. 1  in accordance with one or more implementations of the technology disclosed. 
     
    
    
     DETAILED DESCRIPTION 
     Introduction 
     The following detailed description is made with reference to the figures. Sample implementations are described to illustrate the technology disclosed, not to limit its scope, which is defined by the claims. Those of ordinary skill in the art will recognize a variety of equivalent variations on the description that follows. 
     Production support teams often use real-time graphing systems for monitoring and graphing the performance of computer systems to monitor metrics and graph system performance. Historically, those who develop new dashboard visualizations have faced the challenges of developing and testing the new dashboards before deploying them in a production development. When a designer exports a newly developed dashboard to a non-production server for testing, no production data is available in the development environment, so the designer and reviewers do not know whether the dashboard will function as desired when loaded into production and displaying production data. Also, when a new dashboard is submitted to a production server environment, the new dashboard often overwrites the existing dashboard on the server. Errors—often caused by the inability to test the design with production data prior to deployment—can cause down time when no dashboard is available to production support teams who need to monitor metrics of production systems. 
     If, alternatively, new dashboards are uploaded using different dashboard names—to avoid overwriting the existing dashboard—then extra dashboards accumulate and appear in search results, and potentially clog folders and confuse users of the system. After testing, the new dashboards will need to be renamed and reviewed, approved and uploaded again—a process which adds multiple steps at which errors can be introduced. In some use cases, down time is not an acceptable option for dashboard visualizations that display critical metrics. When monitoring dashboards become unavailable, service engineers tasked with maintaining continuous reliable service for enterprise systems are hampered in their ability to complete their goals. Other users across enterprises and the IoT also need dashboards tested with production data before deployment. 
     A disclosed method of making production data available for development testing includes providing a development server code with app server code that, when run on a processor, serves a development app code that is a modified version of a production app, in response to a request from a developer device; and exposes development APIs that mirror relative addressing of production APIs and relay selected messages between the development app and the production server. The development server code also includes a code manager that accesses the development code in a persistent development code store, in response to a request from the developer app or browser. The method includes receiving at least one data assembly and rendering request from a development server that includes or references a definition of data assembly and rendering requirements; processing each data assembly and rendering request against a data store; and returning results of the processing via the development server. 
     The disclosed method of making production data available for testing in a non-production environment includes receiving an app request from a developer device at a development server, and serving a development app that is a modified version of a production app. The development app replicates access to a production server access tier which is relied on to retrieve and process time series data from a production server data tier, including relative addressing of production APIs. The development app processes user selected JSON dashboards; and deploys selected, approved JSON dashboards to the production server. The disclosed method also includes receiving at a development API a preview request to preview one or more data assembly, transformation and rendering requests. The requests and a definition of data assembly and rendering requirements are forwarded to the production server at production APIs that mirror the development APIs that received the development preview request. The development app receives, from the production server, and returns, to the client, results from the processing—in the form of one or more graphics or as data usable by the client to render a dashboard of graphs. 
     Environment for Dashboard Development with Production Data 
       FIG. 1  illustrates one example implementation for making production data available for testing in a non-production environment  100 , which includes real time graphing engine  102 , production data store  104 , IoT device  108  and global data center  128 . Production data store  104  includes monitoring data for production data centers—including, but not limited to performance metrics for servers and other IoT devices. Environment  100  additionally includes monitoring data store  142  and network  145 . 
     Real time graphing engine  102  comprises a production server that includes WebApp engine  112  for rendering graphs, storage manager  122  and cache memory  132 . When the user submits a graphing request for a metric, as specified via a dashboard, WebApp engine  112  scans files in monitoring data store  142  identifying data whose time range is specified in the dashboard query, and returns a graph of the data that matches the request. In one implementation, a real time graphing engine can be a Graphite production server. 
     Storage manager  122  accepts incoming metric data over various protocols from production systems under monitoring, including but not limited to IoT device  108  and global data center  128 , and writes the metric values and their related timestamps to disk as efficiently as possible. Metric values are stored in cache memory  132  as they are received, and are flushed to monitoring data store  142  in the form of a time stamp and metric value (e.g. CPU for this server=55 at 13:02:15). Storage manager  122  locates the appropriate file in monitoring data store  142  and appends the received metric data to the file. 
     Environment  100  also includes user computing device  158 —with graphical user interface client  168 , dashboard creation engine  178 , dashboard rendering engine  188  and proxy WebApp engine  198 —running in user device  164  or mobile application  165 . Graphical user interface client  168  receives input from users and displays data on the screen of a user&#39;s computing device  158 . Dashboard creation engine  178  runs on user computing device  158  and renders development code definitions of dashboards on demand. The development code definitions of dashboards can be expressed using JSON, XML or another data format. In one example, a developer can use a browser or developer app to select dashboard creation engine  178 , which a developer can use for designing a new dashboard, modifying an existing dashboard and selecting a dashboard to load and test. For some use cases, dashboard templates can be made available for developers. 
     Proxy WebApp engine  198  includes a development server with a development app which is a modified version of a production app. The development app replicates access to the production real time graphing engine  102  access tier relied on to retrieve and process time series data from the production server data tier, including relative addressing of production APIs. In one implementation, the development app is embedded JavaScript which runs on a dashboard developer&#39;s local computer system—user computing device  158 . 
     Dashboard rendering engine  188  implements a rendering of a set of charts received from production real time graphing engine  102 —and displays the rendering on the GUI screen of user computing device  158 —on laptop or workstation  164  or mobile application  165 . For some implementations, data received from production real time graphing engine  102  may be used to render graphics, which can be interactive in some use cases. 
     A dashboard developer or reviewer can preview a dashboard with production data and, if it represents the desired data well, then they can select the publish dashboard button to load and save the dashboard to real time graphing engine  102 —the production server. In one example implementation, the toolbar of the developer app or browser includes a publish dashboard button to implement functionality for deploying tested, approved dashboards from user computing device  158 . 
     Production data store  104  and monitoring data store  142  can be implemented using a general-purpose distributed memory caching system. In some implementations, data structures can store information from one or more tenants into tables of a common database image to form an on-demand database service (ODDS), which can be implemented in many ways, such as a multi-tenant database system (MTDS). A database image can include one or more database objects. In other implementations, the databases can be relational database management systems (RDBMSs), object oriented database management systems (OODBMSs), distributed file systems (DFS), no-schema database, or any other data storing systems or computing devices. Analytical, read-only databases can implement response times of under two seconds when searching over twenty million records and compiling aggregate statistics from selected records. 
     In some implementations, user computing device  164  can be a personal computer, a laptop computer, tablet computer, smartphone or other mobile computing device, personal digital assistant (PDA), digital image capture devices, and the like. In some implementations, user mobile device  165  can be a tablet computer, smartphone or other mobile computing device, personal digital assistant (PDA), digital image capture devices, and the like. 
     User computing device  158  can take one of a number of forms, running in a browser or as an application, including user interfaces, dashboard interfaces, engagement consoles, and other interfaces, such as mobile interfaces, tablet interfaces, summary interfaces, or wearable interfaces. In some implementations, it can be hosted on a web-based or cloud-based server in an on premise environment. In one implementation, graphical user interface client  168 , dashboard creation engine  178 , dashboard rendering engine  188  and proxy WebApp engine  198  can be accessed from a browser running on a computing device. The browser can be CHROME™, INTERNET EXPLORER™, FIREFOX™, SAFARI™, OPERA™, and the like. 
     Other implementations for dashboard development with production data in a non-production environment  100  may not have the same elements or components as those listed above and/or may have other/different elements or components instead of, or in addition to, those listed above, such as a web server and template database. The different elements or components can be combined into single software modules and multiple software modules can run on the same hardware. 
     In one implementation, dashboard creation engine  178  includes a feature for displaying one or more graphs at a time, with the set of graphs showing the same time range. A dashboard developer can select a metric series to show on a graph in the dashboard. One example UI for the dashboard creation engine  178  includes a choice between a completer, for selecting a metric series, or a browser tree. The developer can import existing graphs into a dashboard under development, either from saved graphs or from a library of graphs. In some cases, separate graphs that each show a single metric can be combined to show more than one metric on a single graph. The dashboard developer can also order graphs, either by editing the JSON file directly or using the dashboard creation UI. 
     Dashboard creation engine  178  includes a feature for setting the time range as relative or absolute. Relative times are more commonly used. The current time range can be displayed in the center of the menu bar, as described infra. The developer typically sets dashboards for manual refresh. Alternatively, the dashboard can be set to auto-refresh, which defaults to 60 seconds, for some use cases. 
     Transforms of metric elements can be customized by combining a group of metric series—using sum, average, product, minimum or maximum functionality. Transformations of data include functions that transform the values in a metric series, against either the y axis or (less commonly) the x-axis. Functions for transformations can include, but are not limited to, scale, scale to seconds, offset, derivative, integral, time-shift, and log. Additionally functions can be used to calculate a new metric series based on an existing metric series, including moving average, percentage, Holt-Winters forecast, ratio and difference of two metrics. Further, filter functions that filter metric series from a group can include highest current value, current value above the limit, most deviant, and remove below percentile. Additionally, line colors, widths and styles and graph titles can be customized, as can graph sizes 
     In one use case, a sequence of events can include a developer selecting the JSON development code definition for a specific dashboard “XYZ” from their browser or local developer app, which calls proxy WebApp engine  198  to request the dashboard via REST API call:/dashboard/load/XYZ. The local server, proxy WebApp engine  198  responds with XYZ JSON from the local file system. Dashboard rendering engine  188  processes the XYZ JSON and finds that it specifies N graphs, and parses the JSON definition. 
     Dashboard creation engine  178  calls REST API as listed below. 
     /render/&lt;graph 1 definition&gt; 
     /render/&lt;graph 2 definition&gt; . . . 
     /render&lt;graph N definition&gt; 
     For testing the dashboard, the requests are directed to proxy WebApp engine  198 , which then proxies the render API calls, listed below, to production real time graphing engine  102 . 
     /render/&lt;graph 1 definition&gt; 
     /render/&lt;graph 2 definition&gt; . . . 
     /render&lt;graph N definition&gt; 
     Continuing with the example use case, production real time graphing engine  102  responds back with fully rendered PNG images of the rendered graphs to proxy WebApp engine  198 , which calls the REST API to send the PNG images of the rendered graph to dashboard creation engine  178  for display via graphical user interface client  168 . 
     The result is a dashboard visualization with graphs described in JSON on the local developer&#39;s browser or developer app. The developer can repeat the steps described until they are satisfied with the results, and can then save the dashboard. After review and approval of the look and feel and correctness, the JSON dashboard can be published to production real time graphing engine  102 . Proxy WebApp engine  198  would call /dashboard/save/&lt;JSON&gt;. 
     For some use cases, when the development app connects to the production server, for acquiring production data, proxy WebApp engine  198  requests authentication credentials for a dashboard developer, and stores authentication credentials received from production real time graphing engine  102 . 
     Systems under monitoring, whose metrics need to be displayed on visualization dashboards, can include global data center  128 , shown in  FIG. 2 , which includes super-pods  215 ,  225  and  235 , each of which includes similar components. Super-pod  215  contains POD  214 , POD  224  and POD  234 , and each POD includes a logical grouping of hosts. In one example, the monitoring of a global data center infrastructure tracks one hundred pods using the disclosed disparate monitoring technology. POD  214  includes app server  218 , adjacent services  219  including file system and search services, and database  220 . In one implementation, a company allocates a customer to a single POD where their data resides. In common use, multiple customers rely on a single POD—a self-contained unit that contains everything required to run an instantiation of a company&#39;s services. Maintenance engineers can monitor many different metrics to ensure system reliability for multiple enterprise customers, using graphs of metric data over time. 
       FIG. 3  shows prior art for developing dashboards before the implementation of the disclosed technology for viewing metrics using real time graphs: a dashboard developer device  312  would transmit a developed dashboard  322  for display on a non-production server  316 —with no production data available for testing the dashboard design. The dashboard developer device  312  would transmit a review request  332  to reviewer device  313 , for review  333  on nonproduction server  316 . After a successful review, dashboard developer device  312  would export the reviewed dashboard to file system  317  and release engineer device  314  would obtain a file displaying test data  354 . Release engineer device  314  would deploy  364  the dashboard to production real time graphing server  318 , and reviewer device  313  would enable review of the dashboard with production data  374 . The deployed dashboard would overwrite the existing dashboard on production real time graphing server  318 , so any errors in the newly developed dashboard would cause the service engineers who rely on the graphs of system metrics to assure system reliability would have no access to the needed visualizations of the metrics until the dashboard with errors was corrected, reviewed, exported, deployed and re-reviewed. Instead of viewing the desired graphs, errors can cause display of red Xs in the location on the display where a graph is desired, in one implementation. 
       FIG. 4  shows an example of the process for the disclosed method for developing and reviewing dashboards, using proxy web app  425  to access production data from production real time graphing server  418 —for use by the proxy web app server  415  in testing a new or changed dashboard design. Dashboard developer device  412  submits credentials  422 . Proxy web app server  415  sends an authentication request  425  to production real time graphing server  418  and receives and stores authentication credentials  435  from production real time graphing server  418  for the user of dashboard developer device  412 . Dashboard developer device  412  handles submission of a dashboard design  432  to proxy web app server  415 , which renders the request by proxy  445  to production real time graphing server  418 . Results returned  457  from production real time graphing server  418  include production data for display on the dashboard under review. 
     Dashboard developer device  412  transmits a dashboard review request  442  to reviewer device  413 , which transmits the review request  443  to proxy web app server  415  which sends review request by proxy  455  to production real time graphing server  418 . Results returned  457  can be viewed by a reviewer and approved, using proxy web app server  415 . 
     Reviewer device  413  transmits approval by reviewer  453  for the dashboard, and proxy web server  415  marks approved  465  for the new or changed dashboard. Dashboard developer device  412  can be used to transmit deploy request  462  for the approved dashboard, and proxy web app server  415  transmits publish dashboard  475  to production real time graphing server  418 . 
       FIG. 5  shows an example preview of part of a dashboard  500  with four graphs. The graph in the upper left quadrant shows open file descriptor count  514 , with different colors shown for different days. The upper right quadrant shows non heap memory usage  518 , with different colors for the display of each of five different days, as shown in the legend  558 . The dashboard can be published using publish dashboard  516  after the reviewer approves the design.  FIG. 6  shows an example confirmation dialog  645  to confirm that the dashboard will be published.  FIG. 7  shows the same dashboard, after it has been published: upper left quadrant shows open file descriptor count  714 , with different colors shown for different days and upper right quadrant shows non heap memory usage  718 , with different colors for the display of each of five different days, as shown in the legend  758 . The time range is specified as showing the past two hours  716 . For implementations of the disclosed technology, the reviewer can have a high level of confidence that what they approve is correct, and that the deployed dashboard displayed in the production server is visually the same as the infographic approved by the reviewer, removing the source of many potential errors, and saving development time and energy. 
       FIG. 8  shows a snippet of the JSON representation for the dashboard infographic, shown in  FIG. 5 ,  FIG. 6  and  FIG. 7 , which includes open file descriptor count  844  for the dates 11-1 through 11-5  848  and non-heap memory usage  874  for the dates 11-1 through 11-5  878 . 
       FIG. 9  shows an example dashboard graphical user interface  900  for previewing and deploying dashboards. Users can select a developed dashboard by dashboard name  932  and can choose preview  936  to cause the display of results for the newly developed dashboard. A preview of results can include a specialized background to signify that the user is viewing a local dashboard displaying production data. A user can deploy the dashboard after review and approval of a design—by selecting deploy  938  from the preview screen, which causes the dashboard to be published to production real time graphing server  418  which makes it available to service engineers who select the dashboard on production real time graphing server  418 . Additionally a full set of dashboards can be deployed simultaneously, by selecting publish all  914  option. In one implementation, a deployment manager running on the processor deploys selected development code—typically a JSON dashboard—to the production server, in response to a request from the development app. 
     System Flow 
       FIG. 10  illustrates a flowchart of one implementation  600  of making production data available for testing in a non-production environment. Flowchart  1000  can be implemented at least partially with a database system, e.g., by one or more processors configured to receive or retrieve information, process the information, store results, and transmit the results. Other implementations may perform the steps in different orders and/or with different, fewer or additional steps than the ones illustrated in  FIG. 10 . 
     At action  1010 , a development server serves an app server code  1020  that is a modified version of a production app  1030 , in response to a request from a developer device; and exposes development APIs that mirror relative addressing of production APIs and relay of selected messages between the development app and the production server  1040 . 
     The development server includes a code manager that, when run on the processor, accesses development code definitions in a persistent development code store  1050 , in response to a request from the developer app or browser. 
     At action  1060 , the development code server receives at least one data assembly and rendering request from a development server that includes or references a definition of data assembly and rendering requirements. 
     At action  1070 , development server code processes each data assembly and rendering request against a data store. At action  1080 , development server code returns results of the processing via the development server. 
     The actions described above can be subdivided into more steps or combined into fewer steps to carry out the method described using a different number or arrangement of steps. 
     Multi-Tenant Integration 
       FIG. 11  presents a block diagram of an exemplary multi-tenant system  1100  suitable for making production data available for testing in a non-production environment  100  of  FIG. 1 . In general, the illustrated multi-tenant system  1100  of  FIG. 11  includes a server  1104  that dynamically supports virtual applications  1116  and  1118 , based upon data  1122  from a common database  1132  that is shared between multiple tenants, alternatively referred to herein as a “multi-tenant database”. Data and services generated by the virtual applications  1116  and  1118 , including GUI clients, are provided via a network  1145  to any number of client devices  1148  or  1158 , as desired. 
     As used herein, a “tenant” or an “organization” refers to a group of one or more users that shares access to common subset of the data within the multi-tenant database  1132 . In this regard, each tenant includes one or more users associated with, assigned to, or otherwise belonging to that respective tenant. Stated another way, each respective user within the multi-tenant system  1100  is associated with, assigned to, or otherwise belongs to a particular tenant of the plurality of tenants supported by the multi-tenant system  1100 . Tenants may represent users, user departments, work or legal organizations, and/or any other entities that maintain data for particular sets of users within the multi-tenant system  1100 . Although multiple tenants may share access to the server  1104  and the database  1132 , the particular data and services provided from the server  1104  to each tenant can be securely isolated from those provided to other tenants. The multi-tenant architecture therefore allows different sets of users to share functionality and hardware resources without necessarily sharing any of the data  1122  belonging to or otherwise associated with other tenants. 
     The multi-tenant database  1132  is any sort of repository or other data storage system capable of storing and managing the data  1122  associated with any number of tenants. The database  1132  may be implemented using any type of conventional database server hardware. In various implementations, the database  1132  shares processing hardware with the server  1104 . In other implementations, the database  1132  is implemented using separate physical and/or virtual database server hardware that communicates with the server  1104  to perform the various functions described herein. The multi-tenant database  1132  may alternatively be referred to herein as an on-demand database, in that the multi-tenant database  1132  provides (or is available to provide) data at run-time to on-demand virtual applications  1116  or  1118  generated by the application platform  11111 , with tenant 1  metadata  1112  and tenant 2  metadata  1114  securely isolated. 
     In practice, the data  1122  may be organized and formatted in any manner to support the application platform  1122 . In various implementations, conventional data relationships are established using any number of pivot tables  1113  that establish indexing, uniqueness, relationships between entities, and/or other aspects of conventional database organization as desired. 
     The server  1104  is implemented using one or more actual and/or virtual computing systems that collectively provide the dynamic application platform  1117  for generating the virtual applications. For example, the server  1104  may be implemented using a cluster of actual and/or virtual servers operating in conjunction with each other, typically in association with conventional network communications, cluster management, load balancing and other features as appropriate. The server  1104  operates with any sort of conventional processing hardware such as a processor  1136 , memory  1138 , input/output features  1134  and the like. The input/output  1134  generally represent the interface(s) to networks (e.g., to the network  1145 , or any other local area, wide area or other network), mass storage, display devices, data entry devices and/or the like. User interface input devices  1134  can include a keyboard; pointing devices such as a mouse, trackball, touchpad, or graphics tablet; a scanner; a touch screen incorporated into the display; audio input devices such as voice recognition systems and microphones; and other types of input devices. In general, use of the term “input device” is intended to include possible types of devices and ways to input information into computer system  1117 . 
     User interface output devices can include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem can include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem can also provide a non-visual display such as audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from processor  1136  to the user or to another machine or computer system. 
     The processor  1136  may be implemented using any suitable processing system, such as one or more processors, controllers, microprocessors, microcontrollers, processing cores and/or other computing resources spread across any number of distributed or integrated systems, including any number of “cloud-based” or other virtual systems. The memory  1138  represents any non-transitory short or long term storage or other computer-readable media capable of storing programming instructions for execution on the processor  1136 , including any sort of random access memory (RAM), read only memory (ROM), flash memory, magnetic or optical mass storage, and/or the like. The computer-executable programming instructions, when read and executed by the server  1104  and/or processor  1136 , cause the server  1104  and/or processor  1136  to create, generate, or otherwise facilitate the application platform  1117  and/or virtual applications  1116  and  1118 , and perform one or more additional tasks, operations, functions, and/or processes described herein. It should be noted that the memory  1138  represents one suitable implementation of such computer-readable media, and alternatively or additionally, the server  1104  could receive and cooperate with external computer-readable media that is realized as a portable or mobile component or application platform, e.g., a portable hard drive, a USB flash drive, an optical disc, or the like. 
     The application platform  1117  is any sort of software application or other data processing engine that generates the virtual applications  1116  and  1118  that provide data and/or services to the client devices  1148  and  1158 . In a typical implementation, the application platform  1117  gains access to processing resources, communications interfaces and other features of the processing hardware using any sort of conventional or proprietary operating system  1128 . The virtual applications  1116  and  1118  are typically generated at run-time in response to input received from the client devices  1148  and  1158 . 
     With continued reference to  FIG. 11 , the data and services provided by the server  1104  can be retrieved using any sort of personal computer, mobile telephone, tablet or other network-enabled client device  1148  or  1158  on the network  1145 . In an exemplary implementation, the client device  1148  or  1158  includes a display device, such as a monitor, screen, or another conventional electronic display capable of graphically presenting data and/or information retrieved from the multi-tenant database  1132 . 
     In some implementations, network(s)  1145  can be any one or any combination of Local Area Network (LAN), Wide Area Network (WAN), WiMAX, Wi-Fi, telephone network, wireless network, point-to-point network, star network, token ring network, hub network, mesh network, peer-to-peer connections like Bluetooth, Near Field Communication (NFC), Z-Wave, ZigBee, or other appropriate configuration of data networks, including the Internet. 
     The foregoing description is merely illustrative in nature and is not intended to limit the implementations of the subject matter or the application and uses of such implementations. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the technical field, background, or the detailed description. As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any implementation described herein as exemplary is not necessarily to be construed as preferred or advantageous over other implementations, and the exemplary implementations described herein are not intended to limit the scope or applicability of the subject matter in any way. 
     The technology disclosed can be implemented in the context of any computer-implemented system including a database system, a multi-tenant environment, or a relational database implementation like an Oracle™ compatible database implementation, an IBM DB2 Enterprise Server™ compatible relational database implementation, a MySQL™ or PostgreSQL™ compatible relational database implementation or a Microsoft SQL Server™ compatible relational database implementation or a NoSQL non-relational database implementation such as a Vampire™ compatible non-relational database implementation, an Apache Cassandra™ compatible non-relational database implementation, a BigTable™ compatible non-relational database implementation or an HBase™ or DynamoDB™ compatible non-relational database implementation. 
     Moreover, the technology disclosed can be implemented using two or more separate and distinct computer-implemented systems that cooperate and communicate with one another. The technology disclosed can be implemented in numerous ways, including as a process, a method, an apparatus, a system, a device, a computer readable medium such as a computer readable storage medium that stores computer readable instructions or computer program code, or as a computer program product comprising a computer usable medium having a computer readable program code embodied therein. 
     Particular Implementations 
     Some particular implementations and features are described in the following discussion. 
     In one implementation, a disclosed development server includes a processor, memory coupled to the processor, the processor coupled in communication with a persistent development code store that holds development code, and with the processor coupled in communication with a production server. An app server running on the processor serves a development app that is a modified version of a production app, in response to a request from a developer device; exposes development APIs that mirror production APIs of the production server, and relays messages between the development APIs and the production APIs. The disclosed development server also includes a code manager running on the processor that accesses the development code in the persistent development code store, in response to a request from the development app; and the development app including a user interface for selection among development code definitions of dashboards; and replication of access to production server APIs used to retrieve and process time series data from a data store, the replication including access to development server APIs that mirror relative addressing of production APIs. 
     The disclosed development server can also include the app server, running on the processor, establishing a session by authenticating the developer to the production server. The disclosed development server further includes a deployment manager running on the processor that deploys selected development code to the production server, in response to a request from the development app. The development server can further include a user interface for selection of deployment of a development code definition to the production server. For the disclosed development server, development code definitions of dashboards are expressed in JSON or XML. Additional formats that transmit data objects consisting of attribute-value pairs can be used to express development code definitions of dashboards. 
     In one implementation, a disclosed method of making production data available from a production server for development testing includes providing a development server code, including an app server code that, when run on a processor, serves a development app code that is a modified version of a production app, in response to a request from a developer device; and exposes development APIs that mirror relative addressing of production APIs, and relay of selected messages between the development app and the production server. The disclosed method also includes a code manager that, when run on the processor, accesses development code definitions in a persistent development code store, in response to a request from the development app. The disclosed method further includes receiving at least one data assembly and rendering request from a development server that includes or references a definition of data assembly and rendering requirements; processing each data assembly and rendering request against a data store; and returning results of the processing via the development server. 
     The disclosed method further includes a deployment manager that when run on the processor that accesses the development code in the persistent development code store, deploys selected development code to the production server, in response to a request from the development app. The method disclosed can further include the production server receiving from the development server code, one or more data transformation requests that include a definition of data transformation requirements. 
     This method and other implementations of the technology disclosed can include one or more of the following features and/or features described in connection with additional methods disclosed. In the interest of conciseness, the combinations of features disclosed in this application are not individually enumerated and are not repeated with each base set of features. The reader will understand how features identified in this section can readily be combined with sets of base features identified as implementations. 
     For the disclosed method, the development code in the persistent development code store uses one of JSON or XML to express development code definitions of dashboards. Additional formats that transmit data objects consisting of attribute-value pairs can be used to express development code definitions of dashboards. 
     In some implementations of the disclosed method, the results returned via the development server from the processing include a graphic that illustrates the data assembled in response to the data assembly and rendering request. In other implementations, the results returned via the development server from the processing include data values assembled in response to the data assembly and rendering request, adapted for rendering by the development app. 
     In other implementations, a disclosed method of making production data available for development testing includes, at a development server, receiving an app request from a developer device and serving a development app that is a modified version of a production app. The development app processes user selection among development code definitions of dashboards in a persistent development code store; and replicates access to a production server, including development APIs replicating relative addressing of production APIs that implement access to data assembly from a production data store. The method also includes receiving, at a development API, a preview request from the development app to preview at least one data assembly and rendering request; and forwarding the request and a dashboard code definition to the production server at a production API replicated by the development API that received the preview request. The method further includes receiving from the production server and causing display, at the developer device, of results from the processing. 
     In some implementations of the disclosed method, the results received from the production server include a graphic that illustrates the data assembled in response to the preview request. For other implementations, the results received from the production server include data values assembled in response to the preview request, adapted for rendering at the developer device. The developer device can include an app or a browser. 
     For some implementations, the development API includes a user interface for selection of deployment to the production server of the development code definitions. The persistent development code store uses one of JSON or XML to represent development code definitions of dashboards. 
     In one implementation, a development server code, loaded onto a tangible computer readable medium, includes instructions executable on a processor that implement an app server that serves a development app code that is a modified version of a production app from a production server, in response to a request from a developer device, exposes development APIs that mirror relative addressing of production APIs that implement access to data assembly from a production data store; relays selected messages between the development app and the production server; and returns results from the processing to a developer device. The production APIs process requests for data assembly and rendering, accompanied by dashboard code definitions; and a code manager accesses the development code in a persistent development code store, in response to a request from the development app. The disclosed implementation of the development server code further includes a deployment manager that deploys selected development code to the production server, in response to a request from the development app, and can include a user interface for selection of deployment to the production server of a selected development code definition. 
     In some implementations of the disclosed development server code, the results returned via the development server from the processing include a graphic that illustrates the data assembled in response to the data assembly and rendering request. In other implementations, the results returned via the development server from the processing include data values assembled in response to the data assembly and rendering request, adapted for rendering by the development app. 
     In yet other implementations a development app code, loaded onto a tangible computer readable medium, includes instructions executable on a processor that implement a development app that is a modified version of a production app from a production server, that includes reference to development APIs of a development server that mirror relative addressing of production APIs that implement access to data assembly from a production data store and cause relay of messages between the development app and the production server. The production APIs process requests for data assembly and rendering, accompanied by dashboard code definitions, against a production data store; access to a code manager running on the development server that accesses the development code in persistent development code store, in response to a request from the development app. Further, the development app includes access to a deployment manager that deploys selected development code from the persistent development code store to the production server, in response to a request from the development app. For the disclosed development app code, the deployment manager can include a user interface for selection of deployment to the production server, of selected development code definitions. 
     While the technology disclosed is disclosed by reference to the preferred embodiments and examples detailed above, it is to be understood that these examples are intended in an illustrative rather than in a limiting sense. It is contemplated that modifications and combinations will readily occur to those skilled in the art, which modifications and combinations will be within the spirit of the invention and the scope of the following claims.