Preprocessing and storage of cloud service usage reports

Systems and methods for streaming billing data associated with third-party network resources consumed by a managed network to a remote network management platform are disclosed. Streaming of billing usage data associated with third-party resources may use resource identifiers to associate resources with line items in a service bill generated by the third-party network. A proxy server in the managed network may intermediate streaming from the third-party network to a remote server in a computational instance of the remote network management platform. Billing data may be recorded partly in a metric-base database configured for storing time series data corresponding to usage quantities of third-party resources consumed by the managed network per unit time, and partly in a static records database configured for storing database records associated with the time series data, and including data for identifying the third-party resources and for recording static information relating to billing for the third-party resources.

BACKGROUND

Managed networks may support the missions and operations of organizations or enterprises, and may include various types of computer networks that can be remotely administered. Management may involve one or more computing devices disposed within a remote network management platform collecting information about the configuration and operational states of software applications executing on behalf on the managed network, and then presenting representations of this information by way of one or more user interfaces. The user interfaces may be, for instance, web-based user interfaces. In some instances, remote management of networks may be provided by an entity separate from that of the managed network, such as a service provider or vendor.

In addition to engaging a service provider or vendor for remote network management services, an enterprise may also use a third-party network, sometimes referred to as a cloud or datacenter, to deploy various applications and services for the enterprise's employees, clients, and customers. For instance, the enterprise may use on-demand computational resources from the third-party network, or the enterprise may reserve the right to use the computing resources in advance by way of a reserved instance (RI). On-demand computing resources offer the flexibility to increase or decrease in scale so that these resources match fluctuating needs of the enterprise, and often involve a pay-per-use model. On the other hand, computing resources purchased via a reserved instance often involve an agreement for computational resources across a specified term (e.g., 1 or 3 years) for particular payment (e.g., an upfront payment or a payment plan). For these or other models of third-party computational resource usage, the third-party network owner and/or operator will bill the enterprise on some basis for resources consumed.

SUMMARY

In accordance with example embodiments, remote network management of a managed network may be implemented by a “computational instance” or “customer instance” of a remote network management platform. A computational instance may include various physical and/or virtual servers, databases, and other computing resources that may be dedicated or assigned to managing a network of an organization or enterprise. A computational instance may also include various ready-made network management tasks and services, as well as facilities for customizing tasks and services, for example. The remote management platform may be owned and/or operated by a service provider, which may provide computational instances to organizations or enterprises on a subscription basis, for example. In some deployments, an organization or enterprise may have more than one computational instance.

As noted, an enterprise or organization may, in addition to obtaining network management services from a remote network management platform owner or service provider, also obtain other types of network and/or computing resources from a third-party provider. Such resources may include computing/processing resources of virtual machines, cloud storage, and data center services, among others. For purposes of discussion herein, the term “third-party network” will be used to refer to both a third-party provider of these types of services, as well as the infrastructure operated by the third-party provider to make the services available to customers. An example of a third-party network is Amazon Web Services™ (AWS™). Other third-party network providers may be available as well.

A third-party network will typically track usage of resources consumed by a customer, such as an enterprise, and issue a bill on some timeline for the usage. For example, the customer may have an account with the third-party network provider and be billed for third-party resources consumed on a monthly basis, corresponding to a billing cycle. The bill will generally be separate from any billing or charges by a remote network management platform service provider, as it will reflect a different set of services delivered to the enterprise. In a common arrangement, an enterprise may subscribe to a set of specific third-party network resources (e.g., virtual machine computing cycles, cloud storage, etc.), and the third-party network provider may meter usage of the specific resources during incremental time intervals within a billing cycle, and generate a bill for cumulative usage of each of the specific resources at some time after the end of each billing cycle. For instance, usage for a time-based resource, such as computing cycles, might be metered hourly as a fraction of each hour over which computing cycles were consumed by the enterprise. Each day of a billing cycle may thus be tracked according to 24 hourly rates of usage. Similar metering and tracking could be applied to cloud storage or other third-party computing resources. Other forms of metering and tracking could be used as well.

In practice, usage billing for third-party resources consumed by an enterprise can be complicated. For example, an enterprise may have a number of sub-organizations, such as departments, geographic regions, and/or administrative divisions, some or all of which may utilize third-party resources under separate tracking and metering arrangements. Thus, the total bill associated with an enterprise's account might be broken down not only according to specific resources, but according an enterprises organizational structure. Additionally, hourly (or other incremental interval) metering and tracking may yield expansive ranges and volumes of billing data. In particular, time-based metering and tracking may dictate, at least in part, how billing data are recorded and how they may be made available to the enterprise customer.

It may be common for a third-party network to provide usage billing for an account in a relatively compact form, such as monthly (or other billing cycle) totals for specific resources, or even summarized for resource categories. On the other hand, an enterprise may have an interest in careful scrutiny of billing usage data, not only for verification purposes, but to help identify and understand resource consumption trends in order to help ensure efficient usage and find areas improvement can be made. While detailed billing data may be available from the third-party network provider, the volume and format of the data may make it inconvenient or difficult for an enterprise customer to obtain, much less to process or examine.

The inventors have recognized that the remote network management platform, including one or more customer instances assigned to or associated with an enterprise's managed network, may support services and operations, and may already deploy infrastructure, both for providing efficient and intelligent acquisition and storage of detailed third-party resource-usage billing data for the enterprise, as well as for processing and analysis of detailed billing data. By obtaining the detailed data and storing it in a format that is convenient for processing and analysis, the remote network management platform may thus also support tools and functions that enable the enterprise to manage its utilization of third-party computing resources effectively.

Example embodiments herein are directed at systems and methods for advantageously achieving these valuable capabilities. In particular, by coordinating communications and operations between entities in a managed network and servers in a computational instance associated with the managed network, voluminous detailed billing usage data may be efficiently streamed from a third-party network to specially designed databases in the remote network management platform, where the data may be recorded in formats well-suited for both storage and analysis.

Accordingly, a first example embodiment may involve a system for streaming billing data to a remote network management platform from a third-party network, wherein the billing data are associated with usage of resources provided by the third-party network to a managed network that is associated with a computational instance of the remote network management platform, the system comprising: a metric-base database (MBDB) disposed within the remote network management platform and configured for storing time series data, wherein the time series data comprise data specifying usage quantities of third-party resources consumed by the managed network per unit time over specified time intervals; a static records database (SRDB) disposed within the remote network management platform and configured for storing database records that are associated with the time series data and include data for identifying the third-party resources and for recording static information relating to billing for the third-party resources; and one or more server devices configured to: determine a resource identifier associating a particular third-party resource with a particular line item in a service bill generated by the third-party network for third-party resources consumed by the managed network; based on the resource identifier, carry out download-streaming of billing usage data associated with the particular third-party resource from the third-party network to the one or more server devices; store the download-streamed billing usage data in a time-series file associated with the particular third-party resource in the MBDB; and if the SRDB does not already contain a static record associated with the particular third-party resource, create and store in the SRDB a particular static record that includes a link to the time-series file for the particular third-party resource.

In a second example embodiment may involve a proxy server for streaming billing data to a remote network management platform from a third-party network, wherein the billing data are associated with usage of resources provided by the third-party network to a managed network that is associated with a computational instance of the remote network management platform, and wherein the proxy server is disposed within the managed network and comprises: a communicative connection to the third-party network; a communicative connection to a remote server disposed within the computational instance; one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the proxy server to carry out operations including: downloading, from the third-party network, a service bill for third-party resources consumed by the managed network; determining one or more resource identifiers each associating a respective third-party resource with a respective line item in the service bill; based on the one or more resource identifiers, intermediating download-streaming of billing usage data associated with each respective third-party resource from the third-party network to a metric-base database (MBDB) for storage in a respective time-series file, wherein the MBDB is disposed within the remote network management platform and the download-streaming is by way of the remote server, and wherein each respective time-series file comprises respective time series data specifying usage quantities of the respective third-party resource consumed by the managed network per unit time over a respectively specified time interval; and based on at least the one or more resource identifiers, providing the remote server with information for creating and storing respective static records in a static records database (SRDB) disposed within the remote network management platform, wherein the respective static records are associated with the respective time series data and each includes a respective link to the respective time-series file, and each further includes respective data for identifying the respective third-party resource and for recording respective static information relating to billing for the respective third-party resource.

In a third example embodiment may involve a method for streaming billing data to a remote network management platform from a third-party network, wherein the billing data are associated with usage of resources provided by the third-party network to a managed network that is associated with a computational instance of the remote network management platform, wherein the remote network management platform comprises (i) a metric-base database (MBDB) configured for storing time series data, wherein the time series data comprise data specifying usage quantities of third-party resources consumed by the managed network per unit time over specified time intervals, and (ii) a static records database (SRDB) configured for storing database records that are associated with the time series data and include data for identifying the third-party resources and for recording static information relating to billing for the third-party resources, and wherein the method comprises: determining a resource identifier associating a particular third-party resource with a particular line item in a service bill generated by the third-party network for third-party resources consumed by the managed network; based on the resource identifier, carrying out download-streaming of billing usage data associated with the particular third-party resource from the third-party network to the one or more server devices; storing the download-streamed billing usage data in a time-series file associated with the particular third-party resource in the MBDB; and if the SRDB does not already contain a static record associated with the particular third-party resource, creating and storing in the SRDB a particular static record that includes a link to the time-series file for the particular third-party resource.

In a fourth example embodiment, a system may include various means for carrying out each of the operations of the third example embodiment.

DETAILED DESCRIPTION

In order to achieve this goal, the concept of Application Platform as a Service (aPaaS) is introduced, to intelligently automate workflows throughout the enterprise. An aPaaS system is hosted remotely from the enterprise, but may access data, applications, and services within the enterprise by way of secure connections. Such an aPaaS system may have a number of advantageous capabilities and characteristics. These advantages and characteristics may be able to improve the enterprise's operations and workflow for IT, HR, CRM, customer service, application development, and security.

The aPaaS system may support development and execution of model-view-controller (MVC) applications. MVC applications divide their functionality into three interconnected parts (model, view, and controller) in order to isolate representations of information from the manner in which the information is presented to the user, thereby allowing for efficient code reuse and parallel development. These applications may be web-based, and offer create, read, update, delete (CRUD) capabilities. This allows new applications to be built on a common application infrastructure.

The aPaaS system may support clearly-defined interfaces between applications, so that software developers can avoid unwanted inter-application dependencies. Thus, the aPaaS system may implement a service layer in which persistent state information and other data is stored.

The following embodiments describe architectural and functional aspects of example aPaaS systems, as well as the features and advantages thereof.

FIG. 2depicts a cloud-based server cluster200in accordance with example embodiments. InFIG. 2, operations of a computing device (e.g., computing device100) may be distributed between server devices202, data storage204, and routers206, all of which may be connected by local cluster network208. The number of server devices202, data storages204, and routers206in server cluster200may depend on the computing task(s) and/or applications assigned to server cluster200.

III. EXAMPLE REMOTE NETWORK MANAGEMENT ARCHITECTURE

FIG. 3depicts a remote network management architecture, in accordance with example embodiments. This architecture includes three main components, managed network300, remote network management platform320, and third-party networks340, all connected by way of Internet350.

Managed network300may also include one or more proxy servers312. An embodiment of proxy servers312may be a server device that facilitates communication and movement of data between managed network300, remote network management platform320, and third-party networks340. In particular, proxy servers312may be able to establish and maintain secure communication sessions with one or more computational instances of remote network management platform320. By way of such a session, remote network management platform320may be able to discover and manage aspects of the architecture and configuration of managed network300and its components. Possibly with the assistance of proxy servers312, remote network management platform320may also be able to discover and manage aspects of third-party networks340that are used by managed network300.

In some cases, managed network300may consist of a few devices and a small number of networks. In other deployments, managed network300may span multiple physical locations and include hundreds of networks and hundreds of thousands of devices. Thus, the architecture depicted inFIG. 3is capable of scaling up or down by orders of magnitude.

Remote network management platform320is a hosted environment that provides aPaaS services to users, particularly to the operators of managed network300. These services may take the form of web-based portals, for instance. Thus, a user can securely access remote network management platform320from, for instance, client devices302, or potentially from a client device outside of managed network300. By way of the web-based portals, users may design, test, and deploy applications, generate reports, view analytics, and perform other tasks.

As shown inFIG. 3, remote network management platform320includes four computational instances322,324,326, and328. Each of these instances may represent one or more server devices and/or one or more databases that provide a set of web portals, services, and applications (e.g., a wholly-functioning aPaaS system) available to a particular customer. In some cases, a single customer may use multiple computational instances. For example, managed network300may be an enterprise customer of remote network management platform320, and may use computational instances322,324, and326. The reason for providing multiple instances to one customer is that the customer may wish to independently develop, test, and deploy its applications and services. Thus, computational instance322may be dedicated to application development related to managed network300, computational instance324may be dedicated to testing these applications, and computational instance326may be dedicated to the live operation of tested applications and services. A computational instance may also be referred to as a hosted instance, a remote instance, a customer instance, or by some other designation. Any application deployed onto a computational instance may be a scoped application, in that its access to databases within the computational instance can be restricted to certain elements therein (e.g., one or more particular database tables or particular rows with one or more database tables).

For purpose of clarity, the disclosure herein refers to the physical hardware, software, and arrangement thereof as a “computational instance.” Note that users may colloquially refer to the graphical user interfaces provided thereby as “instances.” But unless it is defined otherwise herein, a “computational instance” is a computing system disposed within remote network management platform320.

The multi-instance architecture of remote network management platform320is in contrast to conventional multi-tenant architectures, over which multi-instance architectures exhibit several advantages. In multi-tenant architectures, data from different customers (e.g., enterprises) are comingled in a single database. While these customers' data are separate from one another, the separation is enforced by the software that operates the single database. As a consequence, a security breach in this system may impact all customers' data, creating additional risk, especially for entities subject to governmental, healthcare, and/or financial regulation. Furthermore, any database operations that impact one customer will likely impact all customers sharing that database. Thus, if there is an outage due to hardware or software errors, this outage affects all such customers. Likewise, if the database is to be upgraded to meet the needs of one customer, it will be unavailable to all customers during the upgrade process. Often, such maintenance windows will be long, due to the size of the shared database.

In some embodiments, remote network management platform320may include one or more central instances, controlled by the entity that operates this platform. Like a computational instance, a central instance may include some number of physical or virtual servers and database devices. Such a central instance may serve as a repository for data that can be shared amongst at least some of the computational instances. For instance, definitions of common security threats that could occur on the computational instances, software packages that are commonly discovered on the computational instances, and/or an application store for applications that can be deployed to the computational instances may reside in a central instance. Computational instances may communicate with central instances by way of well-defined interfaces in order to obtain this data.

In order to support multiple computational instances in an efficient fashion, remote network management platform320may implement a plurality of these instances on a single hardware platform. For example, when the aPaaS system is implemented on a server cluster such as server cluster200, it may operate a virtual machine that dedicates varying amounts of computational, storage, and communication resources to instances. But full virtualization of server cluster200might not be necessary, and other mechanisms may be used to separate instances. In some examples, each instance may have a dedicated account and one or more dedicated databases on server cluster200. Alternatively, computational instance322may span multiple physical devices.

Third-party networks340may be remote server devices (e.g., a plurality of server clusters such as server cluster200) that can be used for outsourced computational, data storage, communication, and service hosting operations. These servers may be virtualized (i.e., the servers may be virtual machines). Examples of third-party networks340may include AMAZON WEB SERVICES® and MICROSOFT® Azure. Like remote network management platform320, multiple server clusters supporting third-party networks340may be deployed at geographically diverse locations for purposes of load balancing, redundancy, and/or high availability.

Managed network300may use one or more of third-party networks340to deploy applications and services to its clients and customers. For instance, if managed network300provides online music streaming services, third-party networks340may store the music files and provide web interface and streaming capabilities. In this way, the enterprise of managed network300does not have to build and maintain its own servers for these operations.

Remote network management platform320may include modules that integrate with third-party networks340to expose virtual machines and managed services therein to managed network300. The modules may allow users to request virtual resources and provide flexible reporting for third-party networks340. In order to establish this functionality, a user from managed network300might first establish an account with third-party networks340, and request a set of associated resources. Then, the user may enter the account information into the appropriate modules of remote network management platform320. These modules may then automatically discover the manageable resources in the account, and also provide reports related to usage, performance, and billing.

FIG. 4further illustrates the communication environment between managed network300and computational instance322, and introduces additional features and alternative embodiments. InFIG. 4, computational instance322is replicated across data centers400A and400B. These data centers may be geographically distant from one another, perhaps in different cities or different countries. Each data center includes support equipment that facilitates communication with managed network300, as well as remote users.

Data centers400A and400B as shown inFIG. 4may facilitate redundancy and high availability. In the configuration ofFIG. 4, data center400A is active and data center400B is passive. Thus, data center400A is serving all traffic to and from managed network300, while the version of computational instance322in data center400B is being updated in near-real-time. Other configurations, such as one in which both data centers are active, may be supported.

FIG. 4also illustrates a possible configuration of managed network300. As noted above, proxy servers312and user414may access computational instance322through firewall310. Proxy servers312may also access configuration items410. InFIG. 4, configuration items410may refer to any or all of client devices302, server devices304, routers306, and virtual machines308, any applications or services executing thereon, as well as relationships between devices, applications, and services. Thus, the term “configuration items” may be shorthand for any physical or virtual device, or any application or service remotely discoverable or managed by computational instance322, or relationships between discovered devices, applications, and services. Configuration items may be represented in a configuration management database (CMDB) of computational instance322.

As noted above, VPN gateway412may provide a dedicated VPN to VPN gateway402A. Such a VPN may be helpful when there is a significant amount of traffic between managed network300and computational instance322, or security policies otherwise suggest or require use of a VPN between these sites. In some embodiments, any device in managed network300and/or computational instance322that directly communicates via the VPN is assigned a public IP address. Other devices in managed network300and/or computational instance322may be assigned private IP addresses (e.g., IP addresses selected from the 10.0.0.0-10.255.255.255 or 192.168.0.0-192.168.255.255 ranges, represented in shorthand as subnets 10.0.0.0/8 and 192.168.0.0/16, respectively).

IV. EXAMPLE DEVICE, APPLICATION, AND SERVICE DISCOVERY

In order for remote network management platform320to administer the devices, applications, and services of managed network300, remote network management platform320may first determine what devices are present in managed network300, the configurations and operational statuses of these devices, and the applications and services provided by the devices, and well as the relationships between discovered devices, applications, and services. As noted above, each device, application, service, and relationship may be referred to as a configuration item. The process of defining configuration items within managed network300is referred to as discovery, and may be facilitated at least in part by proxy servers312.

FIG. 5Aprovides a logical depiction of how configuration items can be discovered, as well as how information related to discovered configuration items can be stored. For sake of simplicity, remote network management platform320, third-party networks340, and Internet350are not shown.

InFIG. 5A, CMDB500and task list502are stored within computational instance322. Computational instance322may transmit discovery commands to proxy servers312. In response, proxy servers312may transmit probes to various devices, applications, and services in managed network300. These devices, applications, and services may transmit responses to proxy servers312, and proxy servers312may then provide information regarding discovered configuration items to CMDB500for storage therein. Configuration items stored in CMDB500represent the environment of managed network300.

Task list502represents a list of activities that proxy servers312are to perform on behalf of computational instance322. As discovery takes place, task list502is populated. Proxy servers312repeatedly query task list502, obtain the next task therein, and perform this task until task list502is empty or another stopping condition has been reached.

To facilitate discovery, proxy servers312may be configured with information regarding one or more subnets in managed network300that are reachable by way of proxy servers312. For instance, proxy servers312may be given the IP address range 192.168.0/24 as a subnet. Then, computational instance322may store this information in CMDB500and place tasks in task list502for discovery of devices at each of these addresses.

FIG. 5Aalso depicts devices, applications, and services in managed network300as configuration items504,506,508,510, and512. As noted above, these configuration items represent a set of physical and/or virtual devices (e.g., client devices, server devices, routers, or virtual machines), applications executing thereon (e.g., web servers, email servers, databases, or storage arrays), relationships therebetween, as well as services that involve multiple individual configuration items.

Placing the tasks in task list502may trigger or otherwise cause proxy servers312to begin discovery. Alternatively or additionally, discovery may be manually triggered or automatically triggered based on triggering events (e.g., discovery may automatically begin once per day at a particular time).

In general, discovery may proceed in four logical phases: scanning, classification, identification, and exploration. Each phase of discovery involves various types of probe messages being transmitted by proxy servers312to one or more devices in managed network300. The responses to these probes may be received and processed by proxy servers312, and representations thereof may be transmitted to CMDB500. Thus, each phase can result in more configuration items being discovered and stored in CMDB500.

In the scanning phase, proxy servers312may probe each IP address in the specified range of IP addresses for open Transmission Control Protocol (TCP) and/or User Datagram Protocol (UDP) ports to determine the general type of device. The presence of such open ports at an IP address may indicate that a particular application is operating on the device that is assigned the IP address, which in turn may identify the operating system used by the device. For example, if TCP port135is open, then the device is likely executing a WINDOWS® operating system. Similarly, if TCP port22is open, then the device is likely executing a UNIX® operating system, such as LINUX®. If UDP port161is open, then the device may be able to be further identified through the Simple Network Management Protocol (SNMP). Other possibilities exist. Once the presence of a device at a particular IP address and its open ports have been discovered, these configuration items are saved in CMDB500.

In the identification phase, proxy servers312may determine specific details about a classified device. The probes used during this phase may be based on information gathered about the particular devices during the classification phase. For example, if a device was classified as LINUX®, a set of LINUX®-specific probes may be used. Likewise if a device was classified as WINDOWS® 2012, as a set of WINDOWS®-2012-specific probes may be used. As was the case for the classification phase, an appropriate set of tasks may be placed in task list502for proxy servers312to carry out. These tasks may result in proxy servers312reading information from the particular device, such as basic input/output system (BIOS) information, serial numbers, network interface information, media access control address(es) assigned to these network interface(s), IP address(es) used by the particular device and so on. This identification information may be stored as one or more configuration items in CMDB500.

Running discovery on a network device, such as a router, may utilize SNMP. Instead of or in addition to determining a list of running processes or other application-related information, discovery may determine additional subnets known to the router and the operational state of the router's network interfaces (e.g., active, inactive, queue length, number of packets dropped, etc.). The IP addresses of the additional subnets may be candidates for further discovery procedures. Thus, discovery may progress iteratively or recursively.

Once discovery completes, a snapshot representation of each discovered device, application, and service is available in CMDB500. For example, after discovery, operating system version, hardware configuration and network configuration details for client devices, server devices, and routers in managed network300, as well as applications executing thereon, may be stored. This collected information may be presented to a user in various ways to allow the user to view the hardware composition and operational status of devices, as well as the characteristics of services that span multiple devices and applications.

Furthermore, CMDB500may include entries regarding dependencies and relationships between configuration items. More specifically, an application that is executing on a particular server device, as well as the services that rely on this application, may be represented as such in CMDB500. For instance, suppose that a database application is executing on a server device, and that this database application is used by a new employee onboarding service as well as a payroll service. Thus, if the server device is taken out of operation for maintenance, it is clear that the employee onboarding service and payroll service will be impacted. Likewise, the dependencies and relationships between configuration items may be able to represent the services impacted when a particular router fails.

In general, dependencies and relationships between configuration items may be displayed on a web-based interface and represented in a hierarchical fashion. Thus, adding, changing, or removing such dependencies and relationships may be accomplished by way of this interface.

Furthermore, users from managed network300may develop workflows that allow certain coordinated activities to take place across multiple discovered devices. For instance, an IT workflow might allow the user to change the common administrator password to all discovered LINUX® devices in single operation.

The discovery process is depicted as a flow chart inFIG. 5B. At block520, the task list in the computational instance is populated, for instance, with a range of IP addresses. At block522, the scanning phase takes place. Thus, the proxy servers probe the IP addresses for devices using these IP addresses, and attempt to determine the operating systems that are executing on these devices. At block524, the classification phase takes place. The proxy servers attempt to determine the operating system version of the discovered devices. At block526, the identification phase takes place. The proxy servers attempt to determine the hardware and/or software configuration of the discovered devices. At block528, the exploration phase takes place. The proxy servers attempt to determine the operational state and applications executing on the discovered devices. At block530, further editing of the configuration items representing the discovered devices and applications may take place. This editing may be automated and/or manual in nature.

The blocks represented inFIG. 5Bare for purpose of example. Discovery may be a highly configurable procedure that can have more or fewer phases, and the operations of each phase may vary. In some cases, one or more phases may be customized, or may otherwise deviate from the exemplary descriptions above.

V. EXAMPLE STREAMING OF CLOUD SERVICE RESOURCE USAGE REPORTS

Managed network300may use a third-party network for a variety of computing and/or network resources to fulfill operational needs or demands without necessarily having to deploy additional hardware or infrastructure. The third-party network may also be referred to as a cloud or datacenter, and the resources they provide may also be referred to as cloud resources. As an example, managed network300may use the computing resources of a third-party network to meet growing or fluctuating computational needs, document control and security, and to enable remote access to applications and services. The third-party provider may dynamically allocate computing resources for managed network300. Examples of such computing resources may include processing power, data storage, and networking services, among others.

A third-party network may provide resources on some form of fee basis that generally falls into one of two categories broadly described as on-demand usage and reserved instances (i.e., purchasing the computing resources via one or more reserved instances). When using computational resources on-demand, managed network300obtains the computational resources as needed. The third-party provider may monitor usage of computing resources allocated to managed network300, which can enable the third-party provider to charge managed network300accordingly. For example, the third-party provider may charge managed network300based on a duration that managed network300used computational resources on-demand and/or based on a quantity of computational resources used on-demand. The manner in which the third-party provider measures usage may vary. As an example, when managed network300uses computational resources for processing, the third-party provider may measure a number of processor cores used. Alternatively, when managed network300uses computational resources for data storage, the third-party provider may measure the amount of data (e.g., gigabytes, terabytes, etc.) was allocated for storage.

When obtaining computational resources through reserved instances, managed network300may enter into an agreement with the third-party provider. The agreement may represent a quantity of computing resources reserved by managed network300over an agreed upon term (e.g., 1 year, 3 years) and may involve an upfront payment or a payment plan between managed network300and the third-party provider. The use of reserved computing resources by managed network300may depend on various factors, including an account maintained by the third-party provider, region of use by managed network300, and instance type, among others.

The usage of computational resources by managed network300can vary depending on the time of day. In some cases, managed network300may only require access to certain resources during specific hours (e.g., hours of operation). Similarly, the computational needs of managed network300can fluctuate throughout the day or week depending on various factors, such as demand from client devices302or customers. As such, management network300may attempt to manage computing resources obtained from a third-party network in order to avoid unwanted costs.

FIG. 6is a simplified block diagram showing high-level relationships involving a managed network300, a remote network management platform320, and a third-party network600, in accordance with example embodiments. As described above in connection withFIG. 3, the managed network300, remote network management platform320, and third-party network600may be communicatively connected by way of an intermediate network, such as an internet350. Accordingly, the managed network300may access network and computing resources608to obtain various cloud resources. Network and computing resources608could include virtual machines and/or cloud storage, among other third-party resources. The third-party network may track or meter resource usage by the managed network300and record the usage in one or another form of database or other data storage arrangement, represented by utilization reports and billing602. By way of example, utilization reports and billing602could be a database or other form of data storage configured for storing on-demand computing resources usage604and/or reserved computing resources usage606.

For either form of usage and billing model, the third-party network may track resources used or consumed by a customer associated with the managed network300. As such, the customer, which could be an enterprise, for example, may have an account with the third-party network. The third-party network may then bill customer's account on a periodic basis, such as monthly billing. Even for reserved instance billing, the third-party network may track detailed usage for the customer and store the data in the reserved computing resources usage606.

In a typical arrangement, the third-party may provide the customer with a summary of usage for a given billing cycle. However, in order for the customer to assess its usage more comprehensively, it needs to be able to obtain the detailed usage and billing data. These data may correspond to relatively high-resolution time-based metering of resource usage, and constitute a large volume of data that makes acquisition by the customer relatively inconvenient and/or cumbersome. In addition, since the customer (e.g., enterprise) may have a billing agreement (e.g., account or accounts) with the third-party network separate from its relationship or subscription with a remote network management owner and/or service provider, access by the remote network management platform to the customer's third-party account information may involve coordination with the customer's managed network.

In accordance with example embodiments, the computational instance322may operate in a coordinated fashion with the managed network300to stream the time-based data to a database610in the remote network management platform320, which may then use its considerable and well-suited analytical computing resources and specialized application programs to carry out a variety of processing and analysis tasks and operations. The customer associated with the managed network may thus obtain useful information for understanding how it is using the third-party resources, and in particular, how it may be more efficient and/or cost-effective in doing so.

Systems and methods for streaming a managed network's third-party usage data from the third-party network to a remote network management platform via the managed network may be described by considering an example format and storage of detailed usage billing data in the third-party network.FIG. 7Aillustrates such an example in which a manifest file700includes informational data, such as metadata, that describe the detailed usage data and specify where they are stored. It will be appreciated that different third-party networks may implement different formats of storage, which in turn may be described differently from one another. Thus, use of a manifest file may be used to account for such differences, as it supplies a mechanism for a form of metadata between the actual billing usage data and access to those data by a managed network and/or a remote network management platform. Accordingly, the manifest file illustrated inFIG. 7A, including the description herein of how it is used, is just one example, and not intended to be limiting with regard to example embodiments herein of systems and methods for streaming.

In the example ofFIG. 7A, the manifest file700includes a header702, and assembly ID704, and usage report filenames706,708,710, . . . , and720, where the ellipses in the figure indicate that there could be additional usage report filenames. The header702may include descriptive date about the usage report files, such as formatting information, column names of tables, and usage data types (e.g., integer, floating point, etc.), among other information that may be common to the usage report files. The assembly ID may be used to uniquely identify a version of the manifest file700that might otherwise pertain to a common filename for the manifest file. For example, a given manifest file may pertain to a specific billing cycle for a particular account, and thus may have a filename reflective of an account name and billing cycle (e.g., date range). At the same time, the third-party network may update billing information on a daily (or other) basis throughout the billing cycle. As such, successive versions of the manifest file may correspond to incrementally more usage data. Each version may therefore have a different assembly ID to distinguish the otherwise commonly-named manifest files.

The usage report filenames706,708,710, . . . , and720may be pointers, or associated with pointers, to actual data files that store the usage data. In the illustrated example, the usage data are stored in third-party resource usage files750, as shown. These files may store large volumes of metered usage data for various third-party resources consumed by the enterprise. The data in each file could correspond to resource usage measurements record every hour over the course of a 24-hour period, for example. In addition to being associated with usage for a specific resource, each file may also be associated with a specific category with the customer's account. For example, distinct categories could be associated with different divisions, different geographic locations, or different functional organizations of an enterprise. Each of the different filenames may identify usage data files for the different account categories and third-party resource types.

FIG. 7Billustrates, by way of example, how different versions of manifest file700may be associated with different date ranges of usage data within a common billing cycle. As shown, two versions of the manifest file700have a common filename that includes an example billing cycle of 2018-03-01 to 2018-03-31. For purposes of illustration, this is identified as a report date “super-range.” Both versions also have a common header702. However, each has a distinct assembly ID, namely assembly ID704-1and assembly ID704-2. The usage report filenames are also different between the two versions of the manifest file. Namely, usage report filenames706-1,708-1,710-1, . . . , and720-1; and usage report filenames706-2,708-2,710-2, . . . , and720-2.

By way of example, the usage report filenames706-1,708-1,710-1, . . . , and720-1point to usage report files723-1with a date “sub-range” of 2018-03-01 to 2018-03-02, or one day. Similarly, and also by way of example, the usage report filenames706-2,708-2,710-2, . . . , and720-2point to usage report files723-2, but with a date “sub-range” of 2018-03-01 to 2018-03-03, or a cumulative two days. Other arrangements could be used, such as storing consecutive one-day sub-ranges. The example names of the usage report files indicate the resource and the sub-range dates. For this example, usage report filenames706-1,708-1,710-1, . . . , and720-1point to usage reports for resources identified as “X,” “Y,” “Z,” and “W,” respectively. The filenames include the resource and sub-range dates, as shown. The usage report filenames706-2,708-2,710-2, . . . , and720-2also point to usage reports for resources identified as “X,” “Y,” “Z,” and “W,” respectively, but with different sub-range dates.

With the example arrangement as shown inFIGS. 7A and 7B, a manifest file may be identified according to dates of a super-range, such as billing cycle dates. Then the assembly ID may be used to determine dates of the sub-range covered by the usage report files listed in the manifest file. In accordance with example embodiments, this information could be used to access the usage report files and to open a streaming connection to stream the detailed usage data from its storage location in the third-party network to one or more databases in the remote network management platform.

As described above, the resource usage files may contain usage measurements metered every hour (or on some other interval) over the course of successive days of a billing cycle. The data may therefore be represented as time series data, which records usage values and/or quantities derived therefrom at each of a series of successive, discrete times (e.g., every hour). The resource usage files may also contain constant or nearly constant data associated with particular sets of time series data, such as data that identify specific resources, account names, customer categories, and so on. Thus a given resource usage file (e.g., associated with a given usage report filename) may include a collection of constant or nearly constant data and an associated collection of time series data that include usage measurements and/or derived quantities at a series of time markers.

In accordance with example embodiments, streaming of third-party resource usage data from a third-party network600to the remote network management platform320may be done in an efficient and flexible manner by separately storing the received time series data and associated constant data in two differently formatted and constructed databases. Specifically, the received time series data may be stored in a specialized database of files structured for time series data, while the constant data may be stored in records of a relational database. For purposes of the discussion herein, the constant or nearly constant data may be referred to as “static data,” and the time series data may be referred to as “metric data.” Accordingly, the database of time series files may be referred to herein as a metric base data base (MBDB), and the database of constant data may be referred to herein as a static records data base (SRDB). It will be appreciated that these terms, while descriptive of time series and constant components of third-party resource usage files, may not necessarily correspond to terms assigned by any given third-party network to describe the data stored in the third-party network.

FIG. 8Adepicts an example architecture involving a managed network300, a third-party network600, and a remote network management platform320, in accordance with example embodiments. As inFIG. 7A, usage data may be stored in third-party resource usage files750in the third-party network600. The managed network may include a proxy server312and a cache612for storing MBDB file IDs, the purpose of which is described below. The remote network management platform may include a computational instance322, an MBDB804, and a SRDB806. The computational instance322may include a remote server802. Other elements and/or components of the managed network300, remote network management platform320, and computational instance322shown, for example inFIG. 3, have been omitted fromFIG. 8Afor the sake of clarity only.

As shown, the proxy server312may be communicatively connected with the third-party resource usage files750, as well as with the cache612. Communicative connections are also shown between the proxy server312and the remote server802, and between the remote server and each of the MBDB804and the SRDB806. As described below, these connections support interactions and coordinated operations between the connected entities in the service of streaming and storage in the remote network management platform320of third-party resource usage data.

FIG. 8Breproduces the architectural components ofFIG. 8A, and adds numbered annotations illustrating example communications and operations, in accordance with example embodiments. The intent ofFIG. 8Bis mainly to identify the various communication paths that are used, and to associate particular operations of the download-streaming process with them. In this vein, however, it is noted that the numbered operationsFIG. 8Bdo not necessarily represent all the steps carried involved in download-streaming of third-party usage data, and the numbering sequence may not necessarily reflect an exact ordering of operations in all instances.

In example operation, download-streaming of third-party resource usage data may be initiated by an end user in the managed network. For example, personnel of the enterprise interested and/or responsible for evaluating billing by a third-party network for resources consumed under some form of service agreement may initiate download-streaming of the usage data as a procedure in preparation for assessment and analysis of the data. As described above, the assessment and analysis may make use of tools and applications also provided by the remote network management platform. During the course of download-streaming of the billing usage data, the enterprise personnel may wish to view download status, such as percentage completion, viewing status and/or error messages, or the like. The personnel may also wish to control the download streaming, such pausing, restarting, and/or canceling one or more download. A third-party network typically does not provide such convenient and flexible download services and operations, if they provide any at all. Accordingly, embodiments described herein support and provide convenient, efficient, and flexible download-streaming operations for obtaining third-party resource usage data, thereby filling a need generally left unmet by third-party networks.

In the operation labeled1, a manifest file is downloaded from the third-party resource usage files750to the proxy server312. As noted, this operation may be carried out in response to, or as part of, an action by an end-user in or of the managed network to initiate the download. The file is then parsed to obtain the assembly ID and to determine the usage report filenames in the downloaded manifest file. The assembly ID may be used to determine if the data associated with this manifest file has already been downloaded. If the associated data have not already been downloaded, or they have but the user indicates that the associated data should be re-downloaded, the processing to set up the download-streaming continues. Otherwise, further processing for download-streaming of these data is skipped.

If processing is not skipped, then the usage reports associated with the usage report filenames are processed to determine identifiers of MBDB files for storing the associated metric data that will be download-streamed from the third-party network. These MBDB file identifiers are referred to herein as MBDB file IDs or just MB file IDs. In an example embodiment, processing of the usage files to determine the MBDB file IDs may be done one file at a time. However, in other example embodiments, processing of the usage files could be done in parallel.

In the operation labeled2, the proxy server312checks which, if any, of the determined MBDB file ID already exists in the cache612. For each MBDB file ID found in the cache612, a time series filed for the metric data already exists in the MBDB804. For each MDBD file ID not found in the cache612, a new MBDB file and file ID needs to be created. The operation labeled2corresponds to either of these alternative actions; that is, retrieving the MB ID from either the cache612or in a transactional query to the remote server802. Note that this operation may be carried out in tandem with each MBDB file ID determination, one at a time, or could be done a batch or parallel manner.

The MB ID may be used to derive or determine links to the MBDB time series file for receiving and recording the metric data of the associated third-party resource usage files. This operation is signified by label3, where the proxy server312communicates with the third-party network to open up one or more download-streaming connections for each of one or more of the third-party resource usage files listed in the downloaded manifest file. In some instances, a download-streaming connection may be opened for each file listed in the manifest file; in other instances, download-streaming connections may be opened for just a subset of the files listed in the manifest file. Multiple download-streaming connections may be opened in series, parallel, or both.

The operation labeled4involves streaming the selected usage files from the third-party resource usage files750to the MBDB804and SRDB805by way of the opened download-streaming connections. As indicated, the path of download-streaming connections is by way of the proxy server312and the remote server802in the computational instance322. Advantageously, the customer of the third-party network (e.g., an enterprise) makes its account information and access to the third-party billing available to the remote network management platform by way of the proxy server312, which already has established connections with one or more remote servers802of the computational instance322.

The operation labeled5involves synchronizing status information relating to the download streaming. In accordance with example embodiments, the synchronization allows an end user to view status of one or more ongoing download, and/or to control one or more ongoing downloads. As shown, the synchronization may involve communications and/or transactions between the remote server802and the proxy server312. Although the label5may suggest that this operation occurs last, it should be noted that synchronization operations could occur at regular and/or user-selected intervals throughout download-streaming of multiple usage files.

It will be appreciated that the operations illustrated inFIG. 8Brepresent just one example of operations that may be carried out in systems and method for downloading third-party resource usage data. Other arrangements of operations are possible as well.

FIG. 9illustrates an example organization and logical associations of data records, in accordance with example embodiments. In particular,FIG. 9illustrates an example relationship between static data and metric data. In accordance with example embodiments, the SRDB806may be configured to store static records, each including static data associated with third-party billing for a specific third-party resource. This represented inFIG. 9by static record903, which includes static data906. By way of example, static data906may be structured to include various data fields including an account ID, a resource ID, and a usage mode, among other possible static data (indicated by ellipses). The resource ID may be an identifier of the specific third-party resource, while the usage mode may be a descriptor of how the resource is (or has been) used by the managed network.

Also in accordance with example embodiments, the resource ID and usage mode may be used jointly (e.g., combined in some algorithmic manner) to derive a link907that points to or is associated with a time series file904in the MBDB804, as indicated. In accordance with example embodiments, the time series file904may include time series data908corresponding to the metric usage data for the specific third-party resource. A conceptual illustration of the time series data (metric data) according to an example embodiment is also shown inFIG. 9.

In particular, the time series data908may correspond to usage measurements or metering by the third-party network an each of a series of periodic times over the course of a billing cycle or some sub-interval thereof. For example, a given time series file904may contain time series data908for a single day of a billing cycle measured or metered every hour. Alternatively, a given time series file904may contain time series data908accumulated over a portion of the billing cycle, such as the first 1, 2, 3, . . . , days of the billing cycle, again measured or metered every hour. Other sub-intervals are possible as well.

The conceptual illustration of example time series data is represented graphically as metered amounts of a resource consumed and/or derived quantities thereof at each of the periodic times. In the figure, resource usage rate for each measurement interval is plotted with crosses, resource usage cost is plotted with triangles, and cumulative cost is plotted with circles. The specific quantities measured and how they are metered may differ depending on the third-party network provider, as well as on the specific resources being tracked or other possible information. In the illustrated conceptual example, usage rate may be measured as a total amount consumed in a given hour, and the cost may then be derived by multiplying the rate by the cost per unit of resource consumed. The cumulative cost may then be determined as a sum of the cost at each measurement time (e.g. each hour).

For example, the rate of virtual machine usage per hour may be calculated as a percentage (or fraction) of each hour that virtual machine cycles are metered as in use divide by the duration (one hour). Cost per hour might be calculated as the metered rate multiplied by the third party's charge for each hour of consumed virtual machine cycles. Other charging models could be used as well by a third-party network. The quantitative description and format of the metric data thus may correspond to the specific form of the resource usage data as defined and stored by the third party network. The specific time series data included in this illustration are shown by way of example, and not intended to be limiting. Other forms of metered usage data that can be represented in time series could be include as well or instead, and any one of the illustrated forms could be omitted.

In accordance with example embodiments, the time series (metric) data stored in the MBDB804may include quantitative values in a numeric format, such as floating point numbers, at each of the measurement times (e.g., per hour). It will be appreciated that floating point number could be stored with varying numerical precision, such as 32-bit, 64-bit, and so on. Other numerical formats could be used for time series data as well.

VI. EXAMPLE OPERATIONS AND METHODS

FIG. 10is a flow chart illustrating of operations1000involved in streaming billing usage data, in accordance with example embodiments.FIG. 11is a flow chart illustrating an example embodiment of a method1100. Both example operations1000and example method1100may be carried out by a computing device, such as computing device100, and/or a cluster of computing devices, such as server cluster200. However, the processes and operations can be carried out by other types of devices or device subsystems. For example, the process could be carried out by a portable computer, such as a laptop or a tablet device. In example embodiments, the operations and method illustrated inFIGS. 10 and 11may be carried out by a computing device disposed within a managed network, such as network300, and/or may be carried out by a computing device disposed within a computational instance, such as instance322, of a remote network management platform, such as platform320, which remotely manages a managed network, such as network300.

Thus, the example operations1000and method1100may be implemented and/or executed by more than one computing device or server. For example, the operations1000and method1100may involve some actions and operations carried out by a computing device in the managed network, such as a proxy server, and may also involve actions and operations of a remote server device in a computational instance. Other actions and operations may involve one or more databases of the remote network management platform as well.

The embodiments ofFIGS. 1000 and 1100may be simplified by the removal of any one or more of the features shown therein. Further, these embodiments may be combined with features, aspects, and/or implementations of any of the previous figures or otherwise described herein.

The operations1000and method1100may be carried out in a system configured for streaming billing data to a remote network management platform from a third-party network. More particularly, the billing data may be associated with usage of resources provided by the third-party network to a managed network. The managed network may be associated with a computational instance of the remote network management platform. The system may include a metric-base database (MBDB) disposed within the remote network management platform and configured for storing time series data. In accordance with example embodiments, the time series data may specify usage quantities of third-party resources consumed by the managed network per unit time over specified time intervals. The system may further include a static records database (SRDB) disposed within the remote network management platform and configured for storing database records that are associated with the time series data. SRDB records may also include data for identifying the third-party resources and for recording static information relating to billing for the third-party resources. The system may also include one or more server devices configured to carry out various operations of the example method1100, possibly as well as other functions and operations not necessarily explicitly described herein.

The example operations1000and method1100may also be embodied as instructions executable by one or more processors of the one or more server devices of the system. For example, the instructions may take the form of software and/or hardware and/or firmware instructions. In an example embodiment, the instructions may be stored on a non-transitory computer readable medium. When executed by one or more processors of the one or more servers, the instructions may cause the one or more servers to carry out various operations of the example method.

The example operations1000ofFIG. 10are described first, followed by a description of the example method1100ofFIG. 11.

Example operation starts at step1002, following which a manifest filename is constructed at step1004. This may involve applying a naming pattern that includes, for example, a date range, such as a billing cycle, and an account name. The operation may be performed by the proxy server or an application on an end user device, in response to an end user initiating a download-streaming operation.

The manifest file is then downloaded to the proxy server from the third party network, and at step1006, the assembly ID is read. At step1008, a determination is made if this assembly ID has already been processed. That is, if the usage data associated with the manifest file have already be download-streamed and stored in the MBDB and SRDB. If the assembly ID has been processed (“Yes” branch from step1008), a completion message indicating completion of operations is issued at step1020, which then ends operations at step1022. Although not explicitly indicated in step1008, a user could choose to re-download data associated with an assembly ID that has already been processed. Allowing for this qualification, the “Yes” branch from step1008could correspond to the assembly ID having already been processed and the user not choosing to re-download the previously-processed data.

If the assembly ID has not be previously processed (“No” branch from step1008), then the usage report filenames are read at step1010, followed by a determination of the total size of all the usage reports at step1012. The sizes may be determined in preparation for receiving the data, for example. Allowing, again, for the user choosing to re-download data associated with an assembly ID that has already been processed, the “No” branch from step1008could correspond to the assembly ID not having already been processed, or the user choosing to re-download previously-processed data.

A loop that download-streams the metric data is then begun at step1014. Specifically, a streaming connection from the third-party network may be opened and download-streaming of the first listed usage file may commence. As noted, streaming from the third-party network to the MBDB is by way of the proxy server in the managed network and remote server in the associated customer instance. Loop control checks if there any remaining usage files from the list of filenames. If there are (“Yes” branch from step1014), then the next usage file is download-streamed to the MBDB at step1016.

The download-streaming step may include or be followed by synchronization and control operations1018, shown with dotted-line block, since this step may not necessarily be executed for every download-streaming operation. Synchronization and control may entail the remote server providing status to the proxy server of the download-streaming process, such as percentage complete. It may also provide an end user with the ability to pause, cancel, and/or restart download-streaming.

If the loop control step1014determines that there are no more usage files to be download-streamed from the filename list (“No” branch from step1014), then a completion message indicating completion of operations is issued at step1020, which then ends operations at step1022.

The example method1100ofFIG. 11is next described.

Block1102may involve the one or more servers determining a resource identifier associating a particular third-party resource with a particular line item in a service bill generated by the third-party network for third-party resources consumed by the managed network.

Block1104may involve, the one or more servers carrying out download-streaming of billing usage data associated with the particular third-party resource from the third-party network to the one or more server devices. The downloading operation may be based, at least in part, on the resource identifier.

Block1106may involve the one or more servers storing the download-streamed billing usage data in a time-series file associated with the particular third-party resource in the MBDB.

Finally, block1108may involve the one or more servers creating and storing in the SRDB a particular static record that includes a link to the time-series file for the particular third-party resource. In particular, creating and storing the SRDB record may be carried out if the SRDB does not already contain a static record associated with the particular third-party resource.

In accordance with example embodiments, the example method may further entail operations and/or steps for download-streaming at least one additional billing usage data file. Specifically, the example method may entail determining a different resource identifier associating a different third-party resource with a different line item in the service bill. Then, based on the different resource identifier, carrying out download-streaming of different billing usage data associated with the different third-party resource from the third-party network to the one or more server devices, and storing the download-streamed different billing usage data in a different time-series file associated with the different third-party resource in the MBDB. If the SRDB does not already contain a static record associated with the different third-party resource, then a different static record may be created and stored in the SRDB. The different static record may include a link to the different time-series file for the different third-party resource.

In further accordance with example embodiments, the service bill may include a billing manifest file covering a specified date range and having a manifest filename common to one or more versions of the billing manifest file. The billing manifest file may include (i) a list of one or more resource-report identifiers, each associated with a respective resource-usage report for a respective third-party resource, and (ii) an assembly identifier unique to a particular version of the billing manifest file. In an example embodiment, each of the one or more versions may correspond to a unique combination of the one or more resource-report identifiers in the list, specific date ranges covered by the respective resource-usage reports associated with the one or more resource-report identifiers in the list, or both. With this arrangement, determining the resource identifier may entail downloading the billing manifest file from the third-party network, and determining from the downloaded billing manifest file a resource-usage identifier associated with a resource-usage report for the particular third-party resource.

In accordance with example embodiments, carrying out download-streaming of the billing usage data based on the resource identifier may entail opening a download-streaming connection from the third-party network for the resource-usage report for the particular third-party resource. The example method may further entail forgoing download-streaming any resource-usage reports associated with resource-usage identifiers included in any given downloaded billing manifest file if the assembly identifier matches that of a previously downloaded and processed billing manifest file. In further accordance with example embodiments, if the assembly identifier matches that of a previously downloaded and processed billing manifest file, one or more of the resource-usage reports may be re-downloaded upon a user indication to do so. That is, automatic skipping of download-streaming of previously-downloaded resource-usage reports may be overridden by a user command or other user preference indication.

In accordance with example embodiments, the method may further entail determining the resource identifier from service bill information downloaded from the third-party network to the proxy server, and determining the MBDB file identifier based at least in part on the resource identifier. Then, if the MBDB file identifier is stored in a local cache associated with the proxy server, the MDBD file identifier may be retrieved from the local cache. If the MBDB file identifier is not stored in a local cache associated with the proxy server, then a request for the MBDB file identifier may be transmitted from the proxy server to the remote server to obtain the MBDB file identifier. The link may be derived from the MBDB file identifier, and the time-series file for storing the download-streamed billing usage may be identified according to the derived link.

In accordance with example embodiments, the example method may further entail monitoring status of the download-streaming of the billing usage data, and transmitting monitored status information to a client computing device communicatively connected in or with the managed network. The status information may include percentage completion, for example. The example method may additionally involve receiving one or more interactive commands from the client computing device to control the download-streaming. The one or more interactive commands may correspond to control actions, such as aborting download, pausing download, and/or restarting download.