Asynchronous search of electronic assets via a distributed search engine

Asynchronous search of electronic assets via a distributed search engine is disclosed herein. An example method includes receiving a request from a user, the request including a query and a query time parameter, the query time parameter defining a time that the user will wait for results to be completed synchronously, determining that the query is incomplete and that the time has been exceeded, issuing the query a unique query identifier, and asynchronously adding results to an index based on the unique query identifier.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is set forth regarding the accompanying drawings. The use of the same reference numerals may indicate similar or identical items. Various embodiments may utilize elements and/or components other than those illustrated in the drawings, and some elements and/or components may not be present in various embodiments. Elements and/or components in the figures are not necessarily drawn to scale. Throughout this disclosure, depending on the context, singular and plural terminology may be used interchangeably.

FIG. 1is a schematic diagram of an example environment where aspects of the present disclosure can be practiced.

FIG. 2is a flowchart of an example method of the present disclosure.

FIG. 3is a flowchart of another example method of the present disclosure.

FIG. 4is a schematic diagram of an example computer system that can be used to implement aspects of the present disclosure.

SUMMARY

According to some embodiments, the present disclosure can be directed to a method comprising receiving a request from a user, the request comprising a query and a query time parameter, the query time parameter defining a time that the user will wait for results to be completed synchronously, executing the query, determining query results are incomplete and that the time has been exceeded, returning initial query results of the query results that have been completed within the time, issuing the query a unique query identifier, and asynchronously adding additional results to the portion of the query results based on the unique query identifier. In some instances, the query results (both synchronous and asynchronous can be aggregated and/or reduced into an index).

According to some embodiments, the present disclosure can be directed to a system comprising one or more processors; and one or more memories, the one or more memories storing instructions that can be executed by the one or more processors to receive a request from a user, the request comprising a query and a query time parameter, the query time parameter defining a time that the user will wait for results to be completed synchronously, execute the query, determine query results are incomplete and that the time has been exceeded, return initial query results of the query results that have been completed within the time as an index that is stored in the one or more memories, issue the query a unique query identifier; and asynchronously add additional results to the index based on the unique query identifier.

According to some embodiments, the present disclosure can be directed to a method comprising receiving a request from a user, the request comprising a query and a query time parameter, the query time parameter defining a time that the user will wait for results to be completed synchronously; determining query results are incomplete and that the time has been exceeded; issuing the query a unique query identifier; and asynchronously adding additional results to an index based on the unique query identifier.

DETAILED DESCRIPTION

Overview

The present disclosure pertains to systems and methods for conducting an efficient asynchronous search of electronic assets, such as electronic documents and/or indices. In exemplary embodiments, the asynchronous search allows a client device to submit a search query at one time and receive full or partial results at a later time, without having to keep the communication channel open to wait indefinitely for a large search query to complete. A user can request the search results later with a unique query identification number (QID), and view either incremental or full search results (when available).

With conventional search engines, a search query is run upon request by a user, and the user is unable to take any other actions until the search query has been completed. This is referred to generally as the user being blocked. When searching a large corpus of electronic assets, it can take a substantial amount of time for the search query to be completed, thus locking the user on the search screen unable to take any other actions while waiting for the search to be completed.

In embodiments disclosed herein, a user can view incremental search results, results can be returned to a credentialed user on any client device, and a user can choose to switch between synchronous and asynchronous searching models for various search queries.

In some instances, the systems and methods can be configured to receive a request from a user, where the request includes a query and a query time parameter. Generally, the query time parameter defines a time that the user will wait for results to be completed synchronously.

The systems and methods can execute the query and determine that query results are incomplete and that the time has been exceeded. The systems and methods can optionally return a portion of the query results that have been completed within the time. In some embodiments, the results are added to an index. The systems and methods can also issue the query a unique query identifier based on exceeding the time set forth in the query time parameter.

In various embodiments, the systems and methods can be configured to asynchronously add additional results to the index based on the unique query identifier. That is, as asynchronous results are obtained, they can be added to the synchronously generated results (and/or other asynchronous results) that have already been returned or indexed.

In some embodiments, the systems and methods can be configured to incrementally reduce query results (synchronous and/or asynchronous) to incrementally reduce the size of the index/results. In additional embodiments, incremental results are linked together using the QID established for the asynchronous portions of the query. Access to the QID can be limited to the user(s) who requested the query, in some instances using authentication credentials provided when the user logged into the search engine/service.

FIG. 1illustrates an example architecture100. The architecture100comprises a distributed search engine102, a client104, and various nodes106A-106N. In general, the distributed search engine102can be implemented as a cloud that comprises one or more processors108(e.g., compute resources) and one or more memories110(e.g., storage resources). The distributed search engine102can implement an asynchronous search service112that implements the asynchronous search features described in greater detail herein. For context, when discussing processes/methods performed by the distributed search engine102, this implicitly includes execution of instructions stored in memory by a processor.

Some or all of these components in the architecture100can communicate with one another using a network. The network can include combinations of networks that enable the components in the architecture100to communicate with one another. The network may include any one or a combination of multiple different types of networks, such as cellular, cable, the Internet, wireless networks, and other private and/or public networks. In some instances, the network may include Wi-Fi or Wi-Fi direct.

In some embodiments, the client104can access and use the distributed search engine102using an exposed application programming interface (API114). The distributed search engine102provides near real-time search for any type of electronic data, including but not limited to, structured text, unstructured text, numerical data, geospatial data, metadata, and indices.

The distributed search engine102allows a user to quickly search across large amounts of data. There are situations where a search executes on many index shards, such as shards116A-116N. The search can be executed against possibly frozen indices and spanning multiple remote clusters, for which results are not expected to be returned in milliseconds. When a user needs to execute long-running searches, synchronously waiting for its results to be returned is not ideal.

A user of the client104can submit a search request that gets executed by the search engine asynchronously. The client104can include any end-user computing device such as a personal computer, laptop, server, smart device, as well as any other virtual and/or physical computing device that would be known to one of ordinary skill in the art.

When the user logs in to access the distributed search engine102, the user can be authenticated using credentials, such as an email address and password provided through the API114. When authenticated, these authentication credentials can be associated with the QID. This ensures that only the user(s) who were authenticated and specified the parameters of the query can use the QID.

It will be understood that in some instances a search can query a slower storage resource, or a large amount of data, for which synchronous search may take a long time; hence an asynchronous approach becomes advantageous. In certain situations, fast search is undesirable. Asynchronous search features allow a user flexibility to balance the speed of search execution, the amount of data that can be searched in a single go, and the cost of the hardware to support their use case. That is, asynchronous search allows a user to make a search go slowly to search over massive amounts of data Asynchronous searching also enables querying of data that is stored on less expensive storage, hence slower to be queried.

In these exemplary situations, the API114is adapted accordingly. In an exemplary embodiment, a communication channel cannot be relied upon to be held open for several minutes at the HTTP layer. Thus, the channel is closed and the results are retrieved in a separate channel at a later time, asynchronously.

Asynchronous search lets a user run potentially long-running queries in the background, allowing the user to track their progress and retrieve partial results as they become available. Users can run an asynchronous search query in the background and continue to do other things in the software platform.

Sometimes users need to be able to search through vast amounts of data (potentially many petabytes of data) no matter how long the search takes. To be sure, there are times when long-running queries are desired. Consider regulatory audits or threat hunting scenarios that need to search through years and years of data—data that has, in fact, likely been moved to more economical (cold) storage. These examples can benefit from asynchronous search.

Asynchronous search makes long-running queries feasible and reliable. Asynchronous search allows users to run long-running queries in the background, track the query progress, and retrieve partial results as they become available. Asynchronous search enables users to more easily search vast amounts of data with no more pesky timeouts.

In exemplary embodiments, an asynchronous search query can be first split into multiple smaller searches per index. Then for each index, sub-searches can be conducted per-shard. In this way, millions (could be more or fewer) of documents can be handled and searched. In this example, a shard is a division of the data (e.g., atomic unit), and each of the shards contains a subset of the entire document set.

Because the search engine can split a search query into these multiple sub searches, once a sub search is finished, the response is provided to the user through the API114, and the user can get a partial view of the response before it completes. In one embodiment, the user can initially receive results obtained during the synchronous part of the query. The results can be placed in an index118stored in the one or more memories of the distributed search engine102(can be stored in long-term storage as well).

That is, the user can receive results collected when the client104was connected to the distributed search engine102, and prior to expiration of the time specified in the query time parameter. Using the example above, search results obtained at a time of up to five seconds may be provided to the client104. Those results obtained after five seconds may be aggregated with these initial results in an index that can be stored in memory or long-term, non-volatile storage by the distributed search engine102. As noted above, the distributed search engine102can tie incrementally received search results together using the QID assigned.

Thus, search responses can be provided incrementally to the user. That is, the results from each shard/sub-search are combined with one another as soon as they are available, instead of being combined once at the end when all sub-searches have finished. In this way, incremental search results can be provided to the user. For example, this incrementalism allows the system provides more feedback to end users about the query they are running. From a user interface perspective, users would be able to see some of the results before the query is completed, or a progress bar indicating how long the query took and how long it is expected to take to complete. This also allows the asynchronous search to reduce the memory needed to keep the partial results available.

In some embodiments, a second API can be used by the client for retrieving asynchronous search results which allow a user to monitor the progress of an asynchronous search request and retrieve its results. An ongoing asynchronous search can be deleted through yet another API for deleting an asynchronous search request.

When submitting a search query, the response can comprise the QID of the search being executed. A user can then use this QID to later retrieve the search's final results. Available (partial) search results may be returned as part of the response object.

When the query is no longer running, an indication is provided to the client104as to whether the search failed or was completed on all index shards. While the query is being executed by the distributed search engine102, is_partial parameter may be set to “true” in the API114. Further, the API114provides information such as how many shards the search will be executed on overall, and how many shards have completed. An additional parameter is provided regarding how many documents are currently matching the search query, belonging to the shards that have already completed the search.

Although a query may be no longer running, search results may be partial. This may occur when the search failed after some shards returned their results, or when the node that is coordinating the asynchronous search terminates.

The user can specify that the distributed search engine102can block and wait to return results until the search is completed up to a certain timeout by providing a wait_for_completion_timeout parameter, which defaults to one second. When the asynchronous search completes within such timeout, the response may not include the QID as the results are not stored in the cluster. The keep_on_completion parameter, which defaults to false, can be set to true to request that results are stored for later retrieval also when the search completes within the wait_for_completion_timeout.

A user can also specify how long the asynchronous search and/or results may be made available through the keep_alive parameter, which defaults to five days in some embodiments. Ongoing asynchronous searches and any saved search results can be deleted after this period.

In exemplary embodiments, when the primary sort of the results is an indexed field, shards get sorted based on minimum and maximum value that they hold for that field, hence partial results become available following the sort criteria that was requested. Further, this allows users to obtain quick feedback on more recent data through the partial results, assuming they are sorting by date. This allows the server to prioritize querying the most interesting portion of the data in order to provide feedback to the end user.

The API114may have similar parameters to a synchronous search submission API provided by the distributed search engine102. However, certain default parameter values may differ, such as batched_reduce_size defaults to five and affect how often partial results become available, which happens whenever shard results are reduced, as will be discussed in greater detail infra. The distributed search engine102can perform a partial, incremental reduction when a coordinating node has received a certain number of new shard responses (five by default). A request_cache variable defaults to true, and a pre_filter_shard_size defaults to one to enforce the execution of a pre-filter roundtrip to retrieve statistics from each shard so that the ones that surely don't hold any document matching the query get skipped. A variable ccs_minimize_roundtrips defaults to false.

The get async search API retrieves the results of a previously submitted asynchronous search request given its QID. If the security features are enabled, access to the results of a specific synchronous search may be restricted to the user that submitted it in the first place. In exemplary embodiments, the get async search API depicts whether the search is still being executed or has been completed. When the query is no longer running, indicates whether the search failed or was successfully completed on all shards. While the query is being executed, an is_partial variable can be set to true.

The distributed search engine102further indicates through the API114when the synchronous search will expire, and how many incremental reductions of the results have been performed. If this number increases compared to the last retrieved results, a user can expect additional results included in the search response. The distributed search engine102through the API114further indicates how many shards have executed the query. Note that in order for shard results to be included in the search response, they can be incrementally reduced first. In addition, partial aggregations results are provided, coming from the shards that have already completed the execution of the query. Incremental reduction may be performed iteratively to condense results from multiple shards into a single shard.

The wait_for_completion_timeout parameter can also be provided when calling the get async search API, in order to wait for the search to be completed up until the provided timeout. Final results will be returned if available before the timeout expires, otherwise, the currently available results will be returned once the timeout expires. By default, no timeout is set, meaning that the currently available results may be returned without any additional wait time.

The keep_alive parameter specifies how long the asynchronous search should be available in the cluster. When not specified, the keep_alive set with the corresponding submit, synchronous request will be used. Otherwise, it is possible to override such value and extend the validity of the request. When this period expires, the search, if still running, is canceled. If the search is completed, its saved results may be deleted from memory or non-volatile storage. A user can use a delete async search API to manually delete an asynchronous search by QID. If the search is still running, the search request will be canceled. Otherwise, the saved search results can be deleted by user request.

According to some embodiments, the distributed search engine102can be configured to obtain and evaluate historical data regarding query responsiveness regarding the shards116A-116N (e.g., query execution parameters of the plurality of nodes). For example, the distributed search engine102can evaluate how long it takes an individual node to execute a query of a given size. The distributed search engine102can also track how many searches are currently queued for each shard. The distributed search engine102can utilize these data to determine how long a query/search may take. The distributed search engine102could provide the user with feedback regarding their particular query and query time parameter. By example, the user can specify a time of five seconds for their query (after which results are provided asynchronously). If the distributed search engine102determines that the query is likely to take fifteen seconds based on analysis of the shards, the distributed search engine102can inform the user that waiting an additional period longer than five seconds would result in the user receiving all the results of the query synchronously. In another example use case, users could receive an estimation of how costly a search is before executing the search.

Also, the distributed search engine102can allow users to be given permissions per node and/or shard. For example, a user may be allowed to view, edit, delete, move, or conduct other action(s) relative to a search result, index, or shard. These permissions can be tailored for each user.

FIG. 2is a flow chart of an example method of the present disclosure. The method can include a step202of receiving a request from a user. As noted above, the request includes not only the query that the user desires to execute but also a query time parameter. The query time parameter is a time value that indicates how long a user will wait for results to be completed synchronously.

The method can include a step204of executing the query. This can include separating the query into shards/sub-parts as noted above. The method can also include a step206of determining that the query is incomplete and that the time has been exceeded. As noted above, the query can be executed against a plurality of shards on a plurality of nodes in a simultaneous manner. When the time value has been exceeded, the method can include a step208of returning initial query results that have been completed within the time. Step208is optional in some instances. The user may opt to receive their results once the entirety of the query is completed, combining both the synchronous and asynchronous portions of the query.

In some embodiments, the method can include a step210of issuing the query a unique query identifier when asynchronous searching is needed to complete the query. The method can also include a step212of asynchronously adding additional results based on the unique query identifier. In one example, the initial query results returned during a synchronous portion of the query can be combined with additional results obtained during an asynchronous portion of the query. To be sure, there may be many iterations performed during the asynchronous portion of the query, leading to adding additional results being added in multiple iterations. As noted above, the query results can be added to an index.

The method can include a step214of incrementally reducing the query results in the index to reduce an overall amount of data stored in memory and/or long-term storage. In some embodiments, the method includes receiving authentication credentials for the user and associating authentication credentials for the user with the unique query identifier (QID).

The method can also include receiving a request from a user to review the index and determining that the user is authenticated to access the index based on the authentication credentials.

In one or more embodiments, the method can include identifying, based on data pertaining to prior queries, and/or query execution parameters (e.g., how busy or queued a node is) of the plurality of nodes, as well as determining an estimation as to whether the query can be completed within the time specified in the query time parameter based on the query execution parameters. The method may also include providing the estimate to the user on how costly the query is, or how long it is expected to take.

FIG. 3is a flowchart of another example method of the present disclosure. The method includes a step302of receiving a request from an authenticated user. As noted above, the request comprises a query and a query time parameter. The method can also include a step304of determining that the query is incomplete and that a time has been exceeded. Next, the method includes a step306of issuing the query a unique query identifier to the authenticated user, along with a step308of asynchronously adding results to an index based on the unique query identifier.

FIG. 4is a diagrammatic representation of an example machine in the form of a computer system1, within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In various example embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The computer system1includes a processor or multiple processor(s)5(e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), and a main memory10and static memory15, which communicate with each other via a bus20. The computer system1may further include a video display35(e.g., a liquid crystal display (LCD)). The computer system1may also include an alpha-numeric input device(s)30(e.g., a keyboard), a cursor control device (e.g., a mouse), a voice recognition or biometric verification unit (not shown), a drive unit37(also referred to as disk drive unit), a signal generation device40(e.g., a speaker), and a network interface device45. The computer system1may further include a data encryption module (not shown) to encrypt data.

The drive unit37includes a computer or machine-readable medium50on which is stored one or more sets of instructions and data structures (e.g., instructions55) embodying or utilizing any one or more of the methodologies or functions described herein. The instructions55may also reside, completely or at least partially, within the main memory10and/or within the processor(s)5during execution thereof by the computer system1. The main memory10and the processor(s)5may also constitute machine-readable media.

The instructions55may further be transmitted or received over a network via the network interface device45utilizing any one of a number of well-known transfer protocols (e.g., Hyper Text Transfer Protocol (HTTP)). While the machine-readable medium50is shown in an example embodiment to be a single medium, the term “computer-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) that store the one or more sets of instructions.

The term “computer-readable medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that causes the machine to perform any one or more of the methodologies of the present application, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such a set of instructions. The term “computer-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media, and carrier wave signals. Such media may also include, without limitation, hard disks, flash memory cards, digital video disks, random access memory (RAM), read only memory (ROM), and the like. The example embodiments described herein may be implemented in an operating environment comprising software installed on a computer, in hardware, or in a combination of software and hardware.

The components provided in the computer system1are those typically found in computer systems that may be suitable for use with embodiments of the present disclosure and are intended to represent a broad category of such computer components that are well known in the art. Thus, the computer system1can be a server, minicomputer, mainframe computer, or any other computer system. The computer may also include different bus configurations, networked platforms, multi-processor platforms, and the like. Various operating systems may be used including UNIX, LINUX, WINDOWS, QNX ANDROID, IOS, CHROME, TIZEN, and other suitable operating systems.

In some embodiments, the computer system1may be implemented as a cloud-based computing environment, such as a virtual machine operating within a computing cloud. In other embodiments, the computer system1may itself include a cloud-based computing environment, where the functionalities of the computer system1are executed in a distributed fashion. Thus, the computer system1, when configured as a computing cloud, may include pluralities of computing devices in various forms, as will be described in greater detail below.

The foregoing detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show illustrations in accordance with exemplary embodiments. These example embodiments, which are also referred to herein as “examples,” are described in enough detail to enable those skilled in the art to practice the present subject matter.

The embodiments can be combined, other embodiments can be utilized, or structural, logical, and electrical changes can be made without departing from the scope of what is claimed. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope is defined by the appended claims and their equivalents. In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one. In this document, the term “or” is used to refer to a nonexclusive “or,” such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. Furthermore, all publications, patents, and patent documents referred to in this document are incorporated by reference herein in their entirety, as though individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls.

While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. The descriptions are not intended to limit the scope of the technology to the particular forms set forth herein. Thus, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments. It should be understood that the above description is illustrative and not restrictive.