Patent ID: 12210522

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG.1is a diagram that illustrates an example of a system for enhanced data indexing and searching. A disclosed system may be used to perform data indexing and data searching. In performing data indexing, the system may generate or update an existing search index that can later be used during data searching. The system may generate or update the search index based on metadata extracted from or generated for data that is to be stored. For example, the search index may indicate the structure of various stored data objects, or structures that exist in the stored data objects such as particular graph elements, field elements, or other data elements. In searching for stored data based on a received query, the system may refer to the search index to identify groupings of data objects, or particular portions of data objects that are relevant to the query.

The system100ofFIG.1includes various clients132,134,136,138, a server architecture103, and a gateway130that provides information between the clients and the server architecture103. The server architecture103includes multiple applications servers110and corresponding data repositories102a-102c. The server architecture103also includes a message broker120, a crawler service124, a search service128, a search backend104, and a search index106. The components of the server architecture103are centralized with respect to the multiple application servers110. The various components of the system100may communicate over a network140.

The clients can include the web client132, the workstation client134, the mobile client136, and/or one or more other clients138. The web client132may be, for example a web browser. The workstation client134may be, for example, a computing device such as a desktop computer or a laptop computer. Similarly, the mobile client136may be a mobile computing device such as a smart phone or a tablet. A user may use one of the clients132,134,136, and138to add, remove, or update files of an application server, and, in response, trigger the system100to perform an index update. If however, the user submits a query through one of the clients132,134,136, and138, the system100may, in response, perform a search and generate corresponding search results.

The clients132,134,136, and138can interact with the gateway130over the network140. The network140may include private and/or public networks. The network140may be or include the internet. The network140may be or include a cellular network.

The gateway130is a network gateway that is used to connect the clients132,134,136, and138to the server architecture103. The gateway130may include various protocols for communicating with the different clients132,134,136, and138. The

The application servers110includes multiple servers designed to run applications. The application servers110includes a first application server110a. The application server110aincludes an object server112, a metadata server114, an index manager116, and an indexing engine118. The indexing engine118may be an Elasticsearch indexing engine. The other application servers of the application servers110may also each include similar components to those described specifically with respect to the application server110a.

The Indexing engine118may perform search and indexing functions from a centralized location. That is, the Indexing engine118may be a centralized component of the server architecture103that is (i) not limited to the application server110a, and/or (ii) capable of serving all of the application servers110.

The indexing engine118may be designed to communicate with the search backend104. As an example, the indexing engine118may receive data from the search backend104indicating when the search index106has been updated. The indexing engine118may notify the index manager118when the search index106has been updated. For example, in response to receiving an indication that the search index106has been updated, the indexing engine118may generate a confirmation that indexing is complete and provide this confirmation to the index manager118.

The indexing engine118may perform indexing functions from a centralized location. That is, the indexing engine118may be a centralized component of the server architecture103that is (i) not limited to the application server110a, and/or (ii) capable of serving all of the application servers110. From this centralized location, the indexing engine118may, for example, notify the index manager of each of the application servers110when the search index106has been updated.

The application servers110can generate event messages, such as indexing event messages (e.g., indicating new or modified data that needs to be indexed) and search event messages (e.g., queries). The application servers110can proceed to send event messages to the message broker120and/or directly to their corresponding service. For example, the application servers110may send index event messages to the message broker120, and search event messages directly to the search service128. As will be discussed in more detail below with respect toFIG.3, in generating a search event message (e.g., a query), the application server110amay retrieve corresponding context data from the data repository102a. This context data may be attached and/or sent along with the corresponding search event messages to the search service128(e.g., the message broker120). One benefit of passing the event messages to the message broker120to be subsequently processed by the crawler service124or the search service128(and/or directly the crawler service124and/or the search service128) is that the servers of the application servers110are relieved from managing and interacting with the search backend104. Accordingly, the performance of the application servers110can be improved, and/or the application servers110are free to process other items. This allows event messages to be processed asynchronously.

The data repositories102a-102ccorresponding to the application servers110can include databases, cloud data warehouses, servers, etc. The data repositories102a-102ccan include a wide range of different types of data. For example, the data repositories102a-102ccan store platform analytics data, user libraries, projects, data sets, etc.

The message broker120is an architectural pattern for event message validation, transformation, and routing. As described above, the message broker120may receive event messages from the application servers110. The message broker120may proceed to add received index event messages to an index event queue122. However, search event messages (e.g., queries) may be sent directly to the search service128. That is, search event messages may bypass the message broker120to be immediately processed by a node (e.g., instance) of the search service128.

In some cases, the message broker120receives and adds search event messages to a search event queue. The message broker120may store context data with corresponding search event messages in the search event queue. Alternatively, the message broker120may attach data to the search event messages when adding them to the search event queue. The search service128may use this data to retrieve the context data corresponding to a search event message retrieved from the search event queue.

The crawler service124can retrieve index event messages from the index event queue122, and proceed to process them. The crawler service124may include multiple nodes where each node is able to retrieve and process an index event message from the index event queue122. Each node of the crawler service124may be capable of processing one index event message at a time. After a node of the crawler service124has finished processing an index event message, it may pull the next index event message in the index event queue122to process. The number of nodes of the crawler service124running may depend on the current indexing demand (e.g., based on the number of event messages in the index event queue122) or estimated indexing demand (e.g., based on the current number of users or clients, the time and/or date, the estimated number of users or clients for the time and/or date, etc.). The crawler service124may spin up additional nodes or spin down running nodes depending on the current indexing demand or the estimated indexing demand. For example, if the number of event messages in the index event queue122increases to exceed a predetermined number messages per current node of the crawler service124, the crawler service124may spin up one or more additional nodes to handle the processing of the index event messages.

The crawler service124can identify various content sources to extract information that the search backend104can use to build and/or update the search index106. The content sources may include the data repositories102a-102cand the other sources126, such as web pages, databases, cloud data warehouses, etc. The crawler service124may be used to retrieve particular data objects from a content source, parse through the data objects to extract information, and use the extracted information to categorize the data objects. The crawler service124may provide the data objects, the extracted information, and/or the categorizations and identifications of the corresponding data objects to the search backend104. As an example, the crawler service124may parse through a particular data object retrieved from the data repository102athrough the application server110ato determine that the data object is a document and contains a number of graph elements.

In processing an index event message, a node of the crawler service124may contact the corresponding application servers to retrieve the data (e.g., data objects) to be indexed from the appropriate repository where the data is stored.

The other sources126may include other content sources that the crawler service124can pull content from and, optionally, parse. The other sources126may include web pages, databases, cloud data warehouses, etc. The content of the other sources126can include platform analytics (PA) data. For example, the other sources126may store network usage, status, and/or performance data (e.g., of the network140).

The search service128can receive or retrieve search event messages (e.g., queries) from the application servers110and/or the gateway130. For example, the search service128may receive a request to process a query from the application server110aor directly from the gateway130. In response to receiving a search event message, the search service128may immediately process the search event message. In processing search event messages, the search service128can communicate with the search backend104to provide search queries and receive corresponding search results. Similar to the crawler service124described above, the search service128may include multiple nodes where each node is able to process one or more search event messages. For example, each node (e.g., instance) of the search service128may be able to process a single search event message at a time.

After a node of the search service128has finished processing a search event message, it may indicate that it is ready to receive a new search event message and/or may retrieve a new search event message. The number of nodes of the search service128running may depend on the current searching demand (e.g., based on the number of event messages it has received and/or is currently processing) or estimated searching demand (e.g., based on the current number of users or clients, the time and/or date, the estimated number of users or clients for the time and/or date, etc.). The search service128may spin up additional nodes or spin down running nodes depending on the current searching demand or the estimated searching demand. As an example, the search service128may spin up an additional node if it receives a query to process and all other running nodes are currently busy processing other queries. Similarly, the search service128may spin down one or more nodes if more than a threshold number of nodes are not currently be used to process queries. For example, if more than one of the nodes are not processing any queries, the search service128may spin down all but one of the nodes that are currently not processing any queries.

In some cases, where a search event queue is used, the number of nodes of the search service128running may depend on the on the number of event messages in the search event queue. For example, if the number of event messages in the search event queue increases to exceed a predetermined number messages per current node of the search service128, the search service128may spin up one or more additional nodes to handle the processing of the search event messages.

The search service128may restrict the searching to particular information. For example, the search service128may restrict a searching using a retrieved query to particular projects (e.g., groupings of data objects, groupings of data objects associated with a particular user, groupings of data objects associated with a particular group of users, etc.), to particular types of data objects (e.g., documents, datasets, projects, platform analytics, etc.), to information associated with particular users or groups of users, to information available in particular sources (e.g., limit searching to a particular database), etc. The search service128may use context data associated with a retrieved query to determine how to restrict the information that is to be searched. This context data may be retrieved from one of the data repositories associated with the search event message. The search service128can use the context data to augment the retrieved search event message before providing it to the search backend104.

In some cases, the search service128directly previews the search index106(e.g., without going through the search backend104). For example, the search service128may be able to preview the search index to view the data types available in the search index. The search service128may use the previewed information to augment (or further augment) the retrieved search event message. For example, the search service128may use the previewed information to limit the search to one or more of the data types that are available, thereby making the search more efficient.

After providing a query or a modified query to the search backend104, the search service128may receive corresponding search results from the search backend104. The search results may be in the form of data object identifications, scores (e.g., scores associated with particular data objects that indicate a similarity between the data objects and the corresponding query), and/or the actual data objects. After obtaining search results, the search service128may provide the search results to the gateway130.

In some cases, search event messages are added to a search event queue. The search service128may retrieve search event messages from the search event queue to process using the techniques described above.

The search backend104is a module designed to perform searching and indexing functions. For example, based on index instructions received from the crawler service124, the search backend104can build or update the search index106. Similarly, based on search instructions received from the search service128, the search backend104can use the search index106to obtain search results. The search backend104may be provided by Elasticsearch. Other options include Google Cloud Search, Amazon CloudSearch, Microsoft Azure Search, IBM Watson Discovery, or Apache Solr. Alternatively, the search backend104may be provided by a database server, Oracle database management, MySQL, or the like.

Search index106is a centralized index for all of the application servers110. The search index106may include information that categorizes different data objects of the data repositories102a-102c. As an example, the search index106may include access control data108that includes an index of corresponding documents, datasets, projects, platform analytics, etc. One benefit of using a centralized search index such as the search index106, is that it is the only index that must be regularly updated. This can help prevents situations where applications servers may otherwise rely on an out-of-date search index. This can also improve efficiency as there may otherwise be significant overlap between different search indexes. Accordingly, using a single search index may help to significantly reduce the number of index updates that are ultimately required. Moreover, using a single index can also help to improve search efficiency since only the one search index would need to be used in order to generate comprehensive search results.

In general, for indexing, the application server110adetects when data in the repository102ahas changed, such as when a file is updated or a new file is created. The application server110asends an index event message to the message broker120indicating the data that needs to be indexed. The message broker120stores the index event message in the index event queue122. The crawler service124extract events from the queue122and process them by contacting the corresponding application servers and retrieving the data to be indexed from the appropriate repository where the data is stored. The crawler service124may parse through the retrieved data to extract information for updating the search index106. The crawler service124provide this information to the search backend104, which proceeds to update the search index106. Once the search index106has been updated, the search backend104may send a notification to the Indexing engine118of the application server110aindicating that the search index106has been updated. The Indexing engine118may, in response, generate a confirmation for the application server110athat indicates that the search index106has been successfully updated.

In general, for processing search queries, a client provides a query, which is passed to the gateway130and then to one of the application servers110. As an example, the query may be passed to the application server110a. The application server110asends the query (e.g., a search event message) to the search service128(or, in some cases, to the message broker120) indicating that a search needs to be performed. The search service128starts to process the query. In processing the query, the search service128may retrieve context data from the appropriate repository. The search service128may use this context data to augment the retrieved query, and provide the augmented query to the search backend104. The augmented query may restrict the search to particular data sources, types of data objects, data objects associated with particular users or groups of users, etc. The search backend104performs the search using the augmented query and the search index106, and obtains search results. The search backend104provides the search results to the search service128. The search service128may provide the results to the gateway130, which passes them to the appropriate client.

FIG.2is an example of actions by the system100to issue and process an index event message. Only a subset of the components of the system100are shown for clarity in illustration.

In the example ofFIG.2, in step (A), a client device202makes changes to data of the data repository102a, and, thereby, triggers the issuance of an event message204. Specifically, the client202may modify an existing data object of the data repository102a(e.g., stored in the data repository102a), add a data object to the data repository102a, and/or delete a data object of the data repository102a. Each of these actions may serve as a trigger for the issuance of the event message204.

The client202may be a workstation client or a mobile client. For example, the client202may be the workstation client134or the mobile client136. The client202may be a computing device. For example, the client202may be a desktop computer, a laptop computer, a smart phone, or a tablet.

The data repository102amay include various different types of data objects. For example, the data repository102amay include platform analytics, user libraries, projects, data sets, etc. Accordingly, the detected change to the data in the data repository102amay include a change to one or more of platform analytics data, user library data, project data, data sets, etc.

In some cases, the triggers for the issuance of an index event message are customized. For example, an administrator may customize the triggers for the issuance of an index event message to changes to particular types of data objects, to data objects associated with a particular project, to changes made by a particular user or a group of users, etc.

In step (B), the application server110a, corresponding to the data repository102a, issues the event message204. The application server110amay issue the event message204in response to detecting the change to the data repository102amade by the client202. The event message204may be an index event message as described above with respect toFIG.1that indicates the changed data in the data repository102athat needs to be indexed.

The event message204may include information that can be used to identify and/or retrieve corresponding data in the data repository102a. For example, the event message204may include one or more data objects IDs (or links) associated with the modified data objects in the data repository102a, assigned to data objects added to the data repository102a, or previously associated with data objects removed from the data repository102a.

The event message204may indicate what kind of change to the data repository102ahas been made by the client202. For example, the event message204may indicate that data has been added to the data repository102a, that data has been modified in the data repository102a, and/or that data has been removed from the data repository102a.

In some cases, the event message204includes an indication of the type of data object that has been changed, added, or removed. For example, the event message204may include an indication that a change occurred with respect to platform analytics, a user library, a project, a data set, etc.

In some cases, the event message204includes an indication of the client and/or user that made the change to the data repository102a. For example, the event message204may include an ID for the client202, a user ID for a user using the client202to make the changes to the data repository102a, etc.

In some cases, the event message204includes an indication of the data repository that experienced the change. For example, the event message204may include an ID or a link to the data repository102a.

In some cases, the event message204includes an indication of the application server that issued the event message204. For example, the event message204may include an ID for the application server110a.

In issuing the event message204, the application server110asends the event message204to the message broker120.

In step (C), upon receiving the event message204, the message broker120places the event message204in the index event queue122. In some cases, the message broker120reformats the event message204before placing it in the queue122. For example, the message broker120may reformat the event message204into a format that can be read by the crawler service124or is otherwise used by the crawler service124.

A significant benefit of this approach is that by offloading the processing of the index event messages to the centralized crawler service124, the processing burden on the application servers110can be significantly reduced. Accordingly, the performance of the application servers110can be improved. Additionally or alternatively, the processing resources of the application servers110can be dedicated to the performance of other and/or additional tasks.

The queue122may be loaded with event messages from multiple application servers of the application servers110. For example, all index event messages generated by the application servers110may be sent to the message broker120to be placed in the index event queue122.

In step (D), the crawler service124retrieves event messages to process from the index event queue122. As shown, the crawler service124has retrieved the event message204from the index event queue122. After retrieving the event message204, the crawler service124may proceed to analyze the retrieved event message204to identify corresponding data in the data repository102athat needs to be indexed. For example, in analyzing the event message204, the crawler service124may identify one or more data object IDs.

In step (E), the crawler service124uses the event message204to retrieve corresponding data objects from the data repository102a. For example, the crawler service124may use data object IDs extracted from the event message204to retrieve the corresponding data objects from the data repository102a(or attempt to retrieve corresponding data objects from the data repository102a).

In some cases where the event message204indicates the type of change that happened to the data repository102a, the crawler service124may use the indication in the event message204to determine whether or not data needs to be retrieved from the data repository102a. For example, if the indication provides that a document was deleted from the data repository102a, the crawler service124may not attempt to retrieve the document. Instead, the crawler service124may send the corresponding data object ID (e.g., previously associated with the document) along with an indication that the data object has been deleted to the search backend104.

In retrieving data objects from the data repository102a, the crawler service124may directly communicate with the data repository102ato retrieve the data objects. Additionally or alternatively, the crawler service124may request the corresponding data objects from the appropriate application server of the application servers110. For example, after identifying the data objects corresponding to the event message204, the crawler service124may generate and send a request for the data objects to the application server110a.

In step (F), after retrieving the data objects, the crawler service124uses the data objects to generate a data package206. The data package206may include the retrieved data objects, e.g., in their original format or after being reformatted by the crawler service124. Additionally or alternatively, the data package206may include information extracted from the data objects.

For example, after retrieving the data objects corresponding to the event message204, the crawler service124may parse through the data objects to extract information. The extracted information may include, for example, metadata of the data objects (e.g., size of the data object, project associations, user associations, data object type, creation dates, modified dates, etc.), an indication of data elements in the data object (e.g., graphs, emails, addresses, etc.), keywords in the data objects, etc. The crawler service124may use the extracted information to categorize the data objects and/or parts of the data objects. The crawler service124may generate the data package206using the data objects, the extracted information, and/or the categorization information.

After generating the data package206, the crawler service124may proceed to transmit the data package206to the search backend104.

As discussed above with respect toFIG.1, the crawler service124may include multiple nodes that each process, for example, a single event message at a time. The number of nodes that the crawler service124decides to run may be dynamically determined based on one or more factors. Similarly, determining to spin up or spin down one or more nodes may be dynamically determined based on one or more facts. These factors may include, for example, the number of event messages in the queue122, the time of day, the date, the number of current users, the number of active users, an anticipated number of event messages, an anticipated number of current or active users, etc.

The index event queue122can be a first-in first-out (FIFO) queue. As another example, the index event queue122can be a priority queue, in which the priorities of search requests or search events in the queue are prioritized based on factors such as the user identifier, user role, application, or other element associated with the index request or index event. The application server110amay determine and assign a priority when it notifies the message broker120of the new index update to be performed.

As an example, if the number of event messages in the queue122drops below a certain threshold, the crawler service124may spin down (e.g., suspend) one or more running nodes to improve processing efficiency. Similarly, if the number of event messages in the queue122reaches a certain threshold, the crawler service124may spin up one or more additional nodes to improve processing performance. The threshold may be dynamically determined using, for example, a formula. That formula may include a first variable corresponding to the number of current nodes running and a second variable corresponding to the current, estimated, or anticipated number of event messages in the index event queue122.

Determining to spin up one or more additional nodes or spin down one or more running nodes may depend on an expected demand. For example, the crawler service124may spin up one or more additional nodes if anticipates an increase in event messages based on the date, the time of day, the number of current users, the number of current active users, an anticipated number users, and/or an anticipated number of active users. Similarly, the crawler service124may spin down one or more additional nodes if anticipates a decrease in event messages based on the date, the time of day, the number of current users, the number of current active users, an anticipated number users, and/or an anticipated number of active users.

Once a node of the crawler service124has finished processing an event message, it may retrieve the next event message in the index event queue122. Accordingly, in using multiple nodes, event messages can, at times, be processed asynchronously which can lead to improved performance and/or efficiency. For example, processing an event message that requires the crawler service124to parse through a large amount of content—and, therefore, will require a relatively large amount of processing time when compared to the processing of a typical event message—will not hold up the processing of other, potentially less time consuming, event messages by other nodes of the crawler service124.

In step (G), in response to receiving the data package206, the search backend104extracts the contents of the data package206and uses the contents to update and/or build the search index106. Here, using the content in the data package206, the search backend104updates the search index106to reflect the changes made to the data in the data repository102a.

Where the data package206includes the data objects (e.g., that have been added to the data repository102aor have been updated in the data repository102a), the search backend104may parse through the data objects (e.g., instead of the crawling services124) to extract information that it can use to update and/or build the search index106. The extracted information may include, for example, metadata of the data objects (e.g., size of the data object, project associations, user associations, data object type, creation dates, modified dates, etc.), an indication of data elements in the data object (e.g., graphs), keywords in the data objects, etc. The search backend104may use the extracted information to categorize the data objects and/or parts of the data objects. The search index106may be updated to reflect these categories and the corresponding data objects or corresponding parts of the data objects.

In some cases, parsing the data objects to extract information (e.g., extracted by the crawler service124and/or the search backend104) includes parsing analytics data to get object usage data and telemetry data. The search backend104can add the object usage data and the telemetry data to the search index106. This data may be used during searching, as described in more detail below with respect toFIG.3, to boost search relevance scores for more heavily used data objects.

The search index106describes contents of each of the respective data repositories of the application servers110. For example, the search index106describes the contents of the data repository102afor the application server110a, as well as the contents of other data repositories for the other application servers of the application servers110. That is, the search index106serves as a centrally located search index for all of the application servers of the applications servers110. As such, when a change to any of the corresponding data repositories is made, the search backend104only needs to update the search index106. The search index106may be part of the search backend104.

In an optional step (H), after updating the search index106, the search backend104generates a notification208(e.g., confirmation) and sends the notification208to the application server110a. The notification208may indicate that the search index106has successfully been updated, e.g., in response to the event message204. The search backend104may refrain from transmitting the notification208to any application server other than the application server110a, e.g., because the update was based on the event message204. The other application servers of the application servers110can assume that the search index106is up-to-date unless they are awaiting confirmation for one or more index event messages that they issued. This has the benefit of improving efficiency by not wasting time and resources notifying each of the application servers110.

As step (H) is optional, the search backend104may refrain from generating the notification208or any other notification for the application servers110. That is, with respect to indexing, the system100may use a “fire-and-forget” framework such that, other than generating the index event messages and/or passing them to the message broker120, no further involvement of or management by the application servers110is required. This can improve efficiency as time and resources do not need to be dedicated to the notification of the application servers110when the index106has been updated. Similarly, the application servers110do not spend time and/or resources for processing of index update notifications. Instead, the application servers110can assume that the search index106is up-to-date.

FIG.3shows an example of actions by the system100to receive and process a query. Only a subset of the components of the system100are shown for clarity in illustration.

In the example ofFIG.3, the client device202sends a query302to be processed, and the search service128augments and enhances the query302in order to obtain better results from the search backend104. For example, the search service128can add limitations to the original query302, add or adjust limitations on which types of data or collections of data are searched, enforce access control restrictions either before or after search results are identified, and perform other enhancements as discussed below. The search service128then provides search results for the query to be provided to the client device202.

In further detail, in step (A), the client device202issues a query302to the gateway130. This query can be the result of an explicit query typed or entered by a user, such as a text string entered by the user into a search query field of a user interface. As another example, the query302may be a structured query that the user built or developed using a user interface. As another example, the query302may be generated by an application without a user specifically requesting a query. For example, upon loading a certain page or view of an application, the application may generate and send the query302to obtain data to populate the page or view. Similarly, a user action such as clicking a user interface element to because display or adjustment of a table or chart may similarly cause an application to send a query302to request data to present.

In step (B), the gateway130provides the query302to the appropriate application server110afrom among the set of application servers110. The query302may have an associated server identifier, network address, or other metadata or routing information to specify the appropriate application server110a, for example, to continue a session of communication that is ongoing between the client202and the application server110a. In other cases, the gateway130or a load balancer may select, from among multiple application servers110, a particular application server to handle the query302, even if the messages from the client202do not specify one.

In step (C), the application server110obtains context data304related to the query302. The context data can include data describing factors such as the user, device, time, location, application, task, project, and other factors related to the issuance of the query302.

The application server can acquire the context data304from any of various sources. For example, the application server110acan use information about a current session of communication with the client device202to identify a user identifier for the user, a device identifier for the client device202, and authentication status of the user, and authorization level or set of privileges or permissions available to the user, an application identifier for an application through which the query302was issued, and so on. Information about the user, including an organization, department, role, preferences, history, and so on may be obtained from a user profile for the user or other records. And so on may be obtained from her user profile for the user or other records. Other information including the time that the query302was requested can be determined from the messaging from the client device202to provide the query302. If the user is associated with a certain project or data collection, or has files of the query or data collection open at the time the query is issued, the application server110acan include a project identifier or other data about the software context. In general, the context data can describe the status or state of the client device and the environment in which the query302was issued. This may include a description of a task, workflow, or other software context. These are only a few of the different types of context data304that the application server110may obtain.

In step (D), the application server110anotifies the search service128that a new search should be performed. The application server110amay generate and send a search request or search event to the search service128. This search request or search event may include the query302and the related context data304, or additionally or alternatively may include a reference indicating where the query302and context data304can be obtained. This direct issuance of search requests to the search service128can allow for low-latency and high-bandwidth processing of queries, without the need for application servers to devote resources to managing or tracking the search requests. In some implementations, there is a separate search service instance or search service processing node allocated for each application server110, in order to process the queries forwarded from that application server. In other cases, multiple servers can be assigned to a single search service instance, or potentially there is no specified assignment between servers and search service instances and requests are routed by a management node of the search service128.

Alternatively, the gateway130may notify the search service128that a new search should be performed. The gateway130may pass the query302along to the search service128. Similar to the application server110a, the gateway130may generate and send a search request or search event to the search service128. This search request or search event may include the query302and the related context data304, or additionally or alternatively may include a reference indicating where the query302and context data304can be obtained. For example, in response to receiving the query302from the gateway130, the search service128may access corresponding context data directly from the data repository102aor may request corresponding context data from the application server110a.

Alternatively, it is possible for searches to be processed in a queue, similar to the index event queue122. In this version, the application server110amay notify the message broker120that a new search should be performed, and the message broker120enters the new search into a search event queue. The application server110amay generate and send a search request or search event to the message broker120. This search request or search event may include the query302and the related context data304, or additionally or alternatively may include a reference indicating where the query302and context data304can be obtained.

The search event queue, if one is used, can be a first-in first-out (FIFO) queue. As another example, the search event queue can be a priority queue, in which the priorities of search requests or search events in the queue are prioritized based on factors such as the user identifier, user role, application, or other element associated with the search request or search event. The application server110amay determine and assign a priority when it notifies the message broker120of the new search to be performed.

In optional step (E), if a search event queue is used, the search service128extracts the search request or search event from the search event queue. For example, different search service instances or processing nodes can operate in parallel to concurrently extract and process search requests from the search event queue. One of the search service instances request the next search request from the search event queue, and receives the record for the query302and its accompanying context data304.

In step (F), the search service128accesses additional information to use in processing the query302prior to execution of a search based on the query302. This can include several different aspects including acquiring data indirectly from a search index106, from the data repository102a, from a semantic graph, or from other data sources.

For example, the search service128can receive an identifier for the user associated with the query302. Based on this user identifier, the search service128can request and receive data indicating the privileges of the user, a current authentication level of the user (e.g., whether the user is currently logged in or authenticated), a status of the users credentials a status of the users credentials (e.g., valid, expired, revoked, etc.), and so on. This information will be useful to the search service128as it tailors the query302further to limit the search to the data types and collections that the user is authorized to access. For example, if the user does not have the appropriate credential to access a data collection, the search service128may later append a limitation that excludes that data collection. As another example, if the user does not have the authentication status needed to execute files, then the search service128may add a constraint to exclude executable files from the search results.

The search service128may indirectly acquire data from the search index106through the search backend104. For example, the search service128may request certain information in the search index106from the search backend104after receiving the query302. The search service128may use this information in generating the augmented query306. Additionally or alternatively, the search service128may have previously requested, received, and cached data from the search index106. The search service128may use this previously cached data in generating the augmented query306.

The search service128can also use the user identifier to obtain information about current or previous data access and data used by the user, which can indicate parameters to focus or to weight search results for the current user. Research service may access usage data, such as access logs, from a platform telemetry service.

In some implementations, the search service128accesses data from a semantic graph. The semantic graph can include records for different data objects representing software elements, people, places, things, concepts, and so on. The semantic graph can include data indicating strengths of relationships between the objects, which can be expressed as edge weights or other parameters within the semantic graph. The search service128can use the identifiers in the context data304to identify related objects, topics, keywords, historical usage, and so on to provide a more contextualized or more detailed or expensive interpretation of the query302. For example, the connections in the semantic graph in the semantic graph between a user object for the user and objects representing items mentioned in the query302can be obtained and used to later generate an enhanced query that incorporates more information about the user preferences, history, and likely intention when issuing the query302.

In step (G), the search service generates an augmented query306and provides it to the search backend104. The augmented query306can be generated based on the query302and using the context data304and other information that the search service128obtained in step (F). For example, the search service can use the context data and additional retrieved data to generate a new, more detailed query that may include additional query constraints. For example, the augmented query306may limit the search results further by limiting results for a particular project, data collection, range of time, location, etc. As another example, the search service128may add a constraint or limitation to enforce access control measures, e.g., to exclude one or more data types, data collections, file types, etc. From query processing and query results. As another example, the search service128may expand the scope of the query302by adding keywords determined from context data, search index data, or other data as synonyms (e.g., alternative keywords not in the original query302that may signal a relevant result just as the keywords of the original query302). The augmented query306can include additional limitations added by the search service128, remove one or more limitations of the original query302or replace original limitations of the query302, may change parameters of structured query operations of the query302and so on.

In some cases, the search backend104receives queries through an application programming interface (API) that may not be customizable. The search backend104, even if run locally or by the same enterprise as the application server, may use third-party developed software that requires queries to be in certain formats. The augmented query306can include the various limitations that the search service decides to include, formatted or formulated in the manner required for processing by the search backend104.

In step (H), the search backend104executes a search based on the augmented query306and returns the search results308to the search service128. The search results can include, for example, a list of identifiers for data elements (e.g., files, documents, data sets, data sources, data collections, portions of databases (such as tables, columns, rows, fields, or values), attributes, metrics, etc.), objects registered in the semantic graph or other meta-data repository for the enterprise system (e.g., objects such as locations, users, applications, products, competitors, vendors, clients, etc.), or other types of results.

In step (I), the search service128performs post processing on the results308. This can include filtering the results308do you remove items that the search service determines are not appropriate given access control restrictions for the current user and context.

The search service128provides the results308(potentially as processed in the post processing) for output in response to the original query302. In some implementations, the search service provides the results to the application server110athrough which the query302was provided. This can enable the application server110ato incorporate the results308into its processing, such as to integrate into a user interface or perform application specific processing. In other implementations, as shown inFIG.3, the results308may be provided to the gateway130which may optionally provide the results308directly to the client device202without first forwarding the results308to the application server110a. In some cases, the gateway130may include or be associated with a web server or other frontend server which can merge or otherwise combine search results308with other data from the application server110ato generate output to the client device202.

FIG.4shows a flow diagram that illustrates an example process400for processing search queries and providing results. The process400can be performed by one or more computers, such as one or more servers, workstations, cloud-computing systems, etc.

As discussed above, the arrangement of the search service with respect to the backend provides index management across different application servers, different processor clusters, and across different user projects. A common, shared index is used for multiple application servers, rather than different application server nodes having their own separate search indexes, and so there is no need to synchronize index contexts across. Application servers are relieved of managing and processing indexing and search operations. Application servers can send events to central indexing and search queues and let other processing resources handle the searching and indexing.

The arrangement also enhances scalability of the system. Many application servers send index events and search events which are combined into a shared index event queue and a shared search event queue. From the central queues, different crawler service instances and search service instances can extract items from the queues and process them independently and in parallel. When index event volume is high, additional crawler service instances can be initialized and run concurrently. When index event volume is low, some crawler service instances can be stopped to free up additional resources. Similarly, when search request volume is high, additional search service instances can be initialized and run concurrently. When search event volume is low, the number of search service instances can be decreased.

The search index itself is enhanced with information beyond an indication of the contents of the documents or data in the repository. For example, the search index associates data items with corresponding access control information and permissions, usage data, attributes or metadata, and more. The search index can also index documents and data with a project identifier field to indicate a related project, or with custom-defined fields. These items allow the search service to act as a query pre-processor, adding context, permissions, adjusting the scope of query statements and the collections and data to be searched.

The process400includes providing, by the one or more computers, a search service configured to receive and process queries from a plurality of different application servers (402). Each application server comprises one or more processing nodes and has an associated data repository for the application server. The search index can be a search index for a combined set of data that includes the content of the data repositories for each of the plurality of application servers. The application servers can be any combination of local or on-premises servers, remote servers, cloud-computing-based servers, etc. Each of the data repositories may comprise at least some unique content not present in the data repositories of other application servers. In other implementations, the data repositories may be different copies or portions of a distributed file system. In some cases, the different repositories may be periodically synchronized to include the same contents, but before synchronization is complete the different copies of the file system may be different, at least temporarily, after changes are committed to individual copies.

The search service can be implemented using multiple processing nodes to provide multiple search service instances. These different search service instances can operate in parallel to concurrently process search requests issued by client devices to the different application servers. In some implementations, the system dynamically adjusts the number of processing nodes or a number of search service instances that are active (e.g., scaling up or down) based on a volume of query requests received by the application servers. The system can detect that search demand satisfies a threshold, and in response, increase a number of instances of the search service that are active or increasing an amount of processing resources allocated for the search service. For example, the number of search service instances that are running can vary (e.g., be increased or decreased) based on the number of search requests yet to be processed that remain in a search request queue of the system.

The process400includes receiving, through the search service, a query associated with a user of a client device (404). For example, the search service can receive a query that was forwarded by a particular application server in response to the particular application server receiving the query from a client device.

In some implementations, the system maintains a search request queue to store search requests issued by the different application servers based on queries received by the application servers from client devices. The various instances or processing nodes of the search service can request or extract search request events from the search request queue and process the search requests. This process can be done asynchronously, e.g., in a non-blocking manner, with respect to processing of the application servers and using separate processing resources from the application servers. As a result, the search service can offload the computational demands of managing and carrying out searches, leaving greater capacity for the application servers to handle other tasks.

The search service can receive and use context information corresponding to the query. Application servers may provide this context information that describes, for example: the source of the query (e.g., the geographical location, network location, user, device, organization, department, etc. from which the query was requested); a software context for the query (e.g., an application, user interface, web page, or other software elements through which the query was received); a workflow or task being performed by the user or client device that initiated the query; telemetry data indicating activity, status, or state of a device that provided the query; and so on. Examples of context data include: a user identifier for the user; data indicating a job role of the user; data indicating an organization or department of the user; a device identifier for the client device; a server identifier for a particular application server that received the query from the client device; an application identifier for an application from which the query was initiated; data indicating a device type for the client device; a location of the client device; a time the query was issued; a user interface from which the query was initiated; telemetry data from the client device indicating a state or status of the client device; and data indicating a task, workflow, or project associated with the user.

The search service uses the context information to generate an augmented query based on the query and the context information. The search service then provides the augmented query to the search backend, which allows the search backend to identify more relevant results and better rank the results that are identified. There are many different types of operations that the search service may perform to generate the augmented query. In general, the search service may act as a query pre-processor, modifying, formatting, and reformulating the query for processing by the search backend (e.g., a search engine). Typically, the search service can generate the augmented query by altering a scope of the query based on the context data or adding additional data for ranking results. For example, the search service can add a limitation to the query based on the context information, where the added limitation restricts a scope of search results obtained. For example, the added limitation may narrow the scope to a project that the user is associated with if the context information indicates that the query is provided through a project management interface. As another example, the search service can add attributes to the query, which can be used to adjust a ranking of search results based on the augmented query. The search service can restrict a scope of the query based on access permissions for the user, potentially adding structured query language (SQL) parameters or elements to narrow the scope of results received. For example, if the user's credentials or permissions are not sufficient for executing files or if the context information indicates that the user is not authenticated, elements can be added to the query to exclude executable content. The search service can limit a set of data sources or data collections to be searched. This may be used to exclude data sources or data collections that the user does not have privileges or authentication to access. It may also be used to focus the search to one or more data sources or data collections that are determined to be of highest relevance to the query based on the context information. The search service can alter the query to indicate a level of access for the user based on a current level of authentication of the user.

In addition to or instead of using context information to augment or modify a query, the search service can use information from the search index, a semantic graph defining metadata and relationships among data objects, or other data. For example, the search service can populate or add fields with values that the search service retrieves from the search index or a semantic graph that indicates relationships among objects defined in a centralized metadata repository. The search service can add information based on one or more prior data accesses by the user. The search service can add keywords determined from the search index or the semantic graph.

Before providing the query to the search backend, the search service can perform pre-processing of the query using data access policies, user authentication data, user identity data for the user, or data access permissions of the user. The pre-processing can generate a reformulated query that aligns with requirements of an application programming interface (API) of the search backend. In some cases, the search backend is run by a third party, or includes software designed by a third party, and so may not be customizable. The search service can formulate and adapt the query for the needs of the search backend, translating parameters or statements into a format that the search backend accepts. This relieves the application servers of the need to adjust their query issuing formats and allows the search backend to be changed or replaced with a different version in a way that is transparent to the application servers.

The process400includes providing, by the search service, the query to a search backend having a search index for the plurality of application servers (406). The search index describes contents of each of the respective data repositories of the application servers.

The process400includes receiving, by the search service, search results that the search backend generated in response to the query using the search index (408). The search results can include search results representing data elements (e.g., data items, data sources, documents, files, records, values, etc.) from different data repositories of the different application servers.

The process400includes providing at least some of the search results as an output of the search service provided in response to the query (410). For example, the search service can provide the search results to the particular application server that forwarded the query from a user, e.g., the server that issued a search request event that the search service acted on to initiate the search.

In some implementations, the search service determines whether the user is authorized to access the data represented by the search results. The search service then filters the results to remove results that correspond to data that the user is not authorized to access. The search service provides the filtered results to the application server, which are then forwarded on to the client that issued the query. In some cases, the search service may provide search results directly to the client, bypassing the application server.

The system can also be used to perform indexing of data to update and maintain the search index. For example, the system can maintain an index event queue to store indexing requests issued by the different application servers in response to changes in data in their respective repositories. Multiple data retrieval module instances (e.g., crawler service instances) are configured to operate in parallel. Each data retrieval module instance is configured to: obtain index events from the index event queue; communicate with the application server that issued the index event to obtain the data item corresponding to the index even from the data repository of the application server that issued the index event; and provide the retrieved data to the search engine backend for processing, the search backend being configured to update the index to include information describing the data item.

In some implementations, the system accesses usage data indicating measures of use of different data objects in the repositories of the different application servers. The measures of use can include items such as: a measure of execution time or loading time for the data item; a measure of a number of users that have used the data item; a total accesses made to the data item; and a rate of errors occurring for uses of the data item. The indexing service incorporates the measures of use of the different data objects into the search index. Later, when processing queries, the search service can use the usage data in the index to modify queries, or the search backend can use the measures of use of the different data objects to boost search relevance scores for more heavily used objects. The search relevance scores can be used to select and/or rank the search results for the query.

The indexing of the data repositories of the different application servers can create a search index that provides a single, unified search index across all of the respective data repositories for the different application servers. Many different types of data can be used to enhance the indexing. For example, indexing can include, in the search index, data associated with respective data items that includes at least one of: measures of usage of the data items; access control data that indicates access policies or access restrictions for the data items; identifiers indicating one or more of organizations, departments, users, or projects associated with the data items; subscription data indicating organizations, departments, users, or user roles that are subscribed to receive the data items; and collaboration data indicating sharing of data items or shared use of data items among users.

In some implementations, the search service or other functionality of the system can generate a data package comprising index data from the search index. The search service provides the data package to a client device to enable the client device to perform local, client-side search (e.g., potentially offline or without network access) of at least some of the contents of the data repositories of the application servers using the index data. Embodiments of the invention and all of the functional operations described in this specification may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the invention may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium may be a non-transitory computer readable storage medium, a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus may include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer may be embedded in another device, e.g., a tablet computer, a mobile telephone, a personal digital assistant (PDA), a mobile audio player, a Global Positioning System (GPS) receiver, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the invention may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user may provide input to the computer. Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, speech, or tactile input.

Embodiments of the invention may be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user may interact with an implementation of the invention, or any combination of one or more such back end, middleware, or front end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Although a few implementations have been described in detail above, other modifications are possible. For example, while a client application is described as accessing the delegate(s), in other implementations the delegate(s) may be employed by other applications implemented by one or more processors, such as an application executing on one or more servers. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other actions may be provided, or actions may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a sub combination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.