Patent Description:
Cloud computing is a computing infrastructure for enabling ubiquitous access to shared pools of servers, storage, computer networks, applications and data resources, which can be rapidly provisioned, often over the Internet.

A data resource in this context may be any form of executable software, data set, or data structure usually, but not exclusively for providing a service, for example a data analysis application, a data transformation application, a report generating application, a machine learning process, a spreadsheet or a database.

Some companies provide cloud computing services for registered customers, for example manufacturing and technology companies, to create, store, manage and execute their own data resources. Sometimes, these data resources may interact with other software resources, for example those provided by the cloud platform provider.

For example, an engine manufacturer may create and store a database relating to spare parts for the different models of engines it produces and services. The database may, for example, be a multi-dimensional relational database. The engine manufacturer may also create one or more data analysis applications for performing certain tasks on data held in the database, for example to analyse and/or transform the data in order to identify trends which may be useful for predicting when certain parts will fail and/or need replacing, and hence when to produce and deploy the spare parts.

The engine manufacturer may create and store one or more other databases, which may or may not be related to the spare parts database. One or more other organisations may store and run their own data resources on the same cloud platform. These other data resources may be partitioned from those of other organisations to ensure data security, confidentiality and integrity.

The cloud platform provider may provide a file system for users, for example external users, to view their own data resources and to perform functions such as data creation, data deletion, data modification, granting or revoking permissions, file viewing, file organisation, file execution, file association, file annotating, adding and/or removing metadata and so on. The file system may have an associated graphical user interface (GUI) which shows files as, for example, a list of selectable items in one or more windows, with associated metadata such as type of file, creation date, last edited date, file size and so on. Metadata is primarily data relating to other data, as will be appreciated. The file system may allow the files to be viewed in hierarchical format indicative of the location of files within higher directories or folders. The file system may also comprise one or more selectable functions which permit files to be viewed, executed and/or functions associated with the file system to be viewed.

The file system may comprise a search tool for users to enter one or more search terms to enable locating information relating to data resources. The search tool may comprise a dedicated text entry box provided as part of the file system GUI. Search terms are processed using a searching algorithm, which can be any known algorithm, against a database of indexed data. Based on the searching algorithm, a list of one or more search results are returned in a search results window.

The conventional method of data indexing is for the platform provider to provide a centralised crawling and indexing system which initiates periodic crawling of its users' data resources for indexing. The conventional method cannot be easily scaled, because each time a new data resource is added to the system, the centralised crawling system needs to be manually updated to address the new data resource. This may take a finite period of time, and introduce delays into the system. It also does not cater well for future types of data which may be stored and implemented on the cloud computing platform. It also places a heavy processing burden on the centralised system which may become slow and/or slow down processing of other services on the cloud computing service.

US Patent Application Publication No. <CIT> discloses methods and systems for providing searchable data associated with enterprise applications. In these disclosed systems and methods, an asynchronous feed may be generated from data stored in a database and searched by search engine crawlers. The feed may be populated searchable object definitions that describe the location of searchable data within the database.

The invention is defined in the appended independent claims.

Advantageous embodiments are defined by the dependent claims.

Example embodiments will now be described by way of non-limiting example with reference to the accompanying drawings, in which:.

In brief, this specification describes methods and systems for data indexing for a search tool, for example a search tool associated with a data processing platform, or for example a cloud-based data processing platform storing and executing multiple data resources for multiple users.

The methods and systems involve providing a predetermined application program interface (API) for implementation in relation to one or more data resources requiring their data to be indexed for subsequent searching using a search tool.

The API is configured such that, when implemented in relation to the data resource, which may be performed by a system associated with the data resource, to send data indicative of one or more definitions of data to be crawled from said data resource to an indexing system of the data processing platform, and to generate an event queue indicative of one or more events associated with said defined data that have occurred in relation to the data resource.

The methods and systems may then identify that one or more data resources have implemented the API, and queries the identified data resources to determine one or more events indicated in their event queue.

The methods and systems may then crawl the one or more identified data resources based on the determined one or more events to obtain data for indexing.

The methods and systems may then index data associated with said one or more events for access by a search tool.

The API, in essence, is provided as a dedicated API which may be exposed to any data resource on a given platform. The data resources may include, for example, any form of executable software, data set, or data structure usually, but not exclusively for providing a service, for example a data analysis application, a data transformation application, a report generating application, a machine learning process, a spreadsheet or a database. The API may be a downloadable stand-alone interface or a web API. The API may be public or private. The API allows data resources to communicate with a centralised crawling function, in accordance with routines and/or protocols of the centralised crawling function defined by the API, informing the centralised crawling function that it wants to be crawled and the type of data that it wants crawled for subsequent indexing and searching. Other data may be specified, as will be explained. The centralised crawling function may thereafter periodically check each identified data resource that implemented the API to determine if events are stored in the event queue. The events need not specify the data itself, but merely enables identification of an event that may be crawled to derive the searchable data for indexing.

In this way, new data resources indicate their readiness for being indexed and searchable simply by implementing the exposed API, which also ensures that routines and/or protocols of the crawling and indexing service are adhered to. No manual updating is needed at the crawling and indexing service.

In the context of the following, the following definitions apply.

A data resource is any form of software, data set, or data structure usually, but not exclusively for providing a service, for example a data analysis application, a data transformation application, a report generating application, a machine learning process, a spreadsheet or a database.

A search service, or search engine, is any form of software function or method by which stored data can be crawled and indexed for subsequent searching by means of a user interface which accepts user inputs, such one or more search terms which may be entered using alphanumeric text, and outputs a list of one or more search results.

Crawling is the process by which search services discover data for indexing, such as any filenames, constituent data, metadata, images, attributes, specifications, author names, dates, files sizes and/or contextual data relating to the data. Crawling is a known method which typically employs crawlers, bots or spiders which follow an algorithmic process to determine what to crawl, and when.

Indexing is the process by which data which is discovered during crawling is stored in one or more databases. Indexing may comprise organising the data in a particular way so that it can be searched, for example by arranging pages of a website in a hierarchical order for determining importance. An indexing service is any processing system which receives crawled data and stores it in an organised or structured way for searching.

A dataset, sometimes used interchangeably with data, holds data on the data processing platform, and usually has an accompanying schema for the dataset in order to make sense, or interpret, the data within the dataset.

A schema defines the structure of the data, e.g. in a dataset or other document such as a website, database or word processing document, spreadsheet, PDF document etc. For example, in a database dataset, the column names of the data set and the type of data in each column may comprise the schema. Sometimes, it is possible to infer or guess the schema, particularly for straightforward datasets or documents.

An event is any occurrence of a change in data, including the creation of new data, deleting of data, modification of existing data, including, but not limited to, a change in file name, adding or changing metadata or schemas etc..

An event queue is a buffer or similar data storage means for recording events in the order in which they occur, e.g. chronologically.

An API is any form of exposed interface which permits computers to communicate with other computers according to predefined rules/protocols. An exposed interface is one that can be accessed without requiring initial permission from the provider of the interface, for example by means of a URL. In the context of this disclosure, an API may be exposed for prompting the input of data indicative of one or more definitions of data to be crawled in relation to a data resource, for subsequently sending to the indexing service and implementation of that API service may expose a predetermined interface to one or more users or user computers for them to define the type of data to be crawled for indexing, and for implementing the user interface which tells a crawling and indexing service that it wants to be crawled and indexed. Implementation may by any known means, for example by user selection of a save, commit or implementation function. The API may be a stand-alone API or a web API, and may be public or private.

The data processing platform for which the data indexing methods and systems may be employed may be an enterprise software platform associated with an enterprise platform provider. An enterprise software platform enables use by multiple users, internal and external to the enterprise platform provider. The users may be users of different respective organisations, such as different commercial companies.

The data resources stored on the software platform may relate to technical data and/or technical processes. For example, an engine manufacturer may create and store a database relating to spare parts for the different models of engines it produces and services. The database may, for example, be a multi-dimensional relational database. Certain analyses may be performed on the database using another application, for example an executable application resource for analysing and/or transforming the data in order to identify trends which may be useful for predicting when certain parts will fail and/or need replacing.

For this purpose, the software platform may comprise enterprise applications for machine-analysis of data resources. For example, an organisation may store on the software platform history data for a machine and use an enterprise application for the processing of history data for the machine in order to determine the probability, or a risk score, of the machine, or a component sub-system of the machine, experiencing a fault during a future interval. The enterprise application may use the fault probabilities or risk scores determined for a machine to select a preventative maintenance task which can reduce the probability and/or severity of the machine experiencing a fault.

History data for a machine may include sensor logs, a sensor log being multiple measurements of physical parameters captured by a sensor and relating to different points in time (a time series). History data for a machine may also include computer readable logs such as maintenance logs, fault logs and message logs corresponding to a machine. The maintenance log corresponding to the machine may record information such as dates and locations of prior maintenance tasks, details of replacement parts, free text notes made by an engineer or mechanic performing a maintenance task and so forth. The fault log corresponding to the machine may record information such as dates and locations of faults, the types of faults, the period of time required to rectify each fault and so forth. The message log corresponding to a machine, such as a ship or construction machinery, may records messages generated by controllers, processors or similar devices which are integrated into the component sub-systems of the machine. The messages may include a date and time, an identifier of a component sub-system, and message content such as, for example, warning information of information identifying a fault.

The above application is mentioned by way of example.

The software platform on which the data resources are stored and executed may be a proprietary or open source platform, which offers advantages in terms of time-to-deploy on the platform provider's hardware, as well as offering partitioning of data and rolling upgrades. An example open source platform is Kubernetes, which is particularly suited for automated deployment, scaling and management of applications. Such software platforms may employ containerised data resources.

In this regard, a containerised data resource comprises "containers" which hold one or more applications, and associated data libraries, that are guaranteed to be co-located on the same host machine and which can share resources.

Such software platforms may also provide a set of primitives which collectively provide mechanisms for deploying, maintaining and scaling applications. A basic scheduling unit may be called a pod. A pod may consist of one or more containers.

The conventional method of data indexing for a search tool is for the data processing platform provider to provide a centralised crawling and indexing system which initiates periodic crawling of its users' data resources, and indexes the crawled data for the subsequent processing of entered search terms. The methodology by which data is crawled and indexed is specified in a crawling and indexing function of the centralised system, which therefore needs to be programmed such that it knows which data resources, e.g. applications, database, services, it needs to crawl, what those data resources are, how the data resources are to be crawled and, in some cases, who can search for data of the particular data resources.

The conventional method is not scalable, because each time a new data resource is added to the system, the crawling and indexing service needs to be manually reprogrammed to identify the new data resource, how to interpret its data, and how to index its data etc. This may take a finite period of time, and introduce delays into the system. It also does not cater well for future types of data which may be stored and implemented on the cloud computing platform. It also places a heavy processing burden on the centralised system which may become slow and/or slow down processing of other services on the cloud computing service.

<FIG> is a network diagram depicting a network system <NUM> comprising a data processing platform <NUM> in communication with a network-based permissioning system <NUM> (hereafter "permissioning system") configured for registering and evaluating access permissions for data resources to which a group of application servers <NUM>-<NUM> share common access, according to an example embodiment. Consistent with some embodiments, the network system <NUM> may employ a client-server architecture, though the present subject matter is, of course, not limited to such an architecture, and could equally well find application in an event-driven, distributed, or peer-to-peer architecture system, for example. Moreover, it shall be appreciated that although the various functional components of the network system <NUM> are discussed in the singular sense, multiple instances of one or more of the various functional components may be employed.

The data processing platform <NUM> includes a group of application servers, specifically, servers <NUM>-<NUM>, which host network applications <NUM> -<NUM>, respectively. The network applications <NUM>-<NUM> hosted by the data processing platform <NUM> may collectively compose an application suite that provides users of the network system <NUM> with a set of related, although independent, functionalities that are accessible by a common interface. For example, the network applications <NUM>-<NUM> may compose a suite of software application tools that can be used to analyse data to develop various insights about the data, and visualize various metrics associated with the data. To further this example, the network application <NUM> may be used to analyse data to develop particular metrics with respect to information included therein, while the network application <NUM> may be used to render graphical representations of such metrics. It shall be appreciated that although <FIG> illustrates the data processing platform <NUM> as including a particular number of servers, the subject matter disclosed herein is not limited to any particular number of servers and in other embodiments, fewer or additional servers and applications may be included.

The applications <NUM>-<NUM> may be associated with a first organisation. One or more other applications (not shown) may be associated with a second, different organisation. These other applications may be provided on one or more of the application servers <NUM>, <NUM>, <NUM> which need not be specific to a particular organisation. Where two or more applications are provided on a common server <NUM>-<NUM> (or host), they may be containerised which as mentioned above enables them to share common functions.

Each of the servers <NUM>-<NUM> may be in communication with the network-based permissioning system <NUM> over a network <NUM> (e.g. the Internet or an intranet). Each of the servers <NUM>-<NUM> are further shown to be in communication with a database server <NUM> that facilitates access to a resource database <NUM> over the network <NUM>, though in other embodiments, the servers <NUM>-<NUM> may access the resource database <NUM> directly, without the need for a separate database server <NUM>. The resource database <NUM> may stores other data resources that may be used by any one of the applications <NUM>-<NUM> hosted by the data processing platform <NUM>.

In other embodiments, one or more of the database server <NUM> and the network-based permissioning system <NUM> may be local to the data processing platform <NUM>; that is, they may be stored in the same location or even on the same server or host as the network applications <NUM>, <NUM>, <NUM>.

As shown, the network system <NUM> also includes a client device <NUM> in communication with the data processing platform <NUM> and the network-based permissioning system <NUM> over the network <NUM>. The client device <NUM> communicates and exchanges data with the data processing platform <NUM>.

The client device <NUM> may be any of a variety of types of devices that include at least a display, a processor, and communication capabilities that provide access to the network <NUM> (e.g., a smart phone, a tablet computer, a personal digital assistant (PDA), a personal navigation device (PND), a handheld computer, a desktop computer, a laptop or netbook, or a wearable computing device), and may be operated by a user (e.g., a person) to exchange data with other components of the network system <NUM> that pertains to various functions and aspects associated with the network system <NUM> and its users. The data exchanged between the client device <NUM> and the data processing platform <NUM> involve user-selected functions available through one or more user interfaces (UIs). The UIs may be specifically associated with a web client (e.g., a browser) or an application <NUM>-<NUM> executing on the client device <NUM> that is in communication with the data processing platform <NUM>. For example, the network-based permissioning system <NUM> provides user interfaces to a user of the client device <NUM> (e.g., by communicating a set of computer-readable instructions to the client device <NUM> that cause the client device <NUM> to display the user interfaces) that allow the user to register policies associated with data resources stored in the resource database <NUM>.

Referring to <FIG>, a block diagram of an exemplary computer system <NUM>, which may comprise the data processing platform <NUM>, one or more of the servers <NUM>-<NUM>, the database server <NUM> and/or the network-based permissioning system <NUM>, consistent with examples of the present specification is shown.

Hardware processor <NUM> can be, for example, a general purpose microprocessor. Hardware processor <NUM> comprises electrical circuitry.

Computer system <NUM> includes a main memory <NUM>, such as a random access memory (RAM) or other dynamic storage device, which is coupled to the bus <NUM> for storing information and instructions to be executed by processor <NUM>. The main memory <NUM> can also be used for storing temporary variables or other intermediate information during execution of instructions by the processor <NUM>. Such instructions, when stored in non-transitory storage media accessible to the processor <NUM>, render the computer system <NUM> into a special-purpose machine that is customized to perform the operations specified in the instructions.

Computer system <NUM> further includes a read only memory (ROM) <NUM> or other static storage device coupled to the bus <NUM> for storing static information and instructions for the processor1 <NUM>. A storage device <NUM>, such as a magnetic disk or optical disk, is provided and coupled to the bus <NUM> for storing information and instructions.

Computer system <NUM> can be coupled via the bus <NUM> to a display <NUM>, such as a cathode ray tube (CRT), liquid crystal display, or touch screen, for displaying information to a user. An input device <NUM>, including alphanumeric and other keys, is coupled to the bus <NUM> for communicating information and command selections to the processor <NUM>. Another type of user input device is cursor control <NUM>, for example using a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to the processor <NUM> and for controlling cursor movement on the display <NUM>. The input device typically has two degrees of freedom in two axes, a first axis (for example, x) and a second axis (for example, y), that allows the device to specify positions in a plane.

Computer system <NUM> can implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system <NUM> to be a special-purpose machine. According to some embodiments, the operations, functionalities, and techniques disclosed herein are performed by computer system <NUM> in response to the processor <NUM> executing one or more sequences of one or more instructions contained in the main memory <NUM>. Such instructions can be read into the main memory <NUM> from another storage medium, such as storage device <NUM>. Execution of the sequences of instructions contained in main memory <NUM> causes the processor <NUM> to perform the process steps described herein. In alternative embodiments, hard-wired circuitry can be used in place of or in combination with software instructions.

The term "storage media" as used herein refers to any non-transitory media that stores data and/or instructions that cause a machine to operate in a specific fashion. Such storage media can comprise non-volatile media and/or volatile media.

Storage media is distinct from, but can be used in conjunction with, transmission media. For example, transmission media includes coaxial cables, copper wire and fibre optics, including the wires that comprise bus <NUM>.

Various forms of media can be involved in carrying one or more sequences of one or more instructions to processor <NUM> for execution. For example, the instructions can initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line or other transmission medium using a modem. A modem local to computer system <NUM> can receive the data on the telephone line or other transmission medium and use an infra-red transmitter to convert the data to an infra-red signal. Bus <NUM> carries the data to the main memory <NUM>, from which the processor <NUM> retrieves and executes the instructions. The instructions received by the main memory <NUM> can optionally be stored on the storage device <NUM> either before or after execution by the processor <NUM>.

Computer system <NUM> also includes a communication interface <NUM> coupled to the bus <NUM>. The communication interface <NUM> provides a two-way data communication coupling to a network link <NUM> that is connected to a local network <NUM>. For example, the communication interface <NUM> can be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface <NUM> can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, the communication interface <NUM> sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

The network link <NUM> typically provides data communication through one or more networks to other data devices. For example, the network link <NUM> can provide a connection through the local network <NUM> to a host computer <NUM> or to data equipment operated by an Internet Service Provider (ISP) <NUM>. The ISP <NUM> in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" <NUM>. The local network <NUM> and internet <NUM> both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link <NUM> and through the communication interface <NUM>, which carry the digital data to and from the computer system <NUM>, are example forms of transmission media.

The computer system <NUM> can send messages and receive data, including program code, through the network(s), network link <NUM> and communication interface <NUM>. For example, a first application server <NUM> may transmit data through the local network <NUM> to a different application server <NUM>, <NUM>.

The data processing platform <NUM> may be a containerised data processing platform.

In this regard, a containerised data platform comprises "containers" which hold one or more applications, and associated data libraries, that are guaranteed to be co-located on the same host machine and which can share resources. Such software platforms may also provide a set of primitives which collectively provide mechanisms for deploying, maintaining and scaling applications. A basic scheduling unit may be called a pod. A pod may consist of one or more containers.

For example, <FIG> is a schematic view of a container <NUM> of a containerised data processing platform. The container <NUM> in this case comprises first to fourth applications <NUM>, <NUM>, <NUM>, <NUM>, each having an associated library of functions, and a kernel <NUM>. One or more of the first to fourth applications <NUM>-<NUM> may form part of a pod. For example, the first to third applications <NUM> - <NUM> may comprise a first pod and the fourth application <NUM> may comprise a second pod. For example, the first pod may comprise a first external organisation's applications, whereas the second pod may comprise a second external organisation's applications. Either way, the first and second pods run in the same kernel, in this example.

In accordance with an example embodiment, and referring now to <FIG>, which shows example architecture <NUM>, one of the application servers <NUM> in the data processing platform shown in <FIG> may comprise a platform search service <NUM>, specific to the data processing platform <NUM>. The platform search service <NUM> may be implemented using software, hardware or a combination of both.

The platform search service <NUM> may have an associated interface <NUM>, for example an API, which may be a stand-alone API or a web API. For the purposes of the foregoing, we assume a web API is employed. A web API is an API for either a web server or a web browser. It exposes one or more endpoints to a defined request-response message system, typically expressed in JavaScript Object Notation (JSON) or Extensible Markup Language (XML), via the web, most commonly by means of an HTTP based web server. The interface <NUM> may effectively be a web-page delivered, when requested, to one or more data resources for them to communicate with the platform search service <NUM> using the routines/protocols defined by the interface.

Associated with the platform search service <NUM> is a proprietary search service <NUM>. The proprietary search service <NUM> may be any off-the-shelf system configured to index data received from the platform search service <NUM>. The proprietary search service <NUM> may also be configured to receive search terms entered using a search user interface <NUM> for providing search results via the platform search service <NUM>. For example, the proprietary search service <NUM> may be Elasticsearch (TM) which is a distributed JSON-based search and analytics engine offered by Elasticsearch Global BV. The algorithms by which the proprietary search service <NUM> indexes and searches data is outside the scope of the present embodiments, which is concerned with providing data for indexing to the proprietary search service.

The interface <NUM>, as mentioned above, may be publicly exposed to data resources on the data processing platform <NUM> shown in <FIG>. More particularly, users of the data resources may, via their own user terminals, which may be external to the data processing platform <NUM>, select to implement the interface <NUM> in relation to one or more of their data resources. This may require authentication and/or authorisation of the user, for example limiting the interface implementation only to particular users associated with the relevant data resource and/or users having particular predefined roles.

For example, as shown in <FIG>, three data resources are shown, comprising a file system <NUM>, a first 'other' data resource <NUM> and second 'other' data resource <NUM>. The file system <NUM> may be the file system of the data processing platform <NUM>, i.e. a windows-based operating system that enables users to view their data resources, their organisation's data resources and to modify data resources by creating, deleting and/or modifying data resources or data within the data resources. These operations may be conditional on the above-mentioned authentication and/or authorisation. The first and second data resources <NUM>, <NUM> may be any other type of data resource, for example any form of executable software, data set, or data structure usually, but not exclusively for providing a service, for example a data analysis application, a data transformation application, a report generating application, a machine learning process, a spreadsheet or a database.

Users wishing to have a new data resource, e.g. the second data resource <NUM>, crawled and indexed by the platform search service <NUM> implement the exposed interface <NUM>. In the context of web APIs, the interface <NUM> may be a web page displayed to the user, requesting identification of the new data resource <NUM> and a definition of the type of data to be crawled and indexed.

The interface <NUM> ensures that the information provided by the user conforms with the routines/protocols required by the platform search service, for example the definition, the type of signalling used and so on. The interface <NUM> may be in the form of a web-page or similar, but this is not essential. By defining the data to be crawled, the platform search service <NUM> can interpret the data in such a way that the data resource can be indexed appropriately.

The interface <NUM> may be implemented for the new data resource <NUM> by applying or sending the data to the platform search service <NUM>, which may be by means of a single-click input or similar.

Once the interface <NUM> is implemented for the new data resource <NUM>, the platform search service <NUM> is configured periodically to query that data resource, e.g. using an automated spider or bot, to acquire data for indexing. That is, the platform search service <NUM> invokes the querying of the new data resource <NUM> and indeed any other of the data resources <NUM>, <NUM> that have implemented the interface <NUM>. Periodic querying may be done on a predetermined basis, e.g. once per day, once per week, or may be randomised.

Further, the interface <NUM> when implemented for the new data resource <NUM>, causes it to generate and update a local event queue <NUM>. The local event queue <NUM> may be a buffer which stores data indicative of events occurring in relation to the data resource <NUM>. An event is any occurrence of a change in data, including the creation of new data, deleting of data, modification of existing data, including, but not limited to, a change in file name, adding or changing metadata or schemas etc..

In <FIG>, each of the data resources <NUM>, <NUM>, <NUM> have associated respective local event queues <NUM>, <NUM>, <NUM>.

When an event occurs, the type of event and what it relates to may be added to the queue. An event queue is a small library for transactionally appending to and reading ordered sequences of items. For example. the queue represents a chronological list of events, and may or may not include date/time information. In some embodiments, the event may simply be a resource locator, pointing to the relevant data, e.g. the new data, the deleted data, the modification of existing data etc..

The platform search service <NUM>, when querying the or each data resource <NUM>, <NUM>, <NUM>, looks at the local event queues <NUM>, <NUM>, <NUM> of each to identify what needs to be crawled for indexable data. In the case of a newly created data resource <NUM>, all data in the local event queue <NUM> is retrieved and subsequently crawled for indexing. In the case of a data resource, e.g. the first data resource <NUM>, the platform search service <NUM> only needs to identify new events that have occurred since the last query operation.

This may be performed by deleting events from the local event queue <NUM>, <NUM>, <NUM>, once identified, so that all events appearing subsequently in said queues are new events. Alternatively, each local event queue <NUM>, <NUM>, <NUM> may maintain a time offset, for identifying to the platform search service <NUM> which events have occurred since the last query operation.

The local event queues <NUM>, <NUM>, <NUM> may be stored locally with the associated data resource <NUM>, <NUM>, <NUM> or may be stored at the platform search service <NUM> location.

Thus, it will be appreciated that a centralised platform search service <NUM> is provided that does not require updating or processing to cater for new data resources <NUM>, <NUM>, <NUM> or changes to data types within existing data resources. Rather, by exposing the interface <NUM>, which defines how data resources or data types are to be defined to the platform search service <NUM> and for configuring local event queue(s), a scalable solution is provided for crawling, indexing and searching which does not require significant resources at the platform search service, and nor does it risk damaging or corrupting the platform search service if improperly used.

In <FIG>, the solid lines represent the indexing operations whereby the data resources <NUM>, <NUM>, <NUM> are crawled for data and indexed, which may use any existing crawling/indexing algorithm and may take place within the platform search service <NUM> or the proprietary search service <NUM>. The dashed lines represent the processing search queries, which are sent from the search UI <NUM> via the platform search service <NUM> to the proprietary search service <NUM>.

In accordance with another example embodiment, and referring now to <FIG>, which shows another example architecture <NUM>, the platform search service <NUM>, the interface <NUM>, the proprietary search service <NUM>, and the search UI <NUM> may be the same as the correspondingly-named elements shown in <FIG>. In this case, the data resources comprise a file system <NUM>, a set of schemas <NUM> and metadata <NUM>. Each of the file system <NUM>, schemas <NUM> and metadata <NUM> represent alternative groupings of data that can be crawled, indexed and searched in a similar manner to that described above.

<FIG> is a flow diagram showing processing operations that may be performed by the platform search service <NUM> when implemented on a processor <NUM> shown in <FIG>. The processing operations may represent program instructions stored on one or more of the ROM <NUM>, main memory <NUM> or ROM <NUM> of, in this example, the server <NUM> of the data processing platform <NUM>. In the foregoing, the numbering of the processing operations is not necessarily indicative of the required order to operations, and re-ordering may be employed. Not every operation may be required in some situations.

A first operation <NUM> comprises providing (or exposing) an interface to one or more data resources.

A second operation <NUM> comprises receiving from the one or more data resources a definition of one or more types of data to be crawled.

A third operation <NUM> comprises identifying one or more data resources having implemented the interface.

A fourth operation <NUM> comprises querying the event queue(s) of the one or more data resources to determine event(s.

A fifth operation <NUM> comprises crawling the one or more identified data resources based on the determined event(s) in the event queue(s.

A sixth operation <NUM> comprises indexing data associated with the event(s) for access by a search tool.

<FIG> is a flow diagram showing processing operations that may be performed by the platform search service <NUM> as part of the operation <NUM> when implemented on a processor <NUM> shown in <FIG>. The processing operations may represent program instructions stored on one or more of the ROM <NUM>, main memory <NUM> or ROM <NUM> of, in this example, the server <NUM> of the data processing platform <NUM>. In the foregoing, the numbering of the processing operations is not necessarily indicative of the required order to operations, and re-ordering may be employed. Not every operation may be required in some situations.

In this case, operation <NUM> may comprise receiving a schema identifying the data to be crawled and the format of that data, so that the platform search service <NUM> may interpret the data to be crawled for indexing.

A first operation <NUM> comprises receiving a schema of data to be crawled.

A second operation <NUM> comprises determining if the schema is already available at the platform search service. In this respect, if the schema is already available locally, then operation <NUM> may comprise getting the schema and operation <NUM> may comprise fetching an API appropriate for that schema. If not, an operation <NUM> may comprise creating a schema mapping in the platform search service <NUM> and an operation <NUM> of fetching an API appropriate for that schema. In this case, the API is appropriate for interpreting the data to be crawled for indexing.

<FIG> is a flow diagram showing processing operations that may be performed by one of the data resources <NUM>, <NUM>, <NUM>. The processing operations may represent program instructions stored on one or more of the ROM <NUM>, main memory <NUM> or ROM <NUM> of, in this example, the server <NUM> of the data processing platform <NUM> or indeed any of the other servers <NUM>, <NUM>, <NUM> that may store the relevant data resources. In the foregoing, the numbering of the processing operations is not necessarily indicative of the required order to operations, and re-ordering may be employed. Not every operation may be required in some situations.

A first operation <NUM> comprises receiving an interface from a platform search service.

A second operation <NUM> comprises providing a definition of the type(s) of data to be crawled to the platform search service via the interface.

A third operation <NUM> comprises implementing the interface. This may be by means of submitting the data provided in the second operation <NUM> or selecting an "apply" or "implement" button on a UI.

A fourth operation <NUM> comprises generating or updating an event queue relating to data to be crawled.

<FIG> is a flow diagram showing processing operations that may be performed by the platform search service <NUM> in response to receiving a query in operation <NUM> of <FIG>. The processing operations may represent program instructions stored on one or more of the ROM <NUM>, main memory <NUM> or ROM <NUM>. In the foregoing, the numbering of the processing operations is not necessarily indicative of the required order to operations, and re-ordering may be employed. Not every operation may be required in some situations.

A first step <NUM> comprises determining when the event queue was last queried.

A second step <NUM> comprises generating an offset indicative of the last query for use by the platform search service.

<FIG> indicates therefore that, rather than data resources pushing notifications to the platform search service <NUM>, as in <FIG>, the platform search service <NUM> may periodically poll the event queue of each data resource.

<FIG> is an example GUI <NUM> of the file system <NUM> which is useful for understanding example embodiments. The GUI <NUM> is presented to registered users of the data processing platform <NUM>, usually after they have entered login details and a password which is verified by the network-based permissioning system <NUM> shown in <FIG>.

The GUI <NUM> comprises an upper menu bar <NUM> comprising of three pull down menus for "home" <NUM>, "files" <NUM>, and "tools" <NUM> and a search UI <NUM> comprising an alphanumeric text entry box. All have their ordinary meaning in the art. A left-hand side <NUM> of the GUI <NUM> comprises a menu for the user to select particular data to view in a main window <NUM>. For example, the left-hand side <NUM> may display "data catalog", "all files", "favourites", "shared" and "trash" all of which have their ordinary meaning in the art. In the main window <NUM> are shown a plurality of data resources <NUM> to which the particular user has access. In this case, the user is presumed to be an employee of an engine manufacturing company and views data resources <NUM> relating to his or her work. These may include any type of data resource, for example databases relating to different types of engine build, schemas defining the format of the engine build databases, and reports relating to analysis results for the engine builds, to give some examples. The main window <NUM> may, in addition to showing the names of the data resources <NUM>, may show metadata such as the last time the relevant data resources were accessed, and the type of resource they comprise, e.g. database, schema, word document, to give some examples. In normal use, the user may select individual data resources to access the underlying data and, if permitted by the network-based permissioning system <NUM>, may edit and/or delete data from within the data resources <NUM>.

The user may select one or more items from the GUI <NUM> by means of a selection tool such as a cursor or arrow <NUM> which is moved in accordance with keyboard input or the movement of a mouse or trackball.

In the shown example, the user may wish to make a newly-created one of the data resources <NUM>, particularly an "Engine <NUM>" database <NUM>, available to the platform search service <NUM> so that data from within the "Engine <NUM>" database can be crawled and indexed for subsequent searching by means of the search UI <NUM>. The user may select the "Engine <NUM>" database <NUM> by clicking or otherwise selecting the relevant item in the GUI <NUM>. In some embodiments, a plurality of such data resources may be selected, for example by holding the shift-key whilst selecting data resources.

In this respect, it may be that a user, e.g. the creator of a data resource may not wish that data resource to be searchable until a later time. This may be because the data contained within the data resource may currently be confidential, sensitive or unverified, for example.

Referring to <FIG>, the user (or any user authorised to do so by the network-based permissioning system <NUM>) may commence the process of making the data resource "Engine <NUM>" <NUM> searchable by first selecting the "tools" item <NUM> from the menu bar <NUM>. The "tools" item <NUM> may comprise a plurality of different selectable tools relating to data resources on the data processing platform <NUM>, one of which is a "make searchable" tool <NUM>. Selection of the "make searchable" tool <NUM> is in this embodiment effective to access the exposed interface <NUM> shown in <FIG>. For example, a URL may be requested from the platform search service <NUM> which corresponds to the exposed interface <NUM>.

Referring to <FIG>, responsive to selection of the "make searchable" tool <NUM>, a web page <NUM> is shown, which may or may not be embedded within the GUI <NUM>. The web page <NUM> represents a web API to the platform search service <NUM> and may be common to all data resources wishing to be made searchable in this context. The web API <NUM> ensures data is sent in the required format for the platform search service <NUM>.

Within the web API <NUM> is provided a number of fields for data entry, one or more of which may be automatically populated.

For example, a first field <NUM> indicates the name of the data resource "Engine <NUM>" which may be automatically populated based on the selection operation shown in <FIG>. Alternatively, a different name may be provided.

For example a second field <NUM> indicates the type of resource "Database" which may be automatically populated based on the selection operation in relation to "Engine <NUM>" which has metadata identifying it as a database.

For example, a third field <NUM> may indicate a schema associated with the "Engine <NUM>" resource. This may be required in order for the platform search service <NUM> to interpret the "Engine <NUM>" database, i.e. to be able to understand what each row and/or column and/or constituent field, amongst other data. For this purpose, a selectable "browse" button <NUM> may be provided for the user to select the relevant path <NUM> to a stored schema corresponding to the "Engine <NUM>" database. Alternatively, the path <NUM> may be manually entered. Alternatively still, in some embodiments, the schema itself may be entered manually.

For example, another part <NUM> of the web API <NUM> may enable the input of indexing preferences. By default, indexing of the selected data resource may comprise indexing the file name and any keywords appearing in the data resource. If additional data is required for indexing, for example metadata associated with the data resource, constituent data of the data resource, or reports deriving from the data resource, then appropriate selection may be made using check boxes as shown.

For example, another part <NUM> of the web API <NUM> may enable the input of search restrictions. By default, users identified as belonging to an organisation by means of the network-based permissioning system <NUM> are able to search all data resources associated with that organisation. However, it may be desirable to impose certain restrictions on what is searchable, depending on the identity of the user, which may be based on their name or their role. The web API <NUM> may enable this. For example, selection of one or more options may permit search results only to be made available to all users (across the data processing platform <NUM>), or only to users having a particular role, or only to a particular subset of users identified by name. Indeed, any method for filtering which users can search the data resource can be employed, and multiple methods can be used in combination. In the shown example, selection of "only users having role(s)" box <NUM> is made, causing a selection box <NUM> to appear in which a subset of predefined user roles can be specified. For example, only administrators, project managers and project <NUM> team roles are permitted to receive search results in the shown example.

Furthermore, within particular role groups, further refinement may be provided by means of selecting the "refine" item <NUM>.

Referring to <FIG>, for example, responsive to selection of the "refine" item <NUM> associated with "Project <NUM> Team" the user is presented with the option of limiting the number of search results to a predefined maximum number, which may be entered in the alphanumeric text box <NUM>. Alternative or additional refinement options may be presented, for example for refining the number of search results for particular members of the "Project <NUM> Team" or applying further restrictions, such as blocking search results for which metadata contains the keyword "Management Only" or "Restricted. " Numerous examples may be conceived in this context.

Selection of an "Apply" button <NUM> will apply the selected refinements to the current web API <NUM> for the current data resource "Engine <NUM>" and returns the GUI shown in Figure <NUM>. Alternatively, selection of the "cancel" button <NUM> will cancel the current selections.

Returning to <FIG>, regardless of what index preferences <NUM> and search restrictions <NUM> are selected, if any, the web API <NUM> for the current data resource "Engine <NUM>" is implemented by selecting the "Apply" button <NUM>.

Responsive to selecting the "Apply" button <NUM>, the submitted information is sent to the platform search service <NUM> which updates a local database with said information and adds the "Engine <NUM>" resource to its list of data resources to be crawled in accordance with its internal crawling and indexing algorithms.

Claim 1:
A method of indexing data for a search tool, wherein the method is performed using one or more processors, the method comprising:
providing an indexing application programming interface (API) for implementation in relation to one or more data resources of a data processing platform requiring data of the data resource to be indexed for searching using a search tool of the data processing platform, the interface being configured, when implemented in relation to a new data resource added to the data processing platform, to send data indicative of one or more definitions of data to be crawled from said new data resource to an indexing system of the data processing platform, and to generate an event queue indicative of one or more events associated with said defined data that have occurred in relation to the data resource;
identifying, by the indexing system, that one or more new data resources have been added to the data processing platform by identifying that the one or more new data resources have implemented the indexing API based on receiving, from each of the one or more identified new data resources, said data indicative of one or more definitions of data to be crawled;
querying the one or more identified new data resources to determine one or more events indicated in their event queue;
crawling the one or more identified new data resources based on the determined one or more events to receive data for indexing; and
indexing the received data associated with said one or more events for access by a search tool.