Single point metadata driven search configuration, indexing and execution

Systems and methods for application search configuration, indexing, and execution. A method includes receiving a metadata definition for search and indexing configuration and generating a transfer mode definition to export objects for indexing. The method includes identifying objects to be indexed according to the metadata definition and extracting data according to the transfer mode definition. The method includes indexing the extracted data. The method can include executing queries according to the meta definition.

TECHNICAL FIELD

The present disclosure is directed, in general, to computer-aided design, visualization, and manufacturing systems, product lifecycle management (“PLM”) systems, and similar systems, that manage data for products and other items (collectively, “Product Data Management” systems or PDM systems).

BACKGROUND OF THE DISCLOSURE

PDM systems manage PLM and other data. Improved systems are desirable.

SUMMARY OF THE DISCLOSURE

Various disclosed embodiments include systems and methods for single-point metadata driven “code-less” automation of application search configuration, indexing, and execution. A method includes receiving a metadata definition for search and indexing, configuration and generating a transfer mode definition to export objects for indexing. The method includes identifying objects to be indexed according to the metadata definition and extracting data according to the transfer mode definition. The method includes indexing the extracted data and can include executing queries.

DETAILED DESCRIPTION

Users expect simplicity and accuracy when searching for data in any application. Enterprise applications, however, are traditionally built on relational databases and the user interaction with these databases requires the user to understand the data model, which makes it cumbersome and difficult to use. To address this need, applications integrate with search engines that index the application data to allow for the expected user experience.

Search engine integrations of enterprise applications have historically been an expensive exercise, and in many cases, the integrations require specific coding to address the needs of an industry or customer. Such integrations are expensive to build, deploy and maintain. Furthermore, solutions did not have enough configurability to allow changes to be made as business requirements evolved, which leads to delays in deploying an enhanced solution.

Indexing application data for search purposes is a non-trivial task as applications can generate terabytes of data which needs to be indexed. Application data models and search engine data models typically tend to be disparate entities. The mapping required in aligning data between applications and search engines can be complex, requiring mappings to be redefined each time the data model changes or different search engines are used.

An indexing process can include, among other processes, identifying the data that needs to be indexed, identifying the properties on objects that the end user can query and filter on, exporting the application data to index into the search engine, converting the exported application data into a specific search engine data model format, and uploading the converted data into the search engine.

At runtime, when a search is performed, the search process can include, among other processes, parsing the user query and translating it to the format the search engine understands, enhancing the query to request facet information (counts etc.) for the properties that the system wants to offer faceting on, executing the search, and returning the matches found and the facet information based on the priority and other criteria that decide the order in which the facets are presented to the user. Facets provide aggregated data based on a search query. Facets correspond to properties of the data being searched, and can be derived by analysis of the text or contents of an item using entity extraction techniques or from pre-existing fields in a database such as author, descriptor, language, and format, or other fields specific to the type of data. Thus, existing web-pages, product descriptions or online collections of articles can be augmented with navigational facets.

The indexing code has knowledge about the types of objects that can be queried for among various object types. When an end-user performs searches, the searches can be across all properties or on specific properties of objects. Users can also filter the results based on property values. This knowledge, about which object properties can be searched on or filtered on, can be integrated into the indexing code. Similarly, to be able to export application data for indexing, the indexing code is able to only export those properties on objects that will actually be indexed. Otherwise, exporting all of an object's data may have an impact on performance. Finally, the exported application data can be mapped to a format that a search engine can recognize and handle.

Until now, enterprise systems, even those that have metadata driven architectures, required manual coding and/or manual configuration in addition to metadata for integrating with search engines. In some systems, the business logic needed to perform these steps is embedded in code by the application developer and makes assumptions about various aspects of the data model, properties, mappings etc. This means that when the search requirements change due to business needs or data model changes or when the need arises to use a different search engine, the code has to be modified to accommodate these changes. This increases the cost of maintaining the solutions and slows down the deployment of improvements as the business learns better ways to model data and present data to their users.

This is a problem for in-house search solutions that businesses develop. For commercial products that are deployed by many customers across different industries, the cost of deployment and ownership grows as these products will need to be customized by writing code for each customer based on their needs. Since the business logic is embedded in code, anytime a different set of objects has to be indexed or the customer wants additional properties to be indexed, there are costs associated with the change. The “code” solution makes it inflexible as different customers will have different requirements when it comes to the data they want to index. Either the original source code of the application has to be changed, which could impact all customers or customers have to develop a customized indexing solution specific to their company's needs. In either case, customers have to wait till a newer version of the indexing solution is deployed, which could take a while in a large production environment. So, there are not only costs associated with change, but the change itself is time dependent and not instantaneous.

Systems and methods disclosed herein provide a single-point metadata driven “code-less” approach to automating the indexing process. This innovative approach allows an administrator of an application to define, prior to deployment and in one place, the object types and the properties on those object types that are relevant to indexing.

This pre-defined metadata can then be leveraged to auto-generate any or all of the following “artifacts” during deployment:The configuration needed to query instances of the object types to be indexed;The property sets that determine which properties on what objects are to be exported when application data is exported;The schema for the exported application data;The search engine specific schema files that correlate to the exported application data; andThe transformer configuration needed to convert the exported application data into specific search engine formats.

Using disclosed techniques, when new object types are added or new properties are added and data pertaining to it needs to be indexed, the user or operator is not required to write or modify any code. The administrator merely has to configure and re-deploy the indexing metadata, and during deployment, any or all of the artifacts listed above is auto-generated by the system. The entire configurability of the indexing process rests in the hands of the application administrator. This is a huge productivity improvement and savings for the customer.

The auto-generated artifacts are used by the indexing process and the search component. Thus, a single metadata definition is used to define, index, and execute a search in a completely “code-less” manner, as described in detail below, that provides such a metadata driven end-to-end solution to configure and execute searches in an application.

FIG. 1depicts a block diagram of a data processing system in which an embodiment can be implemented, for example as a PDM system particularly configured by software or otherwise to perform the processes as described herein, and in particular as each one of a plurality of interconnected and communicating systems as described herein. The data processing system depicted includes a processor102connected to a level two cache/bridge104, which is connected in turn to a local system bus106. Local system bus106may be, for example, a peripheral component interconnect (PCI) architecture bus. Also connected to local system bus in the depicted example are a main memory108and a graphics adapter110. The graphics adapter110may be connected to display111.

Other peripherals, such as local area network (LAN)/Wide Area Network/Wireless (e.g. WiFi) adapter112, may also be connected to local system bus106. Expansion bus interface114connects local system bus106to input/output (I/O) bus116. I/O bus116is connected to keyboard/mouse adapter118, disk controller120, and I/O adapter122. Disk controller120can be connected to a storage126, which can be any suitable machine usable or machine readable storage medium, including but not limited to nonvolatile, hard-coded type mediums such as read only memories (ROMs) or erasable, electrically programmable read only memories (EEPROMs), magnetic tape storage, and user-recordable type mediums such as floppy disks, hard disk drives and compact disk read only memories (CD-ROMs) or digital versatile disks (DVDs), and other known optical, electrical, or magnetic storage devices.

Also connected to I/O bus116in the example shown is audio adapter124, to which speakers (not shown) may be connected for playing sounds. Keyboard/mouse adapter118provides a connection for a pointing device (not shown), such as a mouse, trackball, trackpointer, touchscreen, etc.

Those of ordinary skill in the art will appreciate that the hardware depicted inFIG. 1may vary for particular implementations. For example, other peripheral devices, such as an optical disk drive and the like, also may be used in addition or in place of the hardware depicted. The depicted example is provided for the purpose of explanation only and is not meant to imply architectural limitations with respect to the present disclosure.

LAN/WAN/Wireless adapter112can be connected to a network130(not a part of data processing system100), which can be any public or private data processing system network or combination of networks, as known to those of skill in the art, including the Internet. Data processing system100can communicate over network130with server system140, which is also not part of data processing system100, but can be implemented, for example, as a separate data processing system100.

In the Teamcenter® software products by Siemens Product Lifecycle Management Software Inc., the data model is defined using the Business Modeler Integrated Development Environment (BMIDE). The metadata, which can include the object type definition and properties, is stored in BMIDE templates. Other systems use other specific approaches, and disclosed techniques can be used to create and manage metadata as described herein as adapted for those systems. For consistent reference, these will be referred to as a business modeler (BM) and BM templates, and are not intended to refer to the Siemens Product Lifecycle Management Software Inc. software in particular.

To implement the single-point metadata driven “code-less” approach in specific embodiments, the BM is used to define additional metadata that is necessary to index, search, and filter application data. Each object that needs to be indexed can be tagged with a Business Object constant. Each property that needs to be indexed is tagged with property constants, which help determine if the property is indexed, whether it can be filtered on, whether the values for the property need to be cached in the search engine, and whether there is any additional information required or provided when handling reference-based properties, etc.

The above configuration is persisted in BM templates and allows the administrator to override the configuration that is provided out-of-the-box or define new configurations for custom object types and properties. This allows for a single point of definition using BMIDE templates that can then be used to effortlessly auto-generate the artifacts needed for indexing the application data in a “code-less” manner during deployment.

FIG. 2illustrates metadata definition and model processing for an exemplary embodiment that can be implemented, for example, using Teamcenter® data and the Solr search engine. Solr refers to the Solr open-source search server, but the principles described herein can be applied to other search engines and systems, which are referred to generically as a “search server” below. A data server, as used herein, can be implemented by a Teamcenter® server or other server that performs processes as described.

Note that any of the systems described herein can be implemented as a separate data processing system100, or the functions of multiple systems may be performed by the system physical system.

The following description is one example of how a metadata definition, such as a BM metadata definition200in a BM template202, can be used to configure, index, and execute searches in accordance with disclosed embodiments.

An administrator216or other user identifies the Business Objects and properties on Business Objects to be indexed to build a BM template202, such as objects220. Objects220can each have related properties or parameters. The administrator216deploys the BM template202, which is transmitted to and received by both the BM206and a data server204. As part of the deployment process, the BM206extracts the data model and auto-generates several artifacts. These include a search server schema file208, an XSLT file210, the transfer mode definition212, and the XML schema214, each of which is uploaded and stored in the data server204. Of course, XSLT and XML are examples of suitable files and languages, but other languages could be used.

The search server schema file208can identify properties that need to be in the index, identify the definition source for properties, identify properties on referenced objects that need to be in the index, and contain field definitions in the search server format for data server business objects and properties that need to be indexed.

The XSLT file210maps the data server XML data to a search-server-schema-compliant data format. In most cases, only object types and properties marked for indexing are mapped. Auto-generation of the XSLT file210by the BM206saves time, reduces errors, and eliminates the need to understand XSL, which is particularly advantageous since XSL knowledge generally tends to be limited.

The transfer mode definition212can contain the closure rules and property sets required to export data server business objects and their properties, such as objects220. Auto-generation of the transfer mode definition by the BM206saves significant effort as hand-crafting these is error prone and costly. The transfer mode definition212defines which objects, what properties on those objects, which relations to traverse under what conditions, and which objects should be only traversed and which should be traversed and exported. The closure rules can function as the definition of the group of objects that need to be traversed and exported, while the property sets define what properties are to be exported once an object is selected for exporting.

The data server XML schema214validates that the exported data complies with the server schema prior to indexing.

FIG. 3illustrates an exemplary indexing process in accordance with disclosed embodiments.

During the indexing process, the index orchestrator302can perform several processes, which can be scheduled by or via a scheduler306in some embodiments. In other embodiments, no scheduler306is used. These processes can be performed, for example, by one or more software modules308that have various components as described below.

The index orchestrator302can use a query component310to query data server304for objects320that need to be indexed based on the metadata definition in the BM template. The index orchestrator302can use an extractor312to extract the application data from data server304using the auto-generated transfer mode. The extracted data can correspond to one or more of the objects320, and can include object contents, properties, parameters, or other data.

The index orchestrator302can use a transformer314to transform the extracted data into an appropriate format for search server318using the auto-generated XSLT file and search server schema file; the result is a search-server-schema-compliant data file322. The index orchestrator302can use an index loader316to load the search-server-schema-compliant data file322into the search server318for indexing. The search server318indexes the loaded data.

FIG. 4illustrates an exemplary query execution process in accordance with disclosed embodiments.

During query execution, client system402sends the query, according to the metadata defined in the BM template, to data server404.

Data server404constructs the query in the search server format from the user-provided input, including the metadata definition. This can help eliminate the need for the user to have knowledge about the search server internals. The data server404can add any appropriate query filters.

The data server404can send the query and any filters to the search server418.

Search server418executes the query and constructs any appropriate facet information from the metadata definition to send back to the client402. Search server418sends any results and any facet information back to the data server404, which can then send it to the client402.

These processes illustrate the tremendous value the auto-generation provides for the configuration, indexing, and search execution, among other advantages.

FIG. 5illustrates a process in accordance with disclosed embodiments that can be performed by one or more data processing systems, referred to generically as the “system” below.

The system receives a metadata definition for search and indexing configuration (505). “Receiving,” as used herein, can include loading from storage, receiving from another device or process, receiving via an interaction with a user, or otherwise. This step can include building a BM template according to the metadata definition.

The system generates a transfer mode definition to export objects for indexing (510). The transfer mode definition can include closure rules and property sets required to export data server business objects and their properties. This process can be performed by a BM based on a data model extracted from the BM template. The system can also generate other elements, such as the search server schema file, the XSLT file, and the XML schema described above.

The system identifies objects to be indexed according to the metadata definition (515). This process can include validating the objects and other data to be indexed before exporting them for indexing.

The system extracts data according to the transfer mode definition (520). This data can be application data required by the user, and can be extracted from a data server. This step can also include transforming the extracted data into an appropriate format for the search server using the XSLT file and search server schema file to produce a search-server-schema-compliant data file.

The system loads the extracted data into a search server (525).

The system indexes the extracted data (530). This process can be performed by a separate search server or by a search server application on the same system.

The system can construct a query according to the metadata definition, including adding query filters (535).

The system can execute a query according to the metadata definition (540).

To summarize, benefits, among others, of this single-point metadata driven “code-less” approach include a complete “end-to-end” solution using a single-point metadata driven “code-less” approach. Disclosed embodiments are less error-prone than previous approaches and promote configuration over customization. Techniques as disclosed herein provide faster availability of changes to indexing without having to wait for software updates, and the end user has control over what should be indexed, not the application developer.

This single-point metadata driven “code-less” automation of application search configuration and execution as disclosed herein offer a level of complete automation and configurability, among other advantages.

Disclosed embodiments can identify and configure what needs to be indexed for search purposes directly in the application without having to make changes to the original source code. In some embodiments, artifacts to generate search engine schema, configuration to export application data can be auto-generated directly from the data model metadata definition. Rule-based configurations can be made in a “code-less” manner.

Of course, those of skill in the art will recognize that, unless specifically indicated or required by the sequence of operations, certain steps in the processes described above may be omitted, performed concurrently or sequentially, or performed in a different order.