Document message state management engine

A state management sub-system that assists in transmitting and processing documents and messages between two applications in a sequentially correct order through an integration server is disclosed. The state management subsystem analyzes the message and enters state information about the message into a state management table. Once the message is transformed the state management subsystem check the status of the message in the state management table, and checks all waiting parameters. Depending on the status of the check, the state management subsystem changes the state of message in the table. Only when all messages have passed the waiting parameters check is a message posted to the destination system.

BACKGROUND OF THE INVENTION

The present invention relates to the integration of business applications in an integrated business solutions computing environment. More specifically, the present invention relates to maintaining the order of messages and documents when processed or transformed by an enterprise server.

The current business environment is very different from what it was just a few years ago. Today's organizations embrace the global marketplace, and this dictates a need to be able to efficiently operate at all times. Customers are now more sophisticated which translates into an accelerated pace of business and decision-making processes. Further, business relationships have become highly dynamic, and customers expect businesses to adapt quickly.

Technical and operational challenges abound as well. There is a need to support multiple applications on a variety of platforms, and to integrate with companies using the Internet, extranets, business to business (B2B) exchanges, and other resources. Also, to effectively compete in today's market, there is a need to build new solutions on “Internet time,” utilizing open Internet standards and technology to assure maximum interoperability.

Businesses have typically used a variety of mechanisms to control and analyze business operations such as accounting, payroll, human resources, employee tracking, customer relations tracking, etc. Tools which provide these functions are often implemented using computer software. For example, a software package may manage business accounting, another software package might be responsible for receiving new orders, yet another software package will track warehouse inventory and still another package may handle order fulfillment and shipment. In another example, a business software package operated by one business will need to exchange data with a software package operated by another business to allow a business-to-business transaction to occur.

When business tools are implemented in software, it is not unusual for proprietary software packages to be responsible for each individual business task. However, this implementation is cumbersome and requires the same data to be entered in differing formats among the various business applications. In order to improve efficiency, integration applications have been developed which are used to integrate various elements of one business application with elements of another business application.

For example, if a software package, which is used to obtain new orders, includes data fields (or “entities”) referred to as CustomerNameLast and CustomerNameFirst, it is a relatively straightforward process to map those entries to an accounting software program having the data fields BillingAddressFirst and BillingAddressLast. In such an integration system, the relationship between entities in one system (i.e., computer system or application) and entities in another system can be stored in tables. A system administrator can configure entity mapping between the systems by selecting between the various entities of the two systems.

When an integration system actually executes the transaction between the source application and the destination application as defined by the mapping process, several steps occur. First the source application creates a document or message, which contains data stored as entities, and transmits or publishes that document or message to the integration application. The integration application transforms the document or message according to the transformation processes defined during the mapping. Then the integration application posts the transformed message to the destination application.

However, during the publishing and posting of documents and messages between the source and destination applications several problems often arise that affect the reliability of the system. One notable problem is that messages are published to the integration application in order, but are often processed through the server out of order. The integration application does not provide for the concept of ordered delivery of messages. For example, updates from an application are published in order, however, the integration application often seizes these updates as a batch of multiple messages and this results in the loss of ordering or priority of the messages. Thus, a newer update can process before an older update. This results in the posting in the destination application of out-of-date information.

Another problem that has been observed in integration applications used in a B2B environment is that messages often fail to post successfully, because other information necessary to the successful posting of the message has not been created in the destination system. For example, when a customer places a sales order for a product, certain events must be created prior to the processing of the customer's order. If the customer has chosen a specific method of shipping, this method of shipping must exist in the receiving system for the order to successfully post. If this order is not present in the receiving system, the message will not be able to post, and thus fail. However, if the sales order could have been held and not submitted for posting until the shipping method had been created then the message would have posted successfully.

Another problem that occurs is that messages and documents are published from the source system in an order that makes sense for the source system, but the destination system needs the documents in a different order. This typically occurs when in the source system one entity is a parent entity for another entity, but in the destination system the same entity is a child entity. For example in one system the parent entity is a contact name and the child entity is the business name, whereas in another system the parent entity is the business name and the contact name is the child entity. To integrate documents between the two systems it is desirable to ensure that the parent entity for the destination system posts before the child entity attempts to post.

Therefore, it is desirable to have an integration application or system that takes into account the order in which documents and messages are published to the application, takes into account the relationship between various documents, and considers the relationship between the various entities in the documents and messages when posting the messages to the destination system.

SUMMARY OF THE INVENTION

The present invention includes a state management sub-system that assists in transmitting and processing documents and messages between two applications in a sequentially correct order through an integration server. The integration server includes both messaging services and orchestration services to map and transform the message between the two applications.

Messages are picked up by a receive function from a message queue and processed by a preprocessor. While the message is in the preprocessor, the state management subsystem analyzes the message, and determines when the message was published to the messaging queue, and what messages or entities must exist in the destination system or application. The state management subsystem enters the message into a state management table with a state of submitted. Once the preprocessing is finished the message is returned to the receive function.

The message is then passed from the receive function to the channels, and processed through channels in the integration server. The channels transform the message according to a predefined process. This process transforms the message from a format that is useable by the source system to a format that is useable by the destination system. Once the message is transformed it is delivered to a pipeline.

When the message is in the pipeline the state management subsystem checks the status of the message in a state management table, and verifies that all waiting parameters have been satisfied. If the waiting parameters are not satisfied the message is set to retry, and is resubmitted to the receive function or messaging queues. If the waiting parameters are satisfied the message is set to posting, and the message is allowed to post to the destination system. Following a successful posting of the message the state management system sets the state to posted, otherwise the state is set to failed.

Another feature of the present invention is the set state procedure. This procedure defines the rules for changing the state of the message. The set state procedure checks various conditions stored in the tables of the state management subsystem, such as waiting conditions, ordering conditions, and message states. By checking these states and changing the state of the message only when necessary, the set state procedure ensures that messages are delivered to the destination system in the sequentially correct order. The set state procedure also allows the system to recognize and manage multiple messages going to the same destination system while maintaining the order necessary to reduce the number of messages that fail to post because of improper ordering of the messages.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The present invention deals with managing messaging systems. However, prior to discussing the invention in greater detail, one embodiment of an environment in which the invention can be used will be discussed.

FIG. 2is a block diagram illustrating a network environment in which the present invention can be implemented. Network200includes a source system210, a destination system220, an integration server230, and document transports240and245. Source system210is, in one embodiment, a computer, such as computer110or another system having an application program211that produces documents or messages215. System210can be remote from or local to integration server230. Similarly, system220can be a computer having an application program221that can receive documents or messages215. Documents and messages are transported to and from integration server230via transports240and245. Transports240and245can include hypertext transfer protocol (HTTP), secure hypertext transfer protocol (HTTPS), simple mail transport protocol (SMTP), Microsoft Message Queue (MSMQ) or other known transport protocols. Of course, the transports can be bidirectional but are simply shown in a context in which system210is the source and system220is the destination, by way of example.

Application program211produces a document or message215, such as a sales order which is needed by application program221. Messages are illustratively documents that have additional information attached to them, such as headers or footers which produce information regarding the information contained in the document. However, because application program211and application program221reside on different computers, and may use different formats or structures for data in a document, the document215must be transformed or altered so that application program221can read and understand the incoming document.

Integration server230provides this functionality by converting or transforming the format of the document215from application program211to the format of application program221. Integration server230can be a computer (such as computer110inFIG. 1), a network server or any other information processing system. It should be noted that while only one integration server230is illustrated, systems210and220can be connected through multiple servers230. Further, messages215can be passed through multiple servers230to reach the destination system220.

FIG. 3is a high level illustration of two components of the integration server230ofFIG. 2. In one embodiment integration sever230is the BIZTALK Server from Microsoft Corporation of Redmond Wash. However, the integration sever230can be any other business to business (B2B) server or B2B integration control component or any component for messaging between two systems. The integration server230includes two separate individual server components, a messaging service300and an orchestration engine350. However, other components can be included in server230.

The orchestration engine350is, in one embodiment a COM+ application that manages complex distributed processes that require multiple decisions, loops and actions, such as, for example, processing an insurance policy application. The orchestration engine350is a simplistic graphical means of expressing a business workflow to assist in instructing the integration server230in learning how to transfer a document215from the format of application program211to the format of application program221. The orchestration engine350assists a developer in creating workflow diagrams, or orchestration schedules, through the use of an orchestration designer to visually define and build the business process necessary to convert the formats.

FIG. 4is a block diagram illustrating the components of the orchestration engine350. The orchestration designer400is a tool that facilitates the modeling process. One illustrative embodiment leverages existing graphical design software (such as VISIO 2000 also by Microsoft Corporation or any other graphical design component) to provide a graphical user interface (GUI) to assist the developer in developing a process model for the transformation. The process flow is graphically displayed to the developer as a flow chart463. The orchestration designer400includes process options such as action410, decision420, while430, abort440, fork450and join460. However, additional or different process options can be included in orchestration designer400. Each of these process options allows for a different flow routine to be executed. For purposes of clarity and completeness a brief description of some illustrative examples of these process options is provided below for the sake of discussion.

The action process option410defines a process that typically requires information to be manipulated. For example, it might be an action step that runs the information through a filter, or it might be an action step that reformats data being processed.

The decision process option420defines a process where a yes/no, data query, or other information analysis is conducted, and the result determines which branch to follow to continue with the workflow. It may be determined that if a field is blank to have a branch that enables an action to fill the field with data. However, if the field is not blank, then a branch would allow the process to continue.

The while object process option430allows a process to loop until a specific event occurs. This might be a process that continues until the end of a dataset is reached, or it might be a process that continues until a specific data record is reached, for example. A branch can then be followed to a subsequent decision or action.

The abort object process option440identifies a step where the process needs to be terminated. In many cases, the abort440becomes a fail-safe test that determines that the information received is malformed, signaling data corruption in the transmission or just that incorrect data was transmitted. This step can provide a way to stop a process from continuing with incorrect information

The fork process option450splits a process to conduct simultaneous tasks that can then be joined later through the join process option460. This can, for example, be a process that enables information to be gathered for decisions to be made. For example, a fork in a process can be created to gather price and availability information from multiple vendors. This information can be gathered and processed simultaneously. From this fork, when the information has been analyzed, the system can join back together to continue with the process.

These process options are then linked to each other to generate a flow chart463or flow process for the desired process. The flow chart463is then compiled into a drawing465. This “drawing” is referred to as an XLANG scheduling drawing465. The XLANG scheduling drawing465is a version of flow chart463compiled into XLANG. XLANG is an XML based workflow definition language and is but one example of a definition language that can drive the orchestration engine350.

Once the flow chart463has been converted into XLANG (or other definition language) it is passed to the XLANG scheduler engine470. The XLANG scheduler engine470controls the activation, execution, dehydration, and rehydration of an XLANG schedule465. This process takes the final designed business and technical process and puts it into a format that can be executed by the integration server230.

The XLANG schedule465defines the steps to be performed, the components that must be called, and the data that will pass through the integration server230in order to complete the process generated by the orchestration designer350. When documents or messages215are sent from the XLANG schedule465to the messaging service300, the name of a channel that will receive the documents is defined. After the channel is defined, documents can pass from the messaging service300to an orchestration service480through the channel.

In the orchestration service480, the document215is either processed in XML format for instance, or if the document is not in XML, the XLANG scheduler engine470can embed the document215in the engine's470standard XML wrapper. However, other internal language formats can be used. When inside the XLANG scheduler470, the document215can undergo any modifications defined by the process. After processing the document215, it can then be sent back to messaging service300for posting to a destination, or it can be sent out of the XLANG scheduler470to a private or public queue.

The XLANG scheduler engine470employs two processes to prevent scaling problems caused by multiple XLANG schedules running in parallel over extended periods of time. These two processes are referred to as dehydration and rehydration. Dehydration involves taking a schedule465that is not immediately required and writing the status of the schedule to a status database. As the schedule465is not being processed at that time, more resources are freed. Rehydration involves reloading the schedule465into the main memory of the server with the same status the schedule465had at the time of dehydration.

The orchestration service480includes ports482, which are named locations in the XLANG schedule465. The port482implements a technology, such as COM, MSMQ, or the messaging service, in order to send or receive messages. However, other technologies can be implemented as well. The port482can implement communications either synchronously or asynchronously, and is used to send messages or documents215to, or receive messages from the XLANG schedule465. Orchestration ports482differ from messaging ports used in the messaging service300. Messaging ports will be described in greater detail inFIG. 5.

FIG. 5is a block diagram illustrating the components of messaging service300inFIG. 3. Messaging service300includes messaging objects500, receive functions550, and parsers560. However, other components can be included in the messaging service300such as interchange component551and queue580. Messaging services300are a component in the integration server230that enable the sending, receiving, parsing, and tracking of messages and documents215from outside organizations or from applications, such as application programs211and221inFIG. 2. In addition, messaging services300include the ability to generate receipts for certain file formats, correlate and map data, verify the integrity of documents, and provide secure methods for exchanging documents with outside sources and applications.

Messaging objects500, receive functions550, component object models (COM) and COM+ methods, parsers560, and SQL server databases (such as, for example, Microsoft SQL Server version 7.0 with SP2), can be used to implement the messaging services300. Messaging objects500are used to configure the necessary properties to process and transmit documents submitted to the integration server230. Receive functions550, and in some instances other methods, are used to submit incoming documents or messages215to the integration server230for processing. Once a document or messages215is submitted, the appropriate parser560parses the document and, if necessary, converts the document to another format (such as an XML format). Finally a tracking database561stores document records for both incoming and outgoing documents215that are processed by the integration server230.

Messaging objects500include channels510, messaging ports515, distribution lists520, organizations525, document definitions530, and envelopes535. However, other components can also be included. The integration server230uses these objects500to configure the necessary properties to process and transmit submitted documents215or messages. For purposes of this discussion the terms documents and messages are used interchangeably for data that is submitted to the integration server230for processing. The primary difference between documents and messages is that messages contain additional information such as headers and footers that are not present in documents.

Channel510is a named entity that is used as a reception gateway for documents and messages from either a previously known originating organization or from an open messaging source. An open messaging source is a generic channel that is configured to identify the origin of the message from the content of the message. In one embodiment, channel510is bound to a single messaging port515. Whereas, a messaging port515can have an unlimited number of channels510assigned to it.

A channel510includes a set of properties, which identifies the source organization or application that has sent out the document or message. The channel510also defines the specific steps that are performed by the integration server230before the document is delivered to the associated messaging port515. The channel properties can include a source organization or application, a document definition, a transformation map, field and document tracking settings, and archiving information. Only properly formatted and verifiable data is sent through channel510. In order to ensure this, the channel510defines the format in which it expects to receive messages, and also defines how this data will be forwarded through to the associated messaging port515. Both the inbound and outbound document definitions are references to document specifications that are expressed in a format that is understandable by the integration server230. When a document215is submitted to the channel510, it is verified against the inbound specification. If the document is not compatible with the required format for the channel510, the data is rejected and the message is placed in a suspended queue. The queues580will be discussed in greater detail below.

When a document215is in the channel510the document215can illustratively be altered in two ways by transformation or translation. The document215is altered such that the definition of the document, and the data between records and fields for the source document specification, and the records and fields for destination document specification are correlated. This process of correlating information from one input format to another goes through one of two different processes. The integration server230can alter the schema of the information in a process that is said to transform the information (or be a transformation), or the integration server230can actually alter the data itself in a process that is said to be a translation of the information.

In one embodiment, the transformation process used by the integration server230is handled by script coding using functoids. A functoid is a reusable function built-in to the orchestration service engine480that enables simple as well as complex structural manipulation between source and destination specifications. A transformation of information is common because input and output formats frequently do not match, and need to be altered so that fields from one format match the fields of the other format during the transformation process. The translation process, although by definition alters the data in the document215, does not necessarily delete the information and replace it with completely different data, but instead the information may be analyzed and modified for a new or different format. For example, data translation can replace a “State” definition such as “Calif” with a standard two-letter “CA” postal format. This transformation of the documents occurs in the channel510as the document215is passed through the messaging service300.

Messaging ports515are a set of properties that specify how a message or document215is transported to a destination organization or application. Messaging port properties can include transport services, destination organization or application, security settings, and envelope settings.

Each messaging port515has a primary transport516and an optional backup transport517. These transports516and517are used for binding communications endpoints that link to a remote organization or to an application. The messaging ports515provide a map between an abstract addressing scheme of organization identifiers and transport dependent addresses through channels510. Hence, a multitude of messaging ports515can exist for a single destination organization or application.

The primary and back-up transports516and517are push services that use transport protocols supported by integration server230to deliver documents or messages215to the destination application. These transport protocols can include hypertext transfer protocol (HTTP), secure hypertext transfer protocol (HTTPS), simple mail transport protocol (SMTP), Microsoft Message Queue (MSMQ) or other transport protocols. Further, special transport protocols can be used such as Application Integration Components (AIC), which allow documents and messages215to be submitted directly, or to be carried over proprietary protocols.

When a document215is sent to the destination application221identified by the messaging port515, the integration server230uses the primary transport516. The back-up transport517is used only when there is a transport level error, such as an unreachable or crashed server when multiple servers are used. Therefore, the primary and back-up transports516and517generally use the same application protocols, but point to different channels515to increase robustness of the system.

Distribution lists520are a group of messaging ports515. Distribution lists520are used to send the same document215to more than one organization or application. The distribution list520is sometimes referred to as a port group.

Organizations525are trading partners or other entities with whom the integration server230exchanges messages and documents. These organizations can be internal to the server, such as an application or division within the host organization, or can be external to the server, such as an external business or an application at that external business.

Document definitions530are a set of properties that represents an inbound or an outbound document215. The document definition530can include a property that provides a pointer to a document specification. A document specification531defines the document structure, document type, and version. However, a pointer from the document definition530to the document specification531is not required.

One or more documents215can be packaged together as an interchange, which is submitted to the integration server230. The interchange can be a single document along with header and footer information. However, the interchange can also involve multiple documents having header and footer information. As headers and footers are used in the interchange, the messaging service300needs to be able to identify where in the interchange the document begins and ends. Depending on the file format the messaging service300will need to know how the header and footer information is structured, and hence where the document215is found in the interchange. The messaging service300receives this information through the use of envelopes535.

Envelopes535are a set of properties that represent the transport information for a document. The envelope535wraps data for transport and selects the destination of the data. Envelopes535include headers for messages and function groups. However, other types of headers can be included in the envelope535. In one embodiment, the messaging service300uses six types of envelopes535. These are X12, EDIFACT, Flat-File, Custom, Reliable, and CustomXML. However, other or additional types of envelopes can be used. The type of envelope535used to wrap the document215tells the messaging service300which parser560should be used to process the incoming document. An envelope535associated with an inbound message or document provides the integration server230with the information that the server needs to interpret the submitted document. For example, the envelope535can include a pointer to a document definition530. Envelopes535associated with an outbound message or document give an outside server the information it needs to create the document.

Inbound messages do not necessarily require an envelope535to accompany the document215. Envelopes535are required only for documents that are submitted to the messaging service300that are in a format that the messaging service300cannot able to readily recognize. However, for outbound documents messages an envelope535may be required. The envelope535is created during the configuration of the outbound messaging port515. This allows the messaging service300, through a serializer, to build a complete outgoing message including the header and footer information, so that when the document215is submitted to another messaging service it is able to identify the type of documents being submitted, regardless of the internal format used by the destination server.

Message and documents are submitted into the integration server230through an interchange component551or a receive function550. The interchange component551is configured to support transactions. As a result, the interchange component551adopts the characteristics of the application that is invoking the interchange component551. Once a message or document is submitted, the interchange component551selects the appropriate parser560in integration server230based on information in the envelope535to parse the message or document, unless the message or document is submitted with a pass-through flag enabled. The parser560converts the document215contained in the message from its native format to a format that is understandable by the messaging service of the server. In one example, if the integration server230is configured to operate in an XML format, the parser does not parse documents that are submitted in XML. The integration server230also does not parse messages and documents submitted with the pass-through flag enabled, but passes the document on to the destination in the format it was submitted.

Receive functions550are components of the integration server230that monitor either a message queue580or a directory in a file system directory. When a message arrives in the queue580or a file is placed in the directory, the receive function550takes the file or message and submits it directly for processing by the messaging service300. File receive functions watch a certain directory identified on the logical drive or on a universal naming convention (UNC) file path. The file receive function continuously monitors the identified directory for files that have specified file extensions, and when files having these extensions arrive, the file receive function submits the associated document for processing. Message queue receive functions monitor queues in the shared queue database580and submits arriving items into the queue for processing.

When a message or document215is submitted to the integration server230, the document or message215is stored in a shared queue database580until it is picked up for processing. The shared queue580includes four separate queues, a work queue581, a scheduled queue582, a retry queue583, and a suspended queue584. By accessing these queues580, it is possible to determine what stage of processing a message or document215is in. For example, it is possible to determine if a document has been processed and is waiting for transmission or if the message or document failed processing. Messages and documents215appear in each of the queues in the order of “first in, first out.” That is, the oldest items in the work, retry, scheduled, or suspended queue appear first and the newest items appear last.

The work queue581holds all messages and documents that have been submitted to the integration server230for asynchronous processing, but have not been picked up by the receive function550. If messaging processing is synchronous, the work queue581is not used and the document is processed immediately. The work queue581contains messages and documents215that are currently waiting to process. Messages and documents placed in the work queue581are processed upon arrival, and therefore do not remain in the work queue for long periods of time.

The receive function550polls the work queue581at predetermined time intervals to pickup the next documents in the work queue581for processing. These documents are picked up by the receive function550as batches of documents or messages. The rate of release of documents in the work queue581depends on the configuration of the integration server230. Any message in the work queue581can be moved to the suspended queue584. Once a message or document is moved to the suspended queue584, it can be deleted, resubmitted, or retransmitted to the work queue581to complete processing at a later time.

When a message or document is processed, a channel510and a message port515are associated with the message. The message port definitions can include a service window for processing the message. For example, the service window could be defined as between Midnight and 1:00 am. If the message is processed outside this service window the message is redirected to the scheduled queue582. The scheduled queue582contains messages and documents that have been processed by the integration server230and are waiting for transmission based on the service window. The scheduled queue582holds the message until the service window opens. Once the service window is opened the message service300attempts to resend the message through the previously identified message port515. Like the work queue581, any item in the scheduled queue582can be moved to the suspended queue584.

If a message fails to send or post due to a communications error or for other reasons the message is placed in the retry queue583. The retry queue583contains messages and documents215that are to be resubmitted for delivery and documents that are waiting for reliable messaging receipts. A message can be sent to the retry queue583if during processing a receipt was expected, but was not present. When a receipt is required, a message can remain in the retry queue583until the receipt is received, or it can be treated like all other messages in the retry queue583and automatically resent. The message will continue to be resent according to the frequency and number of attempts configured by the channel510. If the number of attempts to resend the message exceeds a threshold or the document's time to live (TTL) is exceeded, the message215is moved to the suspended queue584.

The suspended queue584stores and displays messages and documents215that have failed processing, or that have failed to post to their destination. Messages can fail because some fatal unrecoverable error occurred in processing. For example, these errors can include parsing errors, serialization errors, and missing channels. Messages in the suspended queue584have an associated error report that indicates why the message was placed in the suspended queue584. Depending on the nature of the failure most messages or documents in the suspended queue584can be deleted, resubmitted, or retransmitted to the server230for processing. However, in one embodiment, some messages and documents cannot be resubmitted, for example, messages that failed parsing cannot be resubmitted or retransmitted.

The integration server230takes documents215from the queues580in batches for processing and transformation through the receive function550. However, once documents are taken from the queues580in batches, any order that may have existed in the documents is lost. It is only by chance that documents and messages215are processed in the correct order. To help ensure that documents215are posted in the correct order, the integration server230also includes a document messaging state management engine that assists in posting a document from a first application program211to a second application program221in an order that is sequentially correct for the second application program221. The document messaging state management engine is illustratively a component module of the messaging service300.

Ordered delivery or sequential delivery of documents is important in at least three general areas of document publishing. First is an instance message, second is a parent/child message, and third are associated messages. Instance messages are messages that are related to an entity instance (i.e. address update for a customer). Child messages are messages that are dependent upon a specific related message or parent (i.e. a customer entity is the parent entity for the customer's address or phone number). Associated messages or associated entity instances are messages that are dependent upon the posting of another unrelated message (i.e. customer A wants shipping method B, which is unrelated to customer A).

FIGS. 6A and 6Billustrate a messaging service600having the state management system of the present invention. Messaging service600includes a pipeline610, channels620, a first application receive function630, a second application receive function640, a first application or source application integration component (AIC)635, a second or destination AIC645, a first application queue633, a second application queue643, a linker650, a preprocessor660, and a state management subsystem670. State management subsystem is illustrated in greater detail inFIG. 6B. The messaging service600is connected to a first application program631and a second application program641.

When a document215is published as a message to the messaging service600it contains information indicating when the document was published from the first application program631prior to it entering the messaging service600. This information can be either explicit information, such as a sequential ordinal indicating the document's order, or it can be implicit information, such as a timestamp in the message's header.

FIG. 7is a block flow diagram illustrating the basic steps that are executed by the state management subsystem670when a document is published to the messaging service600.FIGS. 6A,6B and7will be described together and reference is made to each drawing.

First, the state management subsystem670creates an entry in a state management table671in the system670for the document215that indicates that the document has been submitted, its order, and its associated state in a wait state table672. This is indicated at block700. Following completion of this process the document215is allowed to proceed through the messaging service600as though the state management system670was not present. This is illustrated at block710. When the documents has completed any required transformation through the channels610, the document is examined by the subsystem670, which queries the state management table671to identify the current state of the document in order to determine if the document's state can be changed by using a set state procedure of the set state module690(which will be discussed in greater detail below). This is illustrated at blocks720and725.

If the document's state can be changed, then its state will be changed, and the document will be posted to the second application program. These steps are illustrated at blocks730and740, respectively. The steps executed to change the state will be discussed in greater detail inFIG. 9. If the document cannot be changed for various reasons, the document is placed in a retry state, and resubmitted to the service at a latter time. This is illustrated at block750.

Referring back toFIGS. 6A and 6B, documents in the state management table671are placed in one of eight states. These states include submitted, retry, retriable, ignored, posting, posted, failed, and failed special. However, depending on the specific system requirements other or additional states can be used. Each state represents a specific condition of the associated document215. Illustratively, the system attempts to move all documents215to the state of posting. Posting is the state where the state management system670allows the document215to try and post to the second application program641. The state of the document215is changed by invoking the set state module690. Once a document has successfully posted its state is changed from posting to posted by the system670. If the document is not able to be posted it is set to the state of failed.

In some configurations during the transformation of the document in the channel620, a look-up call can be executed to check to see if a specific entity in the destination application exists. During this look-up call it may be discovered that another event must occur prior to the current message. If this is discovered then the state of the document is set to the state of retriable by the system670. When the document215finishes the transformation process in the channel620, the state management system670changes the state from retriable to retry, so that the document215can be resubmitted for posting later. As the messaging service600resubmits all documents that are currently in the retry state, the state management subsystem670sets the state initially to retriable to prevent the service600from submitting the current document twice.

The ignored state is invoked when a large number of messages are posted to the messaging service600, and these messages cannot be processed right away. For example, a sales order can generate a large number of messages that cannot be processed until another action is taken, such as receiving a confirmation that the order is complete. The state management subsystem670places the current messages requiring this further action in the ignored state. The ignored state indicates to the service600that the message was submitted, but wont allow the service600to pick up the message until the required action has occurred and the state is changed.

Documents placed in the retry state are retried until they are successful. However, it is desirable to stop the processing of documents which have been retried for an extended period of time. Therefore, documents have a time to live (TTL), after which the document is no longer processed and its state is set to fail. However, because nothing went wrong with the processing of the document other than the document timed out, the document's215state is set to the state of failed special, which is a special fail state. This special fail state is provided so that it is possible to later review the reason for the failure and possibly resubmit the document.

As discussed above, ordered delivery of documents from the source application program631to the destination application program641is important. If the document215processes through the channel620, and the state management subsystem670determines that the document will be posted out of order, its state is changed to retry. This order information can be contained in order table692, which the state management subsystem670can be configured to query to determine ordering information. This ordering information can be entered into table693by the preprocessor660during preprocessing. Documents and messages can also be set to retry for numerous other reasons such as waiting on an earlier instance of the same document to post, waiting for the parent document to post before posting the child document, or waiting for an associated entity instance to post.

Messages that are changed from submitted to retry can be resubmitted to the service600in either the original published form, or in the transformed form. However, to ensure that the most current information is used in the documents when they are posted, documents are illustratively resubmitted for transformation in the original published form to account for any updates that may have occurred in the interim to any associated fields.

Documents and messages215are submitted to the receive function630when they are placed in one of the messaging services queues633. Once the receive function630picks up the document215as part of a batch636of documents picked up from the queue633, the document215is provided to the preprocessor660.

The preprocessor660includes several interfaces661that define the structure of the document. Some of these features include header and footer information. Each application program631,641that submits messages or documents to the messaging service has its own interface. The preprocessor660selects the correct interface661based upon the application program that generated the document or message. This interface is then loaded into the preprocessor660. The interface661determines the primary key and the entity type for the current document or message. However, different configurations can require other information from the interface661.

Once the interface661is selected, the message215is loaded into the preprocessor660, and the preprocessor660calls the state management subsystem670and sets the message's state to submitted in column676, and enters its document ID674in column675of the state management table671. The preprocessor660also passes a copy of the message to an archive table665. The preprocessor660then checks the type of message that has been submitted. If the message215is of a type where it is necessary to wait for a parent document or record (e.g. an address where it is known that the parent customer must exist) the preprocessor660writes an entry673into a wait state table672for the message. This entry673indicates that the state management system670will wait for the parent document or message to be created before allowing the child message to post. This entry includes a document ID674, an entry for the channel the message will take677, and information regarding the waiting conditioned in column678. Depending on the performance of the system670, a check can be made of the destination application641to see if the parent already exists, but this check is not necessary. The preprocessor660is configured to identify from the message the required entities that must exist prior to posting the message. This information is then written to the wait state table672as part of entry673. The entry673defines the waiting parameters678for the message.

The wait state table672can include a variety of waiting parameters678. For example, a message215can be entered into the wait state table672if it is dependent upon an event happening, such as a customer waiting for a specific type of shipping. In this example the shipping type entities must be created before the customer's order can be posted.

Messages can be placed into the wait state table672, and not released for posting based on a publication order. For example, a message with a timestamp of 3 cannot be posted until messages having timestamps 1 and 2 have posted. In determining the order in which a series of messages should be published, the preprocessor660first checks for any explicit ordering information, such as priority information or incremental ordinals contained in the message215. If an explicit ordering is not present, the preprocessor660will order the messages based upon the timestamp for when the message215was first published to the messaging service600. This order information can also be stored in an order table692. Finally messages can be placed in the wait state table672, and not allowed to post based upon a user defined set of parameters.

The wait state table672also contains information as to which channel in the channels620the message is to be submitted for transformation. As a message215can be submitted through multiple channels620, each associated channel620has its own individual wait state entry673in the table672. This multiple entry for the same message215is used because different channels may have different orders and may receive information differently.

Once the preprocessor660has written the message to the wait state table672, the message is returned to the receive function630. The receive function630then passes the document to one of the channels620. In the channel620the transformation procedure developed in the orchestration service ofFIG. 3and the XLANG schedule can be used to convert the message from the source application's631format to the destination application's641format. During this transformation process a check for external links can be performed, if an external link is required to exist. If the link is not required to exist then this check is not performed. If a link is required to exist, and the required link does not exist, the state management system670invokes the set state procedure of module690to change the state of the message to retriable. However, no change is made to the associated entry673in the wait state table672.

The state management sub-system670also includes a table695which lists all of the messages215that are currently in the system, all of the channels620that are in use, and the state of all of messages215in the system.

After the channel620processes the message215it is passed to the pipeline610. The pipeline610is an interface that receives an indication that the transformation and posting of a document was successful. The state management system670executes several functions when the document reaches the pipeline610. This process is illustrated by the flow chart ofFIG. 8.

At step800the state management subsystem670checks in the state management table671for the state of the document or message215. At810state management subsystem670checks to determine whether the message state is in the state of retriable. If the message is in the state of retriable, state management subsystem670invokes the set state procedure to change the message to the state of retry. This is illustrated at step815.

If the state was not in the state of retry the state management subsystem670refers back to the wait state table672to determine what conditions or waiting parameters678are required to be completed before the message can be changed to the state of posting. This is illustrated at step860.

While the message is in the pipeline610, state management subsystem670can also check to determine whether any external links that are required exist. This is performed by checking a table generated by linker650. The process of creating this table does not form part of the present invention. One embodiment is described in more detail in application Ser. No. 10/435,629 filed May 9, 2003. If the external links are present, the document's state is changed to posting, otherwise the state is changed to retry.

If the waiting condition and/or external links are in proper form, a successful key is returned to the pipeline610. This is illustrated at step870. If the conditions are not met, the system management subsystem670changes the state of the document from submitted to retry so that the document can be picked up and resubmitted by the messaging service. This is illustrated at step880. At890, the state management subsystem670returns to the pipeline an indication that the document cannot be posted. This indication stops the pipeline610from attempting to post the document to the destination application program.

If the message is in a state other than retry, the state management subsystem670returns to the pipeline610an indication that it may proceed to post the document to the destination program641. The pipeline can submit the document to the destination application641through its AIC645, by using data contained in the message which identifies the AIC645. This is illustrated at block820. At this point the set state procedure in module696is called and the state is changed from submitted or retry to posting. This is illustrated at step830.

Next, the subsystem670determines whether the post was successful, at step835. If the message or document posted successfully, then the state management subsystem670, through the set state procedure, changes the status of the document to posted, at step840. If the message failed to post, then the state management subsystem670changes the state to failed at step850. Once a document or message is placed in the state of posted or failed, no further processing or handling of the document is done by the system600.

Resubmitted documents and messages are submitted directly to the associated channels without having to go through the preprocessor660again. The original information for the document contained in the wait state table672also remains unchanged for the document.

In one embodiment, the state of a document or message can only be changed by one state at a time. Further, the states can only be changed in certain orders. For example, a document cannot go directly from a state of submitted to a state of posted. To ensure that the rules of state change are properly followed, a set state module690having a set state procedure is called by the state management subsystem670when a state change is required.

FIG. 9is a flow chart illustrating the steps executed by the set state procedure when a change in state is desired. At900, the procedure checks to see if the documents time to live (TTL) has expired. If the TTL has expired the set state procedure sets the state to fail special at910. If the TTL has not expired the set state procedure checks to see if the state change desired is a valid change at920. If the requested change is not valid the state is set to fail at930. At940, the procedure checks to see if there are any ordering restraints present on the message. If there are ordering restraints present the system checks to see if there is any priority information at950. If priority information is present documents or messages will be processed in order of their priority for the same entitykey/channel, otherwise they will be processed in timestamp order with the oldest processed first. If priority information is present the procedure at951, checks to see if the message has the highest priority. The procedure continues to the wait state check at block970. If another document must be processed first, the set state procedure changes the state to retry at960. If there is no priority information present, the procedure, at953, checks the timestamp for the message. If the message is not the oldest it is set to retry, at block960. If the message is the oldest then the procedure continues to the wait state check at block970.

Once the priority check is complete, the set state procedure, at970, checks to see if a wait state entry673exists for the document in the wait state table672. If an entry673exists in the wait state table672, the set state procedure checks to see if that condition has been met at step980. If the condition of the wait state is met then the procedure changes the state to posting at690. If the condition is not met the procedure sets the state to retry at695.

In summary, the present invention is directed to a state management sub-system that assists in transmitting and processing documents and messages between two applications or systems in a functionally correct order through an integration server. To achieve this result, messages are picked up from a receive function and processed by a preprocessor. The preprocessor makes a copy of the message and stores the message in an archive table. While the message is in the preprocessor, a state management subsystem analyzes the message and determines when the message was published to the receive function, what messages or entities must exist in the destination system, and enters the document into a state management table with a state of submitted. Once the preprocessing is finished the message is returned to the receive function.

The message is then passed from the receive function to the channels, and processed through channels in the integration server. While in the channels the message is transformed according to a predefined process. This process transforms the message from a format that is useable by the source system to a format that is useable by the destination system. Also while the message is in the channel, a look-up function can examine the message for any external links that may need to exist before the message can successfully post to the destination system. Following transforming the message through the channel, the message is delivered to a pipeline. When the message is in the pipeline the state management subsystem checks the status of the message in the state management table, and verifies that all waiting parameters have been satisfied. If the waiting parameters have not been satisfied the message is set to retry and is resubmitted to the receive function. If the parameters have been satisfied the state of the message is set to posting and the message is allowed to post to the destination system. If the post was successful the state management system sets the state to posted, otherwise the state is set to failed. Most of the processing and control of the ordering by the state management subsystem occurs while the message is in the pipeline.

Another feature of the present invention is the set state procedure. This procedure defines the rules for changing the state of the message. The set state procedure checks the various conditions stored in the tables of the state management subsystem, such as waiting conditions, ordering conditions, and message states. By checking these states and changing the state of the message only when necessary, the set state procedure ensures that messages are delivered to the destination system in the functionally correct order. The set state procedure allows the system to recognize and manage multiple messages going to the same destination system while maintaining the order necessary to reduce the number of messages that fail to post because of improper ordering of the messages.