Abstract:
Systems, methods, and computer program products for processing messages and searching for locating messages in an asynchronous messaging network by using content based message tracking. The message descriptor of each message is extended by a content key generated by the message generating application. The content key provides search information derived from application data contained in the message body. Each message queuing system is extend by a message trace component that administers a message trace queue and adds entries in the message trace queue for each operation of a message, and by a message search component that provides the functionality to search for the current location of a specified message as well as to retrieve a trace of the movement of the message.

Description:
FIELD OF THE INVENTION  
       [0001]     The present invention relates to message queuing in an asynchronous messaging network, and in particular to methods, systems, and computer program products for processing messages and searching for the locations of messages in such a messaging network.  
       BACKGROUND  
       [0002]     It is increasingly common to distribute a data processing operation over a plurality of data processing units, with each of the units communicating over a data communications network such as the Internet. One reason for this is that a particular data processing unit may be able to do a job better than another, so a first unit makes a request for a second unit to do a particular job and then to return the result to the first unit.  
         [0003]     It is also very common to have a large number of intermediate data processing units (also known as “nodes”) between the originating unit that makes a request and the destination unit that is being asked to do the work. Each intermediate unit receives the request, performs some initial processing to determine what to do with it, and then forwards it on to the next unit.  
         [0004]     A popular mechanism for carrying out such distributed data processing is called asynchronous message queuing, where applications communicate with each other by sending messages to queues, which can then be accessed by the receiving application at a time that is convenient for the receiving application. IBM&#39;s WebSphere MQ (trademark) software product, which has been on the market for a number of years, is the most popular example of this type of software.  
         [0005]     Frequently, messages which flow between data processing units in an asynchronous message queuing network are considered “high value” messages. For such messages, it is very important for the originating data processing unit to be able to locate such messages, should they become lost on their way to the destination unit. The term “lost” is taken to mean that the message is safe, but its location is unknown. A message could be lost, for example, if a link is broken between units, or if the target messaging address is not known on one of the intermediate nodes, or the message is waiting to be processed by an application but the application is currently not available or is running with bad performance. In such situations, the message will be unable to advance towards the destination unit in a reasonable time until either the link is repaired, or the routing (i.e., address resolution) configuration on the node in error is corrected, or the performance bottleneck is removed, or the failing applications are restarted.  
         [0006]     The only way known in the prior art to locate such lost messages is to have an operator “visit” (either physically or electronically) each of the nodes of the messaging network and search through the various message queues (e.g., the dead letter queues (DLQs) and the transmission queues (TXQs)). However, this is obviously very time consuming and inefficient. Various prior art teachings have employed a test message which is sent by an originating data processing unit into the network of intermediate units on its way to a destination unit. The test message arrives at various intermediate units in the network on its way to the destination unit, and reports are sent back to the originating unit by each intermediate unit to report the exact path that the test message took on its way through the network. For example, the well known Advanced Peer to Peer Networking (APPN) and TCP/IP (Transmission Control Protocol/Internet Protocol) provide such functionality. U.S. Pat. No. 5,668,800 is another example of such prior art. However, such prior art identifies the path of a test message but does not locate the lost (application) message. Such prior art provides an operator with a possible (but not guaranteed) route that a lost message might have taken.  
         [0007]     U.S. Pat. No. 6,654,805 describes an asynchronous message queuing network. A data processing method of finding a lost message includes the steps of: sending a first message from an originating data processing unit to an intermediate data processing unit, the message including an indication that the message is traceable; sending a tracer message from an originating data processing unit to an intermediate data processing unit, the tracer message identifying the first message as a lost message which the originating data processing unit would like to find; at the intermediate data processing unit, upon receiving the tracer message, checking to determine whether the first message exists within the intermediate data processing unit, and sending a reply message back to the originating unit if the first message is found within the intermediate data processing unit; and at the intermediate data processing unit, if the first message is not found within the intermediate data processing unit, determining whether the first message has passed through the intermediate data processing unit, and if the first message has passed through the intermediate data processing unit, determining a neighboring data processing unit which received the first message from the intermediate data processing unit, and forwarding the tracer message to the neighboring data processing unit.  
         [0008]     All these prior art methods work only on the message identifiers of the messages, which in most cases are unknown to the business domain users, rather than on the message contents.  
       SUMMARY  
       [0009]     The present invention provides systems, methods, and computer program products for processing messages and searching for the locations of messages in an asynchronous messaging network by using a content based message tracking mechanism.  
         [0010]     The message descriptor of a message within the present invention may be extended by a content key generated by the message generating application. The content key provides search information that is derived from the application data contained in the message body. The content key may be encrypted and authenticated as needed. Applications in the same business context are able to create appropriate search requests.  
         [0011]     Furthermore, each message queuing system may be extend by a message trace component that administers a message trace queue and adds entries in that message trace queue for each operation of a message. Each entry in the message trace queue contains at least the message id, content key, the time stamp of the entry, and source queue as far as the message is not transferred to another message queuing system. If the message is transferred to another queuing system, the entry contains additionally identifiers of the target queue and target queuing system.  
         [0012]     Finally, the message queuing system may be extended by a message search component which provides functionality to search for the current locations of messages as well as to retrieve a trace of the movement of a specified message in the messaging network. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0013]     The above, as well as additional features and advantages of the present invention will be apparent in the following detailed written description. The novel features of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives, and advantages thereof, will be best understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:  
         [0014]      FIG. 1A  shows the structure of a message according to the prior art,  
         [0015]      FIG. 1B  shows the structure of a message according to the present invention,  
         [0016]      FIG. 2A  shows a prior art message queuing system,  
         [0017]      FIG. 2B  shows a message queuing system according to the present invention,  
         [0018]      FIG. 3  shows the data model of the inventive message trace queue,  
         [0019]      FIG. 4  shows the basic message processing flow including the entries in the message trace queue according to the present invention,  
         [0020]      FIG. 5A -O show examples of message flow in an asynchronous messaging network with its assigned entries in the message trace queue according to the present invention, and  
         [0021]      FIG. 6  shows a sample message queuing network with 4 message queuing systems and a search for a message in that network. 
     
    
     DETAILED DESCRIPTION  
       [0022]      FIG. 1A  shows the structure of a prior art message. Every message has a message descriptor and a message body. The message descriptor identifies the message (message ID) and contains additional control information such as the type of the message and the priority assigned to the message by the sending application. The message body includes the application data. The content and structure of the application data are defined by the application program that uses them.  
         [0023]      FIG. 1B  shows the new structure of a message  7  that is generated according to the present invention. Compared with the prior art message structure, the message descriptor  8  is extended by a content key  10 . The content key  10  is application and message specific, and has at least one label with its assigned value(s)  11 ,  12 . The label identifies a specific data element derived from the message body  9  or from the message descriptor  8 . The value is the content of this data element, and can be encrypted and/or authenticated for security reasons.  
         [0024]     Content keys  10  are character strings and binary, e.g. encrypted values, and therefore need to be translated into hexadecimal string representations. Applications of the same business domain know how their content keys are 10 built, and are therefore able to create the appropriate content keys for message searches.  
         [0025]     The message descriptor  8  is preferably extended by a variable length field holding a content key  10  created by the application that puts the message in the message queuing system.  
         [0026]     Because creating the content key  10  of a message is business-domain specific, it is possible that there may be messages with the same content key  10  in different business domains. In this case, additional information needs to be specified for the message search, e.g. the origin queuing system and queue name.  
         [0027]     Examples of content keys are: 
        content key: #Fld 1 #100.000.000,00         #Fld 2 #IBMADEFF     content key: #L 1 #FE037A2104BA0C4589DE0E7E#F 3 #Daimler Chrysler#F 2 #Bosch        
 
         [0030]     The value of data element L 1  in the second example of a content key may be encrypted, e.g. using the public key of the business domain. The true value is not then visible to the outside world, but each application which is member of the business domain can, for example, create the same encrypted value for searching for the message with the appropriate value of data element L 1 .  
         [0031]      FIG. 2A  shows a prior art message queuing system  1 / 2  within a messaging network  2 . The application  6  generates messages and sends them to a message queuing system  1 / 2 . The message queuing system  1 / 2  includes at least a Queue Manager  10 , Queues  22 , and communication links  30  to other message queuing systems  1 / 2 . A queue  22  is a data structure used to store messages until they are retrieved by an application. Queues  22  are managed by a queue manager  10 . The queue manager  10  is responsible for maintaining the queues  22  it owns, and for storing all the messages it receives from application programs or other queue managers onto the appropriate queues, and retrieving the messages in response to application requests.  
         [0032]     Queues  22  exist independently of the applications that use them. A queue can exist in main storage if it is temporary, on disk or similar auxiliary storage if it must be kept in case of recovery, or in both places if it is currently being used and must also be kept for recovery.  
         [0033]     A message queuing system  1 / 2  therefore is a system for asynchronous, assured exchange of information in form of messages between applications connected to the system. The message descriptor of the message is known and interpretable by the message queuing system but not the message body whose layout is variable and application dependent. A message queuing system  1 / 2  therefore stores and transports the messages regardless and without knowledge of the message content in the message body.  
         [0034]     Multiple message queuing systems  1 / 2  can be interconnected in a message queuing network  2  which allows applications on different platforms and different physical locations to easily exchange data without knowledge of the underlying infrastructure.  
         [0035]      FIG. 2B  shows a message queuing system extended by the present invention. The new and inventive components are the message tracking component  20 , the message trace queue  25 , and the message search component  15 .  
         [0036]     Message tracking component  20  receives message trace information from the queue manager  10  and stores the received message trace information in the message trace queue  25 .  
         [0037]     Message search component  15  receives message search requests from the queue manager  10  and searches for the appropriate entries in its local message trace queue  25 . Additionally, it generates an internal message search request and sends the request to all of its adjacent queue managers. The message search component  15  of each of the queue managers then searches for the requested message trace entry on its local trace queue, and so on. If an appropriate entry is found, it is returned as a reply to the requesting message queuing system, and from this system back to the requesting search application.  
         [0038]     Message trace queue  25  stores the message trace information. This information includes the message id, the content key, the operation performed with the message (PUT, GET, SND, RCV), the source queue, and, if the message is either sent or received to or from another message queuing system, identifiers of the target queue and target queuing system. Preferably, a time stamp of the trace entry is also part of the message trace information.  
         [0039]      FIG. 3  shows a preferred data model of the inventive message trace queue  25 . The message trace queue  25  is preferably implemented in a database. Each message trace queue is preferably represented by two tables, a Message trace table  26  containing one entry for each operation on a traceable message, and a Message Content Table  27  which stores the label, value, and message id of each message only once. Each label value pair of the content key is a separate entry in this table.  
         [0040]     If a content key is specified in the message, the message queuing system adds an entry in the specific message trace queue  25  for each put and get of the message. An entry in the message trace queue  25  preferably includes the message descriptor, the operation type (GET, PUT, SND, RCV), the queue name, the queuing system identifier, and the date and time (time stamp) of the operation. If a message is transmitted to another message queuing system, this is also recorded, including the destination message queuing system, in the message trace.  
         [0041]     The type of the message trace queue  25  can be configured as linear or circular, which defines how the trace queue content is maintained. With circular operation it is possible that existing entries are overwritten by new trace entries, whereas with linear operation the queue can run out of space. The message trace queue  25  may also be implemented as a file system on a storage medium.  
         [0042]      FIG. 4A -C shows the basic message processing flow with its entries in the message trace queue according to the present invention.  
         [0043]      FIG. 4A  shows the basic message processing flow when a message is sent from an application to a message queuing system according to the present invention.  
         [0044]     The message processing flow starts with the generation of a message by an application  10 . Each message includes a content key which has been generated by the application and added to the message descriptor. The message is sent to a queue manager of another message queuing system  20 .  
         [0045]     The queue manager passes the message descriptor, the target queue name, and the operation type PUT to the new message tracking component  30 . The message tracking component checks whether a valid content key is in the message descriptor (valid means at least one label—value entry). If so, it creates a new entry in the message trace queue consisting of the message id, the content key, the queue name where the message is put on, and the operation type PUT  40 . If the entry is stored successfully on the message trace queue, the queue manager stores the message on the requested queue, and the transaction is committed  50 .  
         [0046]      FIG. 4B  shows the basic message processing flow when an application sends a message GET request. If an application sends a message GET request to the queue manager of a connected message queuing system  60 , the queue manager retrieves the appropriate message from the specified queue and  70 , and passes the message descriptor, the source queue name, and the operation type GET to the new message tracking component  80 . The new message tracking component generates a new entry in its assigned message trace queue, which also includes the message id, the content key, the queue name, and the operation type GET  90 . After a successful storing of this entry, the queue manager sends the message to the requesting application and removes it from the queue  100 .  
         [0047]     If a message needs to be transferred to another message queuing system, the queue manager again provides the message descriptor, the source queue name, the operation type SND, and additionally in this case, identifiers of the target queue and target queuing system to its assigned message tracking component. This again creates a new entry in the message trace queue, which consists of the message id, the content key, the source queue, the operation type SND, and in this case also of the identifiers of the target queue and target queuing system. Then the queue manager passes the message to the appropriate communication link.  
         [0048]     On the receiving message queuing system, the message is passed by the communication link to the queue manager. The queue manager on this system provides the message descriptor, the content key, the target queue, the operation type RCV, and the identifiers of the source queue and source queuing system to the message tracking component. The message tracking component creates a new entry in its message trace queue including the message id, the content key, the name of the target queue, the operation type RCV, and additionally the source queue and source queuing system. After the entry is created successfully, the queue manager stores the message on the specified target queue.  
         [0049]      FIG. 4C  shows the basic message processing flow when a queue manager receives a search request. When a message administrator wants to search for a message with a specific content, it sends a message search request message to the queue manager of the connected message queuing system (see  FIG. 4 ;  10 ). The search request message contains the content key to search for. This message is passed to the queue manager. The queue manager passes the search request to the message search component  15 . The message search component creates an internal search request message  20  from the received message search request and sends it to all its adjacent message queuing systems  25 . This internal search request includes the received content key, a unique search request identifier, and an expiration date after which the request becomes invalid. When a queue manager receives an internal message search request  65 , it passes the message search request to its message search component  70 . The message search component checks the expiration date  75 , and checks whether it already has processed a search request with this identifier. If the expiration date has been exceeded, the search request is discarded without a response  85 . If a search request is a duplicate, the message search component creates a ‘duplicate request’ response, and passes it back to the queue manager  95 . If neither is true, the message search component sends this internal search request to its adjacent message queuing systems, and searches, while waiting for the responses, for entries matching the specified content key in its local message trace queue  105 . When either all responses have been received or the expiration date has passed  115 , the message search component returns the result in an appropriate response message back to the queue manager  120 ,  125 , which sends this response to the originating message queuing system  130 .  
         [0050]     While the originating message search component is waiting for the responses from the adjacent message queuing systems, the message search component searches on the local message trace queue for entries with the specified content key  30 . After either the expiration date has passed, or the responses of all message queuing systems have been received, the message search component creates a response message containing a list of all found (if found) entries  40 . This list  45  includes the message id, the queue name, and the name of the message queuing system where the message has been located during the search. The response message is then passed to the queue manager  50 . The queue manager returns the response to the requesting message search application  55 .  
         [0051]      FIG. 5A  shows a message queuing network that has four single message queuing systems QM  1 - 4  without the present invention.  
         [0052]     Application Appl 1  puts a message on a queue Q 13  on queuing system QM 1 . The queuing system QM 1  transfers this message to QM 2  where it is put on queue Q 21 . Application Appl 2  gets the message from the queue, modifies it, and puts it on queue Q 22  on QM 2 . Application Appl 3  gets the message from queue Q 22 , does its processing, and puts the message on queue Q 23 . QM 2  now transfers the message to message queuing system QM 3 , where it is put on queue Q 32 . Application Appl 4  gets the message from queue Q 32 , modifies it, and puts it on queue Q 33 . QM 3  transfers the message to message queuing system QM 4 , where it is put on queue Q 41 .  
         [0053]     A message administrator who needs to know where a message with a specific content currently is, needs to connect to all participating message queuing systems, and scan all participating queues on each message queuing system for the requested message, as shown in  FIG. 5B . But this is a very time and resource consuming task, and the scanning application also needs to know how the message body has to be parsed to find the appropriate values.  
         [0054]     Additionally, the message search applications need to know how the message travels through the different message queuing systems to avoid the scanning of queues which are not involved in processing this specific message. Assuming that the message is located on queue Q 33  on message queuing system QM 3  at the time the message administrator searches for the message, the administrator needs to scan 6 different queues on  3  different queuing systems. If the message had been passed from Q 33  on QM 3  back to QM 1  on queue Q 11 , the administrator would not find the message with this approach.  
         [0055]      FIG. 5C  shows the same message processing as shown in  FIG. 5A , but now with the present invention. Each queuing system in the message queuing network holds an additional message trace queue.  
         [0056]     The messages in this scenario have the structure shown in  FIG. 5A . Application Appl 1  initially puts a message on Q 13  on message queuing system QM 1 . The content key of the message is assumed to have the following structure in this example: 
        #Fld 1 #IBMADEFF#Fld 2 # FE037A2104BA0C4589DE0E7E 
 
 where value ‘FE037A2104BA0C4589DE0E7E’ is assumed to be the encrypted value 5.000.000          
       
 
         [0058]     The following figures help to explain this scenario in more detail. In  FIG. 5D , the queue manager agent of message queuing system QM 1  receives the put request of application Appl 1 , detects that the content key of the message is set, and creates a trace entry for this message with the specified content key in the message trace queue.  
         [0059]     An entry for this “message-put-operation” in the trace queue of QM 1  is shown in  FIG. 5D .  
         [0060]     In this scenario, Q 13  is defined as the transmission queue. So QM 1  transfers the message to the connected queuing system QM 2 . A second trace entry is done by the queue manager agent on QM 1  for the send operation in  FIG. 5E .  
         [0061]     An entry for this “message-send-operation” in the trace queue of QM 1  is shown in  FIG. 5E .  
         [0062]     On queuing system QM 2 , the message is received and stored in queue Q 21 . A new trace entry is created by the queue manager agent for the receiving operation of this message.  
         [0063]     An entry for this “message-receive-operation” in the trace queue of QM 2  is shown in  FIG. 5F .  
         [0064]     Application Appl 2  gets the message from Q 21  (see  FIG. 5G ) for which the queue manager agent creates a second entry in the message trace table.  
         [0065]     An entry for this “message-get-operation” in the trace queue of QM 2  is shown in  FIG. 5G .  
         [0066]     Application Appl 2  processes the message and puts it on queue Q 22  on QM 2 . The queue manager agent then creates a second entry in the message trace table.  
         [0067]     An entry for this “message-put-operation” in the trace queue of QM 2  is shown in  FIG. 5H .  
         [0068]     A third application Appl 3  gets the message from queue  022  and performs its processing on the message. Another entry is created in the message trace table by the queue manager agent for this message get operation.  
         [0069]     An entry for this “message-get-operation” in the trace queue of QM 2  is shown in  FIG. 51 .  
         [0070]     Because Q 23  is defined as a transmission queue, the queuing systems transfer the message to the connected queuing system QM 3 . The appropriate entry in the message trace table is shown in  FIG. 5J .  
         [0071]     An entry for this “message-send-operation” in the trace queue of QM 2  is shown in  FIG. 5J .  
         [0072]     The queue manager agent of queuing system QM 3  receives the message and puts it on queue Q 32 .  
         [0073]     An entry for this “message-receive-operation” in the trace queue of QM 2  is shown in  FIG. 5K .  
         [0074]     Application Appl 4  gets the message from queue  032  ( FIG. 15 ), performs its processing with the message, and puts it on Q 33  ( FIG. 16 ). Two more entries are created in the message trace table for these two actions by the queue manager agent.  
         [0075]     An entry for this “message-put-operation” in the trace queue of QM 2  is shown in  FIG. 5L .  
         [0076]     Application Appl 4  put the message, after finishing its processing, on queue Q 33 .  
         [0077]     Q 33  is defined as a transmission queue, which causes the queue manager agent to transfer the message to the message queuing system QM 4  ( FIG. 17 ). The last entry for this message is created in the message trace table of QM 3 .  
         [0078]     An entry for this “message-send-operation” in the trace queue of QM 2  is shown in  FIG. 5M .  
         [0079]     The queue manager agent of message queuing system QM 4  receives the message and stores it on queue Q 41  ( FIG. 18 ). The first entry for this message is created in the message trace table of QM 4 .  
         [0080]     An entry for this “message-receive-operation” in the trace queue of QM 4  is shown in  FIG. 5N .  
         [0081]     If a message administrator now needs to determine the location of the message with the content IBMADEFF for the original field LT in the message body and 5.000.000          for the original field  32 A in the message body, the message query application needs to generate the appropriate content key of the specified data. The following translation was defined for this business domain: 
        Original field LT=Fld 1  as label in the content key     Original field  32 A=Fld 2  as label in the content key 
 
 The value of field  32 A may be encrypted by the public key of that business domain, which then results in ‘FE037A2104BA0C4589DE0E7E’. 
       
 
         [0084]      FIG. 6  shows a sample of such a message location query. The message query application on a message administration workstation connects, for example, to the initial queuing system QM 1 , and requests a message query with the content key. 
        ‘#Fld 1 #IBMADEFF#Fld 2 # FE037A2104BA0C4589DE0E7E’. 
 
 It is not necessary to provide the same key, as it has been provided with the message, or to provide all the labels. Only labels are specified which have also been specified with the content key of the message. 
         
         [0086]     Message queuing system QM 1  searches in the message trace queue for the specified key combination, and finds the appropriate entry. Additionally, it creates an appropriate internal search request, and sends the search request to its adjacent message queuing system QM 2 . QM 2  itself searches on its local message trace key for entries with the requested content key, and additionally sends the search request to its adjacent message queuing systems QM 3  and QM 4 . These two queuing systems also search on their local message trace queues and send the search results back to the requesting queuing system QM 2 . According to the previous sample, both queuing system will find one or more entries with this content key in their trace queues. Queuing system QM 2  now consolidates the results, and sends a response back to queuing system QM 1  containing the entry of the latest location of the message with the requested content key. QM 1  itself consolidates its local search result with the results from the received responses, and returns a response to the requesting search application containing the latest location of the message with the requested content key. In this sample the latest location would be Q 41  on QM 4 . If more than one message matches the search criteria, the latest locations of all the matching messages are returned.  
         [0087]      FIG. 6  shows a sample message queuing network with four message queuing systems, and a search for a message in that network.  
         [0088]     A message administrator is connected with a message administration workstation to message queuing system QM 1 . A list of trace entries on the different message queuing systems is shown. For the purpose of illustration, the content key is reduced to one value, representing a label—value pair, and the time stamp is reduced to a continuous number.  
                                                                                                                                                                                                       MsgID   Time stamp   Content key   Q   TQ   TQM   Op                                QM1            123   1   EAF   Qa           PUT       123   2   EAF   Qa   Q2a   QM2   SND            QM2            456   1   CD   Q2b           PUT       123   2   EAF   Q2a       QM1   RCV       456   3   CD   Q2b           GET       123   4   EAF   Q2a           GET       123   5   EAF   Q2c           PUT       123   6   EAF   Q2c   Q30   QM3   SND            QM3            123   1   EAF   Q30       QM2   RCV       123   2   EAF   Q30           GET       987   3   0815   Q33           PUT       987   4   0815   Q33           GET       999   5   EAF2   Q34           PUT       999   6   EAF2   Q34   Q2e   QM2   SND            QM4            556   1   UU   Q4a           PUT       557   2   UU2   Q4a           PUT       558   3   CD   Q2b           GET       557   4   UU2   Q4a           GET       557   5   UU2   Q4b           PUT                 MsgID = message ID            TQ = target queue            TQM = target queue manager            OP = operation applied to the message (e.g. get, put, receive, send)             
 
         [0089]     The message administrator in this example sends a message search request for the current location of a message with the content key EAF to the connected message queuing system. The message search component of QM 1  (Queue Manager  1 ) creates the appropriate internal search request and sends it to QM 2  (Queue Manager  2 ), which is the only adjacent queuing system of QM 1 . QM 2  itself sends the search request to QM 3  (Queue Manager  3 ) and QM 4  (not to QM 1  because it received the request from this queuing system). The search component of each of the queuing searches on its local message trace queue for a matching entry. The result of the search on each queuing system is shown below. 
        QM 1 : Nothing found     QM 2 : Nothing found     QM 3 : Msgld=123, Content key=EAF, Queue=Q 30 , Operation=GET     QM 4 : Nothing found        
 
         [0094]     The message search component of QM 2  creates a response based on the three results (QM 3 , QM 4  and itself), and sends the response to QM 1 . QM 1  creates a response for the requesting search application of its own search result and the received response from QM 2 . So the message administrator will get the following response from QM 1 : 
        Msgld=123, Content Key=EAF, Queue=Q 30 , QMGR=QM 3 , Operation=GET        
 
         [0096]     According to this, the administrator now knows where the message with this content key was located at the time of the search. It has been removed by an application from queue Q 30  on queuing system QM 3 .  
         [0097]     If the message administrator sends a message trace request to the queue manager of queuing system QM 1 , almost the same happens as described above. But the responses now contain the full history of the messages with the content key EAF. The result of each search component is shown below:  
                                                                                                                                                                         MsgId   Time stamp   Content key   Q   TQ   TQM   Op                                QM1            123   1   EAF   Qa           PUT       123   2   EAF   Qa   Q2a   QM2   SND            QM2            123   2   EAF   Q2a       QM1   RCV       123   4   EAF   Q2a           GET       123   5   EAF   Q2c           PUT       123   6   EAF   Q2c   Q30   QM3   SND            QM3            123   1   EAF   Q30       QM2   RCV       123   2   EAF   Q30           GET            QM4                  
 
         [0098]     The search component receives the responses of QM 3  and QM 4 , and assembles a response with its own result and sends the assembled response to QM 1 .  
                                                             QM2            MsgId   Time stamp   Content key   Q   TQ   TQM   Op               123   2   EAF   Q2a       QM1   RCV       123   4   EAF   Q2a           GET       123   5   EAF   Q2c           PUT       123   6   EAF   Q2c   Q30   QM3   SND       123   1   EAF   Q30       QM2   RCV       123   2   EAF   Q30           GET                  
 
         [0099]     QM 1  receives the response of QM 2  and assembles the final response for the requesting application with its result set.  
                                                             QM1            MsgId   Time stamp   Content key   Q   TQ   TQM   Op               123   1   EAF   Qa           PUT       123   2   EAF   Qa   Q2a   QM2   SND       123   2   EAF   Q2a       QM1   RCV       123   4   EAF   Q2a           GET       123   5   EAF   Q2c           PUT       123   6   EAF   Q2c   Q30   QM3   SND       123   1   EAF   Q30       QM2   RCV       123   2   EAF   Q30           GET                  
 
         [0100]     The message administrator now has the whole trace of the message through the messaging network, from the initial entry into the messaging network up to the last location of the message on QM 3 .