Patent Description:
Modern industrial process control and automation systems are typically equipped with a considerable number of field devices which monitor and control the manufacture process during the operation of a manufacturing plant. For example, field devices monitor signals such as temperatures and pressure and a variety of software performance metrics relating to the process being controlled by the industrial process control and automation system. Signals provided by the field devices are used by various process controllers of the automation system to control actuators to adjust various process parameters to control the manufacturing process. Industrial process control and automation systems use managed Ethernet based industrial networks to communicate control and data signals between field devices and controllers of the automation system. The managed industrial Ethernet networks are connected in various network topologies, such as for example, DLR ring and Linear/Star Ethernet networks and may use various data communication protocols, such as for example, an EtherNet/IP and or a Profinet protocol to manage communications between the field devices and the controllers.

There are no currently known method that can detect and pinpoint the root causes of a communication failure in such networks caused by mis-connected wiring, connector/extender shorting, and the formation of unsupported local network loops caused by prohibited loop connections. There is a need in industry for a pro-active mechanism that can detect and diagnose network instabilities such as for example network instability or network failures due to a controller issues, a device issues or network issues and arrive at the root cause of the instability and failure.

An example of a currently used system can be found in <CIT>, which discloses a method of implementing online maintenance in communication network. The method includes recording, in any communication device of the communication network, communication data going through the communication device itself when the communication device has detected online a maintenance request; collecting, by an online maintenance server set in the communication network, online the communication data recorded in all the communication devices, and analyzing the communication data to find out a fault reason of the communication network. In the method, communication devices are triggered via a maintenance request to objectively record communication data, and an online maintenance server is set for analyzing the collected communication data to find out the fault reason. In this way, the communication procedure and maintenance procedure are separately performed, which is convenient for the communication service provider to solve problems.

This disclosure relates to an apparatus and method for identifying the root cause of device communication failures in communication networks.

The figures, discussed below, and the various embodiments used to describe the principles of the present invention in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the invention. Those skilled in the art will understand that the principles of the invention may be implemented in any type of suitably arranged device or system.

<FIG> illustrates a portion of an example industrial process control and automation system <NUM> according to this disclosure. As shown in <FIG>, the system <NUM> includes various components that facilitate production or processing of at least one product or other material. For instance, the system <NUM> can be used to facilitate control or monitoring of components in one or multiple industrial plants. Each plant represents one or more processing facilities (or one or more portions thereof), such as one or more manufacturing facilities for producing at least one product or other material. In general, each plant may implement one or more industrial processes and can individually or collectively be referred to as a process system. A process system generally represents any system or portion thereof configured to process one or more products or other materials or energy in different forms in some manner.

In the example shown in <FIG>, the system <NUM> includes one or more sensors 102a and one or more actuators 102b. The sensors 102a and actuators 102b represent components in a process system that may perform any of a wide variety of functions. For example, the sensors 102a could measure a wide variety of characteristics in the process system, such as temperature, pressure, or flow rate. Also, the actuators 102b could alter a wide variety of characteristics in the process system. Each of the sensors 102a includes any suitable structure for measuring one or more characteristics in a process system. Each of the actuators 102b includes any suitable structure for operating on or affecting one or more conditions in a process system.

At least one input/output (I/O) module <NUM> is coupled to the sensors 102a and actuators 102b. The I/O modules <NUM> facilitate interaction with the sensors 102a, actuators 102b, or other field devices. For example, an I/O module <NUM> could be used to receive one or more analog inputs (AIs), digital inputs (DIs), digital input sequences of events (DISOEs), or pulse accumulator inputs (PIs) or to provide one or more analog outputs (AOs) or digital outputs (DOs). Each I/O module <NUM> includes any suitable structure(s) for receiving one or more input signals from or providing one or more output signals to one or more field devices. Depending on the implementation, an I/O module <NUM> could include fixed number(s) and type(s) of inputs or outputs or reconfigurable inputs or outputs. In the exemplary system of <FIG> I/O modules <NUM> are connected to controllers <NUM> via network <NUM>. The controllers <NUM> serve as an entry and exit point for a device node. Control information as well as data must pass through or communicate with the controller <NUM> prior to being routed from the node. For example, control information from a controller <NUM> can be sent to one or more actuators 102a associated with the controllers <NUM> device node. Data from the sensors 102a is communicated to one or more controllers <NUM> associated with the device node.

For example, a first set of controllers <NUM> may use measurements from one or more sensors 102a to control the operation of one or more actuators 102b. These controllers <NUM> could interact with the sensors 102a, actuators 102b, and other field devices via the I/O module(s) <NUM>. The controllers <NUM> may be coupled to the I/O module(s) <NUM> via Ethernet, backplane communications, serial communications, or the like. A second set of controllers <NUM> could be used to optimize the control logic or other operations performed by the first set of controllers. A third set of controllers <NUM> could be used to perform additional functions.

The controllers <NUM> can be used in the system <NUM> to perform various functions in order to control one or more industrial processes. For example, a first set of controllers <NUM>, that operate as a first network node may use measurements from one or more sensors 102b sent from controllers <NUM> operating as a second and separated network node to control the operation of one or more actuators 102b. These controllers <NUM> could interact with the sensors 102a, actuators 102b, and other devices singularly or via I/O module(s) <NUM>.

The controllers <NUM> may be coupled to the I/O module(s) <NUM> via an Ethernet network <NUM> using various network topologies, such as for example, a device level ring (DLR) topology, a linear bus (LINEAR) topology or star topology (STAR) or any combination of DLR, STAR or LINEAR or the like. A second set of controllers <NUM> could be used to optimize the control logic or other operations performed by the first set of controllers within a network node. A third set of controllers <NUM> could be used to perform additional functions.

The Ethernet network <NUM> may use a managed industrial Ethernet application layer for industrial automation, such as for example, an Ethernet industrial (EtherNet/IP) protocol or a process field net (Profinet) protocol to communicate between the controller and devices connected to the Ethernet network <NUM>. Such managed industrial Ethernet application layers use all the transport and control protocols used in a traditional Ethernet system including the Transport Control Protocol (TCP), the user datagram protocol (UDP), the internet protocol (IP) and the media access and signaling technologies found in off-the-shelf Ethernet interfaces and devices. It allows the user to address a broad spectrum of process control needs using a single technology. EtherNet/IP is currently managed by the Open DeviceNet Vendors Association (ODVA) and Profinet by the Profibus international organization.

Both the managed Ethernet protocols either the EtherNet/IP protocol or Profinet protocol use a comprehensive suite of messages and services for a variety of manufacturing automation applications, including control, safety, synchronization, motion, configuration, and information. Controllers <NUM> and compatible Ethernet devices installed on an EtherNet/IP network can communicate with other EtherNet/IP compliant devices connected on an EtherNet/IP network. Profinet compliant devices connected on Profinet network can communicate with other Profinet compliant devices connected on the Profinet network. Data accessed from devices connected to a managed industrial ethernet protocol (reads and writes) can be used for control and data collection.

Controllers <NUM> are often arranged hierarchically in a system. For example, different controllers <NUM> could be used to control individual actuators, collections of actuators forming machines, collections of machines forming units, collections of units forming plants, and collections of plants forming an enterprise, either directly connected in their network node or to a different network node via a controller <NUM>. A particular example of a hierarchical arrangement of controllers <NUM> is defined as the "Purdue" model of process control. The controllers <NUM> in different hierarchical levels can communicate via one or more Ethernet networks <NUM> and associated switches, firewalls, and other components.

Each controller <NUM> includes any suitable structure for controlling one or more aspects of an industrial process. At least some of the controllers <NUM> could, for example, represent proportional-integral-derivative (PID) controllers or multivariable controllers, such as Robust Multivariable Predictive Control Technology (RMPCT) controllers or other types of controllers implementing model predictive control (MPC) or other advanced predictive control. As a particular example, each controller <NUM> could represent a computing device running a real-time operating system, a MICROSOFT WINDOWS operating system, or other operating system. Operator access to and interaction with the controllers <NUM> and other components of the system <NUM> can occur via various operator stations <NUM> coupled to controllers <NUM> via network <NUM>. An operator station <NUM> can be located in a control room <NUM> that controls the plant or enterprise or may be coupled or assigned locally to a controller <NUM> that could receive and display warnings, alerts, or other messages or displays generated by a particular controller <NUM> or set of controllers.

Each operator station <NUM> could be used to provide information to an operator and receive information from an operator. For example, each operator station <NUM> could provide information identifying a current state of an industrial process to an operator, such as values of various process variables and warnings, alarms, or other states associated with the industrial process. Each operator station <NUM> could also receive information affecting how the industrial process is controlled, such as by receiving setpoints for process variables controlled by the controllers <NUM> or other information that alters or affects how the controllers <NUM> control the industrial process. Each operator station <NUM> includes any suitable structure for displaying information to and interacting with an operator. Each of the operator stations could, for example, represent a computing device running a MICROSOFT WINDOWS operating system.

This represents a brief description of one type of industrial process control and automation system that may be used to manufacture or process one or more materials. Additional details regarding industrial process control and automation systems are well-known in the art and are not needed for an understanding of this disclosure. Also, industrial process control and automation systems are highly configurable and can be configured in any suitable manner according to particular needs.

Although <FIG> illustrates a portion of one example industrial process control and automation system <NUM>, various changes may be made to <FIG>. For example, various components in <FIG> could be combined, further subdivided, rearranged, or omitted and additional components could be added according to particular needs. Also, while <FIG> illustrates one example operational environment in which redundant controllers could be used, this functionality could be used in any other suitable system.

<FIG> illustrates an example device level node <NUM> consisting of a controller <NUM> an I/O or communication (COM) module <NUM>, Ethernet network devices <NUM> and 105a and 105b and actuator driver <NUM>. The devices are arranged in a DLR topology using an Ethernet network <NUM> and communicating using a managed Ethernet protocol. For ease of explanation, the disclosure will be explained using the EtherNet/IP protocol, however, the network <NUM> may also use the Profinet protocol. Additionally, for ease of explanation, the device node <NUM> is described as being used in the automation system <NUM> of <FIG>. However, the device node <NUM> could be used in any other suitable system. The example, node <NUM> operates at Level <NUM> of the Purdue model, and among other things, the example device node <NUM> may use the measurements from the one or more sensors 102a to control the operation of one or more actuators 102b.

As shown in <FIG>, the device node <NUM> controller <NUM> may represent, or be represented by, various ones of the controllers <NUM> of <FIG>. Controller <NUM> would act as the ring supervisor for the device node <NUM> providing communication with the module <NUM> and implementing control logic for controlling the actuator driver <NUM> within the device node <NUM>. The device node <NUM> may also include Ethernet switch devices <NUM> for providing RJ45 cable ports to interconnecting the devices of the device node <NUM>. The device node <NUM> may further include Ethernet tap (ETAP) devices 105a and 105b for converting RJ45 cabling to fiberoptic cabling for single port devices that require optical fiber connections. Controller <NUM> may also be connected to one or more <NUM>/O or COM modules <NUM> that may be connected to a separate device specific bus <NUM>, operating using for example a ControlNet, DeviceNet or a Profibus communication protocol. Bus <NUM> providing device level communication between one or more sensors 102a and actuators 102b connected to COM module <NUM>.

<FIG> illustrates an example of a controller <NUM> according to this disclosure. As shown in <FIG>, the controller <NUM> includes a bus system <NUM>, which supports communication between at least one processing device <NUM>, at least one storage device <NUM>, at least one communications unit <NUM> and at least one input/output (I/O) unit <NUM>.

The processing device <NUM> executes instructions that may be loaded into a memory <NUM>. The processing device <NUM> may include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. Example types of processing devices <NUM> include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discreet circuitry.

The memory <NUM> and a persistent storage <NUM> are examples of storage devices <NUM>, which represent any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information on a temporary or permanent basis). The memory <NUM> may represent a random access memory or any other suitable volatile or non-volatile storage device(s). The persistent storage <NUM> may contain one or more components or devices supporting longer-term storage of data, such as a ready only memory, hard drive, Flash memory, or optical disc.

The communications unit <NUM> supports communications with other systems or devices. For example, the communications unit <NUM> could include a network interface card for communication over the Ethernet network <NUM> or a wireless transceiver facilitating communications over a wireless network (not shown). The communications unit <NUM> may support communications through any suitable physical or wireless communication link(s).

This disclosure will use the DLR topology shown in <FIG> to explain the fault detection tool and method of the present disclosure. It will be appreciated by those skilled in the art that the fault detection tool and method of the present disclosure can also be used in a STAR, LINEAR/STAR, or any hybrid combination of LINEAR, STAR or DLR topologies.

A managed Ethernet network using DLR Ring and Star network topologies, requires a large amount of cable interconnections between the various devices connected in the device node. A miswired connection between the devices can lead to network component malfunctions, device failures and controller failures during commissioning of the device node or during routine operation of the automation system. Moreover, in a DLR based network, misconnected wiring can lead to the creation of unwanted local loops, connector/extender shorting, etc. Misconnected wiring and unwanted local loops may cause network flooding and broadcast storms that eventually leads to the instability of the device node, CPU starvation, component failures and failures in the control of the devices in the DLR.

The present disclosure uses a fault detection tool implemented in the controller <NUM> and executed by processing device <NUM> to diagnose network instabilities in the device node <NUM>. The fault detection tool of the present disclosure provides a diagnosis of a network issue in the device node <NUM> and generates alarms and network notifications to a user or plant operator at an operator station <NUM>.

The fault detection tool can diagnose faulty local loops and wiring in the device node <NUM> due to misconnections. For example, in the device node <NUM> illustrated in <FIG>, ETAP device 105a shows a miswired connection to another ETAP device 105b using an Ethernet cable <NUM> illustrated in broken line. The miswiring of a STAR network topology between devices 105a and 105b sets-up a local switching loop in the device node <NUM> that may lead to network problems such as for example, network flooding and broadcast storms that overwhelm the node with continuous multicast or broadcast traffic. The broadcast storm causes the network devices connected to the device node to rebroadcast the data on the Ethernet network eventually causing failure of the network. The fault detection tool can also monitor and diagnose partial network issues/glitches of a newly deployed DLR topology due to for example, faulty network switch devices. Further, the fault detection tool can diagnose momentary network issues due to power loss and single and multiple device communication failures due to CPU loading and communication resource constraints.

The fault detection tool is arranged to detect the root cause of a network failure in the device node <NUM> and provide a user or plant operator with the suspected faults found in the node. A user or plant operator may use the insights provided by the fault detection tool and take appropriate actions to resolve the network issue and avoid loss of productivity.

The fault detection tool comprises a communication data recording method <NUM> shown in <FIG> and a diagnostic method <NUM> shown in <FIG>. <FIG> illustrates the example communication data recording method <NUM> used in gauging the health of the Ethernet network <NUM> according to this disclosure. For ease of explanation, the method <NUM> is described with respect to the process facility system <NUM> shown in <FIG>. In particular, the method <NUM> is described as being performed by the controller <NUM>. However, the method <NUM> could be used in any suitable system and performed by any suitable device(s) or component(s).

In operation the controller <NUM> records certain communication related data of the Ethernet datagram protocol (UDP) packets transmitted from the controller <NUM> to each device on the device node <NUM>. Additionally, the controller <NUM> records certain communication related data of the UDP packets received by the controller from each device connected on the device node <NUM>. The communication related data is continuously recorded and stored in memory <NUM> of controller <NUM>. The recorded communication data is continuously being updated and recorded to memory <NUM>.

Each device connected to the network <NUM> of device node <NUM> periodically sends UDP packets to the other devices connected in the network. A UDP packet consists of the source and destination ports being used in the communication, the packet length, and a checksum. The origin device adding its own header information to the UDP packet. The UDP packets are then broadcast on the network <NUM> between an origin network device and a network target device.

In operation <NUM> the controller <NUM> records the last sequence number (LASTSEQNUM) of target-to-origin UDP packets received from each device connected on the device node <NUM> broadcast on the node. Receiving the same LASTSEQNUM or a lower LASTSEQNUM is indicative of the data packets local looping in the node.

In operation <NUM> the controller <NUM> records a timestamp of the last origin-to-target UDP packet transmitted by controller <NUM> to each device on the device node <NUM>.

In operation <NUM> the controller <NUM> records the average time interval between two subsequent origin-to-target UDP packets sent from controller <NUM> to a device. When a new device is connected to the device node <NUM>, an origin-to-target data packet is sent by the Ethernet/IP network stack to the operating system TCP/IP stack. If the data packet is successfully sent, a timestamp of when the origin-to-target for the new device data packet is recorded and used to calculate a running average time duration between two subsequent origin-to-target data packets sent by the controller <NUM> to the new device. If the origin-to-target data packet cannot be added to the to the TCP/IP stack a counter that tracks the number of lost packets is updated for the new device connection.

In operation <NUM> the controller <NUM> records the total origin-to-target UDP packets that could not be sent by the controller to a device in the device node <NUM>. The sequence number in the target-to-origin data packet is used to determine if any packets have been lost.

In operation <NUM> the controller <NUM> records a timestamp of the last target-to-origin UDP packets received by the controller from each EtherNet/IP device connected to device node <NUM>.

In operation <NUM> the controller records the average time interval between two subsequent target-origin UDP packets from each device connected to the device node <NUM>. When the controller <NUM> receives a new device target-to origin data packet from a device connected to device node <NUM>, a timestamp of the event is used to calculate the running average time duration between two subsequent such events.

In operation <NUM> the controller <NUM> records the number of lost target-to-origin UDP packets from each Ethernet device connected to the device node <NUM>. If the controller detects that any sequence numbers are missing, a counter that tracks the number of lost packets is updated for the device having the lost UDP packets.

In operation <NUM> the controller records the device connection uptime for each active device connection in the device node <NUM>. When a new device is connected to the control network <NUM> a timestamp for the connection is stored which is used to calculate the uptime for the device.

<FIG> illustrates an example diagnostic method <NUM> for assessing the network and device health of the device node <NUM>. The diagnostic method <NUM> is described with respect to the process facility system <NUM> shown in <FIG>. In particular, the method <NUM> is described as being performed by the controller <NUM>. However, the method <NUM> could be used in any suitable system and performed by any suitable device(s) or component(s).

The controller <NUM> will run diagnostic method <NUM> periodically to analyze the health of the managed Ethernet network <NUM> and the Ethernet devices connected to Ethernet network <NUM>, such as for example, COM module <NUM>, network device <NUM>, ETAP devices 105a and 105b and actuator driver <NUM>. The diagnostic method <NUM> uses the communication data recorded by controller <NUM> in the communication data recording method <NUM> to establish if data packets sent and received by the Ethernet devices are local looping or have been lost. Upon the discovery of local looping or lost data packets over an established threshold sent and received between the devices of network <NUM>, the method generates network alarms and notification to a user or plant operator at an operator station, such as for example, operator station <NUM> notifying the user of the unexpected behavior of the network <NUM>. The diagnostic method <NUM> further providing information to the user of the potential cause of a device malfunction as well as potential location of the problem within the network <NUM>. The user can then use the information and insights provided by diagnostic method <NUM> to investigate and troubleshoot the problem.

In operation <NUM> the diagnostic method <NUM> is initiated by the processing device <NUM> of controller <NUM>. The diagnostic method <NUM> can be run periodically on a schedule, such as timed to operate at certain timed intervals, or on-demand by the user. In operation <NUM> the diagnostic method <NUM> retrieves the recorded data stored in memory <NUM> of controller <NUM> and looks for network instabilities in the Ethernet network <NUM> or the Ethernet devices connected to network <NUM>.

In operation <NUM> the diagnostic method analyzes the LASTSEQNUM of every data packet received from each device connected to the device node <NUM>. If a target-to-origin UDP packet from the same device recorded by operation <NUM> of the communication data recording method <NUM> has a LASTSEQNUM or a lower LASTSEQNUM for <NUM> or more times the method determines that the data packets broadcast on the Ethernet network <NUM> are local looping. Then the method at operation <NUM> generates and sends network alarms and notification to a user or plant operator at an operator station, such as for example, operator station <NUM> notifying the user of network mis-wiring causing local looping that may lead to a network storm. If the data sequence number listing is normal than the method moves to operation <NUM>.

In operation <NUM> the diagnostic method analyzes the recorded data and looks to see if all the devices in the network <NUM> have experiences lost data packets over a set threshold. If all the devices on network <NUM> have experiences lost packets the method at operation <NUM> sends notification to the user warning the user of a potential network configuration issue. The method may provide troubleshooting guides to the user to take specific actions such as, for example, to check the duplex and speed settings of all devices/switches <NUM>, and switch configurations used in network <NUM>.

If not all the devices in the network <NUM> have experienced lost data packets than the method continues to operation <NUM>. In operation <NUM> the diagnostic method reads and analyses the recorded data and looks to see if only a few devices in the network <NUM> have experiences lost data packets over a set threshold. If total lost data packets for only few devices are above a threshold limit, then only part of the network <NUM> is unstable. The system notifies the user in operation <NUM> of network issues in the unstable area or the devices connected to the unstable area of the network <NUM>. For example, if lost data packets exceed a threshold from the DLR ring connecting ETAP device <NUM> and actuator driver <NUM> than the diagnostic method <NUM> identifies that location and its connected devices as potentially having a network problem. The diagnostic method <NUM> may provide troubleshooting guides to the user to take specific actions such as for example to check for any network configuration issues, wiring misconnections in ethernet switches, electro-magnetic interferences, etc., at the location.

If lost data packets between the devices in network <NUM> do not exceed the threshold for all devices or, for a few devices, then the diagnostic method continues to operation <NUM>. In operation <NUM>, the diagnostic method <NUM> calculates the average time interval between two subsequent origin-to-target data packets sent from the controller <NUM>. If the average time interval is at least <NUM>% higher than the request packet interval (RPI) for all devices connected to network <NUM>, then the method <NUM> branches to operation <NUM> warning the user that the controller <NUM> is having a communication issue and that the controller is not sending the data packets as expected. The diagnostic method <NUM> may provide troubleshooting guides to the user to take specific actions such for example, testing and inspecting the operation of the controller for overloading of controller's processing device, communication resources, etc..

If the average time interval sent between two packets by the controller <NUM> is below the RPI then the diagnostic method <NUM> continues to operation <NUM>. In operation <NUM> the diagnostic method <NUM> goes to sleep for a period of time and then awakens to branch back to operation <NUM> to read the recorded communication data and run through the diagnostic operations of diagnostic method <NUM> again.

It may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The term "communicate," as well as derivatives thereof, encompasses both direct and indirect communication. The phrase "associated with," as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, have a relationship to or with, or the like.

Claim 1:
An apparatus comprising:
a communication network (<NUM>) operably connecting one or more devices (<NUM>, <NUM>, 105a, 105b, <NUM>);
a memory (<NUM>); and
one or more processing devices (<NUM>) operably connected to the memory and the communication network (<NUM>), the one or more processing devices configured to:
collect communication data transmitted and received on the communication network (<NUM>) from the one more devices (<NUM>, <NUM>, 105a, 105b, <NUM>) and record the communication data in the memory (<NUM>);
analyze the collected communication data stored in the memory to diagnose if communication data has been local looping or lost between the one or more devices (<NUM>, <NUM>, 105a, 105b, <NUM>),
wherein, the communication data has been local looping in the communication network (<NUM>) when a last sequence number or a lower sequence number for a data packet has been same for three or more data packets, and
wherein, the communication data has been lost between the one or more devices (<NUM>, <NUM>, 105a, 105b, <NUM>) when amount of lost data packets for all of the one or more devices (<NUM>, <NUM>, 105a, 105b, <NUM>) exceed a threshold.