Power distribution virtual networking

An electrical distribution system includes a plurality of electrical substations. Each of the electrical substations includes a plurality of intelligent electronic devices (IEDs), and a communications network interconnecting the plurality of IEDs at that substation. The communications networks at the plurality of substations are configured as at least one virtual network spanning multiple ones of the plurality of electrical substations, and interconnecting at least some of the IEDs within the multiple ones of the plurality of electrical substations, and so that delays experience by messages on the at least one virtual network are below a defined threshold. The virtual networks may be reconfigured when/if the threshold is not met.

TECHNICAL FIELD

This relates to power distribution networks, and more particularly to virtual networking in power substations.

BACKGROUND

In recent years, components used in electric power distribution and transmission systems have become increasingly computerized, facilitating the configuration, control and automation of such systems. Many conventional power transmission and distribution components—circuit breakers, transformers, inverters—and the like now incorporate microprocessors under software controls.

Microprocessor based devices are now typically found in power distribution substations, or otherwise deployed on the power distribution grid (e.g. on power distribution poles). These allow for the intelligent and often automated monitoring and control of power distribution subsystems and ultimately the electric power grid.

Not surprisingly, numerous protocols allowing for the intercommunication of microprocessor based power components have evolved. Notably the, IEC61850 standard for substation automation, the contents of which are hereby incorporated by reference, has been developed, and defines communications protocols, and data models allowing for standardized interoperable communication between microprocessor based devices—referred to as intelligent electronic devices (IEDs).

With the advent of distributed energy resources like photovoltaic, wind power generation, and with the proliferation of electric power storage devices, the number of IEDs in the grid is increasing and the need for communication between IEDs is also increasing.

Computer networking, however, has also separately evolved. With the advent of packet switched networks, many networking protocols and standards have been developed and refined, allowing the more flexible and sophisticated configuration and control of computer networks and the more efficient exchange of traffic on such networks. For example, the time sensitive networking (TSN) and deterministic networking (DetNet) projects have proposed additional payload (i.e. data) handling ability (e.g. data shaping, time synchronization) and controllability (e.g. stream reservation classes, registration, managed objects) as building blocks to conventional computer networks.

Most notably, virtualization in communication networks abstracts the network from the underlying hardware and enables virtual networks in diverse forms built upon fixed, installed hardware, so that hardware does not need to be replaced if network needs change.

Unfortunately, some of the approaches to computer networking are not directly compatible with, or accounted for, in substation automation.

Accordingly, there remains a need to enhance approaches to networking in power distribution automation.

SUMMARY

According to an aspect, there is provided an electrical distribution system comprising: a plurality of electrical substations, each of the electrical substations comprising a plurality of intelligent electronic devices (IEDs), and a communications network interconnecting the plurality of IEDs at that substation; wherein the communications networks at the plurality of substations are configured as at least one virtual network spanning multiple ones of the plurality of electrical substations, and interconnecting at least some of the IEDs within the multiple ones of the plurality of electrical substations, and so that delays experience by messages on the at least one virtual network are below a defined threshold.

According to another aspect, there is provided a method of configuring an electric power distribution system comprising a plurality of substations, the method comprising: interconnecting a plurality of intelligent electronic devices (IEDs) at each of the substations to a local area network; interconnecting the local area networks across at least one wide area network; configuring a plurality of virtual networks using the local area networks at the substations and the at least one wide area network; establishing a message transfer delay threshold defining a minimum acceptable delay for messages exchanged between IEDs over a virtual network; measuring delays in messages exchanged between IEDs over the virtual networks; and reconfiguring the plurality of virtual networks, to meet the message transfer delay threshold.

Other features will become apparent from the drawings in conjunction with the following description.

DETAILED DESCRIPTION

FIG.1illustrates an exemplary power distribution network10. Power distribution network10includes a high voltage distribution grid12, providing electrical power between multiple sources and sinks. A source20may be a combination of hydro-electric power generation plants; nuclear power generation plants; coal generation power plants; and other power generation plants (all not specifically illustrated) that feeds power to grid12. A source20may also be a photovoltaic, wind or similar source located at the premises of a customer.

Typical energy sinks18—in the form of customer homes and factories, or the like are also depicted.

As will be appreciated, some sinks18may act as an energy source, and vice-versa. For example, energy consumers may also operate power generation facilities, in the form of wind, hydro or photovoltaic generation stations. These may be connected downstream of substations14, and may also deliver electric power to the grid. Example combined sinks/source are labelled sink/source20/18.

Each substation14includes one or more intelligent electronic devices (IEDs)100-1,100-2, . . . (individually and collectively IED(s)100). As will be understood, IEDs100are micro-processor based controllers of power systems, and may control circuit breakers, transformers, capacitor banks, protective relays and the like.

A typical IED100is illustrated inFIG.2. Typical IEDs100include a functional block50allowing the IED to act as the power electronic device, a processor52, a network interface56, and memory54interconnected over by one or more suitable buses58. Memory54stores processor executable instructions adapting the IED100to perform in accordance with the instructions—which will of course, depend on the nature of IED100.

For example, IEDs100under processor control receive data from electronic sensors and other power equipment at substation14, (as depicted inFIG.1) and can issue control commands, such as tripping of circuit breakers. Typically, control commands can be transmitted to other devices (including other IEDs100) local to a substation14, or in another substation, that may in turn react to the commands. In this way overall operation of each substation, or multiple substations may be automated.

As will be appreciated, with an increase in the number of sinks/sources on network10, the need for intercommunicating IEDs has increased. IEDs100may, for example, connect and disconnect individual power sinks20and sources18to grid12. They may likewise control the power factor of power provided to each sink20and from each source18. IEDs100thus control or influence overall flow of electric power on grid12.

To facilitate interoperability, example IEDs100may support the IEC61850 standard for substation automation, the contents of which are hereby incorporated by reference. IEC 61850 allows for standardized communication protocols that allow the intercommunication between IEDs100.

In accordance with IEC 61850, IEDs100may communicate using one or more communications protocols including the manufacturing message specification (MMS); generic object oriented substation event (GOOSE); and sampled measured value (SMV) protocols. IEDs100accordingly support one or more of such communications protocols.

As will be appreciated, existing IEC 61850 protocols may be transported using network interfaces at IEDs100, in high speed switched Ethernet frames, or even over internet protocol (IP) provided that the response time allows for fast (e.g. sub 4 ms) response of IEDs100to control messages. In this way, one IED100at a substation14(FIG.1) may provide control signals to other IEDs100at that substation. Real time (or near real time) automated control of substations14may thus be achieved. This, in turn, may allow for overall control of the power grid12.

Each substation14thus further includes one or more physical nodes120that interconnect IEDs100local to that substation14. Physical nodes120are organized to form one or more local area networks (LANs)102or other computer communications networks. LANs102are schematically illustrated inFIG.3. IEDs100may act as nodes on LAN102, and may be integrated with other nodes on network. The physical nodes are organized to form one or more local area networks (LANs)102or other networks.

IEDs100at substation14-1are directly attached to a LAN102-1. IEDs100at substation14-2are connected to LAN102-2, and so on with IEDs100at the nthsubstation14-nconnected to LAN102-n. For ease of understanding, only three substations and LANs are depicted. LANs102-1,102-2, . . . are individually and collectively referred to as LAN(s)102. LANs102may, for example, be a time sensitive network (TSN) enhanced Ethernet as detailed in the IEEE TSN standards (including base standards for TSN: IEEE Std 802.1Q-2018: Bridges and Bridged Networks; IEEE Std 802.1AB-2016: Station and Media Access Control Connectivity Discovery (specifies the Link Layer Discovery Protocol (LLDP)); IEEE Std 802.1AS-2011: Timing and Synchronization for Time-Sensitive Applications in Bridged Local Area Networks; IEEE Std 802.1AX-2014: Link Aggregation; IEEE Std 802.1BA-2011: Audio Video Bridging (AVB) Systems; IEEE Std 802.1CB-2017: Frame Replication and Elimination for Reliability; IEEE Std 802.1CM-2018: Time-Sensitive Networking for Fronthaul, the contents of all of which are hereby incorporated by reference) or Ethernet as per IEEE 802.1Q, or any other suitable LAN.

Multiple LANs102are interconnected by way one or more wide area network (WAN)104and form communication network110that spans the multiple substations14. WAN104may, for example, be the public Internet, a deterministic network (DetNet) enhanced internet, or any other suitable wide area network, and may include multiple physical network nodes101.

Each physical network node120of LAN102and network nodes101of WAN104may take the form of a packet switch, router, computing device, or other conventional network node, known to those of ordinary skill. As such, each physical network node120/101is typically addressable on network110by its own network address, and includes its own networking logic. For example, a network switch may include a network switch fabric, including ports and control logic.

As noted, IEDs100may be nodes120on LANs102, and may be included in other nodes120. In example embodiments, at least some of network nodes120are dynamically reconfigurable. Such configurability, allows network routing and data transport functions of nodes120to be reconfigured. Example network nodes120may, for example, be TSN and DetNet compliant nodes.

In some embodiments, network nodes120are software defined network (SDN) nodes under centralized control, by for example an SDN controller, like network controller140. As will be appreciated, SDN refers software based network configuration and physical or virtual nodes compliant therewith, by way of controller platforms like OpenDaylight, protocols like OpenFlow and other, like those currently standardized by the Open Networking Foundation. Example network nodes120may, for example, be TSN and DetNet compliant nodes that are also SDN nodes under the centralized control of network controller140.

In other embodiments, network nodes120may be under distributed operation control, for example by way of the intermediate system to intermediate system (IS-IS) protocol. Each approach allows for the remote and/or dynamic configuration of routing/forwarding tables used at each of network nodes120to switch or route traffic to adjacent nodes.

One or more network controllers140may be in communication with nodes120. A network controller140for a LAN (e.g. LAN102-1) may be local to the substation14-1for that LAN102-1, or may be located at another substation. As such, there may be fewer network controllers140than LANs102. Alternatively, in the presence of redundancy, there may be more such controllers140with one controller140capable of operating in place of another, in case of failure or unavailability.

A network controller140for a WAN104may be located in WAN104or elsewhere—for example in a substation14. The domain of a network controller140may be a physical network like a LAN102, or WAN104or a virtual network, built upon physical network nodes or built upon another virtual network, as detailed below.

Network controllers140may include software and hardware to allow networking functions of nodes120and101to be configured, by an administrator or in an automated fashion by a network administrator. To that end, network controller140may include software that supports a suitable network configuration/management protocol that is understood by nodes120and101, and allows for reconfiguration of network functions at nodes120and101. Suitable network configuration protocols include OpenFlow, Border Gateway Protocol (BGP) plugins for OpenDaylight, Open Shortest Path First (OSPF) plugins for OpenDaylight, Network configuration protocol (NETCONF) protocols. Network controllers140(and nodes120and101) may, for example, be controllers/nodes compatible with the SDN architecture.

Exemplary of embodiments, a plurality of virtual networks are established on LANs102and WAN104. Example virtual networks111(VNET111), virtual network121(VNET121), virtual network131(VNET131) and virtual network141(VNET141), established on networks102, WAN104and thus network110are depicted inFIG.4. Links between nodes101on WAN104are not depicted.

Virtual links on physical links between nodes120are depicted. As will be appreciated each physical link may carry multiple virtual links, and each physical node120may act as virtual node in one or more virtual networks.

Moreover, virtual networks may span across multiple LANs102and WAN104, and thus across network110. In the example embodiment, each LAN102may be divided into several virtual networks. Additionally, a virtual network111,121and131may span multiple LANs102and WAN104.

As noted, virtual networks are built upon physical nodes120and101. Moreover, virtual networks may be established on other virtual networks.

Put another way, virtual networks may be layered, one over another, and each physical node120or101may form part of one or more virtual networks. In turn each virtual network includes virtual nodes that are founded on physical nodes120. A virtual network usually contains a smaller number of virtual nodes than the number of physical nodes on network110. Further any virtual network build upon an established virtual network includes its own virtual network nodes that are based on the virtual (and thus physical nodes) of the underlying virtual and physical networks. For example, inFIG.4, virtual network141is established on virtual network121.

The configuration of nodes120of each LAN102to establish the virtual networks may, for example, be controlled by network controllers140, under for example, control of software executing at network controllers140. Network virtualization may be established by configuring physical nodes120using known protocols, or extensions of such protocols. For example, each of nodes120involved in traffic redistribution may be compatible with a suitable network virtualization protocol or standard. Example network protocols that enable virtualization include multiprotocol label switching (MPLS), OpenFlow, BGP plugins for OpenDaylight, OSPF plugins for OpenDaylight, NETCONF, IEEE 802.1Q Edge Control protocol, the Multiple VLAN Registration Protocol, Internet Protocol Security (IPsec), or IS-IS.

Firewalls202may further form part of communications network110, and may be found at one or more of substations14, physically interconnected to the respective LAN(s)102at those substations. Optionally, multiple firewalls202may be provided for redundancy. A firewall202may operate as a firewall for one or more of virtual networks111,121,131,141. Each firewall202includes a software function, possibly assisted by hardware, and such function is created, maintained and destroyed based on hardware resources availability in substation networks and WAN104. The location of firewall202may be established dynamically based on available resources in communications network110, and may thus migrate. Firewalls202may intercept traffic destined for an associated virtual network111,121,131,141to ensure only authorized traffic arrives on the virtual network. GOOSE messages from and to a virtual network111,121,131,141may, for example, be transferred through firewall(s)202. OpenFlow and similar protocols maybe used to configure and migrate firewalls. Again, configuration may be initiated by controllers140under software control.

IEDs100communicate using GOOSE messages using services of one or more of VNETs111,121to communicate with other IEDs100on those VNETs or other VNETs (e.g. VNET131). VNETs111,121,131,141provide communication services for these GOOSE messages and for other messages and applications. A VNET121provides communication services by using services of one or more underlying VNETs141, and each such VNET141in turn may use services of other underlying VNETs (not shown) until physical nodes120are used.

For example, there are TSN enhanced physical networks as LANs and DetNet enhanced as one or more physical WANs. Further, a network controller140under software control, using OpenFlow protocols may configure a virtual network121connecting certain IEDs. Such network is isolated from other virtual networks (e.g. VNET131) and other traffic on the physical and virtual network. The resulting virtual network121may be for GOOSE and for MMS traffic, with specific GOOSE type messages having the strictest real-time requirements and the service. Further, the DetNet may have another virtual network configured on top of it. On top of these virtual networks there may be one or more virtual networks carrying virtual local area network (VLAN) labeled GOOSE messages with one VLAN identifier i.e. VLAN label for each GOOSE type.

Now, to be effective GOOSE messages must comply with end-to-end delays maxima for which GOOSE messages are used. Applications that communicate using GOOSE messages may be classified as the strictest real-time applications. In an example embodiments, depending on the type of GOOSE message, and other parameters, the end-to-end delay maxima may be different than for other applications and this may be accomplished by one or more QoS classes, normally by assigning GOOSE messages with the strictest delay requirements (i.e. to be the strictest real-time application) corresponding to the QoS class that provides the strictest delay on the network. IEDs100may host a few such strictest real-time applications using GOOSE messages.

For example, applications residing at an IED100, that communicates with other IEDs100using GOOSE messages, may carry an electric circuit trip command or other data that needs to be transported between IEDs100within defined thresholds—for example threshold A and threshold B for end-to-end delays of the strictest real-time application for which GOOSE messages are used, where threshold A is a desired delay and threshold B is a maximum tolerable delay. Both threshold A and threshold B may be represented as vectors with a multiple parameters that include: latency [measured in time units], frequency, time interval for frequency [measured in time units], and other optional parameters. Such end-to-end delay thresholds for IEDs100are used to determine the thresholds of the same kind for each VNET111,121,131,141. Information exchange and any negotiations in this regard (i.e. to establish VNET thresholds for each VNET) may happen between network controllers140using OpenFlow or other suitable protocol. So, we have the desirable delay threshold i.e. threshold A for each VNET (e.g. VNET111), and the maximum tolerable delay threshold i.e. threshold B for each VNET (e.g. VNET111), both threshold A and threshold B for the strictest real-time application for which GOOSE messages are used.

Frequency denotes the number of occurrence of delay being larger than the latency in the time interval for frequency divided by all occurrences of the delay. Threshold A and threshold B can be communicated e.g. using OpenFlow, Hypertext Transfer Protocol Secure (HTTPS), NETCONF, or other suitable protocols. by an IED100to the network110, e.g. to the network controller140, or network110e.g. network controller140can calculate them based on other data communicated by the IED100or communicated by entity controlling IEDs (not shown) e.g. using HTTPS, NETCONF, IEC 61850, or other suitable protocol.

Different applications or functions at each IED100may also need different qualities of service.

In accordance with example embodiments, as illustrated inFIG.5, IEDs100are interconnected in LANs102and across WAN104in block S502. Virtual networks may be established as described above, in block S504. Suitable message delay thresholds defining a minimum acceptable delay for messages exchanged between IEDs100and over each virtual network (e.g. VNET111,121,131,141) may be established in block S506. These thresholds may be protocol and VNET specific. Delays in messages exchanged between IEDs over the virtual networks may be measured in block S508.

If VNET or IEDs end-to-end thresholds are not met, VNETs111,121,131,141may be reconfigured in block S510to meet the message transfer delay threshold. For example, the VNETs may be reconfigured to meet both threshold A and threshold B for the strictest real-time application for which GOOSE messages are used. The reconfiguration may be done using OpenFlow, NETCONF, IS-IS, or other suitable protocol.

Specifically, the established thresholds B may be used to monitor the network and to trigger reconfiguration/recalculation of VNETs111,121,131,141. To that end, controllers140, under software control may measure and monitor network delays on VNETs111,121,131and141to ensure that internode communication on each of VNETs111,121,131and141meets threshold B. For example, message delays between all node pairs on a VNET may be measured. If threshold B is not met between nodes on a VNET111,121,131, or141SDN controller140may re-configure underlying nodes120of LANs102and101of WAN104so that thresholds are met, for example, both threshold A and threshold B for the strictest real-time application for which GOOSE messages are used.

Specifically, threshold A and threshold B for the end-to-end delays of the strictest real-time application for which GOOSE messages are used, are input in calculation of VNET111,121,131,141using underlying nodes120of networks102and101of WAN104. In one such embodiment, threshold A, which is the desired delay is the direct input to calculations e.g. to the shortest path algorithms for the corresponding VNET (and the measured delays are compared to threshold B which is a maximum tolerable delay)

Optionally, a network controller140of a virtual network (e.g. VNET141) built upon another virtual network (e.g. VNET121) may pass a message to the network controller140of VNET121to reconfigure underlying physical nodes used by both VNET141and VNET121. This may lead to further information exchange and negotiations between controllers140of VNETs. Network controllers140may use OpenDaylight or another suitable platform which accommodates such communication and information exchange between network controllers. OpenFlow or another suitable protocol could be used for such communication.

VNET thresholds are determined and delays measured for a delay that a message experiences in transfer between two end points in the virtual network: e.g. two switch/router ports of nodes120interfacing IEDs100. Delays may be measured directly at IEDs102or at virtual or physical ports of routing nodes120.

Message delays may be measured at each IED100, for example, by including a timestamp in an originated GOOSE message at an IED100, dispatching the message over a VNET (e.g. VNET111,121,131or141) and comparing this timestamp to the received time at a recipient IED. Alternatively, packets may be inspected to identify a GOOSE message and associate with it a time stamp in each point of entry and exit to a VNET. Such time stamps could be used to determine the delay.

Ultimately, receiving and/or sending IEDs100may observe the delays, and may notify to network controller140for the VNET on which the delay is observed. For example, if delay measured at any IED100is larger than end-to-end threshold B for the strictest real-time application, that IED100may originate a message to its network controller140. These messages and the data may be communicated to the network controller140by the IED100or communicated by entity controlling IEDs (not shown) e.g. using HTTPS, NETCONF, IEC 61850, or other suitable protocol. Network controller140may, in turn, react by calculating a new topology, and reconfigure the VNET.

If end-to-end delays for GOOSE messages for a specific VNET111,121,131,141as measured are larger than the threshold B for the strictest real-time application, then a new VNET topology for the applicable specific VNET may be calculated, including the new resource allocation to ensure end-to-end delays are smaller than the threshold A for each of VNET111,121,131,141. Thresholds A and B may have specific, different, values for each VNET111,121,131,141. Optionally, thresholds A and B can be changed in the presence of changing traffic properties or similar with the goal to provide end-to-end thresholds to the strictest real-time application that uses GOOSE. For example, if a VNET—for example VNET131—encounters quality of service degradation, and new thresholds for VNET131are set accordingly e.g. to a somewhat larger frequency of exceeding the minimum tolerable delay in threshold B. Other VNETs121and VNET111may be configured with new thresholds with stricter delay and frequency in threshold B and possibly A.

Conveniently, software at controller140may set thresholds and calculate alternate topologies/routes between nodes120and101on network110, with lower delays. OpenDaylight, IS-IS, protocols and calculations utilizing the shortest path algorithms, MPLS, and other protocols may be used to that end. Alternatively, controller140may initiate distributed reconfigurations of nodes120and101using a suitable protocol, such as IS-IS, using QoS constraints. As such, alternate topologies may be established using another protocol.

Each VNET111,121,131,141may use its controller140to determine the routes and handles allocation of resources and passes information to nodes120(e.g. switches/routers) that install virtual routing/forwarding tables for VNETs111,121,131,141.

Blocks S506-S510may be repeated periodically, or on demand to ensure that established VNETs continue to meet messaging thresholds, as network conditions change.