# Presentation: 235596

## Eastern InterconnectPhasor Project

- TVA’s Super Phasor Data Concentrator Architecture
- April 19, 2005

- Tennessee Valley Authority
- Electric System Operations
- James Ritchie Carroll
- E-mail: _[jrcarrol@tva.gov](mailto:jrcarrol@tva.gov)_
- Phone: (423) 751-2827

**Notes:**
Hello, my name is James Ritchie Carroll.  I work for TVA in Electric System Operations and I was the principal developer of TVA’s Super Phasor Data Concentrator (throughout thiss presentation I’ll just reference this as the Super PDC for short).  This presentation will cover the basic architecture of the Super PDC.

## Objectives for TVA’s Super PDC

- Support an open and scalable architecture for real-time data input and output supporting the most popular protocols for transmission of phasor data:
- 	**PC37.118, IEEE1344, PDCstream & OPC**
- Multiple historical data access options including OPC and XML/web service
- All system components running on fully redundant fail-over capable hardware

**Notes:**
The initial objectives for TVA’s Super PDC were simple.  First and foremost we wanted to create a system with an open and completely scalable architecture that specifically supported the most popular phasor data transmission protocols, in particular:  PC37.118, IEEE1344, the BPA PDCstream and Ole for Process Control (or OPC).  In the current working implementation of TVA’s Super PDC, all of these transmission protocol’s have been developed.  We’ve tested an deployed 1344, PDCstream and OPC.  We are also ready to test our 37.118 protocol but haven’t had access to any PMU hardware as of yet that supports the latest revision.  I did, however, get word on Friday that AEP is ready to start transmitting data from all 5 of their connected PMU’s using the new 37.118 protocol – so this should provide us with a good test bed. <advance> the next objective was to support multiple historical data options including OPC and XML formatted data <advance> and finally we wanted to provide a redundant execution environment for the software.  The currently deployed Super PDC implementation has accomplished these objectives.

## TVA’s Super PDC Architecture

**Notes:**
Here is the basic diagram of the Super PDC architecture.  Many of you have probably seen this, if you would like an electronic copy you can download one from the demographics web site.  I’ll try to briefly cover each of the major sections starting with the PMU demographics <advance> and hopefully save a little time for a few questions at the end.

## TVA EIPP Demographics Web Site

- _[https://](https://eipp.tva.com/)__[eipp.tva.com](https://eipp.tva.com/)__[/](https://eipp.tva.com/)_

**Notes:**
The demographics web site is a controlled access, SSL secured site – only specifically authorized users are allowed access <advance> All data to be sent to or from the Super PDC has to be carefully defined – each phasor, frequency, analog and digital value must be accounted for and described.  From transmission protocols to scaling factors, all PMU information is defined here.  The URL is simple and easy remember – it’s eipp.tva.com.

## PMU Information

**Notes:**
The demographics web site lists PMU’s for all companies, but you can only edit PMU’s for your own company.  The PMU list is sortable and searchable for easy access to the desired PMU’s <advance> this PMU information screen captures all the relevant information needed for a PMU including its longitude and latitude which is used for proper geographic positioning on end-user displays <advance> also captured is network connection information and the data transmission protocol.  Note the validation flag shown here – a PMU will not be considered “connected” until the validation flag is set <advance> Also collected on this screen is detailed phasor information, as well as destination demographics for the phasor <advance> as phasors and other relevant information for the PMU gets defined, the point list for the PMU gets dynamically generated.

## Connected vs. Reporting

- 	The demographics web site maintains current reporting status for all connected PMU’s

*Status Display from TVA’s EIPP Demographics Home Page:*

** ****Green:****	****All PMU’s Reporting**
** ****Yellow:****	****Some PMU’s Not Reporting**
** ****Red:****	****	****No PMU’s Reporting**

**Notes:**
Just because a PMU is connected to the Super PDC doesn’t mean it’s currently reporting.  PMU’s can go offline, network connections can go down - we cannot expect that all PMU’s will always be reporting.  The demographics home page maintains the status of all reporting PMU’s.  Note that green means all PMU’s are reporting, yellow means some PMU’s are not reporting and red means no PMU’s are reporting.

## Automated Notification ofOffline PMU

**Notes:**
When a PMU does go offline - an e-mail will be sent notifying the appropriate persons of the outage.  The notification address is maintained by the PMU owner.  In this case Rajesh of Ameren will get an e-mail notification that Callaway is offline.  We are still trying to determine the appropriate frequency for these e-mails, but it currently set to once per day per PMU for now.

## Simple PMU Data Graphs

**Notes:**
Also available from the front page of the demographics web site are graphs of all the phase angles and frequency values for all connected PMU’s.  Once a PMU is connected, this should be very useful for validating your PMU’s data stream as it reported to the TVA Super PDC.

## Real-time Data Acquisition

**Notes:**
Like I mentioned earlier, the TVA Super PDC currently supports 4 data input protocols:  PDCstream, OPC, 1344 and 37.118.  All of these real-time phasor data acquisition protocols have their own advantages and disadvantages.  For example, 1344 is designed for data transmission from a single PMU;  OPC is a neutral process data protocol that only sends data points (the problem there is that it doesn’t carry any information about a PMU);  PDCstream requires an external INI file to define its scaling factors;  of the four protocols, PC37.118 is the most mature protocol and is the best suited for phasor data – it’s currently pending IEEE approval – last I heard it was very close (if Ken Martin is in the crowd, put him on the spot for current status).  Most PMU and PDC vendors are already making headway to start supporting use of this protocol.

## Phasor Measurement Unit’s

**Notes:**
All real-time data acquisition starts here - with Phasor Measurement Unit’s – PMU’s – the heart of the matter.  They don’t look like much – but despite their humble appearance, these little boxes can and do produce volumes of data (trust me ).  They constantly measure key electrical states, such as phase angles and frequency, at speeds of up to 120 samples per second.  TVA’s Super PDC works with data directly from these units or from their parent concentrators.  The protocol’s implemented by the TVA Super PDC allow it the flexibility to work with most all PMU’s.

## Protocol Development & Testing

- PMU Connection Tester and Sample Rate Calculator

**Notes:**
Much work has been done to develop and test TVA’s implementation of these standard phasor data protocols.  As an example, to help test the IEEE1344 protocol, we developed a PMU connection test and sample rate calculator application – using this tool you can really see that a PMU is reporting phase angle and frequency information – this is orders of magnitude better than a simple ping which just tells you the box is alive <demo application>  (anyone like a  copy of this tool?)  Well, today is your lucky day...  TVA has authorized me to make these tools, and more significantly, the protocol implementations they’re built on, available to the entire EIPP community.  These protocols and tools will be made available without any warranty or liability, but be provided to help promote and further advance the purpose and goals of the EIPP.  This represents a real mindshare of work on my behalf and I’m very excited to be able to share this – I expect this tool and the protocol implementations to be available for download from the demographics web site within two weeks.

## DatAWare 30-day Sub-second Archive Array

**Notes:**
Central to the entire data collection and archival process is TVA’s DatAWare.  Similar in operation and function to OSI’s PI - this process data historian was also developed internally by TVA’s very own Brian Fox.  Brian was able to make DatAWare “bend over backwards” to provide phasor data directly to the Super PDC even before DatAWare begins to archive the data to disk – this crucial step was one of the key factors in making this project a success.  By sending the Super PDC data before waiting on it to get archived, there is minimal lag time between the data source and the Super PDC.

We intend to make all sub-second data archived by DatAWare be available for 30 days or longer depending on disk space consumption – right now there’s over three months of data available with plenty of disk space to spare.  For an average PMU with six phasors and compression turned off, we consume about 3 gigabytes of disk space per month.  The oldest sub-second data will be continually rotated off the main disks and backed-up permanently to tape for storage at an off-site location.  As DatAWare or its server hardware has reached its maximum load, we will simply add another instance of DatAWare to the array and scale the system out.  We are happy to report we have yet to determine what this maximum load is.

## OPC Data Access

**Notes:**
Once a secure connection has been established to TVA’s phasor data network – OPC can be used to access near-real time and historical data from the sub-second archive array.  DatAWare is fully HDA and DA OPC compliant – in English, that just means you can read near real-time and historical sub-second phasor data from DatAWare for as far back as we have disk space available.  OPC is only available in “near real-time” simply because you’ve got to wait for the data hit the disk before it’s available to be read.  A full list of the OPC ID’s and needed connection information can be acquired from the demographics web site.  I would also like to note that Matrikon has some good tools for dealing with OPC data.

## Large Dataset Access via XML

**Notes:**
If you are needing access to a very large dataset, OPC will likely not be your best choice – it will take too long to access and download the desired number of data points.  To accommodate large dataset requests, the demographics web site includes a page that allows you to select the desired points for download over a given timespan.  An e-mail address is requested so that we can notify you when the data export is complete – this e-mail will also contain specific download instructions for the dataset.

## Real-time Phasor

***OPC Input***

***PC37.118, IEEE1344 & PDCstream Inputs***

**Real-time**** Phasor**
**Data Acquisition**
**via TVA’s DatAWare**

**Notes:**
Here is a snapshot of DatAWare in action acquiring, rebroadcasting and archiving phasor data.  The incoming OPC interface is currently connected to Entergy.  The small black command windows connect using the standard phasor data protocols to AEP, Ameren, New York ISO as well as TVA.  Let’s see if I can remote into the live session <attempt remote connection to 152.85.98.5 – show rebroadcast options dialog>.  On average, when all the PMU’s are reporting, DatAWare is rebroadcasting all the points of phasor data to the Super PDC at about 10,000 points per second!

## Super PDC

**Notes:**
Much like the PMU’s, TVA’s Super PDC is like a little black box churning out lot’s of data.  This is a snapshot of a remote session into the Super PDC showing the current concentration status <adavance>  The Super PDC runs as a Windows service on a fully redundant, quad processor, clustered Windows 2003 server system with 2 gigabytes of RAM, a storage area network that provides plenty of reliable disk space and the hardware sits on a secure private subnet for non-competing bandwidth.  However, not much bandwidth is needed: the outgoing phasor data formats are so tightly packed that even when publishing a concentrated stream of phasor data from all 24 PMU’s at 30-samples per second, outgoing network utilization is still under 0.2 megabits per second – well under the 100 megabit capacity of most network cards.

## Real-time Data Broadcast Formats

**Notes:**
The only two phasor data protocols that are well suited to broadcast outgoing concentrated phasor data are the PDCstream and the new PC37.118.  The Super PDC already supports these two outgoing real-time data broadcast formats and any EIPP member can request access to this real-time data stream in either desired format.

## Permanent Aggregated Archive

**Notes:**
Although the archived sub-second data that’s no longer online will always be backed up, restoring the data from a backup tape stored at an offsite location will be an involved procedure that will take time to complete.  To make sure some level of phasor data is always immediately available for historical data analysis, another DatAWare archive is also available.  The separate DatAWare instance is a permanent phasor data archive – permanent meaning that none of the data archived here will ever be rolled offline.  We are able to accommodate the permanent disk-space requirements for this by only storing phasor data aggregated over one-second and enabling DatAWare compression.  The aggregation includes an average as well as a minimum and maximum over a period of one second to help indicate any interesting anomalies that may have occurred during that second.  The permanent archive houses three times the number of points as the sub-second archives (that’s because there’s an average, minimum and maximum value per data point), however, the permanent archive only stores 1/30th of that data volume of the sub-second archives - plus with DatAWare compression enabled much less storage space is required.  Just like the sub-second archives, data is available over OPC and web based XML request.

## EPG’s RTDMS Application

**Notes:**
The Electric Power Group (or EPG) has developed a real-time phasor data monitoring application called the Real Time Dynamics Monitoring System (or RTDMS) that directly connects with TVA’s Super PDC.  This application presents phasor data for operations use in a logical fashion.  There are two ways that this program can access real-time data:  the first uses data directly acquired from the Super PDC’s streaming real-time data output – this option produces the fastest refresh rates but requires the end-user to have an onsite real-time data feed and locally installed server software.  The other data access option works via an SSL connection using an EPG web service hosted here at TVA – this option is not as fast, but is much easier to set up and get going quickly.   Currently displayed is a snapshot of the RTDMS frequency response screen <advance> here’s the path dynamics screen <advance> here is the relative angles and voltage magnitudes screen <advance> and here’s path flow showing megawatts... <if time permits, switch to live demo> There are other screens, but let me try to switch to the live system

## Summary

- Demographics Web Site:
- _[https://eipp.tva.com](https://eipp.tva.com/)_
- Real-time Data Inputs:
- PC37.118, IEEE1344, PDCstream and OPC
- Real-time Data Outputs:
- PC37.118 and PDCstream
- Historical & Near Real-time Data Access Options:
- OPC (DA & HDA) and XML formats for
- Sub-second and Aggregated data archives
- Visualization Tool:
- EPG’s RTDMS Application

## Current Connection Status

- 24 Total Phasor Measurement Units Connected:
- Ameren
*	**2 PMU’s connected*
- American Electric Power
*	**1 PMU connected, 5 online within the month*
- Entergy Energy Services
*	**11 PMU’s connected*
- New York ISO / NYPA
*	**5 PMU’s connected*
- Tennessee Valley Authority
*	**5 PMU’s connected, 12 more being installed*

**Notes:**
So, where do we currently stand? We currently have a total of 24 PMU’s connected to the Super PDC.  Here are the current connection stats.  It’s important to note here that AEP expects five more PMU’s online this month and TVA is in the process of installing 12 new PMU’s around Tennessee valley.  Also, in conclusion, please let me say that there were many, many people involved in making all this happen – this took an unprecedented level of cooperation from many people from many departments in all the companies listed here and more.  There were more involved with this than I could possibly begin to thank, but there are definitely a few that really stand out and I think they deserve some recognition:  Rajesh Pudhota from Ameren; Ray Hayes and John Allemong from AEP; Floyd Galvan, Leonard Chamberlin and Steve Winsett from Entergy; Dean Ellis and Jo Chow from New York ISO; Ken Martin from BPA; Manu Parashar from EPG; and from TVA, the DatAWare master Brian Fox, my hardware confidant Tiffany Smith, the demographics web site creator Mehul Thakkar, and a final special thanks to Ken Newberry and Russell Robertson for giving me the freedom to take this project and run with it.  Thank you...