Enhanced reality system for visualizing, evaluating, diagnosing, optimizing and servicing smart grids and incorporated components

A head-mounted user device is provided for use in a smart grid network. The head-mounted user device includes a signal processor configured to receive signaling containing information about one or more assets in a distributed power grid network from a data collection device configured to receive data from the assets in the distributed power grid network; and determine, based at least partly on the signaling received from the data collection device, signaling containing asset display information about the one or more assets. The head-mounted user device also includes an augmented or mixed reality display configured to display an actual field of view of the user and the asset display information by overlaying the asset display information over the actual field of view.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an augmented reality system for visualizing, evaluating, diagnosing, optimizing and servicing smart grids and incorporated components, and a user device for the same.

2. Brief Description of Related Art

Proliferation of the “Internet of Things” (IoT) is driving interconnected smart systems. In particular, smart grids are following this trend though the establishment of smart energy, gas and water management. Interconnected components are now providing an unprecedented level of intelligence supporting numerous operational actions. This landscape is ushering in vast amounts of unstructured data and the need for intelligent data parsing, analysis and action systems.

Delta's DSGN™ (Delta Smart Grid Network) represents an Interconnected BIG DATA analytics system providing advanced intelligence and synergistic components across power distribution, metering and optimization. The DSGNM incorporates discrete elements in transformer monitoring and communications, residential and commercial metering and analytical, predictive and pre-emptive software algorithms. The hardware associated with the DSGN™ facilitates communications with transformers, residential and commercial meters and other Internet or wirelessly connected devices {commonly referred to as the “Internet of Things”}. The DSGN's geographically disbursed assets support system optimization capabilities, some of which are in logistically difficult areas to reference, re-locate, interrogate and service. Consequently, the opportunity exists to drive substantial efficiencies in data visualization, evaluation, diagnosis, optimization, and servicing using enhanced reality systems across this interconnected smart grid network and similar networks.

SUMMARY OF THE INVENTION

The present invention relates to a head-mounted user device for use in connection with a smart grid network and its component devices. The head-mounted user device incorporates a unique, augmented and/or mixed reality display. In augmented reality, a layer of digital content is overlaid onto the natural world visible to the user. In mixed reality, virtual objects are integrated into and responsive to the natural world. In accordance with the present invention, the device is configured to provide an augmented reality display system, a mixed reality display system, or both in various applications.

In the augmented/mixed reality display of the device of the invention, the user is able to see their actual field of view, either through a transparent display screen or a display screen that is in communication with and displaying real-time images received from a camera device pointed in the user's field of view. Digital content or information is displayed to the user on the display screen by overlaying the digital content or information on top of the user's actual field of view, to create the appearance that the digital content or information may form part of their field of view and, in some cases, interact with elements in their field of view. The digital content or information may be linked with a visual cue in the user's real field of view, such as a particular location or a particular physical element. For example, the user may view digital content relating to an electricity meter or transformer in the user's vicinity and the content may be linked to the location of the meter or transformer in the user's field of view. Alternately, an installation work order may visually queue an operator to select certain physical wiring elements, located within a field of view, for installation and commissioning of a particular electrical element. With the head-mounted user device of the present invention incorporating an augmented/mixed reality display, users, such as field technicians, are able to have free use of their hands to perform work while also retaining their actual field of view to see their actual surroundings, which would not be possible in a virtual reality system or hand-held device.

The head-mounted user device represents a unique pairing of smart grid unstructured and structured data with an enhanced (augmented/mixed) reality construct. The software and hardware of the device according to the invention leverages the capabilities of data collection and evaluation assets deployed and disbursed geographically in a smart grid network. The device spatially places and registers assets within a 360 degree view while associating real-time, relative data visualizations against associated assets and systems, essentially realizing a virtual utility smart network. The spatial system registers movement of the assets within its overlaid visual enhanced reality construct as the user's location and visual reference frame change or provide interactive cues. The device provides an intuitive drill-down on assets and data visualizations using gesture, audible and input device based commands. Direct interaction with augmented/mixed reality elements may be effected using similar gesture, audible and/or input based commands. Further, the device allows real-time interaction with assets using internal interactive interfaces, device internal cameras, voice communication, geographic positioning, accelerometers, and digital gyroscopes as well as those same capabilities on wirelessly connected or paired smart devices.

The head-mounted user device is configured to provide real-time review of asset performance, as defined by the smart grid network's central system software, over-laid spatially on the display screen of the device. The identification and flagging of health indicators with locations represented will be defined spatially. As a user moves, the device will provide advisement on the real-time geographic location of assets with routing to the asset overlaid real-time on the hardware. Noting this capability, current data systems provide limited or no interaction in-field for grid or asset review. Further, many local regulations forbid the inclusion of real-time geographic routing with current systems in vehicles of installers and technicians, which inhibits active routing. Additionally, available in-field data systems are cumbersome, unwieldy and limited in their capability for interaction with human factors, precluding their use by many field technicians. The head-mounted user device resolves this customer pain-point, while providing additional functionality. As noted, the augmented/mixed reality interface allows direct interaction with both overlaid elements and in-field assets using gesture, audible or input device based interaction.

The head-mounted user device provides overlaid instructions and interactive geometric element based representations for asset installation on transformer, residential or commercial locations. These instructions provide interactive and virtual cues overlaid on the surroundings in the user's field of view, while also providing a virtual keyboard for asset set-up. Upon installation, visual verification may be confirmed via a camera on the device, stored in a memory of the device and/or stored remotely via a cloud based data storage structure.

As overheating of assets may occur if improperly installed, the device provides in-field temperature verification via linking with hand-held devices which incorporate temperature profiling capabilities. Current methodologies and devices do not include digital capabilities for overlaying installation instructions upon points of installation. Further, in many cases, installers must record both the location and a picture of the installation for verification purposes. The methodology afforded by the head-mounted user device reduces the equipment required for this procedure while also providing efficiency improvements on the recording, localization and verification of installations against current requirements. During in-field product replacements, similar procedures must be followed for product decommissioning, final reads, picture of product and identifiers, and ultimately disposal. The device of the present invention allows the direct action of these activities, in the field, without the need to return the asset to a central location for review and disposition. Recognizing the temperature profiling capability above, “Hot Socket” situations in-field create a liability associated with potential fires and may be inhibited through the use of hardware temperature profiling capabilities and picture recording. The combination of these activities with the device allows the identification and remediation of potential loose meter socket situations.

Energy diversion represents a major issue within many emerging markets. Data collected from the smart grid network identifies assets exhibiting energy diversion situations. Using spatial overlays, the head-mounted user device guides in-field technicians to points of diversion for scheduled remediation. The device also identifies potential issues associated with the location(s) of diversion. No other system or hardware products offer such capability or flexibility. In some cases, the theft of assets that have been located within a smart grid installation may be visualized, localized and actioned for recovery using the internal GPS units within these devices and the augmented/mixed reality routing capability afforded by the device.

The device is inclusive of either a singular or full eye wrap, head-up display, internal microphones, headphones, one or more cameras and wireless connectivity including but not limited to Bluetooth, 3G, 4G and other, supporting the device's display. An ergonomically tuned head strap and facial support are provided to aid in wearability and comfort. In some cases, the device may be directly integrated into personal protection equipment, e.g. hard hats, protective suits, and similar.

In accordance with a first aspect of the invention, a head-mounted user device is provided. The head-mounted user device comprises a signal processor configured to receive signaling containing information about one or more assets in a distributed power grid network from a data collection device configured to receive data from the assets in the distributed power grid network; and to determine, based at least partly on the signaling received from the data collection device, signaling containing asset display information about the one or more assets. The head-mounted user device also comprises an augmented or mixed reality display configured to display an actual field of view of the user and the asset display information by overlaying the asset display information over the actual field of view; and means for mounting the device to the head of the user such that the display is in the user's field of view. The head-mounted user device may further comprise means for providing user inputs to the head-mounted user device.

In accordance with an embodiment of the head-mounted user device of the first aspect of the invention, the asset display information includes a real-time review of asset performance. Further in accordance with an embodiment of the head-mounted user device of the first aspect of the invention, the received signaling containing information about one or more assets includes information about the geographic position of the one or more assets; and the signal processor of the head-mounted user device is further configured to determine the location of the one or more assets relative to the head-mounted user device based on the received information, and is configured to update the location of the one or more assets relative to the head-mounted user device as the user wearing the head-mounted user device moves or as an asset is relocated to another location, either authorized or non-authorized. The asset display information may include identification and flagging of health indicators of the one or more assets together with the location of the one or more assets. The asset display information includes routing information directing the user to the asset overlaid on the display.

In accordance with a further embodiment of the head-mounted user device of the first aspect of the invention, the asset display information includes overlaid instructions for asset installation on the transformer and residential/commercial metering locations, including interactive and virtual cues to be overlaid over the field of view, including a virtual keyboard for asset set-up.

In accordance further with the first aspect of the invention, the head-mounted user device comprises comprising at least one forward facing camera mounted in front for receiving images and one or more additional cameras mounted around the device periphery. The at least one forward facing camera may be configured to capture real-time images of the user's environment in the user's field of view, and the display may be configured to display the real-time images of the user's environment in the user's field of view, onto which the information about said geo-spatial assets is overlaid. In one embodiment of the head-mounted user device, the means for mounting comprises an adjustable head strap. The head-mounted user device may further comprise a microphone configured to receive voice commands from the user.

In accordance with a further embodiment of the head-mounted user device of the first aspect of the invention, the asset display information includes information about an asset routing identification comprising a map directing the user of the head-mounted device to the asset on a recommended route determined by the signal processor and/or other algorithms as may be available via a wireless connection. The asset display information may further include an asset radar that provides a summary of locations of a plurality of assets, within specified distance increments from the head-mounted user device, which is displayed in the center of the asset radar. The asset display information may further include information about geo-spatial asset overlay that identifies assets by type overlaid on the field of view, including where relative size of the asset display information varies in proximity to the user. The asset display information may further include information about geo-spatial asset selected that provides a complete overview on performance data throughput, health and location. The asset display information may further include information about asset installation, set-up and visual verification, including where a targeted asset is visual through the device's enhanced reality field of view.

In accordance with a further embodiment of the head-mounted user device, the data collection device is a transformer mounted data collection device configured to receive signaling containing information about collected data, including some combination of electrical signaling data related to electrical signaling being processed by the transformer located and arranged in the distributed smart grid network and to which the data collection device is mounted, metered data related to associated electrical signaling being provided from the transformer to a building or structure in the distributed smart grid network, and other wireless network data related to other wireless network communication devices/nodes/end points deployed in the distributed smart grid network.

In accordance with a further embodiment of the head-mounted user device of the present invention, the augmented or mixed reality display comprises a transparent display screen configured to allow the user to see the actual field of view through the transparent display screen and to display the asset display information.

In accordance with a further embodiment of the present invention, the head-mounted user device comprises a hand-held controller device configured to permit user interaction with the head-mounted user device.

In accordance with a second aspect of the invention, an apparatus is provided, comprising a signal processor. The signal processor is configured to receive signaling containing information about geo-spatial assets that is received from transformer and residential/commercial metering locations interconnected by Internet/wireless connected devices in a distributed power grid network, and determine corresponding signaling containing information about the geo-spatial assets for providing on a real-time visual display on a visor of eyewear to be worn by a field technician.

In accordance with the second aspect of the invention, the information allows for interaction in-field for grid and asset review as well as asset interaction and modification. The information may further include a real-time review of asset performance for overlaying spatially on the visor. The information may further include identification and flagging of health indicators with locations represented and defined spatially. The information may further include advisements on the real-time location of the geo-spatial assets with routing overlaid on the visor. The information may include overlaid instructions and interactive geometric element based representations for asset installation on the transformer and residential/commercial metering locations, including interactive and virtual cues to be overlaid over the field of view, including a virtual keyboard for asset set-up.

In accordance further with the second aspect of the invention, the apparatus may comprise the eyewear having the visor for displaying geo-spatial information overlaid on view through glasses. The eyewear may further comprise multiple cameras mounted along the periphery for receiving images. The eyewear may further comprise an adjustable head strap for coupling the eyewear on the head of the field technician. The eyewear may further comprise a microphone for providing audio spoken by the field technician and headphones for providing corresponding audio to the field technician.

In accordance further with the second aspect of the invention, the signaling includes information about an asset routing identification of a targeted asset that provides direct, real-time, auto-updating routing from a user's current location. The signaling may further include information about asset radar that provides a summary of asset location, within specified distance increments, including colored information or icons related to different assets, colored items that represent potential occurrences of energy diversion, green items related to healthy assets, yellow items related to assets that have potential health issues. The signaling may further include information about geo-spatial asset overlay that identifies assets by type overlaid on the field of view, including where relative size varies in proximity to the user, or where assets are user selectable with drill-down capability. The signaling includes information about geo-spatial asset selected that provides a complete overview on performance data throughput, health and location. The signaling includes information about asset installation, set-up and visual verification, including where a targeted asset is visual through the device's enhanced reality field of view.

DETAILED DESCRIPTION OF THE INVENTION

FIGS. 1A and 1Bshow big picture renditions of the overall smart power grid network, e.g., as a configuration/establishment of a baseline, power grid centric, smart utility mesh network, for implementing a pole mounted or pad mounted transformer monitor/smart data collector device according to the present invention to communicate upstream and downstream within the network.

The overall smart power grid network represents an interconnected so-called “BIG DATA” technology system providing advanced intelligence and synergistic components across power metering, distribution, communication, optimization, installation and servicing. The network incorporates discrete elements in the transformer monitoring and communications, residential and commercial metering and analytical, predictive and pre-emptive software algorithms. The hardware associated with the network facilitates communications with transformers, residential and commercial meters, and other Internet/wireless connected devices {commonly referred to as the “Internet of Things”}. The network's geographically disbursed assets support a wireless mesh network communications extension, while aiding system optimization capabilities, noting that many assets are in logistically difficult areas to reference, re-locate, interrogate and service. The overall integrated system drives substantial efficiencies in data visualization, evaluation, diagnosis, optimization, and servicing using enhanced reality systems across this interconnected smart grid network and similar networks. The collective systems provide a synergistic and unique alternative network for BtB (business-to-business) and BtC (business-to-consumer) data receipt and delivery.

The smart grid network according to the present invention represents a singular, standardized, and scalable network, providing the industry's first inclusive solution from a singular supplier. The smart grid network is inclusive of four basic technology elements. The primary hardware and software constituents of the network are as noted and identified below.

1. The pole or pad mounted transformer monitor/smart data collector device is identified herein as element20(AKA as “HyperSprout™” (and formerly known as “iTM™”)), which is the localized data aggregation and power flow investigation; establishing a data capture and delivery capability wherever there is power, e.g., consistent with that set forth in U.S. application Ser. No. 15/160,754, which is hereby incorporated by reference in its entirety.

2. A digital data and delivery and receipt mesh network (AKA “DataVINE™” (formerly known as (“iAMI™”)) is identified herein as element40, which is a ubiquitous mesh network facilitating automated residential and commercial metering while deploying an alternative data delivery capability; enforcing a market leading 100% meter read capability, e.g., consistent with that set forth in U.S. application Ser. No. 62/236,420, as well as U.S. provisional application Ser. No. 62/244,919, and U.S. provisional application Ser. No. 62/299,348, each of which are hereby incorporated by reference in their entireties.

3. A smart node power grid communication protocol (AKA “DataSCAPE™” (formerly known as “iDAP™”)), which provides for a comprehensive nodal exchange analysis of all grid parameters; realizing an inclusive geo-spatial understanding of utility operations, e.g., consistent with that set forth in U.S. provisional application Ser. No. 62/205,358, which is hereby incorporated by reference in its entirety.

4. A head-mounted interaction and servicing device100, which is shown inFIGS. 2-8and described herein.

Taken collectively, this energy and communications portfolio and financial strategy improves over current offerings through its intimate understanding of utility partners' pain points, core needs and anticipated delights. Most importantly, the network hardware and software solution allows for the identification of the purposeful diversion of energy {i.e., theft} and the focused remediation of the offending areas or subjects, subsequently enhancing enterprise revenues.

As noted, the aforementioned overall combination provides an infinitely scalable data delivery and receipt capability for communities with poorly established, historical infrastructure while providing a synergistic network capability to those communities with current cellular capability.

By way of example,FIGS. 1A and 1Bshow examples of smart power grid networks generally indicated as10and10′, in accordance with some embodiments of the present invention. By way of example, the smart power grid networks may take the form of, or may be configured to include, one or more digital data and delivery and receipt mesh networks like element40. Each digital data and delivery and receipt mesh network40includes communication nodes such as the transformer module or device20for exchanging information upstream and downstream between the communication nodes and the central location, which takes the form of the private network50inFIGS. 1A and 1B. Communication nodes are configured to be able exchange such upstream and downstream information between themselves in order to exchange such upstream and downstream information between a respective communication node and the central location. Represented within this figure is the concept of a data storage “cloud.” The cloud represents an external digital storage media and/or series of storage devices, accessible via multiple wireless connectivity methodologies and useable as a vehicle for delivering smart grid parameters to multiple assets within the smart grid, e.g. smart node power grid communication protocol, head-mounted user device, and other associated and non-associated elements. InFIGS. 1A and 1B, similar elements are provided with similar reference labels.

InFIGS. 1A and 1B, the smart power grid networks10,10′ include transformers like elements12,22for providing electric energy to residential homes and commercial buildings like elements16,26, each having a respective electrical meter like elements18,28for measuring the associated electrical energy usage. The smart power grid networks10,10′ also include transformer monitor/data collection devices20configured to collect data about the electrical energy usage in relation to residential homes and commercial buildings16,26from the respective electrical meter like elements18,28. For example, each electrical meter18,28may provide metered data signaling containing information about metered data related to associated electrical signaling being supplied from the transformer12,22to the building or structure16,26in the grid network10,10′. Moreover, transformer monitor/data collection devices20may receive associated signaling containing information about electrical signaling data related to electricity being processed by the transformer12,22located and arranged in the grid network and to which the transformer monitoring device is mounted, as well as other wireless network data related to other communication nodes/end points forming part of other wireless network devices deployed in the grid network. In effect, the collected data received by the transformer monitor device20may include some combination of the electrical signaling data related to the transformer, the metered data related to the electrical meter and/or the other wireless network data related to other communication nodes/end points in the grid network.

The transformer monitor/data collection devices20are also configured to provide suitable signaling30containing information about the collected data to a private network50via the digital data and delivery and receipt mesh network40. The private network50is configured as a central point that processes the collected data, e.g., performing utility analysis that may include one or more of the following: delivery subtraction analysis, proactive asset monitoring, distribution asset utilization, T and D subtraction analysis, energy audits and analysis, load control, and geographic localization. This central point may exist in a cloud environment, accessible via multiple wireless connectivity or wired connectivity methodologies. By way of example, the utility analysis is performed in an effort to increase efficiency, decrease costs, increase profits and/or community engagement related to the operation of the smart grid network.

FIGS. 1A and 1Bshows a pole mounted transformer device20in communication with the electrical meter18associated with the residential home16. By way of example, the electrical meter18may be configured to measure single phase electrical energy provided by the transformer12, mounted to a pole14, along a single phase utility line11to the residential home16.

FIG. 1Aalso shows a pad mounted transformer device20in communication with the electrical meter28associated with the commercial building26. By way of example, the electrical meter28may be configured to measure three (3) phase electrical energy provided by the pad transformer22along a three (3) phase utility line21to the commercial building home26.FIG. 1Balso shows a power utility80configured to provide the electrical energy in the smart grid network10′.

FIG. 1Bshows that the transformer device20may be configured to collect data related to some distribution related functionality, e.g., including determinations related to outage, momentary outage, voltage/VAR, and/or transformer monitoring.FIG. 1Bshows that the transformer device20may be configured to collect data related to some voltage analysis, DRM (digital rights management) functionality and energy theft functionality in relation to its associated residential home or commercial building. The transformer device20provides the suitable signaling30containing information about the collected data to the private network50via the digital data and delivery and receipt mesh network40. The collected data received by the private network50may also be analyzed in relation to conservation, load curtailment and/or a demand response vis-a-vis the power utility80. InFIG. 1B, the private network50may include a private network computer and monitor generally indicated as52for performing or implementing the aforementioned analysis and functionality. In many cases this private network computer and monitor may be accessing the noted smart grid data via a cloud environment, accessible via multiple wireless connectivity or wired connectivity methodologies.FIG. 1Balso shows both the receipt and transmission of digital data across the defined wireless mesh network to a representative IoT device indicated as53, e.g., which may take the form of a smart phone, tablet, computer, laptop, etc.

FIG. 1Ashows that the digital data and delivery and receipt mesh network40may include other transformer devices like element20exchanging information with other meters like elements18i,28iassociated with other buildings or structures like elements16,26.

FIG. 1Aalso shows a relay60coupled between the digital data and delivery and receipt mesh network40and the private network50. By way of example, the relay60is shown as 5 GHz relay for communicating with a corresponding 5 GHZ private network50, although the scope of the invention is not intended to be limited to any particular frequency or transmissions/receipt media for the relay or network.

A head-mounted user device100is provided in accordance with the present invention, which is configured for use with the smart grid network10or10′ and its component elements. The head-mounted user device100is configured for visualizing, evaluating, diagnosing, optimizing and servicing the smart grid network10and the assets contained within the smart grid network10or10′. As used herein, the term “asset” or “assets” may refer to any device that may form part of the smart grid network10or10′ or be used in the maintenance or monitoring of the smart grid network10or10′, including but not limited to, transformer monitor device20, electric meters18, additional heat-mounted user devices100, substation monitors, electromagnetic pulse detection devices, mobile devices identified in the network (e.g., utility vehicles technicians and digital devices associated therewith), geographically fixed items identified in the network (e.g., telephone poles, junction points, commercial or residential structures), IoT devices53, and networks formed by the aforementioned devices, including mesh network40, as well as other items not associated with the smart grid network or other IoT devices, but for which there is established a referenced geo-location or moving/variable coordinate capability.

The head-mounted user device100is configured for communication with assets in the smart grid network10, such as transformer mounted data collection devices20and electric meters18. The head-mounted user device100receives information from and about the assets in the smart grid network10, and displays information on a display screen102of the device100in user's line of sight. The user of the head-mounted device100may be, for example, a field technician or a system installer.

A transformer mounted data collection device20transmits signaling to the head-mounted user device100, which contains information about the performance of assets in the smart grid network10. This transmission may be occurring directly from the referenced private network or indirectly via the referenced cloud environment. The user of the head-mounted user device100is therefore able to track and monitor the performance of assets in the smart grid network10while working in the field. The information transmitted from the transformer mounted data collection device20also may include information about the location of assets in the smart grid network10, such as the GPS coordinates of the asset. The head-mounted user device100comprises a geographic positioning component configured to determine the GPS coordinates of device100. As such, the processor110of the device100is configured to determine the locations of assets relative to the device100. This location information can be displayed to the user in a variety of manners, including for example, a radar-like display which displays to the user the distance and direction to one or more assets, or routing information, which displays to the user a route from the current position of the user wearing the device100to the asset onto a map. The head-mounted user device100is configured to track the movement of the device100, and for any asset that is not in a fixed location, the head-mounted user device100also receives updated positioning information for the asset. As such, the location information displayed by the device100can be updated on a real-time basis to reflect the movement of the user and device100.

While the head-mounted device100is configured to receive information from the transformer mounted data collection device20regarding, for example, multiple electric meters18in a smart grid network10, the head-mounted device100may also be configured to interact directly with assets such as an electric meter18. The head-mounted device100can be configured to interact with the electric meter18, or another asset, during installation or repair of the electric meter18, for example.

The head mounted user device100in accordance with one embodiment of the present invention is shown inFIGS. 3A and 3B.

The head mounted user device100may comprise a display screen102, a camera104, a microphone and headphone106, a means108for mounting the device to the head of the user, a signal processor110and memory112, in addition to other components.FIG. 3Bshows, for example, a block diagram of a head-mounted user device100including a display screen102, a camera104, a microphone and headphone106, a signal processor110and memory112, each of which can be connected to and in communication with each other in the operation of the device100.

The display102is an augmented/mixed reality display configured to display geo-spatial information about assets, overlaid on the environment or surroundings that are seen in the user's field of view. In accordance with one embodiment, the display102is fully transparent, such that the user can see directly through the display102to directly observe his or her surroundings. The augmented reality display102is configured to display information about the assets by presenting the asset information over the user's surroundings visible through the transparency of the display102. In accordance with a further embodiment, the display102is a non-transparent display screen. In this embodiment, the camera104of the device100is positioned on the outer surface of the device100, as shown for example inFIG. 3A. The camera104is configured to capture real-time video images of the user's field of view. The screen of the display102is positioned on an inner surface of the device100, directly in front of the user's eyes, and is configured to receive and display the real-time images being captured by the camera104. The user is therefore seeing the environment in his or her field of view indirectly, by way of the camera104in combination with the screen of the display102. Asset information is then displayed to the user on the screen of the display102by overlaying the asset information on top of the field of view images being captured by the camera104.

An example of the viewpoint120of the user wearing the head-mounted user device100is shown inFIG. 3A. Asset information122is displayed to the user on the display in a manner that positions the information122over the environment and surroundings that are in the user's field of view124.

The camera104may be in the form of a forward facing camera mounted in the front of the device100for capturing images that are within the field of view of the user wearing the device100, as previously described, or multiple cameras around the periphery of the device. The camera104may be further configured to record still or video images, which can be stored in the memory112of the device100and/or transmitted by the device100to an external storage device, in many cases a cloud storage system. The camera104may further be configured to take thermal imagery, which can be used, for example, to determine if an asset is overheated.

A microphone and headphones106may be provided as an integrated component, or as separate components on the device. The microphone and headphones106are configured to allow the user to interact with system, by allowing the user to provide audio input through the microphone and receive audio feedback through the headphones. The microphone106may be configured to receive voice commands from the user, which may, for example, instruct the head-mounted user device100to display a particular type of information for a particular asset. The user provides a voice command, which is received by the microphone106. The voice command is processed by the signal processor110. The signal processor110determines the action that is to be taken in view of the instructed voice command, and then executes the command, instructing the display102to display the requested information. The present invention is not intended to be limited to any particular format or location of the microphone and headphones106, but may take the form of any suitable microphone and headphone known in the art.

The means108for mounting may be in the form of an ergonomic and adjustable head strap, which can be wrapped around the user's head and adjusted manually for fit. In alternative embodiments, where the device100is in the form of glasses, the means108for mounting can be the frame of the glasses and for securing behind the ears of the user. The means108may also be in the form of a helmet, or item of headwear, or head portion of a suit, that is worn on the user's head and which integrate the device100. In certain embodiments, the device100may be in the form of a lens placed directly on the user's eye, in which case the lens also serves as the means108for mounting the device100to the user. The means108for mounting the device100to the head of the user are not intended to be limited to these examples, but may be in the form of other suitable head-mounting means known in the art.

The head-mounted user device100further includes a signal processor110and memory112. The non-transitory memory112stores computer program code, which when executed by the signal processor110, causes the signal processor110to implement various of the functions of the head-mounted user device100described herein, including the augmented/mixed reality display of asset information overlaid onto the user's field of view on the display102and the determination of asset information to be displayed on the display102based on the received signaling information from the transformer mounted data collection device20. The memory112may also be configured to further store information received by the processor110of the head-mounted device100from assets in the smart grid network and user collected data, such as images recorded by the camera104.

Memory112may also be configured to store information and signaling received from another device or a server for loading in the field.

The head-mounted user device100can be powered by means known in the art suitable for such devices, including but not limited to, a rechargeable battery that can be recharged while the device100is not in use by plugging the device100into a standard electrical outlet with a power cord received by an input on the device100, or a replaceable battery.

The head-mounted user device100comprises additional components that are used in the generation of the augmented/mixed reality system shown on the display102, including a gyroscope configured to detect the physical orientation and positioning of the head-mounted user device100, a geographic positioning or GPS component configured to determine the geographic location of the head-mounted user device100, an accelerometer configured to detect the acceleration of the head-mounted user device100when worn by a user in motion and a digital compass configured to determine the direction of movement of the head-mounted user device100.

The head-mounted user device100may comprise a plurality of means for providing user input and enabling user interaction with the device. The head-mounted user device100includes a microphone106, as described previously, which is configured to receive voice input from the user. The user may use the microphone106to provide vocal instructions to the head-mounted user device100to proceed through prompts or to display certain categories of information, for example. A separate, controller device may also be provided. In a preferred embodiment, the controller device is a small device in communication with the head-mounted user device100, which can fit into the pocket of the user. The controller device is hand operated by the user and comprises a plurality of user inputs, and may allow the user to scroll through prompts, select items from prompts, or provide any other type of user input that is received and processed by the head-mounted user device100. The head-mounted user device100is also configured to provide to the user a virtual keyboard126. The virtual keyboard126is displayed on the augmented reality display102, overlaid onto the user's field of view. The user can make selections on the virtual keyboard126with his or her fingers by placing his or her fingers where the particular input button of the keyboard appears in the field of view. The gyroscope of the head-mounted user device100is configured to determine the physical orientation and positioning of the device100, and the camera104is configured to detect the movement and location of the user's fingers. The processor110is configured to receive input from the gyroscope and the camera104about the position of the device and the movement/location of the user's fingers, and determine the appropriate selection from the virtual keyboard126by determining the location of the user's finger relative to the position of the device100and where the user's finger would align with the virtual keyboard126as it would appear to the user.

The head-mounted user device100can take the form of other head-mounted user devices comprising augmented reality displays known in the art. In alternative embodiments of the present invention, the user device100can take alternative forms, which are not head mounted. For example, a user device that is not designed to be worn on the head of the user can be provided including a display, camera, microphone, headphone, signal processor and memory consistent with that described above, such as in the form of a personal computer, tablet computer or mobile device.

FIG. 4shows various types of asset information122that can be displayed on the display102in accordance with the enhanced or augmented reality display system described herein. In particular, the asset information122shown inFIG. 4relates to geo-spatial asset performance, health and location.

As previously described, the exact location of an asset is communicated to the processor110of the head-mounted device100. The head-mounted user device100is configured to determine the location of the asset relative to the head-mounted user device100by determining the location of the asset, determining the location of the head-mounted user device100, comparing the location of the asset with the location of the head-mounted user device100, and determining the distance and/or route from between the two locations. The relative location of the asset may be displayed to the user on the display102in one or more different formats.

A route from the user/head-mounted user device100to a particular asset may be displayed in the form of asset routing information122a. The asset routing information122aincludes an identification of the targeted asset and provides direct routing from the user's current location to the asset, while automatically updating the route as the user/head-mounted user device100changes locations. The memory112of the device100may store map information for a particular area, or may be configured to retrieve map information that is stored externally. The locations of the particular asset and of the head-mounted user device100are identified and placed on to a map, and an appropriate route is determined and displayed to the user on the display102as a recommended route. As the head-mounted user device100moves, the changing location of the head-mounted user device100is detected, and the route modified to show such movement or to change the route, in the event that the user has moved to a location where a different route than the originally recommended route becomes advisable.

The relative location of one or more assets can be further displayed in the form of asset radar information122b. Asset radar information122bprovides a summary of asset locations, within specified distance increments of the user and the direction of the assets from the user's perspective. Asset radar information122bcan be displayed on the display102in a format that includes a series of concentric circles with the position of the head-mounted device100identified in the center, and the various assets positioned around the center in a 360-degree panorama. The compass direction that the user/head-mounted user device100is facing is determined, and the relative distance and direction of the asset from the device100is also determined. The assets are then positioned on the asset radar122baccordingly, with closer assets placed closer to the center of the asset radar122b. To provide the user with additional useful information about the assets shown in the asset radar information122b, different assets can be color coded or noted by discrete icons on the display102. For example, in one embodiment, red items may represent instances of potential energy diversion, green items may represent healthy assets without any problems or issues, and yellow items may represent assets with potential issues.

Geo-spatial asset overlay information122cidentifies assets by type, and is overlaid on the user's field of view. Geo-spatial asset overlay information122ccan be determined by the processor110of the device by defining a geographic area or smart grid network, and identifying the assets in the area or network. The locations of the assets are determined and the assets can be classified by their type (e.g., electricity meter18, transformer mounted data collection device20). A command is provided by the user to display assets of a particular class or type, which is processed by the processor110of the device. The processor110places the assets of the selected class or type on a map of the geographic area or smart grid network, which is displayed to the user on the augmented/mixed reality display102. Alternatively, the assets of the selected class or type can be associated with their geographic location and an identifying icon is displayed over the location of the asset, such that when an asset is in the user's field of view, an identifying icon is displayed to the user over the location. A particular icon may be provided to identify a particular type of asset, and the relative size of the icon may vary depending on proximity of the asset to user (i.e., as the user becomes closer to the asset, the icon becomes larger). All assets are user selectable with drill-down capability in order to obtain more particular information about the asset and its performance data. A selected asset can be noted as such in the asset radar information122b.

Asset selection information122dmay also be displayed. Selecting an asset provides a complete overview on its performance, data throughput, health and location. An asset can be selected by receiving a command from the user to select an asset to display asset selection information122d, determining by the processor110which asset out of a collection of assets in a geographic area or smart grid network has been selected, and displaying the asset selection information122don the augmented reality display102over the user's field of view. All parameters relating to the asset may be drilled down further to provide additional information to the user. Upon selection of an asset, asset routing information122aand asset radar information122bcan be adjusted accordingly to show a route to the asset in asset routing information122aand add the asset to the asset radar information122b, if necessary.

FIG. 5shows various types of asset information122that can be displayed on the display102in accordance with the augmented or mixed reality display system described herein. In particular, the asset information122shown inFIG. 5relates to asset installation and set-up, and visual verification of an installation.

To aid a technician in installing an asset, such as an electric meter18, installation content122eis displayed. The installation content122emay include installation videos, images, interactive geometric element based representations, or other instructions or information relating to the installation of an asset, and may be overlaid on top of the user's field of view at the installation site to aid in the installation. The installation content or information122emay be either static or streaming into the device and actionable via gesture controls or voice commands. Full asset information is summarized on the display102and updated based upon the user's activity. The installation content or information122ecan be received from a transformer mounted data collection device20or other device, or can be pre-stored in the memory112.

Asset set-up and visual verification information122fmay also be displayed. A targeted asset is visual through the device's enhanced reality field of view. The camera104of the device can record installation and take an image of the completed installation at the installer's request. The installer can set up the asset through the use of a virtual keyboard126, overlaid on the user's field of view through the display104. Alternatively, the user can advance through the installation of the asset using other means for providing user input, such as the microphone106or hand-operated controller device.

FIG. 6shows various types of asset information122that can be displayed on the display102in accordance with the enhanced or augmented/mixed reality display system described herein. In particular, the asset information122shown inFIG. 6relates to asset energy diversion identification, prioritization and localization.

Asset selection and statistics information122gcan be displayed, in which energy diversion assets are highlighted geo-spatially and organized via specific filtering requirements of the user. An asset can be selected for the display of asset selection and statistics information122gas described previously herein in connection with the selection of an asset for display of other information relating to the asset. Current and historical data may be selected and displayed for these assets to identify the extent of energy diversion. Information helpful to the user is included regarding installation, set-up, access and local surroundings. Summary data regarding local and regional energy diversion statistics are summarized with this asset, providing the capability to prioritize instances of energy diversion within a region, indicate how much of the diversion in the region is resulting from a particular asset, and correct accordingly. A ranking of assets in a region can be displayed, in addition to, for example, the percentage of total diversion in the region the asset is responsible for and the location of the asset. The status of a location may also be indicated, which would identify the accessibility of the location of the asset and any particular notes pertinent to the location of the diversion. For example, if the particular location is within an area that is inaccessible by roads, the user may modify their route and transportation for reaching the location.

Asset routing122ainformation and asset radar information122bmay also be provided in connection with energy diversification information. The asset radar information122bcan be configured in a manner to allow the user to focus on potential issues or specific cases of energy diversion.

In accordance with a further aspect of the present invention, the head-mounted user device100may be used to guide a user through one or more work orders in the field. An example of the presentation and selection of a work order using the head-mounted user device100is shown inFIG. 7.

After the user initiates the head-mounted user device100, the head-mounted user device100is configured to receive a work order (W/O) list from a server, typically provided from a cloud computing environment, and store the list in the memory112of the head-mounted user device. The work order list is displayed on the display102of the device100. The user selects a particular work order relating to an asset using one of the input methods described herein, and directions to the asset are determined by the device100and displayed, as described previously herein. When the user arrives at the location of the asset, the processor110of the device100is configured to initiate an interactive guide for the work order, which includes a series of interactive directives that are displayed by the device by overlaying the directives on the display102for the user to view while conducting the work required by the work order. The work order directives can be preloaded into the memory112of the device100, together with any notes from a supervisor regarding the work order or asset, or they may be sent to the device100from a remote location as the user is proceeding through the work order, such as downloaded from a cloud based server or sent from another user device at a separate location interacting with the device100as the user proceeds through the work order. When the user is at the asset location, the device100is configured to display to the user information regarding the asset, such as the asset parameters, parameter history and average usage. The device100may be configured to automatically display such information about the asset when it is determined that the device100is within a certain distance from the asset, such as five to ten feet, in accordance with asset location determination methods described herein. After the user proceeds through the work order directives, the work order can be considered complete or incomplete, and the user can proceed to selecting a further work order.

One example of a work order interactive guide is shown inFIG. 8and described below, relating to a user responding to an instance of potential energy diversion. The head-mounted user device100displays a series of prompts to the user to guide the user through the investigation and possible repair.

First, the user is instructed to perform visual inspection of power lines to premise to see if there are visible signs of tampering. The user is instructed to determine if there are wires running from the transformer to a home, bypassing a meter. If so, the user is directed to photograph the lines using the camera104and complete the order. If not, the user is directed to continue with the prompts, which directs the user to inspect the meter and meter socket. If the meter seal is broken, the user is directed to photograph the lines using the camera104and complete the order. If the meter seal has not been tampered with, the user is directed to determine if a circuit breaker box is located outside the home, and if it is, to inspect the box to determine if there are any bypass wires entering from a meter socket. If bypass wires are entering the circuit breaker box from the meter socket, the user is directed to photograph the lines using the camera104and complete the order. If the meter seal has not been broken or if there are not bypass wires entering from the meter socket, the user is directed to contact law enforcement or security personal to attempt to gain access to the premise. If access is granted, the user is directed to inspect at the home, the power lines and meter for tampering or bypass. If a bypass or tampering is found, the user is directed to document it and complete the work order. If no bypass or tampering is found, the user is directed to complete the work order. On completion of the work order, the user is directed to enter any additional notes or comments resulting from the work order.

A further example of an interactive work order guide generated by the head-mounted user device is explained below for a meter inspection process.

After the user arrives to the location of the meter to be inspected, if the meter is inside the premises and the user is not able to gain access to the meter, the user is directed to mark the work order as incomplete. If the user is able to gain access to the meter, the user is directed to proceed with inspection of the meter. The user is first directed to inspect the meter to determine if the meter is functioning. If the meter is not functioning, the user is directed to determine if the meter can be replaced at that time. If the user is able to replace the meter, a new interactive guide is presented to the user for the meter replacement procedure. If the meter cannot be replaced, the work order is determined to be incomplete and additional work is required. If the meter is functioning, the user is directed to inspect the meter box to ensure that the lines are properly connected.

If the meter lines are not properly connected, the user is directed to photograph the improperly connected meter and wiring with the camera104and then fix the installation. After the installation has been fixed, the user is directed to photograph the repaired meter and wiring with the camera104, and mark the work order complete. If the lines are connected properly at the meter box, the user is directed to complete the work order and note that no issue found with the meter.

THE SCOPE OF THE INVENTION