MACHINE LEARNING SYSTEM FOR NATURAL GAS LEAK DETECTION

Various embodiments of the present technology relate to solutions for gas leak detection in natural gas extraction and storage environments. In some examples, a machine learning interface generates feature vectors based on video data that depicts a natural gas storage facility and feeds the feature vectors to a machine learning engine. The machine learning engine ingests the feature vectors and generates a machine learning output that indicates the presence of the gas leak in the natural gas storage facility. The machine learning interface receives the machine learning output that indicates the presence of a gas leak in the natural gas storage facility. The machine learning interface generates and transfers an alert based on the machine learning output.

TECHNICAL FIELD

Various embodiments of the present technology relate to natural gas extraction technologies, and more specifically, to detecting and classifying natural gas leaks using machine learning systems.

BACKGROUND

Natural gas extraction systems comprise machinery and equipment configured to extract natural gas from underground reservoirs for use in energy generation, heating, and chemical production applications. Natural gas extraction systems comprise extraction equipment, transfer equipment, and storage equipment. The extraction equipment is configured to remove natural gas from underground gas reservoirs. Examples of extraction equipment include hydraulic fracturing devices. The transfer equipment is configured to move the natural gas between different geographic locations. Examples of transfer equipment include gas pipelines. The storage equipment is configured to store the natural gas over an extended period of time. Examples of storage equipment include bullet tanks and gasholders. The natural gas extraction systems often comprise manufacturing defects or develop defects over time which result in gas leaks. The gas leaks may result in lost revenue and environmental pollution. For example, a valve in a natural gas pipeline may develop a defect which allows gas to escape from the pipeline. Due to the large amount of extraction, transfer, and storage equipment in the natural gas extraction systems, the natural gas leaks can be difficult to detect allowing the leaks to persist over time. Moreover, natural gas leaks are difficult to view in the visible light spectrum which compounds the difficulty of spotting gas leaks.

Machine learning algorithms are designed to recognize patterns and automatically improve through training and the use of data. Examples of machine learning algorithms include artificial neural networks, nearest neighbor methods, gradient-boosted trees, ensemble random forests, support vector machines, naïve Bayes methods, and linear regressions. Some machine learning models comprise supervised learning models. A supervised machine learning algorithm comprises an input layer and an output layer, wherein complex analyzation takes places between the two layers. Various training methods are used to train machine learning algorithms wherein an algorithm is continually updated and optimized until a satisfactory model is achieved. One advantage of supervised learning machine learning algorithms is their ability to learn by example, rather than needing to be manually programmed to perform a task, especially when the tasks would require a near-impossible amount of programming to perform the operations in which they are used. Unfortunately, natural gas extraction systems do not effectively and efficiently use machine learning algorithms to detect natural gas leaks in their extraction, transfer, and storage equipment.

Overview

Various embodiments of the present technology relate to solutions for leak detection systems in natural gas extraction and storage environments. Some embodiments comprise a method of operating a leak detection system to detect gaseous leaks in a natural gas storage environment. The method comprises generating feature vectors based on video data that depicts a natural gas storage facility. The method further comprises providing the feature vectors as input to a machine learning system wherein the machine learning system indicates a presence or lack thereof of a natural gas leak in the natural gas storage environment. The method further comprises generating and transferring data for rendering a user interface that comprises the indication.

Some embodiments comprise a leak detection system to detect gaseous leaks in a natural gas extraction environment. The leak detections system comprises a machine learning interface and a machine learning engine. The machine learning interface generates feature vectors based on video data that depicts a natural gas storage facility. The machine learning interface feeds the feature vectors to the machine learning engine. The machine learning engine generates a machine learning output that indicates the presence of a gas leak in the natural gas storage facility. The machine learning interface receives the machine learning output that indicates the presence of a gas leak in the natural gas storage facility. The machine learning interface generates and transfers data for rendering a user interface that comprises the indication.

Some embodiments comprise a non-transitory computer-readable medium stored thereon instructions to detect gaseous leaks in a natural gas extraction environment. The instructions, in response to execution, cause a system comprising a processor to perform operations. The operations comprise generating feature vectors based on video data that depicts a natural gas storage facility. The operations further comprise providing the feature vectors as input to a machine learning system wherein the machine learning system indicates a presence or lack thereof of a gas leak in the natural gas storage environment. The operations further comprise generating and transferring data for rendering a user interface that comprises the indication.

DETAILED DESCRIPTION

FIG.1illustrates environment100to detect gaseous leaks in storage, transfer, and extraction equipment. Environment100performs services like natural gas storage, natural gas transfer, natural gas extraction, natural gas leak detection, and natural gas leak notification. Environment100comprises storage tanks101-103, sensors104, gas leak110, camera121, camera mount122, model computer131, model132, user computer141, application142, and cloud services151. In other examples, environment100may include fewer or additional components than those illustrated inFIG.1. Likewise, the illustrated components of environment100may include fewer or additional components, assets, or connections than shown. Model computer131and user computer141may be representative of a single computing apparatus or multiple computing apparatuses.

Storage tanks101-103are representative of natural gas storage devices. Exemplary natural gas storage devices include bullet tanks, Liquified Natural Gas (LNG) storage tanks, gasholders, storage vehicles, and/or other types of natural gas storage systems. In some examples, environment100may comprise additional devices for natural gas extraction and natural gas transfer. For example, environment100may comprise hydraulic fracturing equipment, natural gas pipeline equipment, gas storage and transfer vehicles, and the like.

Sensors104are representative of telemetry devices configured to measure and report storage metrics like tank temperature, tank pressure, vent status, tank volume, tank location, tank type, and/or other types of telemetry for tanks101-103. Exemplary telemetry devices of sensors104include thermometers, pressure gauges, flowrate gauges, on/off indicators, Global Positioning System (GPS) devices, and the like. Sensors104are operatively coupled to tanks101-103. Sensors104interact with tanks101-103to generate telemetry data and report the telemetry data to model computer131. Sensors104also provide environmental telemetry like temperature, pressure, wind speed, wind direction, clouds, visibility, humidity, dew point, and the like. The environmental telemetry metrics improve machine learning model performance (e.g., performance of model132) and help the machine learning model better understand the condition of environments when the gas is leaking.

Gas leak110is representative of a natural gas outflow from storage tank101. In some examples, gas leak110is the result of defect in tank101allowing the natural gas contained within tank101to leak out. The defect may comprise a crack, faulty valve, or some other type of defect that compromises the storage ability of tank. In some examples, gas leak110is the result of intentional gas venting from tank101.

Imaging camera121is representative of one or more imaging systems to view tanks101-103and generate videos depicting tanks101-103. In this example, camera121generates infrared and/or optical video images depicting tanks101-103, however in other examples, camera121may employ a different type of imaging technology. For example, camera121may instead comprise an ultraviolet imaging system. It should be understood that natural gas leaks are difficult to view in the visual light spectrum. As such, camera121typically comprises imaging technology for generating images in non-visible spectrums (e.g., infrared). Although camera121is illustrated as a single imaging device, in some examples camera121may comprise multiple imaging devices. The multiple cameras of camera121may include a combination of optical, infrared, and/or laser cameras and imaging devices to improve the gas leak detection. Imaging camera121may also include distance metric devices like laser rangefinders to estimate the distance between gas leak110and camera121to improve leak estimation. Imaging camera121transfers the videos depicting tanks101-103to model computer131. Camera121is mounted on camera mount122. Although camera mount122is depicted as a pole, camera mount122may comprise a different type of mounting structure or camera121may use no mounting structure at all. Camera mount122may include a pan and tilt system that moves the camera in multiple directions and orientations to cover a wider range and stabilize the field of view. Camera mount122may comprise a controller to move camera121to pre-defined views and control the direction of camera121to provide a 360-degree field of view with camera121. The controller of camera mount122may receive instructions (e.g., from model computer131) and responsively position camera121to find gas leak110and stay on view of gas leak110.

Model computer131is representative of one or more computing devices configured to receive video data from camera121and telemetry data from sensors104and to identify the presence of gas leak110. The one or more computing devices of model computer131host machine learning model132. For example, computer131may comprise an application specific circuit configured to implement a machine learning model. Model computer131may additionally host interfacing applications to receive and preprocess the video and telemetry data from camera121and sensors104. The interfacing applications may vectorize the received data to configure the data for ingestion by model132. For example, vectorization may comprise a feature extraction process to numerically represent the received data. In some examples, computer131may generate feature vectors that represent individual pixels of the video data received from camera121.

Machine learning model132comprises any machine learning model implemented within environment100as described herein to detect the presence of gas leaks. A machine learning model comprises one or more machine learning algorithms that are trained based on historical data and/or other types of training data. A machine learning model may employ one or more machine learning algorithms through which data can be analyzed to identify patterns, make decisions, make predictions, or similarly produce output that can identify the presence of gas leaks in environment100. Model132may comprise algorithms to detect background environments, to detect motion, to detect equipment, to classify gas leaks, and/or other types of machine learning algorithms. Examples of machine learning algorithms that may be employed solely or in conjunction with one another include Three Dimensional (3D) deep leaning models, 3D convolutional neural networks, times series convolutional deep learning, transformers, multi-layer perceptron, long term short memory, and attention based deep learning model. Other exemplary machine learning algorithms include artificial neural networks, nearest neighbor methods, ensemble random forests, support vector machines, naïve Bayes methods, linear regressions, or similar machine learning techniques or combinations thereof capable of predicting output based on input data. Machine learning model132may be deployed on premises in environment100(e.g., proximate to tanks101-103) or at a remote location in the cloud.

Machine learning model132may be trained to detect gas leaks using videos generated by camera121. For example, camera121may transfer the training video images to user computer141. A user may then annotate image frames of the video to create a training data set. The user may also combine environment and equipment information in the training data set. The annotations classify or segment portions of the image frames. For example, the annotations may classify a portion of the images as storage tanks101-103, another portion of the images as a gas leak, and another portion of the images as background environment. User computer141transfers the training data to model computer131to train model132. Computer131receives and vectorizes the training data. Model132ingests the training data and trains its constituent machine learning algorithms to detect gas leak110using the training data.

User computer141is representative of one or more computing devices configured to display application142via a Guided User Interface (GUI). User computer141comprises one or more computing devices, display screens, touch screen devices, tablet devices, mobile user equipment, keyboards, and the like. User computer141is operatively coupled to model computer131. User computer141may be deployed at a remote location, on premises in environment100(e.g., proximate to tanks101-103), or both. User computer141and model computer131may be located at different geographic locations. Alternatively, user computer141may be co-located with model computer131. Application142comprises a user interface application to display gas leak footage (e.g., pictures and/or video), gas leak metrics (e.g., leak probability and/or leak flow rate), and/or other visual/textual elements that characterize gas leaks detected in environment100based on machine learning outputs generated by model132. In this example, application142is illustrated comprising visual elements for leak footage and leak metrics, however in other examples, application142may comprise different or additional visual elements. User computer141may send some or all of the model results to cloud service151to distribute the leak indication results for other use cases including reporting, saving historical data, presentation, and/or combining with different models or databases.

Sensors104, camera121, model computer131, user computer141, and cloud services151communicate over various communication links using communication technologies like Institute of Electrical and Electronic Engineers (IEEE) 802.3 (ENET), IEEE 802.11 (WIFI), Bluetooth, Time Division Multiplex (TDM), Data Over Cable System Interface Specification (DOCSIS), Internet Protocol (IP), General Packet Radio Service Transfer Protocol (GTP), and/or some other type of wireline and/or wireless networking protocol. The communication links comprise metallic links, glass fibers, radio channels, or some other communication media. The links use ENET, WIFI, virtual switching, inter-processor communication, bus interfaces, and/or some other data communication protocols.

Sensors104, camera121, model computer131, user computer141, and cloud services151comprise microprocessors, software, memories, transceivers, bus circuitry, and the like. The microprocessors comprise Central Processing Units (CPUs), Graphical Processing Units (GPUs), Digital Signal Processors (DSPs), Application-Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), analog computing circuits, and/or the like. The memories comprise Random Access Memory (RAM), flash circuitry, Hard Disk Drives (HDDs), Solid State Drives (SSDs), Non-Volatile Memory Express (NVMe) SSDs, and/or the like. The memories store software like operating systems, user applications, networking applications, machine learning applications, and the like. The microprocessors retrieve the software from the memories and execute the software to drive the operation of environment100as described herein.

In some examples, environment100implements process200illustrated inFIG.2. In some examples, environment100implements process300illustrated inFIG.3. It should be appreciated that the structure and operation of environment100may differ in other examples.

FIG.2illustrates process200. Process200comprises a leak detection process in a natural gas extraction and storage environment. In other examples, process200may differ. Process200may be implemented in program instructions in the context of any of the software applications, imaging components, module components, machine learning components, or other such elements of one or more computing devices. The program instructions direct the computing device(s) to operate as follows, referred to in the singular for the sake of clarity.

In operation, process200begins by viewing natural gas storage systems and generating video footage depicting the natural gas storage systems (step201). The operation continues by generating feature vectors that represent the video footage (step202). The operation continues by running machine learning gas leak detection models to generate a machine learning output which may indicate a gas leak (step203). The operation continues by processing a machine learning output that indicates the presence of a gas leak (step204). The operation continues by sending the output to different user interfaces or locations using some communication channels and protocols (step205). The operation continues by displaying the machine learning output on a user interface (step206). The operation continues by transferring an alert (step207).

Referring back toFIG.1, environment100includes a brief example of process200as employed by one or more applications hosted by the various computing, camera, and sensor devices comprising environment100.

In operation, camera121views tanks101-103and their surrounding environment. For example, camera mount122may rotate camera121to focus the field of view of camera121on tanks101-103. Camera121generates video footage by viewing tanks101-103(step201). The video footage comprises a sequence of infrared and/or optical image frames that form a video depicting tanks101-103and their surrounding environment. Camera121transfers video footage to model computer131. Camera121may transfer the video footage to computer131over wired links or over wireless links using wireless networking protocols like Bluetooth. Concurrently, sensors104generate telemetry data like temperature, pressure, and venting status for tanks101-103and generate telemetry data for the environment like windspeed, pressure, temperature, and the like. For example, a valve indicator of sensors104may be attached to tank101and generates telemetry data that indicates when tank101is intentionally venting gas. Sensors104transfer the telemetry data to model computer131.

Model computer131receives video footage and telemetry data from camera121and sensors104. Model computer131vectorizes the received data to configure the video footage and the telemetry data for ingestion by machine learning model132(step202). Model computer131may host an interface application to vectorize the received data. For example, the interface application may implement a feature extraction process on the video footage. The feature extraction processes may comprise assigning numeric values to each pixel in the images that comprise the video footage. The interface application may group the numeric values of corresponding pixels from different ones of the video frames to generate feature vectors. Once generated, the interface application may transfer the feature vectors to machine learning model132.

Machine learning model132ingests and processes the feature vectors representing the video footage and the telemetry data using its constituent machine learning algorithms (step203). Machine learning model132comprises background detection algorithms, motion detection algorithms, equipment detection algorithms, segmentation algorithms, and leak identification algorithms. Machine learning model132generates a machine learning output that indicates the presence or lack thereof of gas leak110on storage tank101. For example, machine learning model132may utilize its background detection algorithms to identify the background environment (e.g., area surrounding tanks101-103) in the video footage and subtract the background environment from the video footage. The background detection algorithms may perform two distinct background environment subtractions for long term and short-term portions of the video. For example, the background detection algorithms may identify the background based on 500 image frames of the video footage to perform the long-term background environment subtraction and identify the background based on 30 image frames of the video footage to perform the short-term background environment subtraction.

Model132utilizes its motion detection algorithms to identify motion within the video footage. The motion detection algorithms identify differences in corresponding pixels of different frames in the video to identify the motion. For example, model132may identify a first pixel in a first image frame and a corresponding first pixel in a consecutive second image frame and identify motion when the corresponding first pixels in the two frames differ. Model132utilizes its equipment detection algorithm to locate tanks101-103in the video footage. Model132utilizes its leak identification algorithms to determine the presence of gas leak110. The gas leak identification algorithm may determine the probability that the detected motion comprises a gas leak based on the long-term background subtraction, the short-term background subtraction, the detected motion, the motion duration, the equipment location, and the telemetry data.

Machine learning model132uses image segmentation and object detection models to identify equipment and objects in the video footage. Object detection algorithms identify part of the image as segments that correspond to specific devices, people, cars, equipment, and the like. For example, model132may identify a group of pixels in a frame that corresponds to a segment that can be identified by a human as a known object. Model132utilizes object detection model to reduce false positive from the gas leak detection and correlate the leaks to specific object that may cause the leak.

When the gas leak probability exceeds a threshold value, the leak identification algorithm classifies the detected motion (in this example representing gas leak110) as a gas leak (step204). For example, the gas leak identification algorithm may determine the motion was not subtracted in the long-term and short-term background subtractions, is co-located with the location of tank101, and that tank101is not currently venting, and in response determine the probability gas leak110exists is very high (e.g., 95% probability). Model132generates a machine learning output that indicates the presence of gas leak110, the probability leak110exists, the location of leak110and tank101, and the estimated flowrate of gas leak110. Model132transfers the machine learning output to user computer141that indicates the presence of gas leak110(step205). In some examples, model132may detect multiple gas leaks. Model132may classify the multiple gas leaks as a single gas leak based on their proximity, duration, and time of observation. The leak can also have probability in time, meaning that if the leak is detected longer in time, then its probability of true detection is higher vs leaks that are detected in very short period of time which will assign as low probability leak.

User computer141receives and processes the machine learning output and application142displays video footage and leak metrics indicated by the machine learning output (step206). For example, the leak footage may comprise a video with portions of the video depicting gas leak110highlighted by model132. The leak metrics may comprise metrics like leak rate and probability of the leak occurring. Application142receives user inputs via user computer141to transfer an alert to respond to the gas leak. Application142transfers the alert (e.g., to on-site personnel) to confirm the presence of and respond to gas leak110on tank101(step207). For example, the alert may comprise the location of tank101, indicate the presence of leak110on tank101, the time/date the leak was detected, and/or other information to facilitate a response to gas leak110. Although model132is configured for gas leak detection, in other examples model132may be configured to detect liquid leaks, detect smoke, detect fire, and/or monitor equipment.

Advantageously, environment100effectively utilizes machine learning systems to detect natural gas leaks in natural gas extraction, transfer, and storage equipment. Moreover, environment100employs machine learning model132to ingest infrared or optical videos depicting storage tanks101-103and identify natural gas leak110based on the infrared or optical videos.

FIG.3illustrates process300to detect gas leaks in a natural gas extraction and storage environment. In other examples, process300may differ. Process300is illustrated as a functional block diagram and includes feature vectors block301, motion detection model block302, background detection model block303, object detection model block304, gas leak detection model block305, and non-linear function block306. Process300may be implemented in program instructions in the context of any of the software applications, imaging components, module components, machine learning components, or other such elements of one or more computing devices. Feature vectors301are representative of machine learning inputs to models302-305that comprise a numerical representation of infrared or optical video data that depicts a natural gas extraction and/or storage environment. Motion detection model302comprises a machine learning model trained to ingest feature vectors301and identify motion in the infrared or optical video data. Background detection model303comprises a machine learning model trained to ingest feature vectors301and identify the background environment in the infrared or optical video data. Object detection model304comprises a machine learning model trained to ingest feature vectors301and segment the image in the infrared or optical video data to identify and classify natural gas storage, extraction, and or transfer equipment. Gas leak detection model305comprises a machine learning model trained to ingest feature vectors301and identify gas leaks in the infrared or optical video data. Nonlinear function306comprises a machine learning model configured to ingest outputs from models302,303,304, and305and generate a gas leak probability indication. Models302-305and function306may be hosted by the same computing device or different computing devices.

In operation, a computing device receives infrared or optical video data depicting the natural gas storage environment. The computing device vectorizes the infrared or optical video data to generate feature vectors301. For example, the computing device may numerically represent the pixels in the image frames that comprise the infrared video data and form the feature vectors using the numeric representations. Motion detection model302, background detection model303, object detection model304, and gas leak detection model305each ingest feature vectors301.

Motion detection model302determines regions of the infrared or optical video data that depict motion, including motion representing a gas leak in the infrared or optical video data. For example, motion detection model302may compare corresponding pixels in sequential video frames of the infrared or optical video data to identify changes between the corresponding pixels (e.g., changes in color). Model302may then classify the identified changes in the pixels as motion depicted in the infrared or optical video data.

As stated above, the infrared or optical video data depicts a natural gas storage environment. The storage environment comprises machinery and equipment that extract, store, and transport natural gas. The machinery and equipment that extract, store, and transport natural gas are viewable in the infrared or optical video data and may develop gas leaks that are also viewable in the infrared or optical video data. Other features depicted in the infrared or optical video data may be classified as background environment. Exemplary background environment comprises the sky, the ground, plant and animal life, equipment not associated with gas extraction (e.g., an automobile), buildings, and the like. Background detection model303determines regions of the infrared or optical video data that depict the background environment. For example, background detection model303may compare corresponding pixels in sequential video frames of the infrared or optical video data to identify regions of the infrared or optical video data that are relatively unchanging to identify the background environment. Background detection model303may perform long-term and short-term background detection. Long-term background detection may comprise a background detection process using 500 frames of infrared or optical video data. The short-term background detection may comprise a background detection process using 30 frames of infrared or optical video data. Background detection model303may combine short and long-term background detection analysis to fully identify the background environment depicted in the infrared or optical video data.

Object detection model304ingests the feature vectors that represent the infrared or optical video data and segments parts of the frames that correspond to a known object in the field of view of the camera. Using object detection helps reduce gas leak detection false positive and relates each leak with some probability to an actual device in the field. Understanding which device is causing the leak helps to better manage and focus on design and deployment of different devices. Object detection model304generates an output that indicates regions of the infrared or optical video data that comprise natural gas extraction, storage, and transfer equipment.

Gas leak detection model305ingests the feature vectors that represent the infrared or optical video data and identifies the movement in the video data that is corresponding to a leak. Gas leak detection model305detects a segment of an image in series of images as a leak based on the similarity of movement to a gas. Gas leak detection model305may calculate the flowrate of the gas leak based on distance of the camera to the leak, speed of leak movement in the video, pixel resolution of the video, and environmental conditions like wind speed and direction.

Motion detection model302, background detection model303, object detection model304, and gas leak detection model305transfer their machine learning outputs to a nonlinear function306. Nonlinear function306is a machine learning model that is configured to combine the machine learning outputs generated by models302-305to determine and infer the presence of a gas leak. Nonlinear function306subtracts the portions of the infrared or optical video data identified as background environment from the portions of the infrared or optical video data identified as depicting motion to identify all regions of the video data that depict motion that is not part of the background environment. For example, this may subtract portions of the video data that depict the motion of clouds. Nonlinear function306then subtracts regions of the infrared or optical video depicting equipment from the remainder resulting from the background subtraction to identify all regions of the video data that depict both motion and natural gas equipment. Nonlinear function306then compares the remainder of the image resulting from the equipment subtraction and the background subtraction to the region of the video data identified by gas leak detection model305as a possible gas leak. When the remainder of the video data resulting from the subtractions overlaps with the region of the video data identified by gas leak detection model305as a possible gas leak, nonlinear function306confirms the presence of a gas leak. Generally, motion depicted by the infrared or optical video data that is not part of the background environment, that is co-located with a piece natural gas equipment, and that has been identified as a possible gas leak by detection model305may be classified as a gas leak. By performing multiple image subtractions, nonlinear function306inhibits false-positive gas leak detection by detection model305. Nonlinear function306outputs an indication of as to whether a gas leak has been detected in the infrared or optical video data. The indication may comprise a probability/confidence metric regarding the existence of the gas leak (e.g., 85% chance detected leak is real). The indication may comprise a gas flow rate estimate for the detected gas leak and equipment identification numbers to indicate the location of the gas leak. Nonlinear function306transfers the gas leak indication to user computing systems for review by human operators.

FIG.4illustrates environment400to detect gaseous leaks in natural gas storage, transfer, and extraction equipment. Environment400comprises an example or environment100, however environment100may differ. Environment400comprises natural gas extraction environment410, leak detection environment420, and user environment430. Natural gas extraction environment410comprises tank411, leak412, sensors413, mount414, and camera415. Leak detection environment420comprises machine learning interface421and machine learning engine423. Machine learning interface hosts feature extraction application422. Machine learning engine hosts machine learning (ML) models424which comprise background detection model425, motion detection model426, object detection model427, leak detection model428, and non-linear function model429. User environment430comprises user computer431and display432. In other examples, environment400may include fewer or additional components than those illustrated inFIG.4. Likewise, the illustrated components of environment400may include fewer or additional components, assets, or connections than shown.

Natural gas extraction environment410is representative storage facility for natural gas. Tank411comprises a bullet tank that stores gas and comprises a defect that allows the stored gas to escape as gas leak412. In this example, the defect comprises a defective gasket. As illustrated inFIG.4, environment410comprises other objects like a building, truck, and human operator. Sensors413is representative of a sensor suite comprising an on/off venting indicator, a tank thermometer, an environment thermometer, a tank pressure gauge, an atmospheric barometer, environmental hygrometer, and a wind gauge. A portion of sensors413are coupled to tank411via wired links. For example, the on/off venting indicator, tank thermometer, and tank pressure gauge may be mounted directly on tank411while the remaining sensors may be positioned in the environment proximate to tank411(e.g., on a weather station).

Camera415comprises a Forward Looking Infrared (FLIR) camera attached to mount414via a pan and tilt system. Camera415records tank411and its surrounding environment including any buildings, vehicles, human operators, and the like. Camera415additionally comprises a laser rangefinder to measure distances. The pan and tilt system may rotate camera415along a horizontal axis perpendicular to mount414and may adjust the roll, yaw, and pitch to focus camera on a desired field of view. The pan and tilt system comprises electric motors, actuators, and the like that operate in response to control signaling received from a device controller. Sensors413and camera415are communicatively coupled to machine learning interface421over wired and/or wireless links. The links may comprise a private Local Area Network (LAN) or may traverse public internet links supported by internet backbone providers.

Machine learning interface421comprises a computing device communicatively coupled to sensors413, camera415, and machine learning engine423that hosts feature extract application422. Feature extraction application422is representative of one or more applications, modules, and the like to process input data received from environment410into a consumable format for machine learning models424. In particular, application422generates numeric representations of video frames recorded by camera415. Application422groups the numeric representations into feature vectors and provides the vectors as input to engine423. Machine learning interface421also hosts applications for rendering user interfaces. In particular, the user interface applications generate interfaces to depict outputs from engine423.

Machine learning engine423comprises a computing device that hosts machine learning models424. Background detection model425comprises algorithms trained to detect background environments in video footage depicting natural gas extraction environment410. Motion detection model426comprises algorithms trained to detect movement in video footage depicting natural gas extraction environment410. Object detection model427comprises algorithms trained to classify natural gas storage equipment in video footage depicting natural gas extraction environment410. Leak detection model428comprises algorithms trained to detect possible gas leaks in video footage depicting natural gas extraction environment410. Non-linear function model429comprises algorithms trained to output probability indications for the presence of a gas leak based on the output of the other models.

User computer431is a computing device communicatively coupled to machine learning interface421that hosts an operating system and one or more user applications to display leak indication outputs received from environment420. User computer431comprises display432. User computer431renders a GUI on display432that comprises the leak indication, the infrared video footage, the predicted leak flowrate, tank ID number, and tank location. User computer431generates and transfers alerts in response to user input to notify human operators in extraction environment410when a gas leak is detected. In some examples, user computer430may host machine learning training applications to annotate training data sets to train machine learning models424to detect gas leaks in extraction environment410.

FIG.5illustrates process500. Process500comprises an exemplary operation of environment400to detect natural gas leak412from tank411. In operation, sensors413generate telemetry data for tank411and transfer the telemetry data to machine learning interface421. Sensors413detect tank pressure, tank temperature, operator-initiated venting, environmental temperature, environmental pressure, environmental humidity, and environmental wind speed. Sensors413generate and transfer telemetry data that comprises the aforementioned sensor measurements. The telemetry data may be reported continuously or discretely. For example, sensors413may continuously transfer their measurements or may generate and transfer telemetry reports periodically. Camera415records infrared video depicting tank411and transfers the video footage to machine learning interface421. Camera415uses its rangefinder to determine the distance between camera415and tank411. At some point in time during the operation of sensors413and camera415, tank411develops leak412. The telemetry data and infrared video footage changes in response to the emergence of leak412. For example, when leak412occurs, the measured tank pressure may decrease, and the leak may be visible in the infrared footage captured by camera415.

Feature extraction application422, hosted by interface421, implements a feature extraction process on the received telemetry data and video footage. Machine learning interface421provides the feature vectors to machine learning engine423. Each of models425-428ingests the feature vectors from vectorization application422. Background detection model425processes the feature vectors using its constituent background detection algorithms to identify regions of the infrared video data that depict background portions of extraction environment410. For example, background detection model425may classify the human operator, truck, and building in environment410as background. Motion detection model426processes the feature vectors using its constituent motion detection algorithms to identify regions of the infrared video data that depict motion in extraction environment410. For example, motion detection model426may classify the movements of human operator, the movement of the truck, and gas leak412in environment410as motion. Object detection model427processes the feature vectors using its constituent object detection algorithms to identify regions of the infrared video data that depict natural gas storage equipment in extraction environment410. For example, object detection model427may classify tank411in environment410as natural gas storage equipment. Leak detection model428processes the feature vectors using its constituent leak detection algorithms to identify regions of the infrared video data that depict natural gas leaks in extraction environment410. For example, object detection model427may classify gas leak412in environment410as a potential gas leak.

Models425-428provide their machine learning outputs to non-linear function model429. Non-linear function model429performs a series of image subtractions to confirm the presence of leak412. Model429subtracts the background detection output, the objection detection output, and the motion detection output from the video footage and compares the remainder to the leak detection output. When the overlap between the leak detection output and the remainder exceeds a threshold value (i.e., 90% overlap), non-linear function model429generates an output confirming the presence of leak412. Model429includes contextual information for leak412like tank ID number, latitude and longitudinal coordinates, leak flowrate, raw video footage, leak probability, and the like. Model429transfers the leak indication to machine learning interface421. Machine learning interface421formats the indication for rendering on the display of user computer431and forwards the indication to user computer431.

User computer431displays the indication on display432. The display includes the indication confirming the existence leak412, a confidence percentage in the indication, tank GPS coordinates, data and time, and tank ID number. User computer431receives a user input to notify extraction environment operators and responsively transfers an alert.

FIG.6illustrates model training input600. Model training input600comprises an annotated image to train a machine learning model (e.g., object detection model427) to identify natural gas storage equipment. Training input600comprises footage601, video frame602, bullet tanks611, marked pixels612, and machine learning object indications613. Video frame602comprises an infrared still frame image from footage601of bullet tanks611. A human operator marks pixels that depict bullet tanks611. For example, marked pixels612may comprise user generated annotations that identify the shape and location of the bullet tanks within the still frame image. The annotations may comprise colored markings. Training input600is provided to a machine learning model to advance its constituent algorithms to classify bullet tanks in video images based on marked pixels612. The model segments the training data to generate machine learning object indications613that identify bullet tanks611. The human operator reviews object indications613and either rejects or accepts indications613based on their accuracy.

FIG.7illustrates model training input700. Model training input700comprises an annotated image to train a machine learning model to identify natural gas storage equipment. Training input700comprises footage701, video frame702, cooler towers711, marked pixels712, and machine learning object indications713. Video frame702comprises an infrared still frame image from footage701of cooler towers711. Similar to training input600, a human operator marks pixels that depict cool towers711with user generated annotations that identify the shape and location of the cooler towers711in video frame702. Training input700is provided to a machine learning model to advance its constituent algorithms to classify cooler towers in video images based on marked pixels712. By providing different types of object detection training inputs to the machine learning model, the machine learning model is able to classify a greater number of natural gas extraction equipment types. In should be appreciated that training inputs600and700may also be used to train background detection machine learning models (e.g., background detection model425). For example, inputs600or700may be provided by a machine learning model to train the model to detect portions of frames602or702that do not comprise the marked pixels to classify background environments.

FIG.8illustrates model training input800. Model training input800comprises an annotated image to train a machine learning model (e.g., motion detection model426) to identify motion. Training input800comprises footage801, video frame802, marked pixels811, and machine learning indications812. Video frame802comprises an infrared still frame image from footage801of natural gas storage site. A human operator marks pixels that depict motion within footage from801. For example, the human operator may mark environmental motion, human movements, vehicle movements, and natural gas leaks as motion. Alternatively, a computing system may annotate video frame802. Once annotated, training input800is provided to a machine learning model to advance its constituent algorithms to classify motion in video images based on marked pixels811. The model segments the training data to generate machine learning motion indications812that identify the motion depicted in footage801. The human operator reviews motion indications812and either rejects or accepts indications812based on their accuracy.

FIG.9illustrates model training input900. Model training input900comprises an annotated image to train a machine learning model (e.g., leak detection model428) to identify natural gas leaks. Training input900comprises footage901, video frame902, marked pixels911, and machine learning object indications912. Video frame902comprises an infrared still frame image from footage901of a gas leak from a natural gas storage equipment. A human operator annotates the pixels that depict the gas leak to identify the shape, motion, size, and/or other leak characteristics within the still frame image. The annotations comprise colored markings. Training input900is provided to a machine learning model to advance its constituent algorithms to classify gas leaks in video images based on marked pixels912. The model segments the training data to generate machine learning leak indications912that identify the gas leak. The human operator reviews leak indications912for accuracy.

FIG.10illustrates model training input1000. Model training input1000comprises an annotated image to train a machine learning model to identify natural gas leaks. Training input1000comprises footage1001, video frame1002, marked pixels1011, and machine learning object indications1012. Video frame1002comprises an infrared still frame image from footage1001of a gas leak from a natural gas storage equipment. Similar to input900, training input is annotated to mark the shape, motion, size, and/or other leak characteristics of the gas leak. In should be appreciated that train inputs600,700,800,900, and1000are exemplary and may vary in other examples.

FIG.11illustrates model output1100. Model output1100comprises an output from a machine learning model (e.g., non-linear function model429) that indicates the presence of a gas leak. Model output comprises footage101, video frame1102, gas leak1111, and machine learning leak indication1112. The machine learning model segments frame1102to identify gas leak1111with leak indication1112. For example, the machine learning model may use the leak detection processes described in the preceding Figures to detect gas leak1111. Model output1100may be presented on a user interface to allow human operators to take corrective action.

FIG.12illustrates user interface1200. User interface1200is an example of application132illustrated inFIG.1and display432illustrated inFIG.4, however application142and display432may differ. User interface1200comprises alert1201, metrics102, and leak footage1203. Alert1201comprises a warning that indicates a natural gas leak has been detected. For example, the computing device that renders user interface1200may receive a leak indication generated by a machine learning model trained to detect natural gas leaks and responsively display alert1201. Alert1201comprises selectable options to transfer an alert, to reprocess, or to ignore the alert. Metrics1202comprise information like date/time, latitude and longitude coordinates, leak flowrate, and prediction confidence that contextualizes the detected gas leak. Leak footage1203comprises video data depicting the detected gas leak. Leak footage1203is marked with a box and highlights by the machine learning model to identify the location of the gas leak within the footage. User interface1200comprises a set of selectable options on the left-side panel that allows a user to customize the view of leak footage1203and the information presented in metrics1202.

FIG.13illustrates computing environment1300according to an implementation of the present technology. Computing environment1300comprises computing system1301. Computing system1301is representative of any system or collection of systems with which the various operational architectures, processes, scenarios, and sequences disclosed herein for detecting natural gas leaks in natural gas extraction, transfer, and storage environments. For example, computing system1301may be representative of model computer131, user computer141, machine learning interface421, machine learning engine423, user computer431, and/or any other computing device contemplated herein. Computing system1301may be implemented as a single apparatus, system, or device or may be implemented in a distributed manner as multiple apparatuses, systems, or devices. Computing system1301includes, but is not limited to, storage system1302, software1303, communication interface system1304, processing system1305, and user interface system1306. Processing system1305is operatively coupled with storage system1302, communication interface system1304, and user interface system1306.

Processing system1305loads and executes software1303from storage system1302. Software1303includes and implements leak detection process1310, which is representative of any of the natural gas leak detection processes described with respect to the preceding Figures, including but not limited to the video imaging, machine learning, leak detection and classification, and user interface operations described with respect to the preceding Figures. For example, leak detection process1310may be representative of process200illustrated inFIG.2and/or process300illustrated inFIG.3. When executed by processing system1305to detect natural gas leaks, software1303directs processing system1305to operate as described herein for at least the various processes, operational scenarios, and sequences discussed in the foregoing implementations. Computing system1301may optionally include additional devices, features, or functionality not discussed for purposes of brevity.

Processing system1305may comprise a micro-processor and other circuitry that retrieves and executes software1303from storage system1302. Processing system1305may be implemented within a single processing device but may also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing system1305include general purpose CPUs, GPUs, DSPs, ASICs, FPGAs, analog computing devices, and logic devices, as well as any other type of processing device, combinations, or variations thereof.

In addition to computer readable storage media, in some implementations storage system1302may also include computer readable communication media over which at least some of software1303may be communicated internally or externally. Storage system1302may be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems co-located or distributed relative to each other. Storage system1302may comprise additional elements, such as a controller, capable of communicating with processing system1305or possibly other systems.

Software1303(including leak detection process1310) may be implemented in program instructions and among other functions may, when executed by processing system1305, direct processing system1305to operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. For example, software1303may include program instructions for generating feature vectors that represent a video depicting a natural gas extraction system and generating a machine learning output to identify and classify natural gas leaks in the extraction system as described herein.

In general, software1303may, when loaded into processing system1305and executed, transform a suitable apparatus, system, or device (of which computing system1301is representative) overall from a general-purpose computing system into a special-purpose computing system customized to detect and classify natural gas leaks in infrared or optical videos using machine learning algorithms as described herein. Indeed, encoding software1303on storage system1302may transform the physical structure of storage system1302. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the storage media of storage system1302and whether the computer-storage media are characterized as primary or secondary storage, as well as other factors.

Communication between computing system1301and other computing systems (not shown), may occur over a communication network or networks and in accordance with various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, internets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variation thereof. The aforementioned communication networks and protocols are well known and an extended discussion of them is omitted for the sake of brevity.

While some examples provided herein are described in the context of computing devices for gas leak detection and classification, it should be understood that the condition systems and methods described herein are not limited to such embodiments and may apply to a variety of other environments and their associated systems. As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method, computer program product, and other configurable systems. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.