Error correction

Correction management techniques are provided. In one embodiment, the techniques involve determining, via a first machine learning model, a first and second data based on a respective first and second raw data obtained from a plurality of sensors, determining, based on a deviation between the first data and the second data, an inaccuracy of the first data, identifying an ambient situation corresponding to the first raw data and the second raw data, selecting, from historical raw data of the plurality of sensors, a group of raw data corresponding to the ambient situation, and correcting, via a second machine learning model, the first data based on the selected group of raw data.

BACKGROUND

The present invention generally relates to error correction. Specifically, the present invention relates to computer-implemented methods, computer-implemented systems and computer program products for correcting an error in data monitored by an ambient sensor.

With the development of real-time monitoring and data processing technology, ambient monitoring has become an important aspect of people's daily life. Usually, various types of ambient sensors are deployed for monitoring data. For example, for the purpose of air quality monitoring or weather forecast, a large number of ambient sensors may be deployed across a wide geography area. Various reasons may result in an error in data monitored by the ambient sensor, and thus error detection and correction becomes a research focus.

SUMMARY

In one aspect, a computer-implemented method is disclosed. According to the method, a deviation may be detected by one or more processors between a first data obtained from a sensor in a plurality of sensors and a second data obtained from other sensors in the plurality of sensors, the first data and the second data being obtained in an identical or similar ambient situation. The ambient situation where the first data and the second data are obtained may be identified by one or more processors. A group of raw data that is monitored under the ambient situation may be selected by one or more processors from historical raw data monitored by the plurality of sensors. The first data may be corrected by one or more processors based on the selected group of raw data.

In another aspect, a computer-implemented system is disclosed. The computing system comprises a computer processor coupled to a computer-readable memory unit, where the memory unit comprises instructions that when executed by the computer processor implements the above method.

In another aspect, a computer program product is disclosed. The computer program product comprises a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by an electronic device to cause the electronic device to perform actions of the above method.

It is to be understood that the summary is not intended to identify key or essential features of embodiments of the present invention, nor is it intended to be used to limit the scope of the present embodiment. Other features of the present embodiment will become easily comprehensible through the description below.

DETAILED DESCRIPTION

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

It should be noted that the processing of correction management96according to embodiments of this invention could be implemented by computer system/server12ofFIG. 1. Hereinafter, reference will be made toFIG. 4toFIG. 8to describe details of the correction management96.

Nowadays, various types of ambient sensors are deployed for monitoring data. For example, for the purpose of air quality monitoring, a large number of ambient sensors may be deployed across a wide geography area. In the context of the present invention, the ambient sensor may be an air quality sensor. For example, the ambient sensor may be a PM 2.5 sensor, a PM 10 sensor, a sulfur dioxide sensor, a formaldehyde sensor, and the like. Usually, high precision sensors may be deployed in large scale monitoring stations such as state stations or city stations. While due to great costs for the high precision sensors, cheaper low precision sensors may be deployed in small monitoring points.

Reference will be made toFIG. 4Afor a general description of the working environment, which figure depicts an example environment in which a plurality of sensors may be deployed according to an embodiment of the present invention.FIG. 4Ashows a plurality of sensors that are located in a geography area, where a sensor410A indicates a high precision sensor and sensors420A,422A,424A,426A and428A indicate low precision sensors. As the low precision sensors usually cannot monitor the ambient parameters with high precision, raw data monitored by the low precision sensors should be corrected by raw data monitored by a nearby high precision sensor. As shown inFIG. 4A, raw data monitored by the sensors420A,422A,424A,426A and428A should be corrected based on a correction model.

Usually, the correction model may be generated for correcting the raw data monitored by the low precision sensors420A,422A,424A,426A and428A. Reference will be made to the sensor420A to describe a general procedure for the correction. Supposing the raw data monitored by the sensor420A is represented by “x,” and the corrected data of the sensor420A is represented by “y,” the correction model may be represented by an Equation 1 as below:
y=a1*x+b1  Equation 1

It is to be understood that the above Equation 1 is just a simple example for indicating an association relationship between the raw data monitored by the low precision sensor and corrected data. In other situation, the association relationship may be represented by more complicated equation(s) comprising more or less parameters. In the above Equation 1, values of the parameters “a1” and “b1” may be obtained from human experience or other historical data. For example, a machine learning method may be implemented for obtaining Equation 1.

Usually, the correction model may be used for correcting raw data monitored by multiple sensors located in an area near the high precision sensor. However, sometimes the corrected data may show diversities and corrected data for some sensors may deviate from those for other sensors.FIG. 4Bdepicts an example diagram of a distribution of corrected data for the plurality of sensors420A,422A,424A,426A and428A inFIG. 4A.

All of the raw data monitored by the sensors420A,422A,424A,426A and428A may be corrected by the above Equation 1. InFIG. 4B, reference numbers420B,422B,424B,426B and428B indicate corrected data for the sensors420A,422A,424A,426A and428A. Although the above sensors are located in the same region and all the raw data are processed by the same correction model, the corrected data may include outlier(s). For example, the data420B may deviate from the other data422A,424A,426A and428A according to various situations: (1) the ambient situation at the sensor420A is different from nearby locations (for example, the sensor420A is near a factory); (2) a hardware failure occurs in the sensor420A; and (3) the correction model is not appropriate for the sensor420A. The above situations (1) and (2) are easy to be detected and corrected, while the situation (3) relates to building a new correction model. At this point, how to correct a potential error in the present correction model and provide accurate results becomes a research focus.

In view of the above, the present invention provides an effective solution for error correction based on historical data. In the present invention, the raw data indicates data that is directly monitored by the sensor, and the data obtained from the sensor indicates the data that is obtained by correcting the raw data with the correction model. Hereinafter, reference will be made toFIG. 5for a general description of embodiments of the present invention.

FIG. 5depicts an example diagram500for correcting an error in data obtained from a sensor according to an embodiment of the present invention. InFIG. 5, a data distribution570shows a distribution of data obtained from a plurality of sensors. In the data distribution570, a first data510deviates from second data520-1,520-2,520-3, and520-4(collectively referred to as second data520). Here, the first data510is determined from first raw data monitored by the sensor420A and a first correction model, and the second data520is determined from second raw data monitored by the sensors422A,424A,426A and428A and the first correction model. Although the same correction model is applied to all the sensors, the first data510deviates from the second data520to a certain extent (for example, 20% or another threshold). At this point, it may be considered that the first correction model is not suitable for the first data510, and the first data510may be corrected according to embodiments of the present invention.

The ambient situation may reflect raw data, which in turn may reflect the obtained data. For example, the data obtained in a windy day may be different from that obtained in a rainy day. Accordingly, an ambient situation530where the first data510and the second data520is obtained may be identified. Further, historical raw data540monitored by the plurality of sensors around the sensor410under a similar ambient situation may be selected for correcting the first data510. InFIG. 5, a group550of raw data that is monitored under the ambient situation may be selected from historical raw data540. Supposing the first data510and the second data520is obtained in a windy day, the raw data that is monitored in a windy day may be selected as the group550of raw data. Further, the group550of raw data and data that is determined from the selected group of raw data may be used to generate a second correction model for correcting the first data510to data560.

With these embodiments, when the current first correction model cannot work well for the first data510, the first data510may be corrected based on the above procedure as described inFIG. 5. As the historical raw data540and the data determined from the raw data may reflect the particular ambient situation and may be considered as reliable, then a new correction model may be built on the reliable basis for correcting a potential error in the first data510.

Reference will be made toFIG. 6for more details about embodiments of the present invention.FIG. 6depicts an example flowchart of a method600for correcting an error in data obtained from a sensor according to an embodiment of the present invention. At block610, a deviation may be detected between the first data510obtained from the sensor420A and the second data520obtained from the sensors422A,424A,426A and428A. If the deviation is detected, then the method600may proceed to block620. Here, the first data510and the second data520may be obtained in an identical or similar ambient situation.

At block620, the ambient situation530where first data510and second data520is obtained may be identified. The ambient attribute530may comprise various aspects of the environment where the sensors are deployed. In some embodiments of the present invention, the ambient attribute530may comprise any of: a wind direction, a wind level, a temperature, an air pressure, a humidity level, a weather condition, and a geography condition.

Here, the wind direction may indicate a direction of the wind in the windy day. The wind direction may be represented by multiple formats such as east wind or west wind. Alternatively, the wind direction may be represented by an angle such as an angle between the wind direction and the north direction. The wind level may be represented according to a definition of the World Meteorological Organization, or the wind level may also be represented by a wind speed. Similarly, the formats of the temperature, the air pressure, the humidity level, the weather condition and the geography condition may be defined in various ways.

At block630, a group550of raw data that is monitored under the ambient situation530may be selected from the historical raw data540monitored by the plurality of sensors. The historical raw data540may be stored in various formats, and an example format is presented in Table 1 for description.

Table 1 shows the historical raw data monitored by a sensor, where the first row indicates the day when the raw data is monitored by the sensor, the second row indicates the ambient situation when the raw data is monitored, and the third row indicates values of the raw data. It is to be understood that the above Table only illustrate an example data structure for storing the raw data. In another example, the raw data may include more information such as a date and time point when the raw data is monitored.

If the ambient situation530indicates that the first data510and the second data520is obtained in a rainy day, then the raw data monitored in Day 2 and Day N may be selected as the group550of raw data. If the ambient situation530indicates that the first data510and the second data520is obtained in a windy day, then the raw data monitored in Day 1 and other windy days may be selected as the group550of raw data.

In some embodiments of the present invention, the ambient situation in Table 1 may be represented in a finer granularity by replacing the “ambient situation” with the above ambient attributes. At this point, the raw data may be stored in an example format of Table 2.

Usually, one or more attributes of the ambient situation530may have a stronger impact on the raw data and in turn affects the first data510more than another attribute in the group of ambient attributes. Therefore, the one or more attributes may be identified from a group of ambient attributes of the ambient situation530.

Further, the group550of raw data may be selected based on the identified ambient attribute530, specifically, the selected group of raw data that is monitored in a time duration having the ambient attribute may be selected. Usually, in a rainy day, the air quality at various locations may be similar and thus the weather condition may be identified and raw data that is monitored in the rainy days may be selected as the group550of raw data. In a windy day, the wind direction may affect the air quality and thus raw data that is monitored in days with similar wind direction may be selected as the group550of raw data. In a windless day, the humidity level may affect the air quality and thus raw data that is monitored in days with similar humidity may be selected as the group550of raw data. In some embodiments of the present invention, the above ambient attributes may be assigned with respective weights to provide a method for selecting the group550of raw data based on a whole picture of all the ambient attributes.

At block640, the first data510may be corrected based on the selected group550of raw data. In these embodiments, the first data510is obtained based on first raw data monitored by the sensor420A and a first correction (such as Equation 1), and the second data520is obtained based on second raw data monitored by the other sensors422A,424A,426A and428A and the first correction. Here, the corrected data may be determined based on the selected group550of raw data and the first correction model. In some embodiments of the present invention, the corrected data may be stored together with the raw data as below.

The first three rows are similar as those in Table 1, and the fourth row indicates values of data that is corrected by the first correction model. Accordingly, data in the rows “Raw Data” and “Corrected Data” may be used to determine a second correction, and then the first data510may be corrected by the second correction model based on the first raw data. Reference will be made toFIG. 7for description.

FIG. 7depicts an example diagram700for training a correction model based on historical raw data and corrected data according to an embodiment of the present invention. The upper part ofFIG. 7illustrates a first correction model710which corrects the first raw data into the first data510. The lower part ofFIG. 7illustrates a procedure for generating a second correction model720for correcting a potential error in the first data510. Supposing the second correction model720may be represented by the equation as below:
y=a2*x+b2  Equation 2

In some embodiments of the present invention, the second correction720may be trained based on the selected group550of raw data, such that the trained second correction represents an association relationship between the group of raw data and the corrected under the ambient situation. Accordingly, the procedure for generating the second correction model720relates to how to determine the values of the parameters a2 and b2. Here, the group550of raw data and corrected data may be involved in the Equation 2. Specifically, the variable “x” represents the raw data and the variable “y” indicated the data corrected by the first correction model. Continuing the above historical raw data in Table 1, if the raw data of Day 2 and Day N are selected, then Equation 3 may be generated by replacing the variable “x” and “y” with “RawValue2” and “Value2” in the Equation 2, and Equation 4 may be generated by replacing the variable “x” and “y” with “RawValueN” and “ValueN” in the Equation 2.
Value2=a2*RawValue2+b2  Equation 3
ValueN=a2*RawValueN+b2  Equation 4

As “Value2,” “RawValue2,” “ValueN,” and “RawValueN” are known from Table 3, the values of “a2” and “b2” may be determined by solving Equations 3 and 4. Accordingly, the second correction model720may be generated. In some embodiments of the present invention, if the group550of raw data relates to more than two days, then the values of “a2” and “b2” may be determined based on an average of the solved values of “a2” and “b2.” Further, the first data510may be corrected with the second correction model720. As the group550of raw data and the corrected data in Table 3 is considered as reliable data, the second correction model720which is generated based on a reliable ground may reflect an accurate association relationship between the raw data and the corrected data. Therefore, the first data510which deviates from the second data520may be corrected in a reliable manner.

In some embodiments of the present invention, locations of the sensors422A,424A,426A and428A may be considered in selecting the group550of raw data. Specifically, an area occupied by the plurality of sensors may be divided into a plurality of regions according to the ambient situation and a spatial relationship between the sensor510and the other sensors422A,424A,426A and428A. Reference will be made toFIG. 8for describing how to divide the area into multiple regions.

FIG. 8depicts an example diagram800for dividing an area occupied by a plurality of sensors into a plurality of regions according to an embodiment of the present invention. If the first data510is obtained in a windy day, the wind direction810may be considered as an important attribute playing a strong impact on the raw data. As shown inFIG. 8, the wind direction810is from the west to east and thus the area may be divided into Regions I, II, III, and IV. In some embodiments of the present invention, the boundary between regions may go through a location of the sensor420A.

Although the area is divided into four regions inFIG. 8, more or less regions may be obtained in other embodiments. For example, if the wind direction810is from the northeast to southwest, then the area may be divided into for example 8 regions. For another example, if the wind direction is represented by an angle of 15 between the north and the wind direction, then the area may be divided into for example 12 regions, each of the regions has an angle of 360°/12=30°.

In some embodiments of the present invention, a region may be selected from the plurality of regions based on the ambient situation, and raw data monitored by the sensor is affected by the selected region more than a further region in the plurality of regions. In some embodiments of the present invention, the ambient situation comprises a wind direction, and a region that locates in an upstream direction of the wind direction may be selected. Referring toFIG. 8, Region I locates in the upstream direction of the wind direction810may be selected. Accordingly, the group of raw data that is monitored by sensors in Region I under the ambient situation530may be selected.

Continuing the above example of Table, the wind directions in Day 2 and Day N are west, and thus the raw data in Day 2 and Day N may be selected. Further, the selected raw data may also be filtered based on the Region I, and only raw data monitored by sensors in Region I may be selected, while raw data monitored by sensors in other regions may be filtered out duo to its small impacts on the first data510. With these embodiments, both of the ambient situation and the spatial relationships between the sensors are considered in building the second correction model, such that the second correction model may be built on raw data and data with a closer relationship.