Patent Description:
Systems for controlling environmental conditions, for example in buildings, are becoming increasingly sophisticated. An environment control system may at once control heating and cooling, monitor air quality, detect hazardous conditions such as fire, carbon monoxide release, intrusion, and the like. Such environment control systems generally include at least one environment controller, which receives measured environmental values, generally from sensors, and in turn determines set-points or command parameters to be sent to controlled appliances.

For instance, a room has current environmental characteristic values, such as a current temperature and a current humidity level, detected by sensors and reported to an environment controller. A user interacts with the environment controller to provide set point(s), such as a target temperature. The environment controller sends the current environmental characteristic values (e.g. current temperature and current humidity level) and the set point(s) (e.g. target temperature) to a controlled appliance. The controlled appliance generates commands for actuating internal components of the controlled appliance to reach the set point(s) based on the current environmental characteristic values. Alternatively, the environment controller directly determines command(s) based on the current environmental characteristic values and the set point(s), and transmits the command(s) to the controlled appliance. The controlled appliance uses the command(s) received from the environment controller to actuate the internal components.

Examples of controlled appliances include a heating, ventilating, and / or air-conditioning (HVAC) appliance, which regulates the temperature, humidity level and CO2 level in an area of a building. Examples of internal components include a motor, an electrical circuit (e.g. for generating heat), a valve (e.g. for controlling an air flow), etc..

Current advances in artificial intelligence, and more specifically in neural networks, can be taken advantage of in the context of building automation. More specifically, a predictive model comprising weights of a neural network is generated during a training phase and used during an operational phase. The neural network uses the predictive model to generate the command(s) for controlling the appliance based on the current environmental characteristic values, the set point(s), and optionally other parameters (e.g. characteristic(s) of an area of a building).

The generation of the predictive model during the training phase is a difficult task, which requires a lot of samples (inputs and outputs of the neural network being trained) for generating the predictive model. Automating the generation of samples for the training phase and allowing an improvement of the predictive model during the operational phase are ways of making the training process more efficient and potentially also more accurate.

<CIT> relates to a method and environment controller for inferring via a neural network one or more commands for controlling an appliance. A predictive model generated by a neural network training engine is stored by the environment controller. The environment controller receives at least one environmental characteristic value (for example, at least one of a current temperature, current humidity level, current carbon dioxide level, and current room occupancy) and at least one set point (for example, at least one of a target temperature, target humidity level, and target carbon dioxide level). The environment controller executes a neural network inference engine, which uses the predictive model for inferring the one or more command for controlling the appliance based on the at least one environmental characteristic value and the at least one set point. The environment controller transmits the one or more commands to the controlled appliance.

Therefore, there is a need for a training server and a method for generating a predictive model of a neural network through distributed reinforcement learning.

According to a first aspect, according to the claimed invention, the present disclosure relates to a training server. The training server comprises a communication interface, memory for storing a predictive model comprising weights of a neural network, and a processing unit comprising one or more processor. The processing unit receives a plurality of training data sets from a plurality of environment controllers via the communication interface. Each training data set comprises inputs, one or more output, and at least one metric. The inputs comprise at least one environmental characteristic value in an area and at least one set point. The one or more output comprises one or more commands for controlling a controlled appliance. The at least one metric is representative of an execution of the one or more command by the controlled appliance. For each received training data set, the processing unit determines a value of a reinforcement signal based on the at least one metric; and executes a neural network training engine to update the weights of the neural network based on the inputs, the one or more output, and the value of the reinforcement signal. The processing unit further transmits an update of the predictive model comprising the updated weights to the plurality of environment controllers via the communication interface.

According to a second aspect, according to the claimed invention, the present disclosure relates to a method for improving a predictive model of a neural network used for performing environment control. The method comprises storing in a memory of a training server a predictive model comprising weights of a neural network. The method comprises receiving a plurality of training data sets from a plurality of environment controllers via a communication interface of the training server. Each training data set comprises inputs, one or more output, and at least one metric. The inputs comprise at least one environmental characteristic value in an area and at least one set point. The one or more output comprises one or more commands for controlling a controlled appliance. The at least one metric is representative of an execution of the one or more command by the controlled appliance. For each received training data set, the method comprises determining, by a processing unit of the training server, a value of a reinforcement signal based on the at least one metric; and executing, by the processing unit of the training server, a neural network training engine to update the weights of the neural network based on the inputs, the one or more output, and the value of the reinforcement signal. The method further comprises transmitting an update of the predictive model comprising the updated weights to the plurality of environment controllers via the communication interface of the training server.

According to a third aspect, not forming part of the claimed invention, the present disclosure relates to a non-transitory computer program product comprising instructions executable by a processing unit of a training server. The execution of the instructions by the processing unit of the training server provides for improving a predictive model of a neural network used for performing environment control, by implementing the aforementioned method.

The following disclosure relates to the first and second aspect according to the claimed invention and to the third aspect according to an example not forming part of the claimed invention.

In one embodiment, the value of the reinforcement signal may be one of a positive reinforcement or a negative reinforcement.

In one embodiment, the determination of the value of the reinforcement signal may be implemented through a set of rules.

In one embodiment, the determination of the value of the reinforcement signal may also be based on the at least one set point.

Alternatively or additionally, the determination of the value of the reinforcement signal may also be based on the at least one environmental characteristic value in the area.

In one embodiment, the at least one metric may comprise at least one updated environmental characteristic value in the area.

Alternatively or additionally, the at least one metric may comprise at least one measurement of a time required for reaching at least one corresponding environmental state in the area.

In one embodiment, the at least one environmental characteristic value in the area may comprise at least one of the following: a current temperature in the area, a current humidity level in the area, a current carbon dioxide (CO2) level in the area, and a current occupancy of the area.

Alternatively, the at least one environmental characteristic value in the area may comprise at least one of the following: a plurality of consecutive temperature measurements in the area, a plurality of consecutive humidity level measurements in the area, a plurality of consecutive carbon dioxide (CO2) level measurements in the area, and a plurality of consecutive determinations of an occupancy of the area.

In one embodiment, the at least one set point may comprise at least one of the following: a target temperature, a target humidity level, and a target CO2 level.

In one embodiment, the inputs may also include at least one characteristic of the area.

The at least one characteristic of the area may comprise at least one of the following: an area type identifier selected among a plurality of area type identifiers, one or more geometric characteristics of the area, and a human activity in the area.

The determination of the value of the reinforcement signal may also be based on the at least one characteristic of the area.

In one embodiment, the controlled appliance may consist of a Variable Air Volume (VAV) appliance.

In one embodiment, the one or more command may include at least one of the following: a command for controlling a speed of a fan, a command for controlling a pressure generated by a compressor, and a command for controlling a rate of an airflow through a valve.

In one embodiment, the neural network corresponding to the predictive model and implemented by the neural network training engine may comprise an input layer for receiving the inputs, followed by one or more intermediate hidden layers, followed by an output layer for outputting the outputs.

Embodiments of the disclosure will be described by way of example only with reference to the accompanying drawings, in which:.

The foregoing and other features will become more apparent upon reading of the following non-restrictive description of illustrative embodiments thereof, given by way of example only with reference to the accompanying drawings.

Various aspects of the present disclosure generally address one or more of the problems related to environment control systems for buildings. More particularly, the present disclosure aims at providing solutions for generating and improving a predictive model of a neural network used by a plurality of environment controllers. The generation and improvement is performed through the use of a training server interacting with the plurality of environment controllers and performing reinforcement learning.

The following terminology is used throughout the present specification:.

Referring now to <FIG> and <FIG>, an environment control system where an environment controller <NUM> exchanges data with other environment control devices (ECDs) is illustrated. The environment controller <NUM> is responsible for controlling the environment of an area of a building. The environment controller <NUM> receives from sensors (e.g. <NUM>, <NUM>, <NUM> and <NUM>) environmental characteristic values measured by the sensors. The environment controller <NUM> generates commands based on the received environmental characteristic values. The generated commands are transmitted to controlled appliances <NUM> (to control the operations of the controlled appliances <NUM>). Although a single controlled appliance <NUM> is represented in <FIG> for simplification purposes, the environment controller <NUM> may be interacting with a plurality of controlled appliances <NUM>.

The area under the control of the environment controller <NUM> is not represented in the Figures for simplification purposes. As mentioned previously, the area may consist of a room, a floor, an aisle, etc. However, any type of area located inside any type of building is considered to be within the scope of the present disclosure. The sensors (<NUM>, <NUM>, <NUM> and <NUM>) and the controlled appliances <NUM> are generally located in the area under control (e.g. a room). The environment controller <NUM> may or may not be located in the area under control. For example, the environment controller <NUM> may remotely control the environment of the area under control, which includes controlling the controlled appliances <NUM> based on the inputs of the sensors <NUM>, <NUM>, <NUM> and <NUM>.

Examples of sensors include: a temperature sensor <NUM> for measuring a temperature in the area and transmitting the measured temperature to the environment controller <NUM>, a humidity sensor <NUM> for measuring a humidity level in the area and transmitting the measured humidity level to the environment controller <NUM>, a CO2 sensor <NUM> for measuring a CO2 level in the area and transmitting the measured CO2 level to the environment controller <NUM>, an occupancy sensor <NUM> for generating occupancy data for the area and transmitting the generated occupancy data to the environment controller <NUM>, a lighting sensor (not represented in the Figures) for measuring a light level in the area and transmitting the measured light level to the environment controller <NUM>, etc..

Each environmental characteristic value measured by a sensor may consist of either a single value (e.g. the current CO2 level measured by the CO2 sensor <NUM> is <NUM> parts per million), or a range of values (e.g. the current CO2 level measured by the CO2 sensor <NUM> is in the range of <NUM> to <NUM> parts per million).

In a first implementation, a single sensor (e.g. CO2 sensor <NUM>) measures a given type of environmental characteristic value (e.g. CO2 level) for the whole area. In a second implementation, the area is divided into a plurality of zones, and a plurality of sensors (e.g. temperature sensors <NUM>) measures the given type of environmental characteristic value (e.g. temperature) in the corresponding plurality of zones. In the second implementation, the environment controller <NUM> calculates an average environmental characteristic value in the area (e.g. an average temperature in the area) based on the environmental characteristic values transmitted by the plurality of sensors (e.g. temperature sensors <NUM>) respectively located in the plurality of zones of the area.

Additional sensor(s) may be deployed outside of the area and report their measurement(s) to the environment controller <NUM>. For example, the area is a room of a building. An external temperature sensor measures an external temperature outside the building and transmits the measured external temperature to the environment controller <NUM>. Similarly, an external humidity sensor measures an external humidity level outside the building and transmits the measured external humidity level to the environment controller <NUM>.

The aforementioned examples of sensors are for illustration purposes only. A person skilled in the art would readily understand that other types of sensors could be used in the context of the environment control system managed by the environment controller <NUM>.

Each controlled appliance <NUM> comprises at least one actuation module, to control the operations of the controlled appliance <NUM> based on the commands received from the environment controller <NUM>. The actuation module can be of one of the following types: mechanical, pneumatic, hydraulic, electrical, electronical, a combination thereof, etc. The commands control operations of the at least one actuation module.

An example of a controlled appliance <NUM> consists of a VAV appliance. Examples of commands transmitted to the VAV appliance include commands directed to one of the following: an actuation module controlling the speed of a fan, an actuation module controlling the pressure generated by a compressor, an actuation module controlling a valve defining the rate of an airflow, etc. This example is for illustration purposes only. Other types of controlled appliances <NUM> could be used in the context of an environment control system managed by the environment controller <NUM>.

Details of the environment controller <NUM>, sensors (<NUM>, <NUM>, <NUM> and <NUM>) and control appliance <NUM> will now be provided.

The environment controller <NUM> comprises a processing unit <NUM>, memory <NUM>, and a communication interface <NUM>. The environment controller <NUM> may comprise additional components, such as another communication interface <NUM>, a user interface <NUM>, a display <NUM>, etc..

The processing unit <NUM> comprises one or more processors (not represented in the Figures) capable of executing instructions of a computer program. Each processor may further comprise one or several cores. The processing unit <NUM> executes a neural network inference engine <NUM> and a control module <NUM>, as will be detailed later in the description.

The memory <NUM> stores instructions of computer program(s) executed by the processing unit <NUM>, data generated by the execution of the computer program(s), data received via the communication interface <NUM> (or another communication interface), etc. Only a single memory <NUM> is represented in <FIG>, but the environment controller <NUM> may comprise several types of memories, including volatile memory (such as a volatile Random Access Memory (RAM), etc.) and non-volatile memory (such as a hard drive, electrically-erasable programmable read-only memory (EEPROM), flash, etc.).

The communication interface <NUM> allows the environment controller <NUM> to exchange data with remote devices (e.g. the sensors (<NUM>, <NUM>, <NUM> and <NUM>), the controlled appliance <NUM>, etc.) over a communication network (not represented in <FIG> for simplification purposes). For example, the communication network is a wired communication network, such as an Ethernet network. The communication interface <NUM> is adapted to support communication protocols used to exchange data over the Ethernet network. Other types of wired communication networks may also be supported by the communication interface <NUM>. In another example, the communication network is a wireless communication network, such as a Wi-Fi network. The communication interface <NUM> is adapted to support communication protocols used to exchange data over the Wi-Fi network. Other types of wireless communication network may also be supported by the communication interface <NUM>, such as a wireless mesh network, Bluetooth®, Bluetooth® Low Energy (BLE), etc. In still another example, the environment controller <NUM> comprises two communication interfaces <NUM>. The environment controller <NUM> communicates with the sensors (<NUM>, <NUM>, <NUM> and <NUM>) and the controlled appliance <NUM> via a first communication interface <NUM> (e.g. a Wi-Fi interface); and communicates with other devices (e.g. a training server <NUM>) via a second communication interface <NUM> (e.g. an Ethernet interface). Each communication interface <NUM> usually comprises a combination of hardware and software executed by the hardware, for implementing the communication functionalities of the communication interface <NUM>.

A detailed representation of the components of the sensors (e.g. temperature sensor <NUM>) is not provided in <FIG> for simplification purposes. The sensor comprises at least one sensing module for detecting an environmental characteristic (e.g. temperature). The sensor further comprises a communication interface for transmitting to the environment controller <NUM> an environmental characteristic value (e.g. value of the temperature) corresponding to the detected environmental characteristic. The environmental characteristic value is transmitted over a communication network and received via the communication interface <NUM> of the environment controller <NUM>. The sensor may also comprise a processing unit for generating the environmental characteristic value based on the detected environmental characteristic. Alternatively, the environmental characteristic value is directly generated by the sensing module. The other types of sensors mentioned previously (e.g. humidity sensor <NUM> and CO2 sensor <NUM>) generally include the same types of components as those mentioned for the temperature sensor <NUM>.

The temperature, humidity and CO2 sensors are well known in the art, and easy to implement types of sensors. With respect to the occupancy sensor, its implementation may be more or less complex, based on its capabilities. For example, a basic occupancy sensor (e.g. based on ultrasonic or infrared technology) is only capable of determining if the area is occupied or not. A more sophisticated occupancy sensor is capable of determining the number of persons present in the area, and may use a combination of camera(s) and pattern recognition software for this purpose. Alternatively, the occupancy sensor is not capable of determining the number of persons present in the area, but is capable of determining the number of persons entering or leaving the area (e.g. an infrared beam sensor using infrared rays to detect people entering or leaving the area).

A detailed representation of the components of the controlled appliance <NUM> is not provided in <FIG> for simplification purposes. As mentioned previously, the controlled appliance <NUM> comprises at least one actuation module. The controlled appliance <NUM> further comprises a communication interface for receiving commands from the environment controller <NUM>. The commands control operations of the at least one actuation module. The commands are transmitted over a communication network via the communication interface <NUM> of the environment controller <NUM>. The controlled appliance <NUM> may also comprise a processing unit for controlling the operations of the at least one actuation module based on the received commands.

A detailed representation of the components of the training server <NUM> is not provided in <FIG> as it will be detailed later. The training server <NUM> comprises a processing unit, memory and a communication interface. The processing unit of the training server <NUM> executes a neural network training engine <NUM>.

The execution of the neural network training engine <NUM> generates a predictive model, which is transmitted to the environment controller <NUM> via the communication interface of the training server <NUM>. The predictive model is transmitted over a communication network and received via the communication interface <NUM> of the environment controller <NUM>.

Also represented in <FIG> is a user <NUM>. The user <NUM> provides at least one set point to the environment controller <NUM>. Examples of set points include target environmental characteristic values, such as a target temperature, a target humidity level, a target CO2 level, a combination thereof, etc. The at least one set point is related to the area where the sensors (<NUM>, <NUM>, <NUM> and <NUM>) and the controlled appliance <NUM> are located. Alternatively, the controlled appliance <NUM> is not located in the area, but the operations of the controlled appliance <NUM> under the supervision of the environment controller <NUM> aim at reaching the at least one set point in the area. The user <NUM> enters the at least one set point via the user interface <NUM> of the environment controller <NUM>. Alternatively, the user <NUM> enters the at least one set point via a user interface of a computing device (e.g. a smartphone, a tablet, etc.) not represented in <FIG> for simplification purposes; and the at least one set point is transmitted over a communication network and received via the communication interface <NUM> of the environment controller <NUM>.

The previous examples of setpoints are for illustration purposes only, and a person skilled in the art would readily understand that other types of set points could be used in the context of an environment control system managed by the environment controller <NUM>. Furthermore, each set point may consist of either a single value (e.g. target temperature of <NUM> degrees Celsius), or a range of values (e.g. target temperature between <NUM> and <NUM> degrees Celsius).

Optionally, the control module <NUM> executed by the processing unit <NUM> of the environment controller <NUM> also determines at least one characteristic of the area.

The characteristic(s) of the area include one or more geometric characteristics of the area (e.g. a room in a building). Examples of geometric characteristics include a volume of the area, a surface of the area, a height of the area, a length of the area, a width of the area, etc. Instead of a given value, the geometric characteristics may be identified as ranges of values. For example, the volume of the area is defined by the following ranges of values: <NUM> to <NUM> cubic meters, <NUM> to <NUM> cubic meters, and more than <NUM> cubic meters. Similarly, the height of the area is defined by the following ranges of values: less than <NUM> meters and more than <NUM> meters.

Alternatively or complementarity, the characteristic(s) of the area include an area type identifier of the current area A plurality of area type identifiers is defined, each area type identifier corresponding to areas having one or more geometric characteristics in common. For example, each area type identifier is an alphanumerical value. The area type identifier of the current area is selected among the plurality of pre-defined area type identifiers based on geometric characteristics of the current area. For instance, the area type identifier R1 is allocated to areas having a volume lower than <NUM> cubic meters; the area type identifier R2 is allocated to areas having a volume between <NUM> and <NUM> cubic meters, and a height lower than <NUM> meters; the area type identifier R3 is allocated to areas having a volume between <NUM> and <NUM> cubic meters, and a height higher than <NUM> meters; and the area type identifier R4 is allocated to areas having a volume higher than <NUM> cubic meters.

Alternatively or complementarity, the characteristic(s) of the area include a human activity in the area. For example, the human activity in the area comprises periods of time when the room is occupied by humans (e.g. during the day or during the night, in the morning or in the afternoon, during the week or the week-end, etc.). Alternatively or complementarity, the human activity in the area defines the type of activity performed by the persons occupying the area; for instance, the area is an office room, a room in a store, a storage room, a workshop room, a room in a house or an apartment, etc..

The aforementioned area type identifier of the area can also be based on the human activity in the area. Furthermore, a person skilled in the art would readily understand that other types of area characteristics could be used in the context of an environment control system managed by the environment controller <NUM>.

<FIG> illustrates examples of the determination of the characteristic(s) of the area by the processing unit <NUM> of the environment controller <NUM>.

The determination of the characteristic(s) of the area comprises receiving the characteristic(s) of the area from a computing device <NUM> via the communication interface <NUM>, and storing the characteristic(s) of the area in the memory <NUM> of the environment controller <NUM>.

Alternatively or complementarily, the determination of the characteristic(s) of the area comprises receiving the characteristic(s) of the area from the user <NUM> via the user interface <NUM> of the environment controller <NUM>, and storing the characteristic(s) of the area in the memory <NUM>.

Alternatively or complementarily, the determination of the characteristic(s) of the area comprises receiving the characteristic(s) of the area from a sensor <NUM> via the communication interface <NUM>, and storing the characteristic(s) of the area in the memory <NUM> of the environment controller <NUM>. The sensor <NUM> is capable of automatically determining characteristic(s) of the area. For example, the sensor <NUM> combines one or more cameras, and a processing unit, capable of automatically determining geometric characteristics of the area. In another example, the sensor <NUM> combines one or more cameras (or sound sensor, motion detector, etc.), and a processing unit, capable of automatically determining a human activity in the area. Alternatively, the sensor <NUM> only transmits collected data (e.g. images of the area) to the processing unit <NUM> of the environment controller <NUM>, and the processing unit <NUM> determines the characteristic(s) of the area based on the data transmitted by the sensor <NUM>.

The characteristic(s) of the area usually do not change over time. Thus, the determination occurs only once, and the characteristics of the area are permanently stored in the memory <NUM> for being used by the neural network inference engine <NUM>, as will be illustrated later in the description.

Reference is now made concurrently to <FIG>, <FIG>, <FIG>, <FIG>, <FIG> and <FIG>; where <FIG>, <FIG>, <FIG> and <FIG> represent a method <NUM>. At least some of the steps of the method <NUM> are implemented by the environment controller <NUM>. The method <NUM> aims at improving a predictive model of a neural network used by the environment controller <NUM> (more specifically by the neural network inference engine <NUM>). The present disclosure is not limited to the method <NUM> being implemented by the environment controller <NUM>, but is applicable to any type of computing device capable of implementing the steps of the method <NUM>.

A dedicated computer program has instructions for implementing at least some of the steps of the method <NUM>. The instructions are comprised in a non-transitory computer program product (e.g. the memory <NUM>) of the environment controller <NUM>. The instructions provide for improving a predictive model of a neural network used by the environment controller <NUM> (more specifically by the neural network inference engine <NUM>), when executed by the processing unit <NUM> of the environment controller <NUM>. The instructions are deliverable to the environment controller <NUM> via an electronically-readable media such as a storage media (e.g. CD-ROM, USB key, etc.), or via communication links (e.g. via a communication network through the communication interface <NUM>).

The instructions of the dedicated computer program executed by the processing unit <NUM> implement the neural network inference engine <NUM> and the control module <NUM>. The neural network inference engine <NUM> provides functionalities of a neural network, allowing to infer output(s) based on inputs using the predictive model, as is well known in the art. The control module <NUM> provides functionalities allowing the environment controller <NUM> to interact with and control other devices (e.g. the sensors (<NUM>, <NUM>, <NUM> and <NUM>) and the controlled appliance <NUM>).

The method <NUM> comprises the step <NUM> of storing a predictive model in the memory <NUM>. Step <NUM> is performed by the processing unit <NUM>. The predictive model comprises weights of a neural network implemented by the neural network inference engine <NUM>.

The method <NUM> comprises the step <NUM> of determining at least one environmental characteristic value in the area. Step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>. The at least one environmental characteristic value includes one or more of the following: a current temperature in the area, a current humidity level in the area, a current CO2 level in the area, and a current occupancy of the area. However, other types of environmental characteristic value may be determined at step <NUM>.

In the case of the current temperature, the measurement of the current temperature is performed by the temperature sensor <NUM> (located in the area) and transmitted to the environment controller <NUM>. Thus, step <NUM> includes receiving the current temperature from the temperature sensor <NUM> via the communication interface <NUM>. Alternatively, functionalities of a temperature sensor are integrated to the environment controller <NUM>. In this case, step <NUM> includes receiving the current temperature from a temperature sensing module (not represented in <FIG>) integrated to the environment controller <NUM>. In still another implementation, step <NUM> includes calculating the current temperature in the area based on temperature measurements respectively received from a plurality of temperature sensors <NUM> located in the area (e.g. calculating the average of the temperature measurements received from the plurality of temperature sensors <NUM>).

In the case of the current humidity level, the measurement of the current humidity level is performed by the humidity sensor <NUM> (located in the area) and transmitted to the environment controller <NUM>. Thus, step <NUM> includes receiving the current humidity level from the humidity sensor <NUM> via the communication interface <NUM>. Alternatively, functionalities of a humidity sensor are integrated to the environment controller <NUM>. In this case, step <NUM> includes receiving the current humidity level from a humidity sensing module (not represented in <FIG>) integrated to the environment controller <NUM>. In still another implementation, step <NUM> includes calculating the current humidity level in the area based on humidity level measurements respectively received from a plurality of humidity sensors <NUM> located in the area (e.g. calculating the average of the humidity level measurements received from the plurality of humidity sensors <NUM>).

In the case of the current CO2 level, the measurement of the current CO2 level is performed by the CO2 sensor <NUM> (located in the area) and transmitted to the environment controller <NUM>. Thus, step <NUM> includes receiving the current CO2 level from the CO2 sensor <NUM> via the communication interface <NUM>. Alternatively, functionalities of a CO2 sensor are integrated to the environment controller <NUM>. In this case, step <NUM> includes receiving the current CO2 level from a CO2 sensing module (not represented in <FIG>) integrated to the environment controller <NUM>. In still another implementation, step <NUM> includes calculating the current CO2 level in the area based on CO2 level measurements respectively received from a plurality of CO2 sensors <NUM> located in the area (e.g. calculating the average of the CO2 level measurements received from the plurality of CO2 sensors <NUM>).

In the case of the current occupancy of the area, the measurement of occupancy data is performed by the occupancy sensor <NUM> (located in the area) and transmitted to the environment controller <NUM>. In a first implementation, the current occupancy of the area directly consists of the occupancy data. Thus, step <NUM> includes directly receiving the current occupancy of the area from the occupancy sensor <NUM> via the communication interface <NUM>. For example, an ultrasonic or infrared sensor determines if the area is occupied or not, and transmits the current occupancy status of the area (occupied or not) to the environment controller <NUM>. In a second implementation, the current occupancy of the area is determined by processing the occupancy data. Thus, step <NUM> includes receiving the occupancy data from the occupancy sensor <NUM> via the communication interface <NUM>, and further processing the occupancy data to generate the current occupancy of the area. For example, a visible or thermal camera transmits picture(s) of the area to the environment controller <NUM>, and a detection software implemented by the environment controller <NUM> analyses the picture(s) to determine the number of persons present in the area. Alternatively, functionalities of an occupancy sensor are integrated to the environment controller <NUM>. In this case, step <NUM> includes receiving the occupancy data from an occupancy sensing module (not represented in <FIG>) integrated to the environment controller <NUM>.

Ultimately, the current occupancy of the area determined at step <NUM> comprises one of the following: an indication of the area being occupied or not, a number of persons present in the area, a number of persons entering or leaving the area. A person skilled in the art would readily understand that other types of occupancy sensors <NUM> may be used in the context of the present disclosure, to determine the aforementioned types of current occupancy of the area, or other types of current occupancy of the area.

The method <NUM> comprises the step <NUM> of receiving at least one set point. Step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>. As mentioned previously, the at least one set point includes one or more of the following: a target temperature, a target humidity level, and a target CO2 level. However, other types of set point may be determined at step <NUM>.

A set point is received from the user <NUM> via the user interface <NUM> (as illustrated in <FIG> and <FIG>). Alternatively, a set point is received from a remote computing device via the communication interface <NUM> (this use case is not represented in the Figures for simplification purposes). For example, the user <NUM> enters the set point via a user interface of the remote computing device (e.g. a smartphone) and the set point is transmitted to the environment controller <NUM>.

The order in which steps <NUM> and <NUM> are performed may vary. The order represented in <FIG> is for illustration purposes only.

The method <NUM> comprises the step <NUM> of executing the neural network inference engine <NUM> using the predictive model (stored at step <NUM>) for generating one or more output based on inputs. The execution of the neural network inference engine <NUM> is performed by the processing unit <NUM>. The neural network inference engine <NUM> implements a neural network using the weights of the predictive model. This step will be further detailed later in the description.

The inputs comprise the at least one environmental characteristic value in the area determined at step <NUM>, and the at least one set point received at step <NUM>.

The inputs used by the neural network inference engine <NUM> at step <NUM> may include additional parameter(s). For example, the method <NUM> comprises the optional step <NUM> of determining at least one characteristic of the area. Optional step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>. The determination of characteristic(s) of the area has been detailed previously in relation to <FIG>. The at least one characteristic of the area includes one or more of the following: an area type identifier selected among a plurality of area type identifiers, one or more geometric characteristics of the area, and a human activity in the area. The inputs used at step <NUM> further include the characteristic(s) of the area. Another example of additional parameter(s) for the inputs include an external temperature measured outside the building (where the area is located) and / or an external humidity level measured outside the building.

The one or more output comprises one or more command for controlling the controlled appliance <NUM>. As mentioned previously, an example of controlled appliance <NUM> is a VAV appliance. Examples of commands for controlling the VAV appliance <NUM> include commands directed to one of the following actuation modules of the VAV appliance <NUM>: an actuation module controlling the speed of a fan, an actuation module controlling the pressure generated by a compressor, an actuation module controlling a valve defining the rate of an airflow, etc. Although the present disclosure focuses on generating command(s) for controlling appliance(s) at step <NUM>, other types of output may be generated in addition to the command(s) at step <NUM>.

The method <NUM> comprises the step <NUM> of modifying the one or more command generated at step <NUM>. Step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>.

Different algorithms may be implemented at step <NUM>. Following are examples of algorithms for modifying the one or more command. However, a person skilled in the art would readily understand that other algorithms may be used in the context of the present disclosure.

In a first implementation, the modification to a command is random. Furthermore, the random modification may be limited to a pre-defined range of modifications. For example, the command consists in adjusting the speed of a fan, and the predefined range of modifications is between -<NUM>% and +<NUM>%. If the speed generated at step <NUM> is <NUM> revolutions per second, then a random value between <NUM> and <NUM> revolutions per second is generated at step <NUM>.

In a second implementation, the modification to a command is selected among a set of one or more pre-defined modification. For example, the command consists in adjusting the speed of a fan, and the predefined modifications consist of +<NUM>%, +<NUM>%, -<NUM>% and -<NUM>%. If the speed generated at step <NUM> is <NUM> revolutions per second, then a value among <NUM>, <NUM>, <NUM> and <NUM> revolutions per second is selected at step <NUM>. The sub-algorithm for selecting one among a plurality of pre-defined modifications is out of the scope of the present disclosure.

In the case where the one or more command generated at step <NUM> includes two or more commands, the modification may affect any combination of the commands (e.g. all the commands are modified or only some of the commands are modified). For example, if the one or more command includes one command for adjusting the speed of a fan and one command for adjusting the pressure generated by a compressor, the modification at step <NUM> includes one of the following: only adjust the speed of the fan, only adjust the pressure generated by the compressor, or simultaneously adjust the speed of the fan and the pressure generated by the compressor. Furthermore, the selection of which commands are modified may vary each time step <NUM> is performed, using a random algorithm or a pre-defined modification schedule.

In an exemplary implementation, the type of modification(s) to be applied at step <NUM> is received via the communication interface <NUM>. For example, the training server <NUM> sends a configuration message to the environment controller <NUM>. The configuration message defines the type of modification(s) to be applied at step <NUM>. As will be illustrated later in the description, this mechanism allows the training server <NUM> to control a plurality of environment controllers <NUM> via configuration messages defining various types of modification(s) to be applied at step <NUM>. Thus, the training server <NUM> drives a fleet of environment controllers <NUM> respectively applying modifications at step <NUM>. Each environment controller <NUM> has its own range of modifications, allowing a wide range of exploratory modifications for the purpose of improving the predictive model. The configuration data (type of modification(s) to be applied) included in the configuration message are stored in the memory <NUM> and used each time step <NUM> is performed. Each environment controller <NUM> can also be reconfigured by the training server <NUM> via a new configuration message defining a new set of modification(s) to be applied at step <NUM>.

The method <NUM> comprises the step <NUM> of transmitting the one or more modified command (generated at step <NUM> and modified at step <NUM>) to the controlled appliance <NUM> via the communication interface <NUM>. Step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>.

The method <NUM> comprises the step <NUM> of receiving the one or more modified command at the controlled appliance <NUM>, via the communication interface of the controlled appliance <NUM>. Step <NUM> is performed by the processing unit of the controlled appliance <NUM>.

The method <NUM> comprises the step <NUM> of executing the one or more modified command at the controlled appliance <NUM>. Step <NUM> is performed by the processing unit of the controlled appliance <NUM>. Executing the one or more modified command consists in controlling one or more actuation module of the controlled appliance <NUM> based on the received one or more modified command.

As mentioned previously, a single command or a plurality of commands is generated at step <NUM> and transmitted at step <NUM> (after modification at step <NUM>) to the same controlled appliance <NUM>. Alternatively, the same command is generated at step <NUM> and transmitted at step <NUM> to a plurality of controlled appliances <NUM>. In yet another alternative, a plurality of commands is generated at step <NUM> and transmitted at step <NUM> to a plurality of controlled appliances <NUM>.

The method <NUM> comprises the step <NUM> of generating at least one metric representative of the execution (at step <NUM>) of the one or more modified command by the controlled appliance <NUM>. Step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>.

The role of the one or more metric is to provide a quantified evaluation of the efficiency of the execution of the modified command(s) (at step <NUM>). More specifically, since the one or more modified command aims at reaching the set point(s) received at step <NUM>, the one or more metric evaluates the efficiency of execution of the one or more modified command for the purpose of reaching the set point(s). The efficiency may be measured according to various criteria, including the time required for reaching an environmental state corresponding to the set point(s), the adequacy of the reached environmental state with respect to the set point(s), the impact on the comfort of the users present in the area, etc..

Examples of metrics include the determination of one or more updated environmental characteristic value in the area following the transmission of the modified command(s), the measurement of one or more time required for reaching one or more corresponding environmental state in the area (e.g. reaching one or more set point) following the transmission of the modified command(s), the measurement of an energy consumption by the execution of the modified command(s), etc..

For illustration purposes, we consider the use case where a target temperature is included in the set point(s). A first example of metric consists of an updated temperature measured by the temperature sensor <NUM> and transmitted to the environment controller <NUM> after a given amount of time (e.g. <NUM> minutes), following the transmission of the modified command(s) at step <NUM>. A second example of metric consists of several updated temperatures measured by the temperature sensor <NUM> and transmitted to the environment controller <NUM> at various interval of times (e.g. respectively <NUM> minutes and <NUM> minutes), following the transmission of the modified command(s) at step <NUM>. This second example allows an evaluation of the trajectory of the variation of temperature in the area from the current temperature (determined at step <NUM>) to the target temperature (received at step <NUM>). A third example of metric consists of a measurement of the time required for reaching the target temperature, following the transmission of the modified command(s) at step <NUM>. In this third example, the environment controller <NUM> starts a timer following the transmission of the modified command(s) at step <NUM>. The environment controller <NUM> receives updated temperatures measured by the temperature sensor <NUM> and transmitted to the environment controller <NUM>. Upon reception of an updated temperature substantially equal to the target temperature, the environment controller <NUM> stops the timer. The measurement of the required time is the difference between the times at which the timer was respectively stopped and started. A fourth example of metric consists of several measurements of the time required for reaching milestones on the trajectory from the current temperature towards the target temperature, following the transmission of the modified command(s) at step <NUM>. For example, a first milestone corresponds to a temperature halfway between the current temperature and the target temperature, and a second milestone corresponds to the target temperature.

The previous exemplary metrics are for illustration purposes only. A person skilled in the art would be capable of implementing other metrics particularly adapted to the specific inputs and outputs used by the neural network inference engine <NUM> at step <NUM>.

The method <NUM> comprises the step <NUM> of transmitting the inputs used by the neural network inference engine <NUM> (at step <NUM>), the one or more output generated by the neural network inference engine <NUM> (at step <NUM>), and the at least one metric (generated at step <NUM>) to the training server <NUM> via the communication interface <NUM>. Step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>. All the data transmitted at step <NUM> are referred to as training data in <FIG>.

A new set of training data is transmitted to the training server <NUM> as soon as it is available (after each execution of steps <NUM>-<NUM>-<NUM>-<NUM>). Alternatively, the transmission of a new set of training of data to the training server <NUM> is delayed until a certain amount of training data has been collected (the transmission of all the collected training data occurs after several executions of steps <NUM>-<NUM>-<NUM>-<NUM>).

The method <NUM> comprises the step <NUM> of receiving the inputs, the one or more output and the at least one metric (transmitted at step <NUM>) at the training server <NUM>, via the communication interface of the training server <NUM>. Step <NUM> is performed by the processing unit of the training server <NUM>.

The predictive model stored by the environment controller <NUM> is also stored by the training server <NUM>.

The method <NUM> comprises the step <NUM> of generating an update of the predictive model. Step <NUM> is performed by the processing unit of the training server <NUM>. The update of the predictive model comprises an update of the weights of the neural network. The update is performed based on the inputs, the one or more output and the at least one metric received at step <NUM>.

The method <NUM> comprises the step <NUM> of transmitting the update of the predictive model (comprising the updated weights) to the environment controller <NUM>, via the communication interface of the training server <NUM>. Step <NUM> is performed by the processing unit of the training server <NUM>.

Steps <NUM>, <NUM> and <NUM> will be detailed later, when providing a detailed description of the functionalities of the training server <NUM>.

The method <NUM> comprises the step <NUM> of receiving the update of the predictive model (comprising the updated weights) from the training server <NUM> via the communication interface <NUM>. Step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>.

Reference is now made more particularly to <FIG>. During a training phase, the method <NUM> is used for generating an operational predictive model based on an initial predictive model. Steps <NUM> to <NUM> are repeated systematically. The initial predictive model is stored at step <NUM>. Then, the repetition of steps <NUM> to <NUM> provides data to the training server <NUM> for improving the initial predictive model. At some point, the training server <NUM> determines that an operational version of the predictive model is ready, and transmits the operational version to the environment controller <NUM>. The operational version is received at step <NUM> and stored at step <NUM>.

Reference is now made more particularly to <FIG>. During an operational phase, the method <NUM> can be used to improve / fine-tune the current predictive model. Steps <NUM>, <NUM> and <NUM> are not performed systematically, but only once in a while (for example, once every ten occurrences of step <NUM>). The rest of the time, the command(s) generated at step <NUM> are not modified. The execution of steps <NUM>, <NUM> and <NUM> provides data to the training server <NUM> for improving the current predictive model. At some point, the training server <NUM> determines that an improved version of the predictive model is ready, and transmits the improved version to the environment controller <NUM>. The improved version is received at step <NUM> and stored at step <NUM>.

The steps of the method <NUM> involving the reception or the transmission of data by the environment controller <NUM> may use the same communication interface <NUM> or different communication interfaces <NUM>. For example, steps <NUM>, optionally <NUM>, and <NUM> use a first communication interface <NUM> of the Wi-Fi type; while steps <NUM> and <NUM> use a second communication interface <NUM> of the Ethernet type. In another example, steps <NUM>, optionally <NUM>, <NUM>, <NUM> and <NUM> use the same communication interface <NUM> of the Wi-Fi type.

In an alternative implementation, for each environmental characteristic value considered at step <NUM>, a plurality of consecutive measurements of the environmental characteristic value is determined at step <NUM> (instead of a single current environmental characteristic value). For example, the inputs used by the neural network inference engine <NUM> at step <NUM> include a plurality of consecutive temperature measurements in the area (instead of a single current temperature in the area), and / or a plurality of consecutive humidity level measurements in the area (instead of a single current humidity level in the area), and / or a plurality of consecutive CO2 level measurements in the area, (instead of a single current CO2 level in the area). For instance, a measurement is determined (e.g. received from a corresponding sensor) every minute and the last five consecutive measurements (the current one, one minute before, two minutes before, three minutes before, and four minutes before) are stored in the memory <NUM>. At step <NUM>, the inputs include the last five consecutive measurements stored in the memory <NUM> (e.g. the last five consecutive temperature measurements and the last five consecutive humidity measurements).

<FIG> is a schematic representation of the neural network inference engine <NUM> illustrating the inputs and the outputs used by the neural network inference engine <NUM> when performing step <NUM>.

<FIG> is a detailed representation of an exemplary neural network implemented by the neural network inference engine <NUM>.

The neural network includes an input layer with four neurons for receiving four input parameters (the current temperature in the area, the current humidity level in the area, the number of persons present in the area, and the target temperature). The neural network includes an output layer with two neurons for outputting two output values (the inferred adjustment of the speed of a fan and the inferred adjustment of the pressure generated by a compressor). The neural network includes three intermediate hidden layers between the input layer and the output layer. All the layers are fully connected. The number and type of inputs (four in <FIG>) and outputs (two in <FIG>) of the neural network are for illustration purposes only. Any combination of inputs and outputs supported by the present description can be applied to the neural network illustrated in <FIG>.

The number of intermediate hidden layers is an integer greater or equal than <NUM> (<FIG> represents three intermediate hidden layers for illustration purposes only). The number of neurons in each intermediate hidden layer may vary. During the training phase of the neural network, the number of intermediate hidden layers and the number of neurons for each intermediate hidden layer are selected, and may be adapted experimentally.

The generation of the outputs based on the inputs using weights allocated to the neurons of the neural network is well known in the art. The architecture of the neural network, where each neuron of a layer (except for the first layer) is connected to all the neurons of the previous layer is also well known in the art.

Reference is now made concurrently to <FIG>, <FIG> and <FIG>, where <FIG> illustrates the usage of the method <NUM> in a large environment control system.

A plurality of environment controllers <NUM> implementing the method <NUM> are deployed at different locations. Only two environment controllers <NUM> are represented in <FIG> for illustration purposes, but any number of environment controllers <NUM> may be deployed. Each environment controller <NUM> represented in <FIG> corresponds to the environment controller <NUM> represented in <FIG>. Each environment controller <NUM> interacts with the same entities as represented in <FIG>, such as the controlled appliance <NUM> (the sensors illustrated in <FIG> are not represented in <FIG> for simplification purposes).

In an exemplary configuration, the different locations are within a building, and the environment controllers <NUM> are deployed at different floors of the building, different rooms of the building, etc. The training server <NUM> is also deployed in the building. Alternatively, the training server <NUM> is deployed at a remote location from the building, for example in a remote cloud infrastructure. In another configuration, the environment controllers <NUM> are deployed at different buildings. The training server <NUM> is deployed in one of the buildings, or at a remote location from the buildings.

Each environment controller <NUM> receives an initial predictive model from the centralized training server <NUM>. The same initial predictive model is used for all the environment controllers <NUM>. Each environment controller <NUM> generates training data when using the initial predictive model, and the training data are transmitted to the training server <NUM>. The training server <NUM> uses the training data from all the environment controllers <NUM> to improve the initial predictive model. At some point, an improved predictive model generated by the training server <NUM> is transmitted to the environment controllers <NUM>, and used by all the environment controllers <NUM> in place of the initial predictive model. Several iterations of this process can be performed, where the environment controllers <NUM> use a current version of the predictive model to generate training data, and the training data are used by the training server <NUM> to generate a new version of the predictive model.

The environment controllers <NUM> control environments having substantially similar characteristics, so that the same predictive model is adapted to all the environment controllers <NUM>. For example, the environment controllers <NUM> control the environment of rooms having substantially similar geometric characteristics, and / or substantially the same type of human activity in the rooms, etc..

Details of the components of the training server <NUM> are also represented in <FIG>. The training server <NUM> comprises a processing unit <NUM>, memory <NUM>, and a communication interface <NUM>. The training server <NUM> may comprise additional components, such as another communication interface <NUM>, a user interface <NUM>, a display <NUM>, etc..

The characteristics of the processing unit <NUM> of the training server <NUM> are similar to the previously described characteristics of the processing unit <NUM> of the environment controller <NUM>. The processing unit <NUM> executes the neural network training engine <NUM> and a control module <NUM>.

The characteristics of the memory <NUM> of the training server <NUM> are similar to the previously described characteristics of the memory <NUM> of the environment controller <NUM>.

The characteristics of the communication interface <NUM> of the training server <NUM> are similar to the previously described characteristics of the communication interface <NUM> of the environment controller <NUM>.

Reference is now made concurrently to <FIG>, <FIG>, <FIG> and <FIG> represents a method <NUM> for improving a predictive model of a neural network used by the environment controllers <NUM> (more specifically by the neural network inference engines <NUM>) through reinforcement learning. At least some of the steps of the method <NUM> represented in <FIG> are implemented by the training server <NUM>. The present disclosure is not limited to the method <NUM> being implemented by the training server <NUM>, but is applicable to any type of computing device capable of implementing the steps of the method <NUM>.

A dedicated computer program has instructions for implementing at least some of the steps of the method <NUM>. The instructions are comprised in a non-transitory computer program product (e.g. the memory <NUM>) of the training server <NUM>. The instructions provide for improving the predictive model of the neural network used by the environment controllers <NUM> (more specifically by the neural network inference engines <NUM>) through reinforcement learning, when executed by the processing unit <NUM> of the training server <NUM>. The instructions are deliverable to the training server <NUM> via an electronically-readable media such as a storage media (e.g. CD-ROM, USB key, etc.), or via communication links (e.g. via a communication network through the communication interface <NUM>).

The instructions of the dedicated computer program executed by the processing unit <NUM> implement the neural network training engine <NUM> and the control module <NUM>. The neural network training engine <NUM> provides functionalities for training a neural network, allowing to improve a predictive model (more specifically to optimize weights of the neural network), as is well known in the art. The control module <NUM> provides functionalities allowing the training server <NUM> to gather data used for the training of the neural network.

An initial predictive model is generated by the processing unit <NUM> of the training server <NUM> and transmitted to the plurality of environment controllers <NUM> via the communication interface <NUM> of the training server <NUM>. Alternatively, the initial predictive model is generated by and received from another computing device (via the communication interface <NUM> of the training server <NUM>). The initial predictive model is also transmitted by the other computing device to the plurality of environment controllers <NUM>.

The generation of the initial predictive model is out of the scope of the present disclosure. Generating the initial predictive model comprises defining a number of layers of the neural network, a number of neurons per layer, the initial value for the weights of the neural network, etc..

The definition of the number of layers and the number of neurons per layer is performed by a person highly skilled in the art of neural networks. Different algorithms (well documented in the art) can be used for allocating an initial value to the weights of the neural network. For example, each weight is allocated a random value within a given interval (e.g. a real number between -<NUM> and +<NUM>), which can be adjusted if the random value is too close to a minimum value (e.g. - <NUM>) or too close to a maximum value (e.g. +<NUM>).

The execution of the method <NUM> by the training server <NUM> and the execution of the method <NUM> by the environment controllers <NUM> provide for improving the initial predictive model (more specifically to optimize the weights of the predictive model). At the end of the training phase, an improved predictive model is ready to be used by the neural network inference engines <NUM> of the plurality of environment controllers <NUM>. Optionally, the improved predictive model can be used as a new initial predictive model, which can be further improved by implementing the aforementioned procedure again.

The method <NUM> comprises the step <NUM> of storing the initial predictive model in the memory <NUM>. Step <NUM> is performed by the processing unit <NUM>. The initial predictive model comprises the weights of the neural network implemented by the neural network training engine <NUM>.

The method <NUM> comprises the step <NUM> of receiving a plurality of training data sets via the communication interface <NUM>. Step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>. The training data sets are received from the plurality of environment controllers <NUM>. Step <NUM> corresponds to step <NUM> of the method <NUM> executed by the environment controllers <NUM>.

Each training data set comprises inputs of the neural network implemented by the neural network training engine <NUM>, one or more output of the neural network implemented by the neural network training engine <NUM>, and at least one metric. The inputs comprise at least one environmental characteristic value in the area under the control of the corresponding environment controller <NUM> (determined at step <NUM> of the method <NUM>) and at least one set point (received at step <NUM> of the method <NUM>). The one or more output comprises one or more command for controlling the controlled appliance <NUM> (generated at step <NUM> by modifying the command generated at step <NUM> of the method <NUM>). The at least one metric (generated at step <NUM> of the method <NUM>) is representative of an execution of the one or more command by the controlled appliance <NUM>.

As mentioned previously, the at least one environmental characteristic value includes one or more of the following: a current temperature in the area, a current humidity level in the area, a current CO2 level in the area, and a current occupancy of the area. Alternatively, the at least one environmental characteristic value includes one or more of the following: a plurality of consecutive temperature measurements in an area, a plurality of consecutive humidity level measurements in the area, a plurality of consecutive carbon dioxide (CO2) level measurements in the area, and a plurality of consecutive determinations of an occupancy of the area. The at least one set point includes one or more of the following: a target temperature, a target humidity level, and a target CO2 level. Examples of the one or more command have also been described previously.

Optionally, the inputs include additional parameters used at step <NUM> of the method <NUM>. For example, the inputs further include at least one characteristic of the area (determined at optional step <NUM> of the method <NUM>). As mentioned previously, the at least one characteristic of the area includes one or more of the following: an area type identifier selected among a plurality of area type identifiers, one or more geometric characteristics of the area, and a human activity in the area. Optionally, the outputs include additional parameters different from command(s).

As illustrated in <FIG>, steps <NUM> and <NUM> of the method <NUM> are repeated for each training data set received at step <NUM>.

The method <NUM> comprises the step <NUM> of determining a value of a reinforcement signal based on the at least one metric of a given training data set (among the plurality of training data sets received at step <NUM>). Step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>.

The value of the reinforcement signal is one of positive reinforcement (also referred to as a positive reward) or negative reinforcement (also referred to as a negative reward). For example, the control module <NUM> implements a set of rules (stored in the memory <NUM>) to determine the value of the reinforcement signal. The set of rules is designed for evaluating the efficiency of the modified command(s) transmitted at step <NUM> of the method <NUM> for reaching the set point(s) received at step <NUM> of the method <NUM>. If the command(s) is evaluated as being efficient, the outcome is a positive reinforcement value for the reinforcement signal. If the command(s) is evaluated as not being efficient, the outcome is a negative reinforcement value for the reinforcement signal. The reinforcement signal takes only two Boolean values: positive reinforcement or negative reinforcement. Alternatively, the reinforcement signal is expressed as a percentage representing a relative efficiency. For example, positive reinforcement includes the values between <NUM> and <NUM>%, while negative reinforcement includes the values between <NUM> and <NUM>%. Alternatively, the reinforcement signal takes one among a pre-defined set of values (e.g. +<NUM>, +<NUM>, +<NUM> for positive reinforcement and -<NUM>, -<NUM>, -<NUM> for negative reinforcement). The neural network training engine <NUM> is adapted and configured to adapt the weights of the predictive model based on values chosen for implementing the reinforcement signals. A person skilled in the art would readily understand that the values of the reinforcement signal are not limited to the previous examples.

The determination of the value of the reinforcement signal may further takes into consideration the at least one set point included in the inputs received at step <NUM>. Alternatively or complementarily, the determination of the value of the reinforcement signal may further takes into consideration the at least one environmental characteristic value in an area included in the inputs received at step <NUM>. Alternatively or complementarily, the determination of the value of the reinforcement signal may further take into consideration the characteristic(s) of the area included in the inputs received at step <NUM> (if optional step <NUM> of the method <NUM> is performed).

Following are exemplary sets of rules for evaluating the efficiency of the command(s) transmitted at step <NUM> of the method <NUM>, based on a target temperature (received at step <NUM> of the method <NUM>) and a metric consisting of one or more updated temperature measurement (determined at step <NUM> of the method <NUM>). The target temperature and the one or more updated temperature measurement are comprised in the training data transmitted at step <NUM> of the method <NUM> and received at step <NUM> of the method <NUM>.

A first exemplary set of rules uses a single updated temperature measurement. The reinforcement signal is positive if the absolute difference between the target temperature and the updated temperature measurement is lower than a threshold (e.g. <NUM> degree Celsius). The reinforcement signal is negative otherwise.

A second exemplary set of rules uses several consecutive measurements of the updated temperature. For instance, the reinforcement signal is positive if the absolute difference between the target temperature and a first measurement of the updated temperature determined <NUM> minutes after transmitting the commands (at step <NUM> of the method <NUM>) is lower than a first threshold (e.g. <NUM> degrees Celsius) AND the absolute difference between the target temperature and a second measurement of the updated temperature determined <NUM> minutes after transmitting the commands (at step <NUM> of the method <NUM>) is lower than a second threshold (e.g. <NUM> degree Celsius). The reinforcement signal is negative otherwise.

A third exemplary set of rules further uses the volume of the area (determined at step <NUM> of the method <NUM>, transmitted at step <NUM> of the method <NUM> and received at step <NUM> of the method <NUM>). The reinforcement signal is positive if the absolute difference between the target temperature and the updated temperature measurement is lower than a first threshold (e.g. <NUM> degree Celsius) AND the volume of the area is lower than <NUM> cubic meters. The reinforcement signal is also positive if the absolute difference between the target temperature and the updated temperature measurement is lower than a second threshold (e.g. <NUM> degree Celsius) AND the volume of the area is higher than <NUM> cubic meters. The reinforcement signal is negative otherwise.

A fourth exemplary set of rules further uses the human activity in the area, and more specifically the type of activity performed by humans occupying the area (determined at step <NUM> of the method <NUM>, transmitted at step <NUM> of the method <NUM> and received at step <NUM> of the method <NUM>). The reinforcement signal is positive if the absolute difference between the target temperature and the updated temperature measurement is lower than a first threshold (e.g. <NUM> degree Celsius) AND the area is an office room. The reinforcement signal is also positive if the absolute difference between the target temperature and the updated temperature measurement is lower than a second threshold (e.g. <NUM> degrees Celsius) AND the area is a storage room. The reinforcement signal is negative otherwise.

A fifth exemplary set of rules also uses the human activity in the area, and more specifically periods of time when the area is occupied by humans (determined at step <NUM> of the method <NUM>, transmitted at step <NUM> of the method <NUM> and received at step <NUM> of the method <NUM>). The reinforcement signal is positive if the absolute difference between the target temperature and the updated temperature measurement is lower than a first threshold (e.g. <NUM> degree Celsius) AND the current time is within a period of occupation of the area (e.g. between 8am and 6pm from Monday to Saturday). The reinforcement signal is also positive if the absolute difference between the target temperature and the updated temperature measurement is lower than a second threshold (e.g. <NUM> degrees Celsius) AND the current time is within a period of inoccupation of the area (e.g. anytime except between 8am and 6pm from Monday to Saturday). The reinforcement signal is negative otherwise.

Following is another exemplary sets of rules for evaluating the efficiency of the command(s) transmitted at step <NUM> of the method <NUM>, based on metric(s) consisting of one or more measurement of the time required for reaching the target temperature (received at step <NUM> of the method <NUM>). The one or more measurement of the required time is determined at step <NUM> of the method <NUM>. The one or more measurement of the required time is comprised in the training data transmitted at step <NUM> of the method <NUM> and received at step <NUM> of the method <NUM>.

A first exemplary set of rules uses a single measurement consisting of the time required for reaching the target temperature. The reinforcement signal is positive if the measurement of the required time is lower than a threshold (e.g. <NUM> minutes). The reinforcement signal is negative otherwise.

A second exemplary set of rules uses several consecutive measurements of the of the time required for reaching the target temperature. For instance, the reinforcement signal is positive if a first measurement of the required time for reaching a temperature halfway between the current temperature measurement (determined at step <NUM> of the method <NUM>) and the target temperature is lower than a first threshold (e.g. <NUM> minutes) AND a second measurement of the required time for reaching the target temperature is lower than a second threshold (e.g. <NUM> minutes). The reinforcement signal is negative otherwise.

In addition to the one or more measurement of the time required for reaching the target temperature, other set of rules may be defined, which further use the characteristics of the area determined at step <NUM> of the method <NUM> (e.g. volume of the area, human activity in the area, periods of time when the area is occupied by humans, etc.), as illustrated previously.

The previous exemplary sets of rules are for illustration purposes only. A person skilled in the art would be capable of implementing other sets of rules particularly adapted to the specific inputs and outputs used by the neural network inference engine <NUM> at step <NUM> of the method <NUM>.

The method <NUM> comprises the step <NUM> of executing the neural network training engine <NUM> to update the weights of the neural network based on the inputs (of the given training data set), the one or more output (of the given training data set), and the value of the reinforcement signal (determined at step <NUM>). The execution of the neural network training engine <NUM> is performed by the processing unit <NUM>.

The neural network training engine <NUM> implements the neural network using the weights of the predictive model stored at step <NUM>. The neural network implemented by the neural network training engine <NUM> corresponds to the neural network implemented by the neural network inference engine <NUM> (same number of layers, same number of neurons per layer). As mentioned previously, <FIG> is a detailed exemplary representation of such a neural network.

Reinforcement learning is a technique well known in the art of artificial intelligence. Having a set of inputs and the corresponding output(s), the weights of the predictive model are updated to force the generation of the corresponding output(s) when presented with the inputs, if the value of the reinforcement signal is a positive reinforcement. Complementarily, having a set of inputs and the corresponding output(s), the weights of the predictive model are updated to prevent the generation of the corresponding output(s) when presented with the inputs, if the value of the reinforcement signal is a negative reinforcement. Thus, having a given set of inputs and a candidate set of corresponding output(s), the neural network training engine <NUM> learns through reinforcement learning which one(s) among the candidate set of corresponding output(s) is (are) the best fit for the given set of input(s). In the context of the present disclosure, the neural network training engine <NUM> learns (through reinforcement learning) which command(s) is / are the best fit for reaching the set point(s), when presented with the current environmental characteristic value(s), the set point(s) and optionally the characteristic(s) of the area.

Additionally, during the training phase, the number of intermediate hidden layers of the neural network and the number of neurons per intermediate hidden layer can be adjusted to improve the accuracy of the predictive model. At the end of the training phase, the predictive model generated by the neural network training engine <NUM> includes the number of layers, the number of neurons per layer, and the weights. However, the number of neurons for the input and output layers shall not be changed.

Although not represented in <FIG> for simplification purposes, the modifications to the weights of the neural network performed at step <NUM> are stored in the memory <NUM> of the training server <NUM>.

<FIG> is a schematic representation of the neural network training engine <NUM> illustrating the inputs, the one or more output and the value of the reinforcement signal used by the neural network inference engine <NUM> when performing step <NUM>.

Optionally, as illustrated in <FIG>, several iterations of steps <NUM>-<NUM>-<NUM> are repeated if a plurality of batches of training data sets are received at step <NUM>. The execution of steps <NUM>-<NUM>-<NUM> is implementation dependent. In a first exemplary implementation, as soon as the training server <NUM> receives training data set(s) from a given environment controller <NUM> at step <NUM>, steps <NUM> and <NUM> are immediately performed. In a second exemplary implementation, the training server <NUM> waits for the reception of a substantial amount of training data sets from environment controller(s) <NUM> at step <NUM>, before performing steps <NUM> and <NUM>. In this second implementation, the received training data steps are stored in the memory <NUM> before being used. Furthermore, some of the received training data sets may be discarded by the training server <NUM> (e.g. a training data set is redundant with another already received training data set, at least some of the data contained in the training data set are considered erroneous or non-usable, etc.).

At the end of the training phase implemented by steps <NUM>-<NUM>-<NUM>, the neural network is considered to be properly trained, and an updated predictive model comprising a final version of the updated weights is transmitted to the environment controllers <NUM>, as illustrated in <FIG>. Various criteria may be used to determine when the neural network is considered to be properly trained, as is well known in the art of neural networks. This determination and the associated criteria is out of the scope of the present disclosure.

The method <NUM> comprises the step <NUM> of transmitting an update of the predictive model (originally stored at step <NUM>) comprising the updated weights (updated by the repetition of step <NUM>) to the plurality of environment controllers <NUM> via the communication interface <NUM>. Step <NUM> is performed by the control module <NUM> executed by the processing unit <NUM>. The update of the predictive model of the neural network generally only involves an update of the weights (the number of layers of the neural network and the number of neurons per layer are generally unchanged). Step <NUM> corresponds to step <NUM> of the method <NUM> executed by the environment controllers <NUM>.

From this point on, the environment controllers <NUM> enter an operational mode, where the updated predictive model is used for managing the environment (generating command(s) for controlling the controlled appliances <NUM>) of the respective areas under the control of the environment controllers <NUM>.

During the execution of the method <NUM> for improving the initial predictive model, only a few environment controllers <NUM> may be operating in a training mode, for the sole purpose of providing the training data sets used by the training server <NUM> when executing the method <NUM>. Once the updated predictive model is available at the end of the training phase, it can be distributed to a larger number of environment controllers <NUM> entering the operational mode. Additionally, the methods <NUM> and <NUM> can be used to further improve the updated predictive model used in the operational mode, as described previously.

Claim 1:
A training server (<NUM>) comprising:
a communication interface (<NUM>);
memory (<NUM>) for storing a predictive model comprising weights of a neural network; and
a processing unit (<NUM>) comprising one or more processor configured to:
receive (<NUM>) a plurality of training data sets from a plurality of environment controllers (<NUM>) via the communication interface, each training data set comprising inputs, one or more output, and at least one metric, the inputs comprising at least one environmental characteristic value in an area and at least one set point, the one or more output comprising one or more commands for controlling a controlled appliance, the at least one metric providing a quantified evaluation of an efficiency of an execution of the one or more commands by the controlled appliance for reaching the at least one set point;
for each received training data set:
determine (<NUM>) a value of a reinforcement signal based on the at least one metric; and
execute (<NUM>) a neural network training engine (<NUM>) to update the weights of the neural network based on the inputs, the one or more output, and the value of the reinforcement signal; and
transmit (<NUM>) an update of the predictive model comprising the updated weights to the plurality of environment controllers via the communication interface.