Patent Description:
Model Predictive Control (MPC) of a matrix converter (MxC) can provide certain advantages over traditional proportional integral (PI) controls. One advantage is that MPC enables simplified application of additional criteria (such as switching loss reduction, common mode voltage reduction, and harmonic reductions or eliminations). One or more additional criteria for controlling the MxC can be applied by adding an objective for each criterion to the cost function. Unlike PI controls, this approach avoids the need to change control architecture or add control loops that can affect system stability. Another advantage is that MPC is very effective during transients because it can adapt its switching frequency at different operation regimes (e.g., transients vs. steady-state) as opposed to a PI controlled pulse width modulation (PWM) that has a fixed switching frequency.

The downside of the MPC is that it requires a greedy search by performing a search at each sampling period over every possible switching state. This greedy search consumes resources and may become infeasible as the number of possible switching states increases with development of future power converter or inverter architectures.

While conventional methods and systems have generally been considered satisfactory for their intended purpose, there is still a need in the art to use artificial intelligence for MxC MPC control that has the advantages of MxC MPC but boosts computation speed and/or reduces the searching computation load performed per sampling period.

The invention is defined by the features of device claim <NUM> and method claim <NUM>. The dependent claims recite advantageous embodiments of the invention.

The purpose and advantages of the below described illustrated embodiments will be set forth in and apparent from the description that follows. Additional advantages of the illustrated embodiments will be realized and attained by the devices, systems and methods particularly pointed out in the written description and claims hereof, as well as from the appended drawings.

To achieve these and other advantages and in accordance with the purpose of the illustrated embodiments, in one aspect, disclosed is a matrix converter system of an industrial plant system. The matrix converter system includes a switching matrix coupled between an input side and an output side. The matrix converter system includes a model predictive controller (MPC) configured to select a switching state of the switching matrix from a plurality of switching states. The MPC is configured to receive an operating condition of the industrial plant system and consult a Q-data structure for reward values associated with respective switching states for an operating state that corresponds to the operating condition. The Q-data structure is trained in a real or simulation environment of the industrial plant system using Q-learning to map till convergence a reward value predicted for respective switching states of the plurality of switching states to respective discrete operating states of a plurality of operating states. The MPC is further configured to sort the reward values predicted for the respective switching states mapped to the operating state that corresponds to the operating condition, select a subset of the set of the mappings as a function of a result of sorting the reward values associated with the switching states of the operating state, evaluate each switching state included in the subset, and select an optimal switching state for the operating condition based on a result of evaluating the switching states of the subset.

In accordance with other aspects of the disclosure, a method of controlling a matrix converter system is provideda control system for a matrix converter is provided, wherein the The matrix converter has a switching matrix coupled between an input side and an output side. The control system includes an MPC configured to select a switching state of the switching matrix from a plurality of switching states, the MPC is configured to receive an operating condition of the industrial plant system and consult a Q-data structure for reward values associated with respective switching states for an operating state that corresponds to the operating condition. The Q-data structure is trained in a real or simulation environment of the industrial plant system using Q-learning to map till convergence a reward value predicted for respective switching states of the plurality of switching states to respective discrete operating states of a plurality of operating states. The control system is further configured to sort the reward values predicted for the respective switching states mapped to the operating state that corresponds to the operating condition, select a subset of the set of the mappings as a function of a result of sorting the reward values associated with the switching states of the operating state, evaluate each switching state included in the subset, and select an optimal switching state for the operating condition based on a result of evaluating the switching states of the subset.

These and other features of the systems and methods of the subject disclosure will become more readily apparent to those skilled in the art from the following detailed description of the embodiments taken in conjunction with the drawings.

A matrix converter (MxC) control system is disclosed that uses model predictive control (MPC) in the inner control loop for controlling an MxC of a power converter or inverter. The MxC control system uses machine learning to boost computation speed by reducing search space from all of the possible switching states to a subset selected from the possible switching states. This reduction in search space reduces computation load and time and allows for increased efficiency of control.

The machine learning applies reinforcement learning (RL) to create, using Q-learning, a Q-data structure. The Q-data structure maps expected predicted effects, referred to as rewards (which is the inverse of costs) for each of the possible switching states of the MxC in each respective operating state of the power converter. Each of the operating states corresponds to a different set of operating conditions (also referred to as an operating condition) of the power converter or inverter.

During real-time operation, at each sampling period, the current operating condition is used to determine a corresponding operating state. For the operating state, possible switching states are ranked as a function of their associated award obtained from the Q-data structure. A subset of the switching states is selected based on ranking. Each of the selected switching states is evaluated.

Turning now descriptively to the drawings, in which similar reference characters denote similar elements throughout the several views, <FIG> depicts an exemplary plant system <NUM> in which below illustrated embodiments may be implemented. Plant system <NUM> includes an MxC controller <NUM> that controls an MxC of an industrial plant system <NUM> (referred to as plant <NUM>). Plant <NUM> is an industrial plant that includes a generator (e.g., an input side), and a load (e.g., an output side), shown together as component <NUM>. The generator provides an electrical signal (voltage and/or current) to the MxC <NUM>. The MxC <NUM> includes a matrix of switches controlled by the MxC controller <NUM>. The load receives an electrical signal (voltage and/or current) from the MxC controller <NUM>. In an example embodiment, without limitation to the particular embodiment, the generator is an aircraft generator and the load is an aircraft motor.

MxC controller <NUM> includes a processing device that receives an operating condition from plant <NUM> via signal path <NUM> each sampling period, consults a trained Q-table <NUM> for selecting a subset of possible switching states to analyze, controls the MxC <NUM> to analyze the selected switching states, and selects a switching state to use for the sampling period.

The processing device included with the MxC controller <NUM> can include, for example, a programmable logic device (PLOD), microprocessor, digital signal processor (DSP), a microcontroller, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and/or other discrete or integrated logic circuitry having similar processing capabilities.

The Q-table <NUM> is a data structure trained by Q-learning reinforcement learning (RL) algorithm till convergence. The RL enables mapping conditions of the MxC <NUM> to the most important switching states. Using RL, the Q-table is trained and applied by focusing on learning by the MxC controller <NUM>, via direct interaction between the industrial plant <NUM> and its environment without the need to rely on complete models of the environment. Q-table <NUM> provides an entry for each switching state at each operating condition. Each entry can be initialized with an initial reward value, and then updated with an estimated predictive value as the Q-table <NUM> is trained. In this way, Q-table <NUM> provides a simple representation of converter control configurations using artificial intelligence.

The data structure for representing Q-table <NUM> is referred to as a table, but it is not limited to a particular data structure. In the example embodiment shown, Q-table <NUM> is a lookup table (LUT). Q-table <NUM> is stored in a storage module <NUM>. Storage module <NUM> can be remote from or integrated into MxC <NUM>. Application of Q-table <NUM> reduces computation time to compute fast control of switching states of the MxC <NUM> and enables applying MPC algorithms with an expanded prediction horizon h (h><NUM>). This approach can learn cause and effect relationships between loads, input voltages and currents, and output voltages and currents, converge based on explicit objectives, and extended search horizons. Furthermore, this approach reduces or minimizes uncertainty and nondeterminism associated with different approaches that use models.

A mathematical model behind Q-table structure and table values is Markov decision process (MDP), by defining in each entry of the Q-table operating state, switching state, and reward (wherein reward is the inverse of cost) according to the available control variables (switching states). Even with an unknown dynamic evolution of load during operation of the industrial plant <NUM>, RL algorithm can learn an optimal control policy from training data.

A challenge arises in applying RL to industrial operating plants, since it is known that in settings involving high-dimensional or continuous state spaces, conventional RL approaches suffer from a so-called `curse of dimensionality,' that discourages their employment (see <NPL>). As a solution to this problem, the Q-table is provided with defined discrete operating states, wherein the operating states are determined by applying space vector modulation (SVM) techniques.

In an example application of Q-table <NUM> to an industrial plant <NUM>, the MxC <NUM> is controlled with inner and outer loops. The outer loop control provides references from the MxC controller to be followed. The inner control loop is the MxC controller that uses MPC with RL. Voltage at high-voltage end vABC is connected to capacitors or a voltage source and voltage at low-voltage end vabc is connected to inductors or a current source of the MxC <NUM>, wherein vABC and vabc are three phase voltages.

A switching state variable SXy represents a switching state of the matrix of switches for switching coupling between the high-voltage and the low-voltage end, wherein for a three-phase input and three-phase output, SXy has <NUM><NUM> = <NUM> switching states, but only <NUM> possible switching states are allowed to avoid damaging devices with over voltages (open circuit) or over currents (short circuit) in the plant.

For any given configuration, the space vector definition of the low-voltage side vector is as follows: <MAT> with <MAT>, which represents the <NUM>° phase displacement between the phases and Va, Vb, and Vc, the phase-to-neutral voltages of the MxC low-voltage side. While the input given configurations has <NUM> states, vabc may or may not have <NUM> distinct vectors, because different configurations might map to the same vector. For example, all three switching (<NUM>,<NUM>,<NUM>), (<NUM>,<NUM>,<NUM>), and (<NUM>,<NUM>,<NUM>), map to the same zero-vector, i.e., vabc = <NUM>. This is expected, because in those cases the phases would only carry common-mode voltage. The voltage vector possible outcome is a function of the phase of the voltage waveforms at the high-voltage side.

Possible voltage vectors and switching states generated by the MxC <NUM> change over time and can be plotted at different time instants. A low-voltage reference vector can be plotted anywhere in the operating state space. SVM with pulse width modulation (PWM) can be applied to choose a few operating states around the low-voltage reference vector to generate the reference in the inner control loop. This process would work well for a power converter in which a high voltage is fixed and not changing with time and the number of switching states is small. However, in a converter in which the high-voltage side is AC (as with the matrix converter), or having a complicated topology with a high number of switching states, classical PWM control becomes more complicated and challenging. This challenge can be addressed using MPC with RL, in accordance with embodiments of the disclosure. Once the operating conditions of the system states are determined, RL can be used by dynamically applying actions and determining reward values to the operating states, defining an MDP for which Q-learning is an algorithm to compute optimal subset of actions.

The operating states are a discrete approximation summarizing the continuous currents and voltages in the industrial plant <NUM>. The discretization is based on two continuous inputs.

With reference to <FIG> and <FIG>, <FIG> shows an example application where plots <NUM>, <NUM>, and <NUM> show determination of operating states from two continuous inputs. Plot <NUM> shows a first continuous input associated with a high-voltage ABC side 110A which is one of an input side or output side <NUM> of plant <NUM>. The first continuous input is represented as VABC, shown in plot <NUM> as phase voltages varying over angles <NUM>-<NUM> degrees, divided into multiple uniform sections <NUM>. In the example shown, the sections <NUM> are <NUM>-degree sections. At any given time instant, there are <NUM> possible sections <NUM> referred to as Tables T E {<NUM>,. ,<NUM>}, formally, T = k + <NUM> where <MAT> and k is an integer.

The second continuous input is associated with low-voltage abc side 110b, which is the other of the input side or output side <NUM> of plant <NUM>. The second continuous input is represented as vabc, having a phase and magnitude controlled by an outer loop control that uses system specification and load conditions. Plot <NUM> shows the low-voltage side reference vector at a time instant divided into multiple sectors. In the example, at any given time instant, six sectors for <MAT> are shown, each sector is <NUM> degrees. Plot <NUM> shows an example sector <NUM> at a time instant, with <MAT> divided into multiple regions <NUM>. In the example shown, <MAT>, is divided into three regions <NUM>, labeled R1, R2, and R3. At any given time instant, each sector <NUM> has at least <NUM> possible switching configurations vectors. In the sector <NUM> shown in plot <NUM>, region <NUM> R<NUM> is composed of the first <NUM> switching configurations smallest in magnitude, region <NUM> R2 is composed of <NUM> switching configurations ranked <NUM>th -<NUM>th in increasing magnitude values, and region <NUM> R3 is composed of <NUM> switching configurations ranked <NUM>th -<NUM>th in magnitude (meaning the largest <NUM> possible voltage vector). As described, some of the voltage vectors can appear in more than one sector at a given time.

Applying the example, shown in <FIG>, there are T × S × R = <NUM> discrete operating states H, N = {<NUM>,.

Turning to <FIG>, a plot <NUM> shows a vector <NUM> representing magnitude and phase of <MAT> at a given instant plotted relative to each of the <NUM> switching states <NUM>. Since the input high voltage side is AC, the switching states <NUM> in plot <NUM> are not fixed, and circumferential line <NUM> and spoke lines <NUM> represent trajectories of the switching states.

With reference to actions, RL actions are defined as the switching states of the MxC. A switching state represents an On-Off state of switches of the MxC at any time instant, which can be chosen to generate the reference voltage vector <MAT>. In the MxC control example there are <NUM> possible switching states A, A = {<NUM>,. Accordingly, each reward value in Q-table corresponds to one action a of the actions A and one discrete state s of the finite set H.

In RL, an immediate reward value is a measure for the quality of an action given a state. At each time instant, a reward value is determined based on the environment of the industrial plant. An MxC controller's (e.g., MxC controller <NUM> shown in <FIG>) objective is to maximize the expected total reward value it receives over the long run, which can be expressed mathematically as follow: maximize <MAT>, where <IMG> denotes an expectation, reward value rk is a quantitative measure that defines a good and bad switching state for the matrix converter <NUM> to meet control objectives, and λ. (the discount factor) is a number between <NUM> and <NUM> (<NUM> ≤ λ ≤ <NUM>). λ has the effect of valuing rewards received earlier higher than those received later.

The algorithm, represented as Q-table (e.g., Q-table <NUM> shown in <FIG>) calculates the quality of a state-action combination: <MAT> where H is the discrete operation state of the converter, A is the switching states of the converter, and <IMG> is the set of real numbers.

Before learning begins, values in the Q-table are initialized to possibly arbitrary values. Then, at each time t, an action ak is selected by selecting a switching state from the <NUM> available switching states, the reward value rk is computed from the control objectives, a new state st+<NUM> is entered (depending on both the previous state st and the selected action ak), and Q is updated. The core of the algorithm is a simple value iteration update, using the weighted average of the old value and the new information using Equation (<NUM>): <MAT> where.

<FIG> and <FIG> show exemplary and non-limiting flowcharts illustrating methods for controlling an MxC of an industrial plant in accordance with certain illustrated embodiments. The methods can be performed by an MxC controller, such as MxC controller <NUM>. Before turning to description of <FIG> and <FIG>, it is noted that the flowchart in <FIG> and <FIG> show an example in which operational steps are carried out in a particular order, as indicated by the lines connecting the blocks, but the various steps shown in this diagram can be performed in a different order, or in a different combination or sub-combination. It should be appreciated that in some embodiments some of the steps described below may be combined into a single step. In some embodiments, one or more additional steps may be included. In some embodiments, one or more of the steps can be omitted.

With reference to <FIG>, the figure shows a method of controlling the MxC during operation of the industrial plant once the Q-table has been trained. At block <NUM>, an operating condition of the industrial plant is received, as described in greater detail below. The operating condition can include multiple conditions, such as a multi-phase high-voltage side (HVS) voltage/current signal and a multi-phase low-voltage side reference voltage (LVSR) signal. At block <NUM>, a discrete operating state is determined that corresponds to the operating condition, such as by applying space vector modulation (SVM).

At block <NUM>, the trained Q-table is consulted for reward values associated with respective switching states for an operating state that corresponds to the operating condition. At block <NUM>, a subset of the switching states is selected as a function of a result of 416ed reward values associated with the switching states of the operating state.

At block <NUM>, each switching state included in the subset is evaluated. Evaluating the switching states included in the subset can include determining a reward value for the operating condition based on one or more objectives. Each switching state of the selected subset is evaluated as a function of the computed reward value and the reward value in the Q-table mapped to the switching state of the operating condition. At block <NUM>, an optimal switching state is selected for the operating condition based on a result of evaluating the switching states of the subset.

<FIG> shows an example method of training the Q-table using Q-learning. The Q-table is trained either in a real environment of the industrial plant system or offline with a simulation software using Q-learning to map, till convergence, a reward value predicted for respective switching states of the plurality of switching states to respective discrete operating states of a plurality of operating states. The reward values that are mapped to the switching states are a predicted future reward value based on at least one objective. In one or more embodiments, the reward values are based on multiple control objectives.

At block <NUM>, time is initialized (t=<NUM>), the Q-table is initialized, and the MxC controller is operated. The Q-table provides an entry for each switching state at each operating condition. Each entry can be initialized with an initial reward value (e.g., the initial reward can be set to be equal to <NUM> for all entries in Q-table). The MxC controller is operated with an industrial plant (such as industrial plant <NUM> shown in <FIG> and <FIG>) in actual environmental conditions over time at regular time intervals t. Operation in the actual environmental conditions can include, for example, applying electrical signals by the generator to an MxC (such as MxC <NUM> shown in <FIG>) as controlled by the MxC controller, wherein a load operates by using a signal output by the MxC. This can be performed online using the real MxC system hardware or offline using a simulation software (e.g., MATLAB/SIMULINK). At block <NUM>, a current operating state is determined by determining a discrete state that corresponds to continuous operating conditions of the industrial plant. At block <NUM>, a discrete operating state is determined that corresponds to the operating condition. At block <NUM>, the current reward is determined for each switching state at the operating state. The current reward can be a function of one or more control objectives. At block <NUM>, the entries of the Q-table for the switching states associated with the operating condition are updated with a reward, using a weighted average of old and current rewards determined at block <NUM>. At block <NUM>, an optimal switching state is determined for the operating state using standard control without RL.

At block <NUM>, a determination is made whether convergence of the Q-table is achieved. The convergence is determined if changes to values in the Q-table are within acceptable margins (i.e., Bellman error is below a predefined convergence threshold). If convergence is determined at block <NUM> to be achieved, then at block <NUM>, the reward values associated with the switching states for each of the operating states are sorted and the method ends, meaning the Q-table is trained and ready to be applied during operation, such as in the method shown in <FIG>. If convergence is determined at block <NUM> to be incomplete, the method continues at block <NUM> by advancing to the next time interval t=t+<NUM>, after which the method continues at block <NUM>.

Aspects of the present disclosure are described above with reference to block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. Features of the methods described include operations, such as equations, transformations, conversions, etc., that can be performed using software, hardware, and/or firmware. Regarding software implementations, it will be understood that individual blocks of the block diagram illustrations and combinations of blocks in the block diagram illustrations, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the block diagram block or blocks.

With reference to <FIG>, a block diagram of an example computing system <NUM> is shown, which provides an example configuration of the MxC controller <NUM> or one or more portions of the MxC controller <NUM>. Computing system <NUM> is only one example of a suitable system and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the disclosure described herein. Computing system <NUM> can be implemented using hardware, software, and/or firmware. Regardless, computing system <NUM> is capable of being implemented and/or performing functionality as set forth in the disclosure.

Computing system <NUM> is shown in the form of a general-purpose computing device. Computing system <NUM> includes a processing device <NUM>, memory <NUM>, an input/output (I/O) interface (I/F) <NUM> that can communicate with an internal component <NUM>, and optionally an external component <NUM>.

The processing device <NUM> can include, for example, a programmable logic device (PLOD), microprocessor, digital signal processor (DSP), a microcontroller, a field programmable gate array (FPGA), an application specific integrated circuit (ASCI), and/or other discrete or integrated logic circuitry having similar processing capabilities.

The processing device <NUM> and the memory <NUM> can be included in components provided in the FPGA, ASCI, microcontroller, or microprocessor, for example. Memory <NUM> can include, for example, volatile and non-volatile memory for storing data temporarily or long term, and for storing programmable instructions executable by the processing device <NUM>. I/O I/F <NUM> can include an interface and/or conductors to couple to the one or more internal components <NUM> and/or external components <NUM>.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flow diagram and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational operations to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the block diagram block or blocks.

Embodiments of the MxX controller <NUM> may be implemented or executed by one or more computer systems, such as a microprocessor. Each computer system <NUM> can implement controller <NUM>, or multiple instances thereof. In various embodiments, computer system <NUM> may include one or more of a microprocessor, an FPGA, application specific integrated circuit (ASCI), microcontroller. The computer system <NUM> can be provided as an embedded device. All or portions of the computer system <NUM> can be provided externally, such by way of a mobile computing device, a smart phone, a desktop computer, a laptop, or the like.

Computer system <NUM> is only one example of a suitable system and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the disclosure described herein. Regardless, computer system <NUM> is capable of being implemented and/or performing any of the functionality set forth hereinabove.

Computer system <NUM> may be described in the general context of computer systemexecutable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types.

While the invention has been described with reference to an exemplary embodiment(s), it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment(s) disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the illustrated embodiments, exemplary methods and materials are now described. All publications mentioned herein disclose and describe the methods and/or materials in connection with which the publications are cited.

It must be noted that as used herein and in the appended claims, the singular forms "a", "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a stimulus" includes a plurality of such stimuli and reference to "the signal" includes reference to one or more signals and equivalents thereof known to those skilled in the art, and so forth.

Claim 1:
A matrix converter system of an industrial plant system (<NUM>) comprising:
a matrix converter having a switching matrix coupled between an input side and an output side; and
a model predictive controller, MPC, configured to select a switching state of the switching matrix from a plurality of switching states, the MPC configured to:
receive an operating condition of the industrial plant system;
determine a discrete operating state of a plurality of discrete operating states that corresponds to the operating condition by applying space vector modulation, SVM;
consult a Q-data structure to access reward values that are associated with respective switching states of the switching matrix for the discrete operating state that corresponds to the operating condition, wherein the Q-data structure has been trained in a real or simulation environment of the industrial plant system using Q-learning to map until convergence reward values predicted for the respective switching states of the plurality of switching states and respective discrete operating states of the plurality of discrete operating states and generate a set of mappings;
sort the reward values predicted for the respective switching states for the discrete operating state that corresponds to the operating condition;
select a subset of the set of the mappings as a function of a result of sorting the reward values associated with the switching states for the discrete operating state;
evaluate each switching state included in the subset as a function of the reward value for that switching state; and
select an optimal switching state for the operating condition based on a result of evaluating the switching states of the subset;
wherein the operating condition includes a multi-phase high-voltage side, HVS, voltage signal and a multi-phase low-voltage side reference voltage, LVSR signal, and application of the SVM causes the MPC to:
divide the HVS voltage signal at any time instant into M even phase segments over a full cycle;
divide phase of the LVSR voltage signal into N even sectors; and
divide magnitude of each sector into P regions, wherein the plurality of discrete operating states includes MxNxP states.