Patent Publication Number: US-2022230243-A1

Title: Method and apparatus for renewable energy allocation based on reinforcement learning

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application claims the priority benefit of Taiwan application serial no. 110102148, filed on Jan. 20, 2021. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification. 
     BACKGROUND 
     Technical Field 
     The disclosure relates to a method and an apparatus of reinforcement learning, particularly to a method and an apparatus for renewable-energy allocation based on reinforcement learning. 
     Description of Related Art 
     In recent years, to reduce our dependence on fossil fuels and to response to the rise of environmental awareness, the renewable energy of the total generation of power has increased. Since the amount of electricity generated by renewable energy changes based on the environment, this uncertainty, such as intermittence, of renewable energy result in sharp fluctuations in market price. Currently, there is a liberalized electricity trade structure, in which energy aggregators coordinate the allocation of power supply and its price between energy consumers and energy generation industries. The energy aggregator, who is on behalf of the energy consumers, negotiates with the energy generation industries to coordinate the supply-demand allocation of renewable energy between the energy supplier and the energy demander. 
     Conventionally, game theory, genetic algorithm, or fuzzy logic is usually adopted to allocate the supply and demand of renewable energy. However, these algorithms become less suitable in today&#39;s environment due to their design. To solve this problem, in recent years, mainstream researchers come to adopt reinforcement learning algorithms to determine the pricing strategy. Still, this method still spotlights fossil fuels and does not take into account the uncertainty, such as intermittence, of renewable energy. In addition, energy aggregators of today often use the attached energy storage system (ESS) to mitigate to a certain extent the energy loss and the price loss caused by overbought or oversold. However, this method needs to take additional consideration of the charge and discharge controls of the ESS (where the upper and lower bounds are not fixed values), which increases the overall complexity of the system. 
     SUMMARY 
     The disclosure provides a method and an apparatus for renewable-energy allocation based on reinforcement learning, which are adapted to reduce the intermittent uncertainty of renewable energy and accelerate the learning speed of reinforcement learning. 
     The disclosure provides a method for renewable-energy allocation based on reinforcement learning, and the method is adapted for an energy aggregator having an energy storage system (ESS) to coordinate renewable-energy allocation between multiple energy suppliers and multiple energy demanders. This method includes the following steps: collecting historical power-generation data of each energy supplier to generate a renewable-energy index representing uncertainty of renewable energy; collecting at least one market index related to renewable-energy market dynamics and integrating the market index with the renewable-energy index and electricity information of the ESS into multiple states of a Markov decision process; inputting these states to a reinforcement learning model, so as to determine a price bid for each of the energy suppliers and the energy demanders through computation of the reinforcement learning model; and according to a supply amount and a demand amount proposed by the energy suppliers and the energy demanders in response to the price bids, adjusting the ESS to coordinate the supply and demand of the renewable energy between the energy suppliers and the energy demanders, and updating the reinforcement learning model. 
     The disclosure provides an apparatus for renewable-energy allocation based on reinforcement learning, and the apparatus includes an ESS, a connection device, a storage device, and a processor. The ESS is configured to store renewable energy. The connection device is configured to connect an energy supplier and an energy demander with each other. The storage device is configured to store computer programs. The processor is coupled to the ESS, the connection device, and the storage device, and is configured to load and execute the computer programs so as to: collect historical power-generation data of each energy supplier to generate a renewable-energy index representing uncertainty of renewable energy; collect at least one market index related to renewable-energy market dynamics and integrate the market index with the renewable-energy index and the electricity information of ESS into multiple states of a Markov decision process; input these states to the reinforcement learning model to determine price for the energy supplier and the energy demander through computation of the reinforcement learning model; and according to a supply amount and a demand amount proposed by the energy supplier and the energy demander in response to the price, adjust the ESS to coordinate the supply and demand of renewable energy between the energy supplier and the energy demander, and update the reinforcement learning model. 
     Based on the above, the apparatus and the method for renewable-energy allocation based on reinforcement learning of the present disclosure process and group the historical power-generation data of renewable energy, and use the indexes obtained through processing for the state design of the reinforcement learning algorithm, thereby integrating the reinforcement learning model and the charging-and-discharging strategy of the ESS to accelerate the learning speed. 
     In order to make the above features of the present disclosure more comprehensible, the following embodiments are described in detail with the drawings attached. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a structural diagram of a renewable-energy zonal grid according to an embodiment of the present disclosure. 
         FIG. 2  is a block diagram of an apparatus for renewable-energy allocation based on reinforcement learning according to an embodiment of the present disclosure. 
         FIG. 3  is a flowchart of a method for renewable-energy allocation based on reinforcement learning according to an embodiment of the present disclosure. 
         FIG. 4  is a schematic diagram of a method for renewable-energy allocation based on reinforcement learning according to an embodiment of the present disclosure. 
         FIG. 5  is a flowchart of a method for generating a renewable-energy index based on reinforcement learning according to an embodiment of the present disclosure. 
     
    
    
     DESCRIPTION OF THE EMBODIMENTS 
     The embodiments of the present disclosure extract features of power-generation data of renewable energy within a certain period of time by using a statistical method, and obtain the degree of uncertainty of the current acquired energy by grouping the feature data and assigning weights to each group. In addition, the embodiments of the present disclosure control the charging and discharging of an energy storage system (ESS) by designing specific rules, and coordinate the purchase and sale of the renewable energy. Finally, the two methods above are integrated into a reinforcement learning algorithm, such as deep deterministic policy gradient, to improve the learning speed of the reinforcement learning algorithm. 
       FIG. 1  is a structural diagram of a renewable-energy zonal grid according to an embodiment of the present disclosure. In  FIG. 1 , there are multiple energy suppliers, which are also known as producers, in a zonal grid  1  of renewable energy. These may be, for example, citizen power plants, household solar generators, and small hydropower plants that supply renewable energy. There are also multiple competing energy aggregators in the zonal grid  1 , which, for example, act as energy aggregators that announce electricity prices regularly to attract the producers to sell the renewable energy. There are also multiple energy demanders in the zonal grid  1 , which are also called consumers. They are, for example, household consumers or factories, who propose their demand of electricity to the energy aggregators. And the energy aggregators, for example, evaluate the uncertainty of the renewable energy provided by the producers, and integrate them with the current electricity stored in their own ESSs and other market features into multiple states in a Markov decision process, which are input to a reinforcement learning model, and through computation of the neural network and other learning model, the price bids for the producers and the consumers are determined, thereby optimizing the interests of both parties as well as the aggregators themselves. 
       FIG. 2  is a block diagram of an apparatus for renewable-energy allocation based on reinforcement learning according to an embodiment of the present disclosure. Please refer to  FIG. 1  and  FIG. 2  at the same time. An apparatus for renewable-energy allocation  20  of the embodiment of the present disclosure, for example, is adapted as the energy aggregator in  FIG. 1 , and may coordinate the supply-demand allocation of the renewable energy between the energy suppliers and the energy demanders. In some embodiments, the apparatus for renewable-energy allocation  20  is, for example, a computer apparatus, such as a file server, a database server, an application server, a workstation, or a personal computer with computing capabilities, which includes an ESS  22 , a connection device  24 , a storage device  26 , and a processor  28 . The functions of these components are described as follows. 
     The ESS  22  includes, for example, a high-efficiency energy storage battery, such as lithium-iron batteries, all-iron flow batteries, solid-state lithium-ion batteries, and zinc-nickel batteries, which, for example, combines the battery management system (BMS), temperature control system, background management system, and other technologies, to play the role of reconciliation in the renewable energy grid to provide real-time power buffering, absorption, and compensation, thereby improving the power quality of the grid, increasing the power reserve capacity, and/or stabilizing the intermittent output of the renewable energy, so that the supply and load of the grid maintains a stable balance. 
     The connection device  24  is, for example, any wired or wireless interface device capable of connecting the energy suppliers and the energy demanders. For wired methods, the connection device  24  may be a universal serial bus (USB), a RS232, a universal asynchronous receiver/transmitter (UART), an internal integrated circuit (I2C), serial peripheral interface (SPI), display port, thunderbolt, or local area network (LAN) interface, but the disclosure is not limited thereto. For wireless methods, the connection device  24  may support wireless fidelity (Wi-Fi), RFID, Bluetooth, infrared, near-field communication (NFC), device-to-device (D2D) communication, and/or other communication protocol devices, and the disclosure is not limited thereto. In some embodiments, the connection device  24  may also include a network interface card that supports Ethernet or wireless network standards, such as 802.11g, 802.11n, 802.11ac, etc., so that the apparatus for renewable-energy allocation  20  can be connected to the energy suppliers and the energy demanders through the Internet, so as to upload the price bids and receive the supply amount and the demand amount proposed by the energy suppliers and the energy demanders in response to the price bids. 
     The storage device  26  is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, similar components, or a combination thereof, and is configured to store computer programs executable by the processor  28 . 
     The processor  28  is, for example, a central processing unit (CPU), other programmable general-purpose or special-purpose microprocessor, microcontroller, digital signal processor (DSP), programmable controller, application specific integrated circuit (ASIC), programmable logic device (LD), other similar devices, or a combination thereof, and the present disclosure is not limited thereto. In this embodiment, the processor  28  loads a computer program from the storage device  26  to execute the method for renewable-energy allocation based on reinforcement learning of the embodiment of the present disclosure. 
     The method for renewable-energy allocation of the embodiment of the present disclosure adopts, for example, a reinforcement learning algorithm. In this algorithm, an action of the energy aggregator may be defined as a price bid p k,t   P  of the energy suppliers and a price bid p k,t   C  of the energy demanders, which is shown as follows: 
     
       
         
           
             
               a 
               
                 k 
                 , 
                 t 
               
             
             = 
             
               [ 
               
                 
                   
                     
                       p 
                       
                         k 
                         , 
                         t 
                       
                       P 
                     
                   
                 
                 
                   
                     
                       p 
                       
                         k 
                         , 
                         t 
                       
                       C 
                     
                   
                 
               
               ] 
             
           
         
       
     
     Among the above formula, p min ≤p k,t   P ≤p max , and p min ≤p k,t   C ≤p max , where p min  and p max  are the lower limit and upper limit of the price bids, and its value may be defined suitably based on the market environment, to which the present disclosure is not limited. 
     The states observed by the energy aggregator may include but is not limited to: the electricity information of the ESS  22 , renewable-energy index, and market indexes, and the states are defined as follows: 
     S k,t =[ζ t−1 , ζ t , ξ k,t−1 , ξ k,t , p k,t−1   C , p k,t−1   P , e k,t−1 , p k,t−1   C     oppo   , p k,t−1   P     oppo   , H k,t−1   ν     high   , H k,t−1   ν     low   ] 
     Among the above formula, ζ t  is the market rationality index for determining whether a market is rational. In some embodiments, the processor  28 , for example, determines whether a minimum value of a price bid p k,t−1   C  for the energy demanders is greater than or equal to a maximum value of a price bid p k,t−1   P  for the energy supplier, and determines the market rationality index ζ t  according to the determination result, and the definition of the market rationality index ζ t  is as follows: 
     
       
         
           
             
               ζ 
               t 
             
             = 
             
               { 
               
                 
                   
                     
                       1 
                       , 
                     
                   
                   
                     
                       
                         if 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             min 
                             
                               ∀ 
                               
                                 k 
                                 ∈ 
                                 K 
                               
                             
                           
                           ⁢ 
                           
                             p 
                             
                               k 
                               , 
                               
                                 t 
                                 - 
                                 1 
                               
                             
                             C 
                           
                         
                       
                       ≥ 
                       
                         
                           max 
                           
                             ∀ 
                             
                               k 
                               ∈ 
                               K 
                             
                           
                         
                         ⁢ 
                         
                           p 
                           
                             k 
                             , 
                             
                               t 
                               - 
                               1 
                             
                           
                           P 
                         
                       
                     
                   
                 
                 
                   
                     
                       0 
                       , 
                     
                   
                   
                     
                       
                         if 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             min 
                             
                               ∀ 
                               
                                 k 
                                 ∈ 
                                 K 
                               
                             
                           
                           ⁢ 
                           
                             p 
                             
                               k 
                               , 
                               
                                 t 
                                 - 
                                 1 
                               
                             
                             C 
                           
                         
                       
                       &lt; 
                       
                         
                           max 
                           
                             ∀ 
                             
                               k 
                               ∈ 
                               K 
                             
                           
                         
                         ⁢ 
                         
                           p 
                           
                             k 
                             , 
                             
                               t 
                               - 
                               1 
                             
                           
                           P 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     In addition, ξ k,t  is a supply-demand balance index of the energy aggregator itself. In some embodiments, the processor  28 , for example, determines whether the total supply amount Ψ k,t−1   P  of renewable energy supplied by the energy suppliers is greater than the total demand Ψ k,t−1   C  of renewable energy demanded by the energy demander, so as to determine the supply-demand balance index ξ k,t  according to the determination result, and the definition of the supply-demand balance index ξ k,t  is as follows: 
     
       
         
           
             
               ξ 
               
                 k 
                 , 
                 t 
               
             
             = 
             
               { 
               
                 
                   
                     
                       1 
                       , 
                     
                   
                   
                     
                       
                         if 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             ψ 
                             
                               k 
                               , 
                               
                                 t 
                                 - 
                                 1 
                               
                             
                             P 
                           
                           
                             ψ 
                             
                               k 
                               , 
                               
                                 t 
                                 - 
                                 1 
                               
                             
                             C 
                           
                         
                       
                       &gt; 
                       1 
                     
                   
                 
                 
                   
                     
                       0 
                       , 
                     
                   
                   
                     
                       
                         if 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             ψ 
                             
                               k 
                               , 
                               
                                 t 
                                 - 
                                 1 
                               
                             
                             P 
                           
                           
                             ψ 
                             
                               k 
                               , 
                               
                                 t 
                                 - 
                                 1 
                               
                             
                             C 
                           
                         
                       
                       = 
                       1 
                     
                   
                 
                 
                   
                     
                       
                         - 
                         1 
                       
                       , 
                     
                   
                   
                     
                       
                         if 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             ψ 
                             
                               k 
                               , 
                               
                                 t 
                                 - 
                                 1 
                               
                             
                             P 
                           
                           
                             ψ 
                             
                               k 
                               , 
                               
                                 t 
                                 - 
                                 1 
                               
                             
                             C 
                           
                         
                       
                       &lt; 
                       1 
                     
                   
                 
               
             
           
         
       
     
     The above state also includes the price bids p k,t−1   C     oppo    and p k,t−1   P     oppo    of other competing aggregators for the energy demanders and the energy suppliers, which are defined as follows: 
     p k,t−1   C     oppo   =[p 1,t   C , . . . , p k−1   C , P k+1,t   C , . . . , p K,t   C ]
 
p k,t−1   P     oppo   =[p 1,t   P , . . . , p k−1   P , P k+1,t   P , . . . , p K,t   P ]
 
     In addition, the above state may also include other elements, such as the current electricity e k,t−1  of the ESS  22 , and the renewable-energy indexes H k,t−1   ν     high    and H k,t−1   ν     low   . Their definitions and computation method are described in detail later. 
       FIG. 3  is a flowchart of a method for renewable-energy allocation based on reinforcement learning according to an embodiment of the present disclosure.  FIG. 4  is a schematic diagram of a method for renewable-energy allocation based on reinforcement learning according to an embodiment of the present disclosure. Please refer to  FIG. 2 ,  FIG. 3 , and  FIG. 4  at the same time. The method of this embodiment is applicable to the apparatus for renewable-energy allocation  20  mentioned above, and may be divided into stages of data collection, state integration, and model update. The following description recounts in detail the steps of the method for renewable-energy allocation of this embodiment with various components of the apparatus for renewable-energy allocation  20 . 
     In the stage of data collection, the apparatus for renewable-energy allocation  20  serves as a target aggregator to observe, for example, its own state, market state, and competitors&#39; price bids to obtain market indexes of the renewable-energy market. 
     Specifically, in step S 32 , the processor  28 , for example, connects each of the energy suppliers by the connection device  22  so as to collect historical power-generation data, and use the same to generate a renewable-energy index representing uncertainty of renewable energy (for example, in  FIG. 4 , data collected from a market  40  is used to generate renewable-energy index  202 ). 
     In some embodiments, the processor  28  adopts the historical power-generation data to construct a feature matrix and executes a clustering algorithm to divide multiple elements in the feature matrix into multiple groups, and calculates the distance between the center of each group and the zero point, thereby generating the renewable-energy index. In some embodiments, the processor  28  further counts the number of the energy suppliers that select the energy aggregator in each group, and multiplies the number of each group by a corresponding weight to generate the renewable-energy index. 
     Specifically,  FIG. 5  is a flowchart of a method for generating a renewable-energy index based on reinforcement learning according to an embodiment of the present disclosure. Please refer to  FIG. 2  and  FIG. 5  at the same time. This embodiment illustrates the process of step S 32  of  FIG. 3  in detail. 
     In step S 321 , the processor  28 , for example, collects the historical power-generation data E t   P  of all energy suppliers within a certain period of time, and constructs feat a feature matrix E t   P,feat  with the historical power-generation data, and they have the following definitions: 
     
       
         
           
             
               E 
               t 
               P 
             
             = 
             
               
                 [ 
                 
                   
                     
                       
                         
                           e 
                           → 
                         
                         
                           1 
                           , 
                           t 
                         
                         P 
                       
                     
                   
                   
                     
                       
                         
                           e 
                           → 
                         
                         
                           2 
                           , 
                           t 
                         
                         P 
                       
                     
                   
                   
                     
                       ⋮ 
                     
                   
                   
                     
                       
                         
                           e 
                           → 
                         
                         
                           M 
                           , 
                           t 
                         
                         P 
                       
                     
                   
                 
                 ] 
               
               = 
               
                 [ 
                 
                   
                     
                       
                         e 
                         
                           1 
                           , 
                           
                             t 
                             - 
                             1 
                           
                         
                         P 
                       
                     
                     
                       
                         e 
                         
                           1 
                           , 
                           
                             t 
                             - 
                             2 
                           
                         
                         P 
                       
                     
                     
                       … 
                     
                     
                       
                         e 
                         
                           1 
                           , 
                           
                             t 
                             - 
                             
                               T 
                               h 
                             
                           
                         
                         P 
                       
                     
                   
                   
                     
                       
                         e 
                         
                           2 
                           , 
                           
                             t 
                             - 
                             1 
                           
                         
                         P 
                       
                     
                     
                       
                         e 
                         
                           2 
                           , 
                           
                             t 
                             - 
                             2 
                           
                         
                         P 
                       
                     
                     
                       … 
                     
                     
                       
                         e 
                         
                           2 
                           , 
                           
                             t 
                             - 
                             
                               T 
                               h 
                             
                           
                         
                         P 
                       
                     
                   
                   
                     
                       ⋮ 
                     
                     
                       ⋮ 
                     
                     
                       ⋱ 
                     
                     
                       ⋮ 
                     
                   
                   
                     
                       
                         e 
                         
                           M 
                           , 
                           
                             t 
                             - 
                             1 
                           
                         
                         P 
                       
                     
                     
                       
                         e 
                         
                           M 
                           , 
                           
                             t 
                             - 
                             2 
                           
                         
                         P 
                       
                     
                     
                       … 
                     
                     
                       
                         e 
                         
                           M 
                           , 
                           
                             t 
                             - 
                             
                               T 
                               h 
                             
                           
                         
                         P 
                       
                     
                   
                 
                 ] 
               
             
           
         
       
       
         
           
             
               E 
               t 
               
                 P 
                 , 
                 fear 
               
             
             = 
             
               [ 
               
                 
                   
                     
                       
                         e 
                         → 
                       
                       
                         1 
                         , 
                         t 
                       
                       
                         P 
                         , 
                         feat 
                       
                     
                   
                 
                 
                   
                     
                       
                         e 
                         → 
                       
                       
                         2 
                         , 
                         t 
                       
                       
                         P 
                         , 
                         feat 
                       
                     
                   
                 
                 
                   
                     ⋮ 
                   
                 
                 
                   
                     
                       
                         e 
                         → 
                       
                       
                         M 
                         , 
                         t 
                       
                       
                         P 
                         , 
                         fear 
                       
                     
                   
                 
               
               ] 
             
           
         
       
     
     Among the above formulas, t is the current moment, T h  is the duration of the historical data, and M is the total number of the energy suppliers. 
     For this feature matrix E t   P , in step S 322 , the processor  28  executes a clustering algorithm (a clustering method), for example, to divide the elements in the feature matrix into multiple groups. For example, the processor  28  may adopt, but not limited to, the k-means algorithm to divide the elements in the feature matrix into 4 groups, as shown below: 
     G t   1 , G t   2 , . . . , G t   4 ←CLUSTER(E t   P,feat ) 
     Among the above formula, G t   n  is the n-th group at the current moment. 
     In step S 323 , the processor  28  labels for each group the level of intermittent uncertainty of the total amount of its recent power generation, for example, by calculating the distances between the centers of all groups and the zero point, as shown below: G t   I , G t   II , . . . , G t   IV ←LABEL (G 1   1 , G t   2 , . . . , G t   4 ) 
     In some embodiments, the processor 28 directly adopts the aforementioned uncertainties G t   I , G t   II , G t   III , and G t   IV  as the renewable-energy index for the reinforcement learning algorithm. 
     In step S 324  of this embodiment, the processor  28 , for example, counts the number of the energy suppliers that select the energy aggregator in each group at the moment, and multiplies in step S 325  the number of each group by different weights, thereby using the obtained indexes as the renewable-energy indexes. In this way, the dimensionality occupied by the renewable-energy index in the defined state is reduced, thereby reducing the amount of computation for the subsequent execution of the reinforcement learning algorithm. The computation formulas for the above renewable-energy indexes H t   1  and H t   2  are as follows: 
     H t   1 ←α*num(G t   I )+(1−α)*num(G t   II )
 
H t   2 ←β*num(G t   III )+(1−β)*num(G t   IV )
 
     Among the above formulas, 0≤α≤1 and 0≤β≤1. 
     Then, it is moved on to the stage of state integration. In this stage, the apparatus for renewable-energy allocation  20  serves as the target aggregator, where the processor  28  integrates the observed market indexes mentioned above, the calculated renewable-energy index, and the electricity information of the ESS  22  itself into multiple states of the Markov decision process, and inputs the states to the reinforcement learning model, so as to determine price bids p k,t   P  and p k,t   C  for the energy suppliers and the energy demanders through computation of the reinforcement learning model. 
     And in step S 34  of  FIG. 4 , for example, the apparatus for renewable-energy allocation  20  may integrate, for example, in the state integration  204  the electricity information output by an ESS control strategy  201 , the calculated renewable-energy index  202 , and the observed market index  203  into a state S of the Markov decision process, and then in step S 36 , inputs the state S to an reinforcement learning model  205 , which adopts an arbitrary neural network algorithm, such as the deep deterministic policy gradient algorithm, to calculate and output the price bids p k,t   P  and p k,t   C . The price bids p k,t   P  and p k,t   C  are provided as an action a for the energy suppliers and the energy demanders on the market  40 . The energy suppliers and the energy demanders respond to the price bids and propose the supply amount of renewable energy they can provide or the demand amount for renewable energy they need. 
     Finally, it enters the stage of model update. In this stage, the apparatus for renewable-energy allocation  20  that serves as the target aggregator learns the selection (that is, the supply amount and demand amount aforementioned) of the energy suppliers and the energy demanders, and thereafter obtain the final supply amount and demand amount through, for example, the control of the ESS  22 . After adjusting the ESS  22 , the apparatus for renewable-energy allocation  20  learns the reward of the price bids and obtains a new market observation to update the parameters of the reinforcement learning model (for example, in  FIG. 4 , the ESS control strategy  201  outputs the reward r to the reinforcement learning model  205  to update the parameters of the reinforcement learning model  205 ). 
     Specifically, in step S 38 , the processor  28  adjusts the ESS  22  to coordinate the supply and demand of renewable energy between the energy suppliers and the energy demanders according to the supply amount and the demand amount proposed by the energy suppliers and the energy demanders in response to the price bids, and updates the reinforcement learning model. 
     In some embodiments, the processor  28  determines whether the total supply amount of renewable energy supplied by the energy supplier is greater than or equal to the total demand of renewable energy demanded by the energy demander. Among the above, when the total supply amount is greater than or equal to the total demand amount, the processor  28  selects the minimum value from the difference between the total supply amount and the total demand amount and the difference between the maximum energy storage capacity and the currently stored electricity of the ESS  22 , so as to determine the supply amount of the energy suppliers and the demand amount of the energy demanders; when the total supply amount is less than the total demand amount, the minimum value is selected from the difference between the total demand amount and the total supply amount and the difference between the currently stored electricity and the minimum energy storage capacity of the ESS  22 , so as to determine the supply amount of the energy suppliers and the demand amount of the energy demanders. Their formulas are described as follows. 
     When the total supply amount Ψ k,t   P  is greater than or equal to the total demand amount Ψ k,t   C , the control u k,t  of the ESS  22  is as follows: 
         u   k,t =min(Ψ k,t   P −Ψ k,t   C   , e   max   −e   k,t )
 
     When the total supply Ψ k,t   P  is less than the total demand Ψ k,t   C , the control u k,t  of the ESS  22  is as follows: 
         u   k,t =min(Ψ k,t   C   −Ψ   k,t   P   , e   k,t   −e   min )
 
     Among the above formulas, e max  and e min  are respectively the maximum energy storage capacity and minimum energy storage capacity of the ESS  22 . 
     In some embodiments, the processor  28  calculates the reward for the determined price bids according to the adjustment made to the ESS  22 , and then collects and integrates the market indexes of the energy market in response to the price bids with the renewable-energy index and the electricity information of the ESS  22  into the states, and updates the reinforcement learning model using the states and the reward. 
     The processor  28 , for example, takes a profit obtained by deducting a loss produced by adopting the price bids from an income obtained by adopting the price bids as a reward r, and its formula is as follows: 
     
       
         
           
             
               r 
               
                 k 
                 , 
                 
                   t 
                   + 
                   1 
                 
               
             
             = 
             
               { 
               
                 
                   
                     
                       
                         
                           p 
                           
                             k 
                             , 
                             t 
                           
                           C 
                         
                         ⁡ 
                         
                           ( 
                           
                             
                               ψ 
                               
                                 k 
                                 , 
                                 t 
                               
                               C 
                             
                             + 
                             
                               u 
                               
                                 k 
                                 , 
                                 t 
                               
                             
                           
                           ) 
                         
                       
                       - 
                       
                         
                           p 
                           
                             k 
                             , 
                             t 
                           
                           P 
                         
                         ⁢ 
                         
                           ψ 
                           
                             k 
                             , 
                             t 
                           
                           P 
                         
                       
                       - 
                       
                         Φ 
                         
                           k 
                           , 
                           t 
                         
                       
                     
                   
                   
                     
                       
                         
                           if 
                           ⁢ 
                           
                             
                                 
                             
                             ⁢ 
                             
                                 
                             
                           
                           ⁢ 
                           
                             ψ 
                             
                               k 
                               , 
                               t 
                             
                             C 
                           
                         
                         ≥ 
                         
                           ψ 
                           
                             k 
                             , 
                             t 
                           
                           P 
                         
                       
                       ⁢ 
                       
                           
                       
                     
                   
                 
                 
                   
                     
                       
                         
                           p 
                           
                             k 
                             , 
                             t 
                           
                           C 
                         
                         ⁢ 
                         
                           ψ 
                           
                             k 
                             , 
                             t 
                           
                           C 
                         
                       
                       - 
                       
                         
                           p 
                           
                             k 
                             , 
                             t 
                           
                           P 
                         
                         ⁢ 
                         
                           ψ 
                           
                             k 
                             , 
                             t 
                           
                           P 
                         
                       
                       - 
                       
                         Φ 
                         
                           k 
                           , 
                           t 
                         
                       
                     
                   
                   
                     
                       
                         if 
                         ⁢ 
                         
                           
                               
                           
                           ⁢ 
                           
                               
                           
                         
                         ⁢ 
                         
                           ψ 
                           
                             k 
                             , 
                             t 
                           
                           C 
                         
                       
                       &lt; 
                       
                         ψ 
                         
                           k 
                           , 
                           t 
                         
                         P 
                       
                     
                   
                 
               
             
           
         
       
     
     Among the above formula, a loss function Ψ is follows: 
     
       
         
           
             
               Φ 
               
                 k 
                 , 
                 t 
               
             
             = 
             
               { 
               
                 
                   
                     
                       
                         ( 
                         
                           
                             ψ 
                             
                               k 
                               , 
                               t 
                             
                             C 
                           
                           - 
                           
                             ψ 
                             
                               k 
                               , 
                               t 
                             
                             P 
                           
                           - 
                           
                             u 
                             
                               k 
                               , 
                               t 
                             
                           
                         
                         ) 
                       
                       ⁢ 
                       
                         ϕ 
                         - 
                       
                     
                   
                   
                     
                       
                         
                           if 
                           ⁢ 
                           
                             
                                 
                             
                             ⁢ 
                             
                                 
                             
                           
                           ⁢ 
                           
                             ψ 
                             
                               k 
                               , 
                               t 
                             
                             C 
                           
                         
                         ≥ 
                         
                           ψ 
                           
                             k 
                             , 
                             t 
                           
                           P 
                         
                       
                       ⁢ 
                       
                           
                       
                     
                   
                 
                 
                   
                     
                       
                         ( 
                         
                           
                             ψ 
                             
                               k 
                               , 
                               t 
                             
                             C 
                           
                           - 
                           
                             ψ 
                             
                               k 
                               , 
                               t 
                             
                             P 
                           
                           - 
                           
                             u 
                             
                               k 
                               , 
                               t 
                             
                           
                         
                         ) 
                       
                       ⁢ 
                       
                         ϕ 
                         + 
                       
                     
                   
                   
                     
                       
                         if 
                         ⁢ 
                         
                           
                               
                           
                           ⁢ 
                           
                               
                           
                         
                         ⁢ 
                         
                           ψ 
                           
                             k 
                             , 
                             t 
                           
                           C 
                         
                       
                       &lt; 
                       
                         ψ 
                         
                           k 
                           , 
                           t 
                         
                         P 
                       
                     
                   
                 
               
             
           
         
       
     
     Among the above formula, ϕ −  and ϕ +  are the penalty prices respectively for undersupply and oversupply. 
     In some embodiments, the processor  28  adopts reinforcement learning to update the parameters in the reinforcement learning model V(s): 
         V ( s )← V ( s )+α[ r ( s )+γ V ( s ′)− V ( s )]
 
     Among the above formula, r(s) is the reward, α is the learning rate, and γ is the discount factor. 
     Taking reinforcement learning (Q-Learning) as an example, the update formula is as follows: 
         Q ( S, A )← Q ( S, A )+α[ r +γmax a   Q ( S′, A )− Q ( S, A )]
 
     Among the above formula, Q(S, A) is the current learning value (i.e., Q value), Q(S′, A) is the learning value of the next period of time, r is the reward, α is the learning rate, and γ is the discount factor. 
     Taking deep learning network (Deep Q-Network) as an example, the update formula is as follows: 
         L   i (θ i )=   (s,a,r,s′)˜U(D) [( r +γmax a′   Q ( s′, a′; θ   i−1 )− Q ( s, a; θ   i )) 2 ]
 
     Among the above formula, L i (θ i ) is the loss function value, θ i  is the neural network parameter,   is the expected value of the cumulative cost, Q(s, a; θ i ) is the current learning value, Q(s′, a′; θ i−1 ) is the learning value of the next period of time, r is the reward, α is the learning rate, and γ is the discount factor. 
     By repeating the above steps of data collection, state integration, and model update, a reinforcement learning model optimized for the market environment and the charging and discharging controls of the ESS may finally be obtained, and through this reinforcement learning model, the price bids may be calculated and the supply-demand allocation of renewable energy between the energy suppliers and the energy demanders is coordinated accordingly, thus optimizing the utilization of renewable energy and the profit of the energy aggregator. 
     To sum up, the apparatus and the method for renewable-energy allocation based on reinforcement learning according to the embodiments of the present disclosure integrate the statistical characteristics of renewable energy; by processing and grouping the historical power-generation data of renewable energy, and controlling the charge and discharge of the ESS based on specific rules, the control of the ESS transcends the framework of reinforcement learning and reduces the impact of the intermittent uncertainty of renewable energy on learning, thereby accelerating the speed of reinforcement learning and maximizing the total profit of the energy aggregator. 
     Although the disclosure has been described with reference to the above embodiments, it is apparent to one of ordinary skill in the art that modifications to the described embodiments may be made without departing from the spirit of the disclosure. Accordingly, the scope of the disclosure is defined by the attached claims.