Patent Description:
The disclosure herein generally relates to electric vehicles, and, more particularly, to method and system to generate pricing for charging electric vehicles.

Electric vehicle charging stations connect electric vehicles (e.g., electric battery powered vehicles, plug in hybrid electric vehicles, etc.) to the electric power supply network for the purpose of charging batteries (or other electrical storage devices) of electric vehicles (EV). Multi-unit residential building (MURB) residents are the upcoming segment of electric vehicle owners and potential buyers in several countries. Such MURB residents mostly rely on public chargers, which currently handle only <NUM>% of the EV charging needs. Early adopters of EV typically had private charging infrastructure and with increase in EV owners residing in MURB access to public EV supply equipment (EVSE) which is fast becoming important. However, access to charging through EVSE is the most essential condition for EV adoption in wide network. Most importantly, EVSE deployment has a feedback bootstrapping dependency with EV adoption and either may lead or lag the other. To sustain a virtuous cycle in the joint adoption of EVs and EVSEs, efficient management of existing public EVSE is critically important. With EVs becoming more mainstream, public chargers may not be able to match such operations scaled without extensive deployments. Also, this may not only lead to a demand-supply mismatch in the short-term but impacts the growth of EV adoption in long term. For managing such demand supply mismatch, dynamic pricing is a widely used control tool, but often lacks in making informed pricing decisions to maximize aggregator revenue for business growth.

Most of the conventional systems need efficiency in EVSE particularly during peak demand and even with the presence of fast charging, a single user typically takes around <NUM> minutes for one charging session and so, the EVSE can typically serve only <NUM> EVs during normal working hours. From a user's perspective, an ideal experience would be zero wait availability of public charging at a reasonable cost. From the EVSE operator's perspective, there would be high utilization of the infrastructure that maximizes revenue and margin. To maximize the overall system utility from both these user and system perspectives, a demand management (DM) approach in dynamic or surge pricing for EV charging is essentially required. In general, surge pricing can either reduce the demand (demand-response) to meet the supply; or augment the supply to meet the demand; or both. Most of the conventional techniques have considered pricing for EV charging to varying degrees of abstraction and assumptions/knowledge/models about both EV users (e.g., state of charge or SoC) and operators (e.g., availability). <NPL>" presents an online reinforcement learning based application which increases the revenue of one particular electric vehicles (EV) station, connected to a renewable source of energy. Moreover, the proposed application adapts to changes in the trends of the station's average number of customers and their types. Most of the parameters in the model are simulated stochastically and the algorithm used is a Q-learning algorithm. A computer simulation was implemented which demonstrates and confirms the utility of the model (Abstract). <CIT>) discloses according to one embodiment of the present invention, provided is a charging system, which comprises: a learning control module obtaining a status parameter regarding a charging station environment, updating a compensation value based on the obtained status parameter, and determining a service area based on the updated compensation value; and a communication system broadcasting service information based on the determined service area (Abstract). <NPL>" To mitigate global warming and energy shortage, integration of renewable energy generation sources, energy storage systems, and plug-in electric vehicles (PEVs) have been introduced in recent years. The application of electric vehicles (EV) in the smart grid has shown a significant option to reduce carbon emission. However, due to the limited battery capacity, managing the charging and discharging process of EV as a distributed power supply is a challenging task. Moreover, the unpredictable nature of renewable energy generation, uncertainties of plug-in electric vehicles associated parameters, energy prices, and the time-varying load create new challenges for the researchers and industries to maintain a stable operation of the power system. The EV battery charging management system plays a main role in coordinating the charging and discharging mechanism to efficiently realize a secure, efficient, and reliable power system. More recently, there has been an increasing interest in data-driven approaches in EV charging modeling. Consequently, researchers are looking to deploy model-free approaches for solving the EV charging management with uncertainties. Among many existing model-free approaches, Reinforcement Learning (RL) has been widely used for EV charging management. Unlike other machine learning approaches, the RL technique is based on maximizing the cumulative reward. This article reviews the existing literature related to the RL-based framework, objectives, and architecture for the charging coordination strategies of electric vehicles in the power systems. In addition, the review paper presents a detailed comparative analysis of the techniques used for achieving different charging coordination objectives while satisfying multiple constraints. This article also focuses on the application of RL in EV coordination for research and development of the cutting edge optimized energy management system (EMS), which are applicable for EV charging (Abstract).

Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. For example, in one embodiment, a system for generating pricing for charging electric vehicles is set out in the appended claim <NUM>.

In another aspect, a method for generating pricing for charging electric vehicles is provided in set out in the appended claim <NUM>.

In yet another aspect, a non-transitory computer readable medium provides one or more non-transitory machine-readable information storage mediums comprising one or more instructions, which when executed by one or more hardware processors perform actions includes an I/O interface and a memory coupled to the processor is capable of executing programmed instructions stored in the processor in the memory to receiving by an electric vehicle (EV) charging aggregator having an RL agent, a user request comprising an EV charging demand request and a time of day to generate an EV charging price to maximize revenue of the EV charging aggregator is set out in the appended claim <NUM>. Further, model using a state generator, a state of the RL agent for processing the user request and assigning a reward to the RL agent for the performed action. Then, dynamically generating by the RL agent, the EV charging price for the user request to maximize revenue of the EV charging aggregator by, computing (i) an actual demand pool (P<NUM>) based on a next time step of actual demand pool size ( <MAT>), (ii) a service pool (S) based on a next time step of service pool size ( <MAT>), (iii) a total number of currently available EV chargers (Kt), and (iv) a time of day (td) of the user request.

Embodiments herein provide a method and system to generate pricing for charging electric vehicles (EV). The method disclosed, enables generation of dynamic pricing policies to maximize aggregator revenue based on a stochastic model constraints and user behavioral models. The system includes an EV charging aggregator having a reinforcement learning (RL) agent which processes user request. Each user request comprises an EV charging demand request to charge EVs from electric vehicle supplier equipment (EVSE). The state of the RL agent is spatially tagged with the EV demand request and the availability of EVSE. The RL agent learns to map actions of the user request to maximize aggregator revenue for which a reward is assigned to the RL agent based on the performed action. EV charging pricing policy dynamically adapts to user behaviors at a demand side and a supply side. Realistic management of EVSE, spatio temporal variations for the user request demands to the driving patterns for accurate estimation and adapting the demand elasticity of EV charging. Based on the state space representation, the reward is assigned dynamically to the RL agent for price management. The RL agent is well trained and implemented using a Deep Q-Network (DQN). The present disclosure generates pricing dynamically appropriately to the user request maximizing the aggregator revenue for demand elasticity of the user population.

To manage the demand supply mismatch, dynamic pricing is a widely used control tool, but it is often difficult to make informed pricing decisions: (i) when there is variability (both) in demand and supply; (ii) when the user's spatiotemporal behavior and price elasticity is unknown; (iii) when charging preconditions (such as state-of-charge) are not freely available. The present disclosure utilizes the RL agent to overcome these challenges of dynamic pricing for EV charging acting as the service aggregator. The method of the present disclosure is evaluated on real-world traffic patterns for the city of example (Luxembourg) by augmenting the Luxembourg SUMO (Simulation of Urban Mobility) traffic scenario (LuST) simulator with EV charging demand models. The method matches the performance of popular price-based control approaches, without making any explicit assumptions. In the case of SoC based charging completion policy, represent the present disclosure obtains <NUM>% more revenue than other compared baselines.

<FIG> illustrates an exemplary block diagram of a system (alternatively referred as EV charging management system), in accordance with some embodiments of the present disclosure. In an embodiment, the EV charging management system <NUM> includes processor (s) <NUM>, communication interface (s), alternatively referred as or input/output (I/O) interface(s) <NUM>, and one or more data storage devices or memory <NUM> operatively coupled to the processor (s) <NUM>. The system <NUM>, with the processor(s) is configured to execute functions of one or more functional blocks of the system <NUM>. Referring to the components of the system <NUM>, in an embodiment, the processor (s) <NUM> can be one or more hardware processors <NUM>. In an embodiment, the one or more hardware processors <NUM> can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) <NUM> is configured to fetch and execute computer-readable instructions stored in the memory. In an embodiment, the system <NUM> can be implemented in a variety of computing systems, such as laptop computers, notebooks, <NUM> hand-held devices, workstations, mainframe computers, servers, a network cloud, and the like.

The I/O interface(s) <NUM> can include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like and can facilitate multiple communications within a wide variety of networks N/W and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite. In an embodiment, the I/O interface (s) <NUM> can include one or more ports for connecting a number of devices (nodes) of the system <NUM> to one another or to another server. The memory <NUM> may include any computer-readable medium known in the art including, for example, volatile memory, such as static random-access memory (SRAM) and dynamic random-access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. The modules <NUM> can be an Integrated Circuit (IC) (not shown), external to the memory <NUM>, implemented using a Field-Programmable Gate Array (FPGA) or an Application-Specific Integrated Circuit (ASIC). The names (or expressions or terms) of the modules of functional block within the modules <NUM> referred herein, are used for explanation and are not construed to be limitation(s).

<FIG> illustrates a public EV supply equipment for the demand EV profiles in discrete time steps using the system of <FIG>, in accordance with some embodiments of the present disclosure. <FIG> describes the challenge of pricing EV charging from a different perspective that drives any technology adoption in practice user behavior. EV users prefer to align their charging with convenient points in their driving patterns unless they are forced to charge immediately due to a critically low stochastic model constraints. A complementary aspect was considered in augmenting the supply-side of EVSE by incentivizing private garage owners to allow public charging. The key insight was that the behavior of private garage owners to charge at night; could be leveraged to augment supply during peak demand for EVSE charging during the day. The present disclosure dynamically generates EV charging pricing policy that can adapt to user behaviors at the demand-side and potentially the supply- side without any a priori knowledge. Pricing of EV charging for dynamic management is nontrivial for several reasons. Firstly, the demand for EV charging varies in space and time. If private garages participate in public charging, the supply-side also varies in space and time. Second, explicit, and accurate modeling of user behavior to price is challenging. Specifically, some users might prefer the nearest EVSE at any price; while others might be more sensitive to price and willing to drive around; and so on. Third, state of charge (SoC) in EVs may not be known to the EVSE operators till the start of charging. In other words, the propensity of the user to charge due to critically low SoC is not known when setting the price. Finally, the EVSE operator may also need to consider any time-of-day variations in pricing when procuring electricity in bulk from the utility company.

<FIG> illustrates a functional architecture to generate pricing for charging EV using the system of <FIG>, in accordance with some embodiments of the present disclosure. The system includes an EV charging aggregator having an RL agent, a tentative demand pool, an actual demand pool, and a service pool. The system involves three primary stake holders comprising firstly a demand side having EV users requesting efficient charging at lowest possible price, secondly a supply side which includes a public or private EVSE operators, and thirdly an EV charging aggregator which acts as intermediator between the demand side and the supply side. The RL agent acts as an EV aggregator to a set of EVSEs spread across a city. The RL agent addresses the problem of dynamic pricing for EV charging by observing the state and decides a control action. The system implements the action and provides the reward to the RL agent. The RL agent learns to map actions to the observed state maximizes the rewards earned over the training period which complements several existing approaches that use assumptions or models about user behaviors. The action taken by the RL is to decide the price to be offered to users. Users are assigned to the nearest available EVSE separately outside the system. The reward to the RL agent would be the total revenue obtained in that timestep of the control action. The action taken by the RL agent is to decide the price to be offered to user request. Further, the reward to the RL agent would be the total revenue obtained in the timestep of the control action. Here, a low price may be accepted by many users, but the revenue would be limited by the number of EVSEs. A high price may be rejected by many users, and the revenue and utilization would substantially fall. In such scenarios, the RL agent learns to price appropriately to maximize revenue based on the demand elasticity of the user population.

<FIG> illustrates a flow diagram illustrating the method for maximizing EV aggregator revenue by generating pricing for EV charging using the system of <FIG>, in accordance with some embodiments of the present disclosure. In an embodiment, the system <NUM> comprises one or more data storage devices or the memory <NUM> operatively coupled to the processor(s) <NUM> and is configured to store instructions for execution of steps of the method <NUM> by the processor(s) or one or more hardware processors <NUM>. The steps of the method <NUM> of the present disclosure will now be explained with reference to the components or blocks of the system <NUM> as depicted in <FIG> and the steps of flow diagram as depicted in <FIG>. Although process steps, method steps, techniques or the like may be described in a sequential order, such processes, methods and techniques may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps to be performed in that order. The steps of processes described herein may be performed in any order practical. Further, some steps may be performed simultaneously.

Referring now to the steps of the method <NUM>, at step <NUM>, the one or more hardware processors <NUM> receive, via an electric vehicle (EV) charging aggregator having an RL agent, a user request comprising an EV charging demand request and a time of day to generate an EV charging price to maximize revenue of the EV charging aggregator. Considering an example, where EV users go about their daily routines make the EV charging demand request to the EV charging aggregator depending upon the time of day and the stochastic model constraints which is unknown to the RL agent. On receiving the EV charging demand request, the RL agent responds with a price. The EV users can decide either accept the offered price and proceed to charging or reject the price and try again later in the day. Again, the model for users accepting the offered model is parametrized, stochastic, and hidden from the EV charging aggregator. The EV charging demand requests would be GPS location stamped; and the EV charging aggregator obtains the current availability status of all participating EVSEs.

Referring now to the steps of the method <NUM>, at step <NUM>, the one or more hardware processors <NUM>, models, by using a state generator, a state of the RL agent for processing the user request and assigning a reward to the RL agent for the performed action. Referring now to the above example, a city is considered as one large block, a single RL agent can serve the target environment based on the system load. Intuitively, to get a sense of the load on the system, the system state includes the details about the actual demand pool and the service pool. Once the user makes the request, they finish charging or decide to leave the charging process users can be tracked. Therefore, estimate of the demand pool sizes NX for the actual demand pool and the service pools can be collected. The state of the RL agent includes (i) the actual demand pool size ( <MAT>), (ii) the service pool size ( <MAT>), (iii) the total number of currently available EV chargers (Kt) at current time step, and (iv) the user request time of day (td). At any time t, EVs user requests fall into one of three demand queues: (<NUM>) a tentative demand pool (P<NUM>), (<NUM>) an actual demand pool (P<NUM>), and (<NUM>) a service pool S. The respective queue sizes evolve from t to t+<NUM> i.e., from the current time step to the next time step and the variable definitions are described below in Table <NUM>,.

The time-of-day(td) as part of the system state, so that any time-of-day effects in user behavior can be learnt over time. The state of the RL agent is defined as denoted below in equation <NUM>, <MAT> The state of the RL agent makes no assumptions about the stochastic model constraints of EVs as this information is potentially available only after connecting to the EVSE.

Referring now to the steps of the method <NUM>, at step <NUM>, the one or more hardware processors <NUM>, dynamically generate, via the RL agent, the EV charging price for the user request to maximize revenue of the EV charging aggregator by, computing (i) an actual demand pool (P<NUM>) based on a next time step of actual demand pool size ( <MAT>), (ii) a service pool (S) based on a next time step of service pool size ( <MAT>), (iii) a total number of currently available EV chargers (Kt), and (iv) a time of day (td) of the user request. Referring now to the above example, to generate the EV charging price for the user request, the following steps are performed by the system,.

The action chosen (At) is simply the price (pt) and the instantaneous reward Rt obtained by the action (At) is the revenue over the next timestep.

<FIG> and <FIG> illustrate a behavior model of EV users and private EVSE suppliers using the system of <FIG>, in accordance with some embodiments of the present disclosure. In the experimental analysis set-up, all EV users whose stochastic model constraints is less than <NUM>% can raise a charging request. Such users are referred to as the tentative demand pool. Users who request charging are added to the actual demand pool; and offered a price (pt). The user either accepts (pt) and moves to the service pool; or rejects (pt) and goes back to the tentative demand pool. The acceptance or rejection is a probabilistic event that is modeled as shown in <FIG>. The X-axis is the offered price (between <NUM>-<NUM> EUR/kWh); and the Y-axis shows the probability of acceptance. If the SoC is not in critical state (i.e., SoC > <NUM>%), the probability of acceptance at a lower price (pt) is higher. As the SoC depletes and goes below a critical level (i.e., SoC < <NUM>%), users would accept an arbitrarily high offered price. <FIG> shows a similar probability of supply for private garages where the acceptance to join the system is higher when the price is above an expected threshold (i.e., <NUM>% more than the minimum charging cost at public chargers).

In one embodiment, the transport simulator acts as an integrator for the EV model for Tesla Model-S (<NUM> kWh RWD) and the system model is integrated with
SUMO (Simulation of Urban Mobility). LuST (Luxembourg SUMO traffic scenario) that simulates the traffic in the city of Luxembourg. The Luxembourg city map and its household demographics from OpenStreetMap (state of the art technique), and public charging locations from Open Charge Map (state of the art technique). The LuST scenario inserts roughly <NUM>, <NUM> cars into the simulation over its <NUM> hours duration. The method randomly chooses <NUM>% of cars as EVs, in order to maintain the population of simulated EVs consistent with the statistics reported for Luxembourg city. Out of this pool, the randomly marked <NUM>% as garage-orphaned electric cars, which charge on-the-go using public chargers.

The RL agent is implemented using the TensorFlow framework. The details about the neural network architecture and the hyperparameters used in the learning and testing are as follows: (i) DQN (Deep Q-Network) architecture:(<NUM>,<NUM>,<NUM>,<NUM>); (ii) optimizer: Adam; (iii) learning rate: <NUM>; (iv) discount factor: <NUM>; (v) mini batch size: <NUM>; (vi) replay buffer size: <NUM>% of training epochs; (vii) exploration factor: <NUM> to <NUM> (decay in steps of <NUM>); (viii) training epochs per episode: <NUM>. The training consisted of <NUM> episodes and testing <NUM> episodes, where <NUM> episode is the same as <NUM> day.

<FIG> and <FIG> illustrate graphical representation of system load vs predicted price maximizing EV aggregator revenue using the system of <FIG>, in accordance with some embodiments of the present disclosure. <FIG> shows a basic test of the price predicted by the RL agent against the charging load experienced by the system. From the <FIG> it is observed that the price varies with the load; except for.

<FIG> illustrates performance of the RL agent with various (business) metrics compared during peak hours and on average for generating pricing for EV charging aggregators for maximizing revenue using the system of <FIG>, in accordance with some embodiments of the present disclosure. <FIG> shows the performance of the RL agent in terms of business metrics. <FIG> plot the total revenue, revenue per charger, and utilization, respectively, for the average case, while the same metrics are shown in <FIG> for the peak case. For the SOC-based charging policy, on average, it is observed that the RL agent gets <NUM>% more revenue than all other baselines. Since the utilization of the chargers is almost similar
across all cases (<FIG>), the revenue increase is not only due to a greater number of chargers in the system, but also from a higher average revenue per charger (<FIG>). However, this does not hold good in the peak case, where the higher revenue of the RL agent is primarily from using a larger pool of available chargers. For the time-based charging policy where every EV can charge for a predefined short time-period, the system is able to service charging requests at a faster rate. Therefore, for the same demand profile, it brings down the load on the system by making chargers more available. In this case, compared to other baselines, the RL agent gets an additional revenue of <NUM>-<NUM>%.

The embodiments of present disclosure herein address unresolved problem of charging electric vehicles. The embodiments, thus provide method and system to generate pricing for charging electric vehicles. Moreover, the embodiments herein further provide dynamically generates pricing policies to maximize aggregator revenue based on stochastic model constraints and user behavioral models. The EV charging aggregator having an RL agent receives user requests to generate pricing based on a tentative demand pool, an actual demand pool, and a service pool. The method learns the pricing policies that maximize the aggregate revenue but without making any assumptions about the stochastic dynamics of the system, or data availability, or user behavioral models.

Thus, the means can include both hardware means, and software means.

Claim 1:
A processor implemented method to generate pricing for charging electric vehicles characterized in that:
receiving by an electric vehicle, EV, charging aggregator having an Reinforcement Learning, RL, agent via one or more hardware processors (<NUM>), a user request comprising an EV charging demand request and a time of day to generate an EV charging price to maximize revenue of the EV charging aggregator;
modelling using a state generator, via the one or more hardware processors (<NUM>), a state of the RL agent for processing the user request and assigning a reward to the RL agent for the performed action, wherein the state of the RL agent includes i. the actual demand pool size <MAT>, ii. the service pool size <MAT>, iii. the total number of currently available EV chargers Kt at current time step, and iv. the time of day td of the user request so that time-of-day effects in user behavior are learnt over time, wherein the state of the RL agent is spatially tagged with the EV charging demand request and availability of EV supply equipment, EVSE, wherein the EV charging demand request is Global Positioning System, GPS, location stamped and the EV charger aggregator obtains current availability of participating EVSEs, wherein the RL agent learns to map action to observed state and maximizes the reward earned over training period, wherein the RL agent is trained and implemented using a Deep Q-network, DQN; and
dynamically generating by the RL agent, via the one or more hardware processors (<NUM>), the EV charging price for the user request, without a priori knowledge, to maximize revenue of the EV charging aggregator by, computing i. an actual demand pool P<NUM> based on a next time step of actual demand pool size <MAT>, ii. a service pool S based on a next time step of service pool size <MAT>, iii. a total number of currently available EV chargers Kt, and iv. a time of day td of the user request.