# Optimizing integrated lot sizing and production scheduling in flexible flow line systems with energy scheme: A two level approach based on reinforcement learning

Mohamed Habib Jabeur, Sonia Mahjoub, Cyril Toublanc, Veronique Cariou

► To cite this version:

Mohamed Habib Jabeur, Sonia Mahjoub, Cyril Toublanc, Veronique Cariou. Optimizing integrated lot sizing and production scheduling in flexible flow line systems with energy scheme: A two level approach based on reinforcement learning. Computers & Industrial Engineering, 2024, 190, pp.110095. 10.1016/j.cie.2024.110095. hal-04534236

HAL Id: hal-04534236

https://hal.science/hal-04534236

Submitted on 24 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Optimizing integrated lot sizing and production scheduling in flexible flow line systems with energy scheme: A two level approach based on reinforcement learning

Mohamed Habib Jabeur<sup>a, *</sup>, Sonia Mahjoub<sup>b</sup>, Cyril Toublanc<sup>c</sup>, Veronique Cariou<sup>a</sup>

<figure><img src="image_1.png" /></figure>

## ARTICLE INFO

Keywords:

Lot sizing

Production scheduling

Flexible flow line

Renewable energy

Time-of-use prices

Multi-agent system

## ABSTRACT

Many production environments are faced with the need to simultaneously determine the planning of lot sizing and the scheduling of production sequences while ensuring cost minimization. This issue becomes even more complex when integrating multiple energy sources with the goal of a low-carbon economy. To address this challenge, this paper proposes an integrated lot sizing and flexible flow line production scheduling model under a time-of-use pricing scheme. In addition, the model takes into account conventional grid power, on-site renewable energy sources, and an energy storage system. The associated objective function is solved adopting a two-level approach to optimize energy costs while maintaining production throughput and meeting customer demand. The implementation relies on reinforcement learning capabilities to tackle complexity issues. The proposed approach is evaluated on a benchmark case and its results are compared with those obtained with First-In-First-Out heuristic, genetic algorithm and CPLEX. These results highlight the promising aspect of the proposed approach in terms of performance.

## 1. Introduction

Within the scope of industry 4.0, improving industrial energy efficiency has become a key focus for many companies aiming to achieve their carbon-neutrality goals and sustain their competitive advantage. According to Eurostat (2019), the industrial sector accounted for 25 % of the total energy consumption in the European Union (EU). To reduce energy costs and decrease greenhouse gas (GHG) emissions, demand side management (DSM) and on-site renewable energy appear to be promising solutions. DSM is an energy policy implemented by utilities to modify users' energy consumption patterns, ultimately enhancing power grid efficiency (Duarte et al., 2020). DSM can be effectively implemented through demand response (DR) programs, which seek to alter energy demands in response to economic incentives (Wang et al., 2020). The DR program has emerged as an effective policy for industries to reduce their energy costs without compromising their production throughput. In practice, this program involves an increase in production rates during off-peak electricity demand hours and a decrease in

production rates during peak grid demand hours (Basán et al., 2020). The DR can be considered as a production scheduling problem whose objective is to minimize both the energy consumed and the associated operational costs by prioritizing production during off-peak periods (Kelley, Baldick, & Baldea, 2019, 2020; Kelley, Pattison, Baldick, & Baldea, 2018). Consequently, a flexible and dynamic manufacturing process supported by an efficient energy system is of paramount interest to adapt to electricity market price signals. For this purpose, on-site renewable energy offers substantial flexibility and opportunity for manufacturers to reduce energy costs in the face of time-varying electricity prices. Furthermore, the integration of renewable energy sources (RES) into the manufacturing process is an effective sustainable strategy for reducing GHG emissions. The adoption of renewable energy technologies in the EU more than doubled between 2004 and 2017, and renewable energy production is expected to continue increasing in the coming years (Eurostat, 2018; 2019). RES generation, such as wind and solar energy, exhibits high variability due to its dependence on weather conditions. The inherent uncertainties associated with these renewable

<footnote><sup>a</sup> Oniris, INRAE, STATSC, 44300 Nantes, France</footnote>

<footnote><sup>b</sup> Oniris, Nantes université, LEMNA, CS 82225, 44322 Nantes, France</footnote>

<footnote><sup>c</sup> Oniris, Nantes université, CNRS, GEPEA, UMR 6144, F-44000 Nantes, France</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail addresses: mohamed-habib.jabeur@oniris-nantes.fr (M.H. Jabeur), sonia.mahjoub@oniris-nantes.fr (S. Mahjoub), cyril.toublanc@oniris-nantes.fr (C. Toublanc), veronique.cariou@oniris-nantes.fr (V. Cariou).</footnote>

<footnote>https://doi.org/10.1016/j.cie.2024.110095</footnote>
<figure><img src="image_2.png" /></figure>

# Optimizing integrated lot sizing and production scheduling in flexible flow line systems with energy scheme: A two level approach based on reinforcement learning

Mohamed Habib Jabeur<sup>a, *</sup>, Sonia Mahjoub<sup>b</sup>, Cyril Toublanc<sup>c</sup>, Veronique Cariou<sup>a</sup>

## ARTICLE INFO

Keywords:

Lot sizing

Production scheduling

Flexible flow line

Renewable energy

Time-of-use prices

Multi-agent system

## ABSTRACT

Many production environments are faced with the need to simultaneously determine the planning of lot sizing and the scheduling of production sequences while ensuring cost minimization. This issue becomes even more complex when integrating multiple energy sources with the goal of a low-carbon economy. To address this challenge, this paper proposes an integrated lot sizing and flexible flow line production scheduling model under a time-of-use pricing scheme. In addition, the model takes into account conventional grid power, on-site renewable energy sources, and an energy storage system. The associated objective function is solved adopting a two-level approach to optimize energy costs while maintaining production throughput and meeting customer demand. The implementation relies on reinforcement learning capabilities to tackle complexity issues. The proposed approach is evaluated on a benchmark case and its results are compared with those obtained with First-In-First-Out heuristic, genetic algorithm and CPLEX. These results highlight the promising aspect of the proposed approach in terms of performance.

## 1. Introduction

Within the scope of industry 4.0, improving industrial energy efficiency has become a key focus for many companies aiming to achieve their carbon-neutrality goals and sustain their competitive advantage. According to Eurostat (2019), the industrial sector accounted for 25 % of the total energy consumption in the European Union (EU). To reduce energy costs and decrease greenhouse gas (GHG) emissions, demand side management (DSM) and on-site renewable energy appear to be promising solutions. DSM is an energy policy implemented by utilities to modify users' energy consumption patterns, ultimately enhancing power grid efficiency (Duarte et al., 2020). DSM can be effectively implemented through demand response (DR) programs, which seek to alter energy demands in response to economic incentives (Wang et al., 2020). The DR program has emerged as an effective policy for industries to reduce their energy costs without compromising their production throughput. In practice, this program involves an increase in production rates during off-peak electricity demand hours and a decrease in

production rates during peak grid demand hours (Basán et al., 2020). The DR can be considered as a production scheduling problem whose objective is to minimize both the energy consumed and the associated operational costs by prioritizing production during off-peak periods (Kelley, Baldick, & Baldea, 2019, 2020; Kelley, Pattison, Baldick, & Baldea, 2018). Consequently, a flexible and dynamic manufacturing process supported by an efficient energy system is of paramount interest to adapt to electricity market price signals. For this purpose, on-site renewable energy offers substantial flexibility and opportunity for manufacturers to reduce energy costs in the face of time-varying electricity prices. Furthermore, the integration of renewable energy sources (RES) into the manufacturing process is an effective sustainable strategy for reducing GHG emissions. The adoption of renewable energy technologies in the EU more than doubled between 2004 and 2017, and renewable energy production is expected to continue increasing in the coming years (Eurostat, 2018; 2019). RES generation, such as wind and solar energy, exhibits high variability due to its dependence on weather conditions. The inherent uncertainties associated with these renewable

<footnote><sup>a</sup> Oniris, INRAE, STATSC, 44300 Nantes, France</footnote>

<footnote><sup>b</sup> Oniris, Nantes université, LEMNA, CS 82225, 44322 Nantes, France</footnote>

<footnote><sup>c</sup> Oniris, Nantes université, CNRS, GEPEA, UMR 6144, F-44000 Nantes, France</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail addresses: mohamed-habib.jabeur@oniris-nantes.fr (M.H. Jabeur), sonia.mahjoub@oniris-nantes.fr (S. Mahjoub), cyril.toublanc@oniris-nantes.fr (C. Toublanc), veronique.cariou@oniris-nantes.fr (V. Cariou).</footnote>

<footnote>https://doi.org/10.1016/j.cie.2024.110095</footnote>
energy resources can lead to inaccuracies in potential scheduling solu-
tions. To address the intermittent nature of renewable energy, an energy
storage system (ESS) can store excess renewable energy and deploy it
when needed.

In the light of the aforementioned work, this study proposes a dy-
namic sustainable production scheduling model that aims to optimize
energy costs while maintaining production throughput. Specifically, the
model considers an integrated lot sizing on a flexible flow line (FFL)
scheduling problem with sequence-dependent setups under a time-of-
use (TOU) pricing scheme. The proposed model considers a conven-
tional grid, on-site photovoltaic solar panels and an ESS as energy
sources. To the best of our knowledge, only a few research studies
investigated the integration of lot sizing and sustainable production
scheduling on a FFL with energy considerations. Given the relevance of
such integration for many energy-intensive manufacturing industries (e.
g. steel-making, chemical and food industries), a novel optimization
framework is proposed to fill this gap, integrating energy management
policies and renewable energy into the FFL environment. In addition,
the proposed optimization model takes into account two interrelated
decision levels, namely lot sizing and scheduling. A new resolution
approach based on an interactive two-level algorithm is implemented to
circumvent the complexity issues. At the lot sizing decision level, the
algorithm aims at generating optimal lot sizes while trading off RES, ESS
and inventory costs and economic benefits derived from energy saving.
Regarding the scheduling decision level, an innovative and dynamic
cooperative multi-agent approach is developed to address the
complexity of machine allocation and product sequencing problems.
This approach, based on Q-learning methodology, leads to consider that
the Q-agents are trained in a dynamic FFL environment with the
objective of meeting customer's demand while optimizing energy and
operational costs.

The rest of the research is organized as follows. In Section 2, a
concise overview of the literature is provided. Section 3 presents the
problem formulation and its mathematical model. Section 4 outlines the
proposed resolution approach based on reinforcement learning. In Sec-
tion 5, results obtained by applying the proposed approach are detailed.
In particular, performance aspects, including optimization outcomes
and computational efficiency, are analyzed and discussed. Section 6 is
dedicated to managerial insights. Finally, conclusions are presented in
section 7.

## 2. Literature review

### 2.1. Integrated lot sizing and production scheduling (ILSPS) problem on flexible flow lines (FFL)

Operational production planning aims to determine the number of product lots to be produced (lot sizing) and the sequence of operations for their production (scheduling). These two critical decisions are generally considered independently in cases where there is no interdependence between machine setup and operation sequencing (Carvalho et al., 2022). However, when costs and machine setup time are influenced by the sequence of operations, the scheduling decision directly affects the lot sizing decision. Thus, integrating lot sizing and production scheduling decisions into a unified problem may generate better production planning. Within this scope, some studies investigated the integrated lot sizing and production scheduling (ILSPS) problem with different types of machine architectures including single machines, parallel machines, flow shops, and job shops. For example, Carvalho and Nascimento (2022) studied the ILSPS problem given non-identical parallel machines with non-triangular sequence-dependent setup costs and times. The authors proposed different matheuristics based on relax-and-fix, relax-and-optimize, path relinking and kernel search methods to efficiently solve the ILSPS problem on parallel machines. The computational results highlighted the better performance of the proposed algorithms compared to the CPLEX solver. Xiao et al. (2015) investigated a

lot sizing and production scheduling problem on parallel machines with
sequence-dependent setup times. To this end, the authors advocated a
hybrid heuristic that combines simulated annealing with a Lagrangian-
based algorithm. This hybrid approach has proven to outperform other
heuristic algorithms. Mahdieh et al. (2011) also developed mathemat-
ical models for different integrated lot sizing and production scheduling
on flexible flow lines (FFL). The experimental results highlighted the
complexity of the problem, as optimality tests could only be performed
on small instances. Babaei et al. (2014) examined the capabilities of a
genetic algorithm applied on a multi-product integrated lot sizing and
production scheduling problem on a flow line. The computational ex-
periments demonstrated the effectiveness of this algorithm in problem
solving. In the same vein, the present study addresses the ILSPS problem
on FFL, which is commonly encountered in various production systems
e.g. food processing and textile industries. Despite the importance of this
approach in meeting the needs of the industries, the literature on ILSPS
problem on FFL remains scarce even if it proposes various optimization
methods, including exact methods, heuristics and *meta-heuristics*. Exact
mathematical optimization methods aim to find an optimal solution in
the entire solution space. However, due to the NP-hardness of those
methods, they are unfortunately unable to solve larger problems within
a reasonable period (Li, Pan and Liang, 2010, Lei et al., 2022).
Conversely, heuristic and *meta-heuristic* methods have shorter compu-
tation times at the expense of less optimal scheduling results compared
to exact methods. To make a balance between the quality result and
computational time, several artificial intelligence approaches such as
reinforcement learning (RL) (Van Moffaert and Nowe, 2014; Zou et al.,
2021) have been investigated to solve complex combinatorial optimi-
zation problems (Mazyavkina et al., 2021). For instance, Lei et al. (2022)
developed a deep reinforcement framework for solving a flexible job
shop scheduling problem. The proposed framework involved two sub-
policies designed either for job operation actions or for a machine ac-
tion. The computational experiment results have proven that this deep
reinforcement learning method outperforms heuristic and *meta-heuristic*
methods in solution quality and running time, respectively. Wang et al.
(2021) proposed a dynamic scheduling approach based on deep rein-
forcement learning to deal with uncertainties and dynamic events in a
job shop scheduling environment. Reyna et al. (2019) explored a flow
shop scheduling problem with an adapted reinforcement learning
approach involving a heuristic to initiate the search space with a po-
tential solution. Their approach provided high-quality solutions within
short computational times. In another study, Lin et al. (2022) developed
a Q-learning technique to guide the selection of a heuristic from a pre-
designed low-level heuristic for solving a semiconductor final testing
scheduling problem. Yan et al. (2022) employed a combination of
reinforcement learning and digital twin technology to solve a dynamic
flexible job shop-scheduling problem with flexible maintenance policies.
Numerical experiments validated the feasibility of the obtained sched-
uling results and highlighted the associated performance compared to
three competitor algorithms. Finally, Cheng et al. (2022) proposed a Q-
learning-based genetic algorithm to obtain near-optimal solutions for a
complex production-scheduling problem.

As detailed above, several studies have already proven the relevance
of RL techniques to solve production-scheduling problems such as flow
shop and job shop scheduling problems. Notwithstanding, these tech-
niques have not yet been considered to address the ILPSP in an FFL
environment. In response to this research gap, this study proposes to
implement a RL approach with an innovative architecture to tackle the
ILSPS problem in an FFL environment. The efficiency of this approach is
further demonstrated through comparisons with several conventional
methods.

### 2.2. Production scheduling with energy consideration

To address the issue of energy costs, several studies were conducted
on production scheduling with energy considerations to optimize energy
energy resources can lead to inaccuracies in potential scheduling solu-
tions. To address the intermittent nature of renewable energy, an energy
storage system (ESS) can store excess renewable energy and deploy it
when needed.

In the light of the aforementioned work, this study proposes a dy-
namic sustainable production scheduling model that aims to optimize
energy costs while maintaining production throughput. Specifically, the
model considers an integrated lot sizing on a flexible flow line (FFL)
scheduling problem with sequence-dependent setups under a time-of-
use (TOU) pricing scheme. The proposed model considers a conven-
tional grid, on-site photovoltaic solar panels and an ESS as energy
sources. To the best of our knowledge, only a few research studies
investigated the integration of lot sizing and sustainable production
scheduling on a FFL with energy considerations. Given the relevance of
such integration for many energy-intensive manufacturing industries (e.
g. steel-making, chemical and food industries), a novel optimization
framework is proposed to fill this gap, integrating energy management
policies and renewable energy into the FFL environment. In addition,
the proposed optimization model takes into account two interrelated
decision levels, namely lot sizing and scheduling. A new resolution
approach based on an interactive two-level algorithm is implemented to
circumvent the complexity issues. At the lot sizing decision level, the
algorithm aims at generating optimal lot sizes while trading off RES, ESS
and inventory costs and economic benefits derived from energy saving.
Regarding the scheduling decision level, an innovative and dynamic
cooperative multi-agent approach is developed to address the
complexity of machine allocation and product sequencing problems.
This approach, based on Q-learning methodology, leads to consider that
the Q-agents are trained in a dynamic FFL environment with the
objective of meeting customer's demand while optimizing energy and
operational costs.

The rest of the research is organized as follows. In Section 2, a
concise overview of the literature is provided. Section 3 presents the
problem formulation and its mathematical model. Section 4 outlines the
proposed resolution approach based on reinforcement learning. In Sec-
tion 5, results obtained by applying the proposed approach are detailed.
In particular, performance aspects, including optimization outcomes
and computational efficiency, are analyzed and discussed. Section 6 is
dedicated to managerial insights. Finally, conclusions are presented in
section 7.

## 2. Literature review

### 2.1. Integrated lot sizing and production scheduling (ILSPS) problem on flexible flow lines (FFL)

Operational production planning aims to determine the number of product lots to be produced (lot sizing) and the sequence of operations for their production (scheduling). These two critical decisions are generally considered independently in cases where there is no interdependence between machine setup and operation sequencing (Carvalho et al., 2022). However, when costs and machine setup time are influenced by the sequence of operations, the scheduling decision directly affects the lot sizing decision. Thus, integrating lot sizing and production scheduling decisions into a unified problem may generate better production planning. Within this scope, some studies investigated the integrated lot sizing and production scheduling (ILSPS) problem with different types of machine architectures including single machines, parallel machines, flow shops, and job shops. For example, Carvalho and Nascimento (2022) studied the ILSPS problem given non-identical parallel machines with non-triangular sequence-dependent setup costs and times. The authors proposed different matheuristics based on relax-and-fix, relax-and-optimize, path relinking and kernel search methods to efficiently solve the ILSPS problem on parallel machines. The computational results highlighted the better performance of the proposed algorithms compared to the CPLEX solver. Xiao et al. (2015) investigated a

lot sizing and production scheduling problem on parallel machines with
sequence-dependent setup times. To this end, the authors advocated a
hybrid heuristic that combines simulated annealing with a Lagrangian-
based algorithm. This hybrid approach has proven to outperform other
heuristic algorithms. Mahdieh et al. (2011) also developed mathemat-
ical models for different integrated lot sizing and production scheduling
on flexible flow lines (FFL). The experimental results highlighted the
complexity of the problem, as optimality tests could only be performed
on small instances. Babaei et al. (2014) examined the capabilities of a
genetic algorithm applied on a multi-product integrated lot sizing and
production scheduling problem on a flow line. The computational ex-
periments demonstrated the effectiveness of this algorithm in problem
solving. In the same vein, the present study addresses the ILSPS problem
on FFL, which is commonly encountered in various production systems
e.g. food processing and textile industries. Despite the importance of this
approach in meeting the needs of the industries, the literature on ILSPS
problem on FFL remains scarce even if it proposes various optimization
methods, including exact methods, heuristics and *meta-heuristics*. Exact
mathematical optimization methods aim to find an optimal solution in
the entire solution space. However, due to the NP-hardness of those
methods, they are unfortunately unable to solve larger problems within
a reasonable period (Li, Pan and Liang, 2010, Lei et al., 2022).
Conversely, heuristic and *meta-heuristic* methods have shorter compu-
tation times at the expense of less optimal scheduling results compared
to exact methods. To make a balance between the quality result and
computational time, several artificial intelligence approaches such as
reinforcement learning (RL) (Van Moffaert and Nowe, 2014; Zou et al.,
2021) have been investigated to solve complex combinatorial optimi-
zation problems (Mazyavkina et al., 2021). For instance, Lei et al. (2022)
developed a deep reinforcement framework for solving a flexible job
shop scheduling problem. The proposed framework involved two sub-
policies designed either for job operation actions or for a machine ac-
tion. The computational experiment results have proven that this deep
reinforcement learning method outperforms heuristic and *meta-heuristic*
methods in solution quality and running time, respectively. Wang et al.
(2021) proposed a dynamic scheduling approach based on deep rein-
forcement learning to deal with uncertainties and dynamic events in a
job shop scheduling environment. Reyna et al. (2019) explored a flow
shop scheduling problem with an adapted reinforcement learning
approach involving a heuristic to initiate the search space with a po-
tential solution. Their approach provided high-quality solutions within
short computational times. In another study, Lin et al. (2022) developed
a Q-learning technique to guide the selection of a heuristic from a pre-
designed low-level heuristic for solving a semiconductor final testing
scheduling problem. Yan et al. (2022) employed a combination of
reinforcement learning and digital twin technology to solve a dynamic
flexible job shop-scheduling problem with flexible maintenance policies.
Numerical experiments validated the feasibility of the obtained sched-
uling results and highlighted the associated performance compared to
three competitor algorithms. Finally, Cheng et al. (2022) proposed a Q-
learning-based genetic algorithm to obtain near-optimal solutions for a
complex production-scheduling problem.

As detailed above, several studies have already proven the relevance
of RL techniques to solve production-scheduling problems such as flow
shop and job shop scheduling problems. Notwithstanding, these tech-
niques have not yet been considered to address the ILPSP in an FFL
environment. In response to this research gap, this study proposes to
implement a RL approach with an innovative architecture to tackle the
ILSPS problem in an FFL environment. The efficiency of this approach is
further demonstrated through comparisons with several conventional
methods.

### 2.2. Production scheduling with energy consideration

To address the issue of energy costs, several studies were conducted
on production scheduling with energy considerations to optimize energy
consumption in manufacturing systems. In this regard, Shrouf et al. (2014) developed a mathematical model to minimize energy consumption costs for a single machine production scheduling considering variable energy prices. Masmoudi et al. (2017) proposed two heuristics to solve a flow shop scheduling problem with energy consideration. Similarly, Rodoplu et al. (2020) developed a hybrid scheduling model for a single-item flow shop production scheduling, aiming to reduce energy consumption. Li et al. (2018) investigated a hybrid flow shop scheduling problem with setup energy consumptions using an energy-aware multi-objective optimization algorithm. Alternatively, making production scheduling sustainable can also be achieved using onsite renewable energy. Indeed, the use of renewable energy resources not only provides a solution to meet peak electricity demands but also helps to limit GHG emissions. The integration of renewable energy sources into production systems has been addressed several times in recent literature. Golari et al. (2017) formulated a lot sizing problem that integrates a renewable energy source as an optimization programming model. Wang et al. (2020) investigated a production scheduling problem that incorporates onsite renewable energy, the main grid, and an energy storage system (ESS). They elaborated a two-stage multi-objective stochastic model for flow shops with sequence-dependent setups to simultaneously minimize total weighted completion time and energy costs. Trevino-Martinez et al. (2022) introduced an optimization framework that incorporates renewable energy sources and an ESS into a job shop scheduling problem. Duarte et al. (2020) addressed a multi-process production planning problem that incorporates intermittent renewable energy sources and an ESS. They proposed a mathematical programming model and an algorithm to optimize production scheduling and energy management system decisions by considering grid power, onsite renewable energy sources, an ESS, and intermediate buffers. Their results show that the integration of these policies ensures a flexible manufacturing process. In this context, Peinado-Guerrero and Villalobos (2022) considered the use of an ESS and intermediate buffers in a manufacturing facility to reduce energy consumption during peak demand periods. Sun et al. (2014) developed a mathematical model that aims to maintain a good tradeoff between the economic costs of intermediate buffer systems and the potential cost savings generated from TOU pricing.

In summary, this study focuses on addressing the ILSPS problem in an FFL (Flexible Flow Line) environment while considering energy efficiency. Compared to previous research, this study makes the following main contributions:

* A novel mixed-integer linear model is developed to accurately represent the ILSPS problem in the FFL environment under a Time-of-Use (TOU) pricing scheme. The mathematical model is specifically adapted to incorporate an onsite photovolatic solar power supply.

* To achieve a balance between the costs associated with buffers, Energy Storage Systems (ESS), Renewable Energy Sources (RES), and TOU pricing, a two-level method based on reinforcement learning is proposed.

* The reinforcement learning approach is developed to effectively handle the complexity of the model and provide good solutions.

* Numerical experiments are conducted to demonstrate the effectiveness of both the proposed model and the resolution approach.

### 3. Problem description and modeling

#### 3.1. Problem description

This paper addresses the ILSPS problem within the FFL scope for a multi-products processing system powered by different energy sources: grid power, solar power generated by onsite PV panels, and an ESS (Fig. 1). To simplify the planning framework, the availability of solar power is considered here as a certain parameter over the planning horizon. This means that the planning process assumes constant and predictable solar energy generation over all micro-periods. The system comprises multiple consecutive processing stages, where each process stage has one or more non-identical parallel machines. The scheduling horizon is finite and divided into *T* macro-periods, which are subdivided into a finite number of micro-periods, denoted as *F*. To ensure that all product demands are met within the planning horizon, there is no micro-period delay between different processes. This means that a product produced at process *p* becomes immediately available for production at the next process. The manufacturing system also involves buffering routines, where products can be stored at process *p* and made available for the next process in the next micro-period. Buffer deployment is crucial for reducing product demands during peak energy periods, considering the application of TOU pricing schemes.

In this study, both sequence-dependent setup times and costs are considered. The cost and time necessary to set up for product *j* after product *i* on machine *m* are assumed to be different from those needed to set up for product *j* if it is produced before product *i* on the same machine. Furthermore, these setups are supposed to be cost and time

<figure><img src="image_2.png" /><figcaption>Fig. 1. The flexible flow line with energy sources.</figcaption></figure>
consumption in manufacturing systems. In this regard, Shrouf et al. (2014) developed a mathematical model to minimize energy consumption costs for a single machine production scheduling considering variable energy prices. Masmoudi et al. (2017) proposed two heuristics to solve a flow shop scheduling problem with energy consideration. Similarly, Rodoplu et al. (2020) developed a hybrid scheduling model for a single-item flow shop production scheduling, aiming to reduce energy consumption. Li et al. (2018) investigated a hybrid flow shop scheduling problem with setup energy consumptions using an energy-aware multi-objective optimization algorithm. Alternatively, making production scheduling sustainable can also be achieved using onsite renewable energy. Indeed, the use of renewable energy resources not only provides a solution to meet peak electricity demands but also helps to limit GHG emissions. The integration of renewable energy sources into production systems has been addressed several times in recent literature. Golari et al. (2017) formulated a lot sizing problem that integrates a renewable energy source as an optimization programming model. Wang et al. (2020) investigated a production scheduling problem that incorporates onsite renewable energy, the main grid, and an energy storage system (ESS). They elaborated a two-stage multi-objective stochastic model for flow shops with sequence-dependent setups to simultaneously minimize total weighted completion time and energy costs. Trevino-Martinez et al. (2022) introduced an optimization framework that incorporates renewable energy sources and an ESS into a job shop scheduling problem. Duarte et al. (2020) addressed a multi-process production planning problem that incorporates intermittent renewable energy sources and an ESS. They proposed a mathematical programming model and an algorithm to optimize production scheduling and energy management system decisions by considering grid power, onsite renewable energy sources, an ESS, and intermediate buffers. Their results show that the integration of these policies ensures a flexible manufacturing process. In this context, Peinado-Guerrero and Villalobos (2022) considered the use of an ESS and intermediate buffers in a manufacturing facility to reduce energy consumption during peak demand periods. Sun et al. (2014) developed a mathematical model that aims to maintain a good tradeoff between the economic costs of intermediate buffer systems and the potential cost savings generated from TOU pricing.

In summary, this study focuses on addressing the ILSPS problem in an FFL (Flexible Flow Line) environment while considering energy efficiency. Compared to previous research, this study makes the following main contributions:

* A novel mixed-integer linear model is developed to accurately represent the ILSPS problem in the FFL environment under a Time-of-Use (TOU) pricing scheme. The mathematical model is specifically adapted to incorporate an onsite photovolatic solar power supply.

* To achieve a balance between the costs associated with buffers, Energy Storage Systems (ESS), Renewable Energy Sources (RES), and TOU pricing, a two-level method based on reinforcement learning is proposed.

* The reinforcement learning approach is developed to effectively handle the complexity of the model and provide good solutions.

* Numerical experiments are conducted to demonstrate the effectiveness of both the proposed model and the resolution approach.

# 3. Problem description and modeling

## 3.1. Problem description

This paper addresses the ILSPS problem within the FFL scope for a multi-products processing system powered by different energy sources: grid power, solar power generated by onsite PV panels, and an ESS (Fig. 1). To simplify the planning framework, the availability of solar power is considered here as a certain parameter over the planning horizon. This means that the planning process assumes constant and predictable solar energy generation over all micro-periods. The system comprises multiple consecutive processing stages, where each process stage has one or more non-identical parallel machines. The scheduling horizon is finite and divided into *T* macro-periods, which are subdivided into a finite number of micro-periods, denoted as *F*. To ensure that all product demands are met within the planning horizon, there is no micro-period delay between different processes. This means that a product produced at process *p* becomes immediately available for production at the next process. The manufacturing system also involves buffering routines, where products can be stored at process *p* and made available for the next process in the next micro-period. Buffer deployment is crucial for reducing product demands during peak energy periods, considering the application of TOU pricing schemes.

In this study, both sequence-dependent setup times and costs are considered. The cost and time necessary to set up for product *j* after product *i* on machine *m* are assumed to be different from those needed to set up for product *j* if it is produced before product *i* on the same machine. Furthermore, these setups are supposed to be cost and time

<figure><img src="image_3.png" /><figcaption>Fig. 1. The flexible flow line with energy sources.</figcaption></figure>
dependent on the machine. It is supposed that: 1) for each beginning of the scheduling horizon, machines are set up for a specific product and this setup can be maintained if there is no other product to process; 2) each machine *m* can produce at most two different products in one micro-period; 3) each machine *m* is limited in the number of products it can produce within a single micro-period. The machine's capacity during a micro-period is proportional to its processing time and the duration of that micro-period.

## 3.2. Mathematical model

In this section, we present a Mixed Integer Linear Program (MILP) to address the IILSPS problem FFL, taking into account energy constraints. The primary objective is to minimize both production and energy costs. To achieve this goal, two crucial decisions are optimized simultaneously. Firstly, we determine the best production lot sizes required to meet the demand. Secondly, we identify the most efficient production sequence that minimizes setup costs, holding costs, and energy costs. Table 1 depicts the parameters including sets and indexes for the scheduling horizon, machines, products, and processes. Table 2 defines machine capacities, setups, costs related to production and energy consumption, and energy capacities. Finally, the decision variables, including production, buffers, setup, and energy variables are depicted in Table 3.

## 3.3. Objective function

The objective function is divided into two parts, the first one corresponding to the production costs and the second one to the energy costs. The production costs, denoted as $C_{sched}$(Eq. (1a)), consist of setup costs and holding costs. In $C_{sched}$, no associated cost with an empty setup ($y_{m,p,t_f}^{i,i}$) is assumed. The energy costs, denoted as $C_{energy}$(Eq. (1b)), take into account the costs associated with energy consumption from various sources, including grid power under the TOU pricing scheme, onsite solar power, and ESS. The objective function defined in Eq. (1c) aims to minimize the total sum of production and energy costs of the manufacturing system during the scheduling horizon.

$$C_{sched} = \sum_{f \in F} \sum_{t \in T} \sum_{p \in P} \sum_{m \in M_p} \sum_{i \in N} \sum_{j \in N \setminus \{i\}} \left( C_{m,p}^{i,j} \cdot y_{m,p,t_f}^{i,j} \right) + \sum_{t \in T} \sum_{p \in P} \sum_{i \in N} \left( C_{i,p,t}^{\text{buffer}} \cdot Q_{p,t_f}^{i} \right) \quad (1a)$$

$$C_{energy} = \sum_{f \in F} \sum_{t \in T} \left( g_t \cdot ec_{t_f} + r_{t_f} \cdot er_{t_f} + C^+ \cdot r_{t_f}^+ + C^- \cdot r_{t_f}^- \right) \quad (1b)$$

$$C = \min C_{energy} + C_{sched} \quad (1c)$$

To achieve the above objective, the following two types of constraints need to be satisfied:

### 3.3.1. Lot sizing and production scheduling constraints

In this part of model, we consider constraints related to lot sizing and production scheduling features, such as demand meeting, process and buffer flow, cycle time, buffering capacities and finally setup constraints. An illustrative representation of the production flow over processes and micro-periods is depicted in Fig. 2. Specifically, the quantity

<table><caption>**Table 1**
Sets and indexes.</caption><thead><tr><th>Symbol</th><th>Description</th></tr></thead><tbody><tr><td>T<sub>t</sub></td><td>Set and index of macro-periods in production horizon.</td></tr><tr><td>F<sub>f</sub></td><td>Set and index of micro-periods</td></tr><tr><td>P<sub>p</sub></td><td>Set and index of processes.</td></tr><tr><td>M<sub>p,m</sub></td><td>Set and index of machines in given process p</td></tr><tr><td>N<sub>i,j</sub></td><td>Set and indexes of products.</td></tr></tbody></table>

<table><caption>**Table 2**
Parameters.</caption><thead><tr><th>Symbol</th><th>Description</th></tr></thead><tbody><tr><td>D<sub>i,t</sub></td><td>Demand for product <i>i</i> at the end of macro-period <i>t</i> (<i>f</i> = |<i>F</i>|).</td></tr><tr><td>r<sup>i</sup><sub>m,p</sub></td><td>Processing time of product <i>i</i> with machine <i>m</i> of process <i>p</i>.</td></tr><tr><td>s<sup>i</sup><sub>m,p</sub></td><td>Setup time from product <i>i</i> to product <i>j</i> with machine <i>m</i> of process <i>p</i>.</td></tr><tr><td>C<sup>i,j</sup><sub>m,p</sub></td><td>Setup cost from product <i>i</i> to product <i>j</i> with machine <i>m</i> of process <i>p</i>.</td></tr><tr><td>C<sup>buffer</sup><sub>i,p,t</sub></td><td>Unit cost per item buffered in process <i>p</i> at macro-period <i>t</i>.</td></tr><tr><td>d</td><td>Micro-period duration.</td></tr><tr><td>M</td><td>A big real number.</td></tr><tr><td>N<sub>p</sub></td><td>buffer capacity for process <i>p</i>.</td></tr><tr><td>s<sub>max</sub>/s<sub>min</sub></td><td>Maximum / Minimum storage capacity for the ESS.</td></tr><tr><td>s<sub>0</sub></td><td>Initial energy level of the ESS.</td></tr><tr><td>R<sup>i</sup>/R</td><td>Charging and discharging capacities of the ESS.</td></tr><tr><td>C<sup>i</sup>/C<sup>-</sup></td><td>ESS charging and discharging cost.</td></tr><tr><td>E<sup>on</sup><sub>i,m,p</sub></td><td>Required power to produce one unit of product <i>i</i> in machine <i>m</i> of process <i>p</i>.</td></tr><tr><td>E<sup>off</sup><sub>m,p</sub></td><td>Consumed power in one time unit when machine <i>m</i> of process <i>p</i> is in setup mode.</td></tr><tr><td>G<sub>t_f</sub></td><td>Available renewable energy at micro-period <i>f</i> from macro-period <i>t</i>.</td></tr><tr><td>g<sub>t</sub></td><td>MWh Conventional energy price at macro-period <i>t</i>.</td></tr><tr><td>ĝ<sub>t</sub></td><td>Nominal value of conventional energy price at macro-period <i>t</i>.</td></tr><tr><td>r<sub>t_f</sub></td><td>MWh PV power cost at micro-period <i>f</i> from macro-period <i>t</i>.</td></tr><tr><td>η<sup>i</sup>/η<sup>-</sup></td><td>Charging/discharging efficiency of the ESS.</td></tr><tr><td>α</td><td>The learning rate</td></tr><tr><td>γ</td><td>The discount factor</td></tr></tbody></table>

<table><caption>**Table 3**
Variables.</caption><thead><tr><th>Symbol</th><th>Description</th></tr></thead><tbody><tr><td>x<sup>i</sup><sub>m,p,t_f</sub></td><td>Quantity of product <i>i</i> produced by machine <i>m</i> of process <i>p</i> at micro-period <i>f</i> of macro-period <i>t</i>.</td></tr><tr><td>XT<sup>i</sup><sub>p,t_f</sub></td><td>Amount of product <i>i</i> produced in process <i>p</i> at micro-period <i>f</i> of macro-period <i>t</i> and transferred to next process.</td></tr><tr><td>XS<sup>i</sup><sub>p,t_f</sub></td><td>Amount of product <i>i</i> produced in process <i>p</i> at micro-period <i>f</i> of macro-period <i>t</i> and waiting for production at next process.</td></tr><tr><td>Q<sup>i</sup><sub>t_f</sub></td><td>Amount of product <i>i</i> buffered in process <i>p</i> in micro-period <i>f</i>.</td></tr><tr><td>QT<sup>i</sup><sub>p,t_f</sub></td><td>Amount of product <i>i</i> buffered in process <i>p</i> and progress to next process <i>p</i>+1 at micro-period <i>f</i> + 1.</td></tr><tr><td>QS<sup>i</sup><sub>p,t_f</sub></td><td>Amount of product <i>i</i> buffered in process <i>p</i> and still buffered in process <i>p</i> at the next micro-period <i>f</i> + 1.</td></tr><tr><td>y<sup>i,j</sup><sub>m,p,t_f</sub></td><td>1, if there is a setup from product <i>i</i> to product <i>j</i> in machine <i>m</i> of process <i>p</i> in the micro-period <i>f</i> of macro-period <i>t</i>, otherwise = 0.</td></tr><tr><td>w<sup>i</sup><sub>m,p,t_f</sub></td><td>1, if machine <i>m</i> of process <i>p</i> is setup for product <i>i</i> at the beginning of micro-period <i>f</i>, otherwise = 0.</td></tr><tr><td>ESS<sub>t_f</sub></td><td>ESS energy level at micro-period <i>f</i> of macro-period <i>t</i>.</td></tr><tr><td>r<sup>i</sup><sub>t_f</sub>/r<sup>-</sup><sub>t_f</sub></td><td>Amount of power to charge /discharge the ESS at <i>f</i> of macro-period <i>t</i>.</td></tr><tr><td>er<sub>t_f</sub></td><td>Consumed energy from onsite solar panels at micro-period <i>f</i> of macro-period <i>t</i>.</td></tr><tr><td>ec<sub>t_f</sub></td><td>Grid conventional energy consumed at micro-period <i>f</i> of macro-period <i>t</i>.</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Process and buffer flow.</figcaption></figure>
dependent on the machine. It is supposed that: 1) for each beginning of the scheduling horizon, machines are set up for a specific product and this setup can be maintained if there is no other product to process; 2) each machine *m* can produce at most two different products in one micro-period; 3) each machine *m* is limited in the number of products it can produce within a single micro-period. The machine's capacity during a micro-period is proportional to its processing time and the duration of that micro-period.

## 3.2. Mathematical model

In this section, we present a Mixed Integer Linear Program (MILP) to address the IILSPS problem FFL, taking into account energy constraints. The primary objective is to minimize both production and energy costs. To achieve this goal, two crucial decisions are optimized simultaneously. Firstly, we determine the best production lot sizes required to meet the demand. Secondly, we identify the most efficient production sequence that minimizes setup costs, holding costs, and energy costs. Table 1 depicts the parameters including sets and indexes for the scheduling horizon, machines, products, and processes. Table 2 defines machine capacities, setups, costs related to production and energy consumption, and energy capacities. Finally, the decision variables, including production, buffers, setup, and energy variables are depicted in Table 3.

## 3.3. Objective function

The objective function is divided into two parts, the first one corresponding to the production costs and the second one to the energy costs. The production costs, denoted as $C_{sched}$(Eq. (1a)), consist of setup costs and holding costs. In $C_{sched}$, no associated cost with an empty setup ($y_{m,p,t_f}^{i,i}$) is assumed. The energy costs, denoted as $C_{energy}$(Eq. (1b)), take into account the costs associated with energy consumption from various sources, including grid power under the TOU pricing scheme, onsite solar power, and ESS. The objective function defined in Eq. (1c) aims to minimize the total sum of production and energy costs of the manufacturing system during the scheduling horizon.

$$C_{sched} = \sum_{f \in F} \sum_{t \in T} \sum_{p \in P} \sum_{m \in M_p} \sum_{i \in N} \sum_{j \in N \setminus \{i\}} \left( C_{m,p}^{i,j} \cdot y_{m,p,t_f}^{i,j} \right) + \sum_{t \in T} \sum_{p \in P} \sum_{i \in N} \left( C_{i,p,t}^{\text{buffer}} \cdot Q_{p,t_f}^{i} \right) \quad (1a)$$

$$C_{energy} = \sum_{f \in F} \sum_{t \in T} \left( g_t \cdot ec_{t_f} + r_{t_f} \cdot er_{t_f} + C^+ \cdot r_{t_f}^+ + C^- \cdot r_{t_f}^- \right) \quad (1b)$$

$$C = \min C_{energy} + C_{sched} \quad (1c)$$

To achieve the above objective, the following two types of constraints need to be satisfied:

### 3.3.1. Lot sizing and production scheduling constraints

In this part of model, we consider constraints related to lot sizing and production scheduling features, such as demand meeting, process and buffer flow, cycle time, buffering capacities and finally setup constraints. An illustrative representation of the production flow over processes and micro-periods is depicted in Fig. 2. Specifically, the quantity

<table><caption>**Table 1**
Sets and indexes.</caption><thead><tr><th>Symbol</th><th>Description</th></tr></thead><tbody><tr><td>T<sub>t</sub></td><td>Set and index of macro-periods in production horizon.</td></tr><tr><td>F<sub>f</sub></td><td>Set and index of micro-periods</td></tr><tr><td>P<sub>p</sub></td><td>Set and index of processes.</td></tr><tr><td>M<sub>p,m</sub></td><td>Set and index of machines in given process p</td></tr><tr><td>N<sub>i,j</sub></td><td>Set and indexes of products.</td></tr></tbody></table>

<table><caption>**Table 2**
Parameters.</caption><thead><tr><th>Symbol</th><th>Description</th></tr></thead><tbody><tr><td>D<sub>i,t</sub></td><td>Demand for product i at the end of macro-period t (f = |F|).</td></tr><tr><td>r<sup>i</sup><sub>m,p</sub></td><td>Processing time of product i with machine m of process p.</td></tr><tr><td>s<sup>i</sup><sub>m,p</sub></td><td>Setup time from product i to product j with machine m of process p.</td></tr><tr><td>C<sup>i,j</sup><sub>m,p</sub></td><td>Setup cost from product i to product j with machine m of process p.</td></tr><tr><td>C<sup>buffer</sup><sub>i,p,t</sub></td><td>Unit cost per item buffered in process p at macro-period t.</td></tr><tr><td>d</td><td>Micro-period duration.</td></tr><tr><td>M</td><td>A big real number.</td></tr><tr><td>N<sub>p</sub></td><td>buffer capacity for process p.</td></tr><tr><td>s<sub>max</sub>/s<sub>min</sub></td><td>Maximum / Minimum storage capacity for the ESS.</td></tr><tr><td>s<sub>0</sub></td><td>Initial energy level of the ESS.</td></tr><tr><td>R<sup>i</sup>/R</td><td>Charging and discharging capacities of the ESS.</td></tr><tr><td>C<sup>i</sup>/C<sup>-</sup></td><td>ESS charging and discharging cost.</td></tr><tr><td>E<sup>on</sup><sub>i,m,p</sub></td><td>Required power to produce one unit of product i in machine m of process p.</td></tr><tr><td>E<sup>off</sup><sub>m,p</sub></td><td>Consumed power in one time unit when machine m of process p is in setup mode.</td></tr><tr><td>G<sub>t_f</sub></td><td>Available renewable energy at micro-period f from macro-period t.</td></tr><tr><td>g<sub>t</sub></td><td>MWh Conventional energy price at macro-period t.</td></tr><tr><td>ĝ<sub>t</sub></td><td>Nominal value of conventional energy price at macro-period t.</td></tr><tr><td>r<sub>t_f</sub></td><td>MWh PV power cost at micro-period f from macro-period t.</td></tr><tr><td>η<sup>i</sup>/η<sup>-</sup></td><td>Charging/discharging efficiency of the ESS.</td></tr><tr><td>α</td><td>The learning rate</td></tr><tr><td>γ</td><td>The discount factor</td></tr></tbody></table>

<table><caption>**Table 3**
Variables.</caption><thead><tr><th>Symbol</th><th>Description</th></tr></thead><tbody><tr><td>x<sup>i</sup><sub>m,p,t_f</sub></td><td>Quantity of product i produced by machine m of process p at micro-period f of macro-period t.</td></tr><tr><td>XT<sup>i</sup><sub>p,t_f</sub></td><td>Amount of product i produced in process p at micro-period f of macro-period t and transferred to next process.</td></tr><tr><td>XS<sup>i</sup><sub>p,t_f</sub></td><td>Amount of product i produced in process p at micro-period f of macro-period t and waiting for production at next process.</td></tr><tr><td>Q<sup>i</sup><sub>p,t_f</sub></td><td>Amount of product i buffered in process p in micro-period f.</td></tr><tr><td>QT<sup>i</sup><sub>p,t_f</sub></td><td>Amount of product i buffered in process p and progress to next process p + 1 at micro-period f + 1.</td></tr><tr><td>QS<sup>i</sup><sub>p,t_f</sub></td><td>Amount of product i buffered in process p and still buffered in process p at the next micro-period f + 1.</td></tr><tr><td>y<sup>i,j</sup><sub>m,p,t_f</sub></td><td>1, if there is a setup from product i to product j in machine m of process p in the micro-period f of macro-period t, otherwise = 0.</td></tr><tr><td>w<sup>i</sup><sub>m,p,t_f</sub></td><td>1, if machine m of process p is setup for product i at the beginning of micro-period f, otherwise = 0.</td></tr><tr><td>ESS<sub>t_f</sub></td><td>ESS energy level at micro-period f of macro-period t.</td></tr><tr><td>r<sup>i</sup><sub>t_f</sub>/r<sup>-</sup><sub>t_f</sub></td><td>Amount of power to charge /discharge the ESS at f of macro-period t.</td></tr><tr><td>er<sub>t_f</sub></td><td>Consumed energy from onsite solar panels at micro-period f of macro-period t.</td></tr><tr><td>ec<sub>t_f</sub></td><td>Grid conventional energy consumed at micro-period f of macro-period t.</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Process and buffer flow.</figcaption></figure>
of product i processed in process p during micro-period f ($\sum_{m \in M_p} x_{m,p,t_f}^i$) can follow one of two paths; it can either be forwarded to process p+1 during micro-period f+1 ($XT_{p,t_f}^i$), or it can be held in buffer of process p, awaiting further processing in subsequent micro-periods ($XS_{p,t_f}^i$). The buffer flow follows the same logic as the production flow using variables $QT_{p,t_f}^i$ and $QS_{p,t_f}^i$. The production process described above is represented in constraints (2a)-(2n).

$$ \sum_{f \in F} \sum_{m \in M_p} x_{m,|P|,t_f}^i + Q_{|P|,t-1|_F}^i - Q_{|P|,|_F}^i \geq D_{i,t}, \forall i \in N, \forall t \in T \quad (2a) $$

$$ \sum_{m \in M_p} x_{m,p,t_f}^i = XT_{p,t_f}^i + XS_{p,t_f}^i, \quad \forall i \in N, \forall p \in P, \forall t \in T, \forall f \in F \quad (2b) $$

$$ Q_{p,t_f}^i = QT_{p,t_f}^i + QS_{p,t_f}^i, \quad \forall i \in N, \forall p \in P, \forall t \in T, \forall f \in F \quad (2c) $$

$$ \sum_{m \in M_p} x_{m,p,t_f}^i = XT_{p-1,t_f-1}^i + QT_{p-1,t_f-1}^i, \quad \forall i \in N, \forall p \in \{2, \dots, |P|\}, \forall t \in T, \forall f \in F \quad (2d) $$

$$ Q_{p,t_f}^i = QS_{p,t_f-1}^i + XS_{p,t_f}^i, \quad \forall p \in P, \forall i \in N, \forall t \in T, \forall f \in F \quad (2e) $$

$$ Q_{p,t_f}^i = Q_{p,t-1|_F}^i + XS_{p,t_f}^i, \quad \forall p \in P, \forall i \in N, \forall t \in \{2, \dots, |T|\} \quad (2f) $$

$$ \tau_{m,p}^i x_{m,p,t_f}^i + \sum_{j \in N} (s_{m,p}^{ij} x_{m,p,t_f}^{j,j} + \tau_{m,p}^i x_{m,p,t_f}^j) \leq d, \forall i \in N, \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (2g) $$

$$ \sum_{i \in N} Q_{p,t_f}^i \leq N_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (2h) $$

$$ x_{m,p,t_f}^i \leq M \left( w_{m,p,t_f}^i + \sum_{j \in N} y_{m,p,t_f}^{j,i} \right), \forall i \in N, \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F, \quad (2i) $$

$$ \sum_{i \in N} \sum_{j \in N} y_{m,p,t_f}^{j,j} \leq 1, \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (2j) $$

$$ \sum_{i \in N} w_{m,p,t_f}^i = 1, \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (2k) $$

$$ w_{m,p,t_f}^i + \sum_{j \in N} y_{m,p,t_f}^{j,j} + \sum_{j \in N} y_{m,p,t_f+1}^{j,i} \leq 1 + w_{m,p,t_f+1}^i, \quad \forall i \in N, \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (2l) $$

$$ x_{m,p,t_f}^i, XT_{p,t_f}^i, XS_{p,t_f}^i, Q_{p,t_f}^i, QT_{p,t_f}^i, QS_{p,t_f}^i, I_{t_f}^i \geq 0, \forall i \in N, \forall p \in P, \forall m \in M_p, \forall t \in T, \forall f \in F \quad (2m) $$

$$ y_{m,p,t_f}^{j,j}, w_{m,p,t_f}^i \in \{0, 1\}, \forall i \in N, \forall p \in P, \forall m \in M_p, \forall t \in T, \forall f \in F \quad (2n) $$

In this model, constraint (2a) ensures that all final demands are met by the end of the scheduling horizon. Constraint (2b) models the production of product i on machine m in process p during micro-period f. Then, $x_{m,p,t_f}^i$ is partitioned into segments: one is earmarked for storage, while the other will progress to next process in the upcoming micro-period. Constraint (2c) employs a similar rationale to (2b) and reflects the buffering level of the process p at micro-period f. Constraint (2d) maintains the production flow balance in process p at micro-period f, while constraints (2e) and (2f) ensure the flow balance of buffering in each process p. However, they apply over distinct timeframes.

Constraint (2e) focuses on maintaining balance within the same macro-period, whereas (2f) ensures equilibrium across two consecutive macro-periods. Constraint (2g) models the production cycle time capacity for each machine and its setup times within a micro-period. In this sense, it states that a given machine m is limited to producing a maximum of two distinct items within a single micro-period f. Constraint (2h) defines the buffering capacity of each process p. Constraint (2i) ensures that the appropriate setup of a product i is performed before production, while constraint (2j) specifies that in a single micro-period i, only one changeover is allowed on a machine m. Constraint (2k) ensures that only one setup state can be defined at the beginning of each period. It also guarantees that each machine m has an initial setup configuration ($w_{m,p,t=1|_{f-1}}^i = 1$). Constraint (2l) relates the changeover and setup state variables. Thus, the setup state of the machine switches immediately from i to j when a changeover occurs from product i to product j. This transition happens in three possible cases: at the beginning of the micro-period, during the micro-period, or at the end of the micro-period. Fig. 3 provides an overview of how the changeover and the setup state variables interact and change in response to these three cases. Finally, non-negativity and binary requirements are stated in (2m) and (2n). Overall, these constraints collectively aim to ensure efficient production planning and scheduling, while balancing the demand for final products with the capacity of the production system.

### 3.3.2. Energy constraints

The integration of renewable energy sources into an electrical distribution network may prove insufficient without the use of an ESS. Indeed, its purpose is to facilitate the integration of PV power into the manufacturing electrical network and to provide power during subsequent micro-periods. Charging and discharging transactions of the ESS, as well as the dynamic of the energy supply system are expressed in (3a)-(3h).

$$ S_{min} \leq ESS_{t_f} \leq S_{max}, \quad \forall t \in T, \forall f \in F \quad (3a) $$

$$ ESS_{t_f} = ESS_{t_f-1} + \eta^+, r_{t_f}^+ - r_{t_f} - \frac{1}{\eta^-}, \quad \forall t \in T, \forall f \in F \quad (3b) $$

$$ ESS_{t_f} = ESS_{t_{f-1}|_F} + \eta^+, r_{t_f}^+ - r_{t_f} - \frac{1}{\eta^-}, \quad \forall t \in \{2, \dots, |T|\} \quad (3c) $$

$$ r_{t_f}^+ \leq R^+, \quad \forall t \in T, \forall f \in F \quad (3d) $$

$$ r_{t_f}^- \leq R^-, \quad \forall t \in T, \forall f \in F \quad (3e) $$

$$ \sum_{i \in N} \sum_{p \in P} \sum_{m \in M_p} \left( E_{m,p}^{on} x_{m,p,t_f}^i \tau_{m,p}^i + \sum_{j=1}^{N} y_{m,p,t_f+1}^{j,j} s_{m,p}^{j,j}, E_{m,p}^{off} \right) = ec_{t_f} + er_{t_f} + r_{t_f}^-, \quad \forall t \in T, \forall f \in F \quad (3f) $$

<figure><img src="image_4.png" /><figcaption>Fig. 3. Relations between binary variables according to changeover.</figcaption></figure>
of product i processed in process p during micro-period f ($\sum_{m \in M_p} x_{m,p,t_f}^i$) can follow one of two paths; it can either be forwarded to process p+1 during micro-period f+1 ($XT_{p,t_f}^i$), or it can be held in buffer of process p, awaiting further processing in subsequent micro-periods ($XS_{p,t_f}^i$). The buffer flow follows the same logic as the production flow using variables $QT_{p,t_f}^i$ and $QS_{p,t_f}^i$. The production process described above is represented in constraints (2a)-(2n).

$$ \sum_{f \in F} \sum_{m \in M_p} x_{m,|P|,t_f}^i + Q_{|P|,t-1|_F}^i - Q_{|P|,|_F}^i \geq D_{i,t}, \forall i \in N, \forall t \in T \quad (2a) $$

$$ \sum_{m \in M_p} x_{m,p,t_f}^i = XT_{p,t_f}^i + XS_{p,t_f}^i, \quad \forall i \in N, \forall p \in P, \forall t \in T, \forall f \in F \quad (2b) $$

$$ Q_{p,t_f}^i = QT_{p,t_f}^i + QS_{p,t_f}^i, \quad \forall i \in N, \forall p \in P, \forall t \in T, \forall f \in F \quad (2c) $$

$$ \sum_{m \in M_p} x_{m,p,t_f}^i = XT_{p-1,t_f-1}^i + QT_{p-1,t_f-1}^i, \quad \forall i \in N, \forall p \in \{2, \dots, |P|\}, \forall t \in T, \forall f \in F \quad (2d) $$

$$ Q_{p,t_f}^i = QS_{p,t_f-1}^i + XS_{p,t_f}^i, \quad \forall p \in P, \forall i \in N, \forall t \in T, \forall f \in F \quad (2e) $$

$$ Q_{p,t_f}^i = Q_{p,t-1|_F}^i + XS_{p,t_f}^i, \quad \forall p \in P, \forall i \in N, \forall t \in \{2, \dots, |T|\} \quad (2f) $$

$$ \tau_{m,p}^i x_{m,p,t_f}^i + \sum_{j \in N} (s_{m,p}^{ij} x_{m,p,t_f}^{j,j} + \tau_{m,p}^i x_{m,p,t_f}^j) \leq d, \forall i \in N, \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (2g) $$

$$ \sum_{i \in N} Q_{p,t_f}^i \leq N_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (2h) $$

$$ x_{m,p,t_f}^i \leq M \cdot (w_{m,p,t_f}^i + \sum_{j \in N} y_{m,p,t_f}^{j,i}), \quad \forall i \in N, \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F, \quad (2i) $$

$$ \sum_{i \in N} \sum_{j \in N} y_{m,p,t_f}^{j,j} \leq 1, \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (2j) $$

$$ \sum_{i \in N} w_{m,p,t_f}^i = 1, \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (2k) $$

$$ w_{m,p,t_f}^i + \sum_{j \in N} y_{m,p,t_f}^{j,j} + \sum_{j \in N} y_{m,p,t_f+1}^{j,i} \leq 1 + w_{m,p,t_f+1}, \quad \forall i \in N, \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (2l) $$

$$ x_{m,p,t_f}^i, XT_{p,t_f}^i, XS_{p,t_f}^i, Q_{p,t_f}^i, QT_{p,t_f}^i, QS_{p,t_f}^i, I_{t_f}^i \geq 0, \forall i \in N, \forall p \in P, \forall m \in M_p, \forall t \in T, \forall f \in F \quad (2m) $$

$$ y_{m,p,t_f}^{j,j}, w_{m,p,t_f}^i \in \{0, 1\}, \forall i \in N, \forall p \in P, \forall m \in M_p, \forall t \in T, \forall f \in F \quad (2n) $$

In this model, constraint (2a) ensures that all final demands are met by the end of the scheduling horizon. Constraint (2b) models the production of product i on machine m in process p during micro-period f. Then, $x_{m,p,t_f}^i$ is partitioned into segments: one is earmarked for storage, while the other will progress to next process in the upcoming micro-period. Constraint (2c) employs a similar rationale to (2b) and reflects the buffering level of the process p at micro-period f. Constraint (2d) maintains the production flow balance in process p at micro-period f, while constraints (2e) and (2f) ensure the flow balance of buffering in each process p. However, they apply over distinct timeframes.

Constraint (2e) focuses on maintaining balance within the same macro-period, whereas (2f) ensures equilibrium across two consecutive macro-periods. Constraint (2g) models the production cycle time capacity for each machine and its setup times within a micro-period. In this sense, it states that a given machine m is limited to producing a maximum of two distinct items within a single micro-period f. Constraint (2h) defines the buffering capacity of each process p. Constraint (2i) ensures that the appropriate setup of a product i is performed before production, while constraint (2j) specifies that in a single micro-period i, only one changeover is allowed on a machine m. Constraint (2k) ensures that only one setup state can be defined at the beginning of each period. It also guarantees that each machine m has an initial setup configuration ($w_{m,p,t=1|_F}^i = 1$). Constraint (2l) relates the changeover and setup state variables. Thus, the setup state of the machine switches immediately from i to j when a changeover occurs from product i to product j. This transition happens in three possible cases: at the beginning of the micro-period, during the micro-period, or at the end of the micro-period. Fig. 3 provides an overview of how the changeover and the setup state variables interact and change in response to these three cases. Finally, non-negativity and binary requirements are stated in (2m) and (2n). Overall, these constraints collectively aim to ensure efficient production planning and scheduling, while balancing the demand for final products with the capacity of the production system.

### 3.3.2. Energy constraints

The integration of renewable energy sources into an electrical distribution network may prove insufficient without the use of an ESS. Indeed, its purpose is to facilitate the integration of PV power into the manufacturing electrical network and to provide power during subsequent micro-periods. Charging and discharging transactions of the ESS, as well as the dynamic of the energy supply system are expressed in (3a)-(3h).

$$ S_{min} \leq ESS_{t_f} \leq S_{max}, \quad \forall t \in T, \forall f \in F \quad (3a) $$

$$ ESS_{t_f} = ESS_{t_f-1} + \eta^+, r_{t_f}^+ - r_{t_f} - \frac{1}{\eta^-}, \quad \forall t \in T, \forall f \in F \quad (3b) $$

$$ ESS_{t_f} = ESS_{t-1|_F} + \eta^+, r_{t_f}^+ - r_{t_f} - \frac{1}{\eta^-}, \quad \forall t \in \{2, \dots, |T|\} \quad (3c) $$

$$ r_{t_f}^+ \leq R^+, \quad \forall t \in T, \forall f \in F \quad (3d) $$

$$ r_{t_f}^- \leq R^-, \quad \forall t \in T, \forall f \in F \quad (3e) $$

$$ \sum_{i \in N} \sum_{p \in P} \sum_{m \in M_p} \left( E_{m,p}^{on} x_{m,p,t_f}^i \tau_{m,p}^i + \sum_{j=1}^{N} y_{m,p,t_f+1}^{j,j}, s_{m,p}^{ij}, E_{m,p}^{off} \right) = ec_{t_f} + er_{t_f} + r_{t_f}^-, \quad \forall t \in T, \forall f \in F \quad (3f) $$

<figure><img src="image_5.png" /><figcaption>Fig. 3. Relations between binary variables according to changeover.</figcaption></figure>
$$er_{t_f} + r_{t_f}^{+} \le G_{t_f}, \forall t \in T, \forall f \in F \quad (3 \text{ g})$$

$$ESS_{t_f}, r_{t_f}^{+}, r_{t_f}^{-}, ec_{t_f}, er_{t_f} \ge 0, \forall t \in T, \forall f \in F \quad (3 \text{ h})$$

Constraint (3a) ensures that the level of charge in the ESS remains within specified lower and upper bounds. Constraints (3b) and (3c) update the power level of the ESS in each micro-period. Constraints (3d) and (3e) express the charge and discharge capacities of the ESS, respectively. Constraint (3f) models the energy requirements of the manufacturing system and specifies the sources of energy, including PV solar power, grid energy under TOU policies, and ESS energy. Constraint (3g) shows how the generated on-site renewable energy is distributed between the manufacturing process and the ESS. Finally, a non-negativity requirement is indicated in constraint (3h).

## 4. Solution approach: A two-level method

The formulated mathematical model aims to optimize the integrated lot sizing and production scheduling, taking into account several features, including energy conditions, TOU prices, production and inventory capacities, satisfying final demand, and changeover metrics on production machines. These various constraints impose limits on feasible solutions, which can significantly increase the computational time required to reach the optimum solution. In addition, the optimal solution must balance energy and production strategies at each macro-period. Furthermore, the integration of buffers, RES, and ESS into production has a significant impact on overall production cost, which, in turn, affects lot sizes and production schedules. To meet these challenges and achieve a good solution within a reasonable timeframe, we propose a novel two-level framework, as summarized in Fig. 4.

The first level optimization (FLO) is designed to handle the integrated lot sizing and production scheduling. It focuses on optimizing the allocation of production quantities and schedules while taking into account the constraints mentioned above. The objective is to determine a good production plan that optimizes costs and meets demands. However, the challenge in the FLO resides in determining the trade-off between TOU energy prices, RES, ESS, and buffer use, when solving the proposed model. All these factors have complex interactions, and striking the appropriate compromise may be challenging. To simplify the problem, we assume that there is no buffer ($N_p = 0$) and no TOU policy ($g_t = \hat{g}_t$). Using this type of approximation may make it easier to

<figure><img src="image_5.png" /><figcaption>Fig. 4. Two-level approach framework.</figcaption></figure>

solve the MP1, within the FLO framework. After obtaining the initial production schedule, we introduce the second level optimization (SLO) which is devoted to optimizing the balance between peak electricity prices, buffers, ESS, and RES. It looks for the economic strategy to utilize energy resources, taking advantage of peak electricity prices and ensuring efficient management of buffers, ESS, and RES. Once, the best trade-off is determined, the FLO is then resolved to check the schedules obtained in the first level and adjust them as necessary. The proposed resolution framework combines these two levels of optimization to overcome problem complexity. The following subsections describe the development of this framework.

### 4.1. First level optimization FLO: Production scheduling

The primary objective of the FLO is to determine the best sequence and assign the most suitable items to each machine, while considering stable lot sizes. Although the assumptions outlined in Section 4 were introduced to streamline the FLO resolution process, the optimization problem remains complex and is identified as NP-Hard (Alves et al., 2021). To cope with NP-hardness, a reinforcement learning (RL) approach based on a multi-agent system (MAS) is introduced to solve the FLO. In particular, the RL method aims to determine the best values for the binary variables $w_{m,p,t_f}^i$ and $Y_{m,p,t_f}^{ij}$ within reasonable computational time.

#### 4.1.1. Multi-agent system

The first-level model corresponds to a Multi-Agent System (MAS), where agents collaborate to achieve specific goals by interacting within a shared environment. In our case, the environment is related to the FFL system, which defines the production parameters. These include the number of agents, the connections between them, the number of products, and the characteristics of each machine, in particular its setup time and cost. By exchanging experiences and knowledge among agents, the MAS aims to optimize costs and improve the quality of solutions. Agents interact with the FFL environment through perception and action, learning how to map different situations to appropriate actions. Since the FFL problem consists of two sub-problems, namely the assignment problem and the sequencing problem, the learning process is divided into two stages. In the first stage, the agents learn how to select the most suitable machine for the processing of each product. In the second stage, the agents learn the best processing sequence for each product. With respect to these stages, the MAS consists of two types of agents: process agents and machine agents.

In situations where outcomes are influenced by both random elements and decisions made by individuals, the interaction process can be effectively modeled using a Markov Decision Process (MDP). Following this strategy, the MAS system is therefore modeled as a factored m-agent Decentralized Markov decision Process (DEC-MDP) (Fig. 5) consisting of a tuple of ($A_g, S, U, T, R, \delta, O$). The set of agents, denoted as $A_g$, includes both process agents $A_p$ and machine agents $A_m$. Each agent is associated with a specific resource within the FFL system. The state space $S$, is factored into $S_m$ and $S_p$, representing the states of the machine agents and process agents, respectively. The action space $U$ is two-dimensional, with $U_m$ representing the sub-action set of the machine agents and $U_p$ representing the sub-action set of the process agents. Each agent has its own observation $o \in O$ based on the observation function $\delta(\text{agent}, u): A_g \times U \to O$. The transition function $T(s, u, s')$ captures the probability of the system transitioning from state $s$ to $s'$ after executing a particular action $u$. It describes the dynamics of the system as it evolves over time. It is important to note that the reward function $R$ assigns different rewards to each agent, addressing the issue of various agents having distinct objectives or goals.

#### 4.1.2. First learning phase: assignment problem

During the learning phase, the focus is on solving the assignment
$$er_{t_f} + r_{t_f}^{+} \le G_{t_f}, \forall t \in T, \forall f \in F \quad (3 \text{ g})$$

$$ESS_{t_f}, r_{t_f}^{+}, r_{t_f}^{-}, ec_{t_f}, er_{t_f} \ge 0, \forall t \in T, \forall f \in F \quad (3 \text{ h})$$

Constraint (3a) ensures that the level of charge in the ESS remains within specified lower and upper bounds. Constraints (3b) and (3c) update the power level of the ESS in each micro-period. Constraints (3d) and (3e) express the charge and discharge capacities of the ESS, respectively. Constraint (3f) models the energy requirements of the manufacturing system and specifies the sources of energy, including PV solar power, grid energy under TOU policies, and ESS energy. Constraint (3g) shows how the generated on-site renewable energy is distributed between the manufacturing process and the ESS. Finally, a non-negativity requirement is indicated in constraint (3h).

## 4. Solution approach: A two-level method

The formulated mathematical model aims to optimize the integrated lot sizing and production scheduling, taking into account several features, including energy conditions, TOU prices, production and inventory capacities, satisfying final demand, and changeover metrics on production machines. These various constraints impose limits on feasible solutions, which can significantly increase the computational time required to reach the optimum solution. In addition, the optimal solution must balance energy and production strategies at each macro-period. Furthermore, the integration of buffers, RES, and ESS into production has a significant impact on overall production cost, which, in turn, affects lot sizes and production schedules. To meet these challenges and achieve a good solution within a reasonable timeframe, we propose a novel two-level framework, as summarized in Fig. 4.

The first level optimization (FLO) is designed to handle the integrated lot sizing and production scheduling. It focuses on optimizing the allocation of production quantities and schedules while taking into account the constraints mentioned above. The objective is to determine a good production plan that optimizes costs and meets demands. However, the challenge in the FLO resides in determining the trade-off between TOU energy prices, RES, ESS, and buffer use, when solving the proposed model. All these factors have complex interactions, and striking the appropriate compromise may be challenging. To simplify the problem, we assume that there is no buffer ($N_p = 0$) and no TOU policy ($g_t = \hat{g}_t$). Using this type of approximation may make it easier to

<figure><img src="image_6.png" /><figcaption>Fig. 4. Two-level approach framework.</figcaption></figure>

solve the MP1, within the FLO framework. After obtaining the initial production schedule, we introduce the second level optimization (SLO) which is devoted to optimizing the balance between peak electricity prices, buffers, ESS, and RES. It looks for the economic strategy to utilize energy resources, taking advantage of peak electricity prices and ensuring efficient management of buffers, ESS, and RES. Once, the best trade-off is determined, the FLO is then resolved to check the schedules obtained in the first level and adjust them as necessary. The proposed resolution framework combines these two levels of optimization to overcome problem complexity. The following subsections describe the development of this framework.

### 4.1. First level optimization FLO: Production scheduling

The primary objective of the FLO is to determine the best sequence and assign the most suitable items to each machine, while considering stable lot sizes. Although the assumptions outlined in Section 4 were introduced to streamline the FLO resolution process, the optimization problem remains complex and is identified as NP-Hard (Alves et al., 2021). To cope with NP-hardness, a reinforcement learning (RL) approach based on a multi-agent system (MAS) is introduced to solve the FLO. In particular, the RL method aims to determine the best values for the binary variables $w_{m,p,t_f}^i$ and $Y_{m,p,t_f}^{ij}$ within reasonable computational time.

#### 4.1.1. Multi-agent system

The first-level model corresponds to a Multi-Agent System (MAS), where agents collaborate to achieve specific goals by interacting within a shared environment. In our case, the environment is related to the FFL system, which defines the production parameters. These include the number of agents, the connections between them, the number of products, and the characteristics of each machine, in particular its setup time and cost. By exchanging experiences and knowledge among agents, the MAS aims to optimize costs and improve the quality of solutions. Agents interact with the FFL environment through perception and action, learning how to map different situations to appropriate actions. Since the FFL problem consists of two sub-problems, namely the assignment problem and the sequencing problem, the learning process is divided into two stages. In the first stage, the agents learn how to select the most suitable machine for the processing of each product. In the second stage, the agents learn the best processing sequence for each product. With respect to these stages, the MAS consists of two types of agents: process agents and machine agents.

In situations where outcomes are influenced by both random elements and decisions made by individuals, the interaction process can be effectively modeled using a Markov Decision Process (MDP). Following this strategy, the MAS system is therefore modeled as a factored m-agent Decentralized Markov decision Process (DEC-MDP) (Fig. 5) consisting of a tuple of ($A_g, S, U, T, R, \delta, O$). The set of agents, denoted as $A_g$, includes both process agents $A_p$ and machine agents $A_m$. Each agent is associated with a specific resource within the FFL system. The state space $S$, is factored into $S_m$ and $S_p$, representing the states of the machine agents and process agents, respectively. The action space $U$ is two-dimensional, with $U_m$ representing the sub-action set of the machine agents and $U_p$ representing the sub-action set of the process agents. Each agent has its own observation $o \in O$ based on the observation function $\delta(\text{agent}, u): A_g \times U \to O$. The transition function $T(s, u, s')$ captures the probability of the system transitioning from state $s$ to $s'$ after executing a particular action $u$. It describes the dynamics of the system as it evolves over time. It is important to note that the reward function $R$ assigns different rewards to each agent, addressing the issue of various agents having distinct objectives or goals.

#### 4.1.2. First learning phase: assignment problem

During the learning phase, the focus is on solving the assignment
<figure><img src="image_6.png" /><figcaption>Fig. 5. Overview of the flexible flow line framework.</figcaption></figure>

problem by considering various factors such as processing time, machine availability, and energy consumption. Each agent is trained to determine the most suitable processing machine for each product, as depicted in Fig. 6. The objective is to achieve a near-optimal allocation of products to parallel machines. To facilitate this optimization, each process *p* is associated with a process agent *k*. The process agent is responsible for boosting the reward by utilizing information about the energy consumption of the machines. The agent can ensure the appropriate allocation of products to machines, taking into account the overall energy efficiency of the system.

**State representation:** The state representation for each process is denoted by the local information about the current scheduling environment and the remaining products waiting to be processed. Each process agent has a local view of the available machines and the products waiting to be processed.

**Action representation:** The action space of a process agent in the FFL context is defined as the set of possible actions that the agent can take. This set typically includes selecting which machine to use from the available machines, considering their capacities. The primary objective of the process agent is to optimize the production process by achieving maximum efficiency while fulfilling quality standards and meeting customer demands.

**Reward representation:** the primary goal is to reduce the overall expenses associated with energy consumption and machine setups. Consequently, the reward system is designed to reflect this objective by assigning a lower value to actions that result in lower costs. More specifically, during this learning phase, the agent's task at each micro-

period is to identify, for a product, the machine with the lowest corresponding energy consumption rate. Consequently, the process agent's reward consists in the power required to produce the given item on a machine. To ensure that demands are satisfied without any backorders at each macro-period, the reward is designed to penalize situations where the total processing time exceeds the duration of the macro-period (|*F*|). In such cases, a large reward value is assigned to discourage actions that lead to excessive processing time. The reward function is formulated as follows:

$$R_p(m, i) = \begin{cases} E_{i,m,p}^{om} \times x_{m,p,t_f}^i, & f + \tau_{m,p}^i \times x_{m,p,t_f}^i \le |F| \\ \text{big number } M, & f + \tau_{m,p}^i \times x_{m,p,t_f}^i > |F| \end{cases}$$

### 4.1.3. Second learning phase: sequencing problem

This learning phase addresses the sequencing problem, which aims to minimize the setup costs. Each machine within the process is associated with a corresponding machine agent. These machine agents are responsible for optimizing the sequencing of products to minimize costs (Fig. 7).

**State and action representation:** The state and action spaces for these machine agents are defined based on available products and the corresponding actions that can be taken in order to select from among the remaining products, respectively. In other words, the state space $S_m$ is defined as the set of remaining products, while the action space $U_m$ corresponds to the selection of a product.

<figure><img src="image_7.png" /><figcaption>Fig. 6. Illustration of the action space of the process agent.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 7. Illustration of the action space of the machine agent.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>Fig. 5. Overview of the flexible flow line framework.</figcaption></figure>

problem by considering various factors such as processing time, machine availability, and energy consumption. Each agent is trained to determine the most suitable processing machine for each product, as depicted in Fig. 6. The objective is to achieve a near-optimal allocation of products to parallel machines. To facilitate this optimization, each process p is associated with a process agent k. The process agent is responsible for boosting the reward by utilizing information about the energy consumption of the machines. The agent can ensure the appropriate allocation of products to machines, taking into account the overall energy efficiency of the system.

**State representation:** The state representation for each process is denoted by the local information about the current scheduling environment and the remaining products waiting to be processed. Each process agent has a local view of the available machines and the products waiting to be processed.

**Action representation:** The action space of a process agent in the FFL context is defined as the set of possible actions that the agent can take. This set typically includes selecting which machine to use from the available machines, considering their capacities. The primary objective of the process agent is to optimize the production process by achieving maximum efficiency while fulfilling quality standards and meeting customer demands.

**Reward representation:** the primary goal is to reduce the overall expenses associated with energy consumption and machine setups. Consequently, the reward system is designed to reflect this objective by assigning a lower value to actions that result in lower costs. More specifically, during this learning phase, the agent's task at each micro-

period is to identify, for a product, the machine with the lowest corresponding energy consumption rate. Consequently, the process agent's reward consists in the power required to produce the given item on a machine. To ensure that demands are satisfied without any backorders at each macro-period, the reward is designed to penalize situations where the total processing time exceeds the duration of the macro-period (|F|). In such cases, a large reward value is assigned to discourage actions that lead to excessive processing time. The reward function is formulated as follows:

$$R_p(m, i) = \begin{cases} E_{i,m,p}^{om} \times x_{m,p,t_f}^i, & f + \tau_{m,p}^i \times x_{m,p,t_f}^i \le |F| \\ \text{big number } M, & f + \tau_{m,p}^i \times x_{m,p,t_f}^i > |F| \end{cases}$$

### 4.1.3. Second learning phase: sequencing problem

This learning phase addresses the sequencing problem, which aims to minimize the setup costs. Each machine within the process is associated with a corresponding machine agent. These machine agents are responsible for optimizing the sequencing of products to minimize costs (Fig. 7).

**State and action representation:** The state and action spaces for these machine agents are defined based on available products and the corresponding actions that can be taken in order to select from among the remaining products, respectively. In other words, the state space $S_m$ is defined as the set of remaining products, while the action space $U_m$ corresponds to the selection of a product.

<figure><img src="image_8.png" /><figcaption>Fig. 6. Illustration of the action space of the process agent.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7. Illustration of the action space of the machine agent.</figcaption></figure>
**Reward representation:** The reward function for the machine agents is designed to minimize energy and setup costs, reflecting the objective of minimizing these costs during the sequencing process. The reward function can be expressed as follows:

$$R_m(i, \text{remaining products}) = \begin{cases} S_{m,p}^{ij} E_{m,p}^{\text{eff}}, & f + \tau_{m,p}^{ij} \text{quantity} \le |F| \\ \text{big number } M, & f + \tau_{m,p}^{ij} \text{quantity} > |F| \end{cases}$$

### 4.1.4. Action selection strategy

In the reinforcement learning process, the selection of the action strategy plays a critical role. When the agent reaches a certain state S, it must make a decision between exploiting its previous experiences by selecting the best action based on the associated Q-value or exploring new paths by selecting a non-explored action. However, choosing the latter option prevents the agent from utilizing its prior knowledge. On the other hand, selecting the best action may lead to a known or sub-optimal path, hindering the agent's ability to minimize the cumulative reward and establish an efficient learning process. Therefore, it is necessary to strike a balance between exploration and exploitation to achieve the best results. A ε-greedy strategy is adopted here as the action selection method. Indeed, such a strategy has already been successfully applied in multi-agent scenarios (Gomes and Kowalczyk, 2009). The ε-greedy policy implies that the agent mainly exploits its existing knowledge by selecting the best action most of the time. However, occasionally, the agent selects a random action to explore new possibilities. The probability of selecting a random action is determined by the value of ε. In practice, to increase the learning rate and make use of prior knowledge effectively, less exploration is required for smaller scheduling problems. In contrast, larger problems necessitates a wider exploration process to discover better solutions and avoid being stuck in suboptimal paths.

### 4.1.5. MAS architecture

The MAS architecture presented in the context of the production scheduling problem in the FFL aims to minimize cumulative reward through agent-environment interactions. This architecture enables real-time scheduling and adaptive planning strategies based on incoming events, which can be classified into two types. The first type of events

occurs at each micro-period f when a machine produces a certain quantity of a product and transfers it to the next process. Thereafter, process agents assess the product quantity and the available machines, and assign the products to a machine based on problem constraints. Products can then proceed for immediate production or wait temporarily, depending on the decision made by the machine agent. These events continuously update the system's information and status. The second type of event occurs when the production of a product is completed, resulting in a change in the agent and system status. This change affects the set of actions that the agent can choose from, requiring the agent to select another action from among the remaining actions. Overall, these outcomes lead to an exchange of information between agents, fostering interaction areas between them.

The proposed MAS architecture presented in Fig. 8 consists of two main parts: agent interaction and agent training. In the interaction part, agents operate with respect to the constraints of the framework and provide functionalities to establish the interaction network. This network enables communication and coordination between agents, facilitating their joint resolution of lot sizing and scheduling tasks. The training part of the architecture governs the dynamic behavior of the agents. Agents aim to minimize the cumulative reward by interacting with the environment, learning from their experiences and adjusting their behaviors over time to enhance their performance. Overall, this proposed MAS architecture shows promise for tackling complex manufacturing tasks that require coordination and cooperation among multiple agents.

### 4.1.6. Agent interactions

As seen previously, the proposed MAS architecture relies on, two types of agent interactions: process agents interacting with their associated machines, and interactions between consecutive processes. The first type of interaction occurs between process and the machines they are associated with. At each micro-period, process agents gather information from their associated machines to explore the environment. Then, this information is used as input to the policy model of the process agent to determine the action to take in order to minimize long-term cumulative rewards. Similarly, machine agents also communicate with process agents to gather information about the current state of the environment. The observations made by the agents are represented by

<figure><img src="image_9.png" /><figcaption>Fig. 8. MAS architecture.</figcaption></figure>
**Reward representation:** The reward function for the machine agents is designed to minimize energy and setup costs, reflecting the objective of minimizing these costs during the sequencing process. The reward function can be expressed as follows:

$$R_m(i, \text{remaining products}) = \begin{cases} S_{m,p}^{ij} E_{m,p}^{\text{eff}}, g_i + C_{m,p}^{ij}, & f + \tau_{m,p}^{ij} \text{quantity} \le |F| \\ \text{big number } M, & f + \tau_{m,p}^{ij} \text{quantity} > |F| \end{cases}$$

### 4.1.4. Action selection strategy

In the reinforcement learning process, the selection of the action strategy plays a critical role. When the agent reaches a certain state S, it must make a decision between exploiting its previous experiences by selecting the best action based on the associated Q-value or exploring new paths by selecting a non-explored action. However, choosing the latter option prevents the agent from utilizing its prior knowledge. On the other hand, selecting the best action may lead to a known or sub-optimal path, hindering the agent's ability to minimize the cumulative reward and establish an efficient learning process. Therefore, it is necessary to strike a balance between exploration and exploitation to achieve the best results. A ε-greedy strategy is adopted here as the action selection method. Indeed, such a strategy has already been successfully applied in multi-agent scenarios (Gomes and Kowalczyk, 2009). The ε-greedy policy implies that the agent mainly exploits its existing knowledge by selecting the best action most of the time. However, occasionally, the agent selects a random action to explore new possibilities. The probability of selecting a random action is determined by the value of ε. In practice, to increase the learning rate and make use of prior knowledge effectively, less exploration is required for smaller scheduling problems. In contrast, larger problems necessitates a wider exploration process to discover better solutions and avoid being stuck in suboptimal paths.

### 4.1.5. MAS architecture

The MAS architecture presented in the context of the production scheduling problem in the FFL aims to minimize cumulative reward through agent-environment interactions. This architecture enables real-time scheduling and adaptive planning strategies based on incoming events, which can be classified into two types. The first type of events

occurs at each micro-period f when a machine produces a certain quantity of a product and transfers it to the next process. Thereafter, process agents assess the product quantity and the available machines, and assign the products to a machine based on problem constraints. Products can then proceed for immediate production or wait temporarily, depending on the decision made by the machine agent. These events continuously update the system's information and status. The second type of event occurs when the production of a product is completed, resulting in a change in the agent and system status. This change affects the set of actions that the agent can choose from, requiring the agent to select another action from among the remaining actions. Overall, these outcomes lead to an exchange of information between agents, fostering interaction areas between them.

The proposed MAS architecture presented in Fig. 8 consists of two main parts: agent interaction and agent training. In the interaction part, agents operate with respect to the constraints of the framework and provide functionalities to establish the interaction network. This network enables communication and coordination between agents, facilitating their joint resolution of lot sizing and scheduling tasks. The training part of the architecture governs the dynamic behavior of the agents. Agents aim to minimize the cumulative reward by interacting with the environment, learning from their experiences and adjusting their behaviors over time to enhance their performance. Overall, this proposed MAS architecture shows promise for tackling complex manufacturing tasks that require coordination and cooperation among multiple agents.

### 4.1.6. Agent interactions

As seen previously, the proposed MAS architecture relies on, two types of agent interactions: process agents interacting with their associated machines, and interactions between consecutive processes. The first type of interaction occurs between process and the machines they are associated with. At each micro-period, process agents gather information from their associated machines to explore the environment. Then, this information is used as input to the policy model of the process agent to determine the action to take in order to minimize long-term cumulative rewards. Similarly, machine agents also communicate with process agents to gather information about the current state of the environment. The observations made by the agents are represented by

<figure><img src="image_10.png" /><figcaption>Fig. 8. MAS architecture.</figcaption></figure>
$o_m^f$ for machine agents and $o_p^f$ for process agents. For machine agents, the observation includes information about the current product, the current micro-period $f$, and any affected products. For process agents, the observation includes information about available machines, the current micro-period $f$, and remaining products for production. The second type of interaction involves interactions between consecutive processes. At each micro-period, the process agent communicates with its associated machines (communication 1) to check if any quantities were produced during the previous micro-period and are ready to be transferred to the next process at the current micro-period ($om_p^f$). The process agent then passes this information to the next process. In turn, the agent linked to the latter communicates with its associated machines to make decisions. The observation for this type of interaction is represented by $om_p^f$, which includes information about the available products to be transferred to the next process, quantities, and the current micro-period.

### 4.1.7. Agent training

The agents' policy model in the proposed MAS architecture is based on Q-learning, which is a popular reinforcement-learning algorithm. Q-learning learns by building an action-value function that estimates the expected cumulative reward for taking a particular action in a particular state. The Q-value of a state-action pair ($s, u$) is updated based on the reward received after selecting an action $u$ in state $s$, using the update rule given by equation (4a).

$$Q(s, u) = (1 - \alpha)Q(s, u) + \alpha[r + \gamma*\min_u(Q(s', u') - Q(s, u))] \quad (4a)$$

Here, $\alpha$ corresponds to the learning rate that determines the weight given to the new information obtained by the agent, while $\gamma$ is the discount factor that determines the importance given to future rewards. The term $r$ represents the immediate reward received by the agent after taking the action $u$ in state $s$. The term $\min_u(Q(s', u') - Q(s, u))$ represents the estimate of the maximum future reward that can be obtained after transitioning to the next state $s'$ and taking the optimal action $u'$ based on the current Q-values. The objective in this study is to minimize production costs, and the Q-learning algorithm is designed to find the optimal policy that achieves this objective by minimizing the expected cumulative reward over time. The input of the models consists of agent observation and action. Agent interaction and all the proposed Q-learning algorithms are summarized in

Algorithm 1. Process agent QL

Initialize:

$Q(s, u) = zeros\_matrix(\text{products}_{\text{number}}, \text{machines}_{\text{number}})$

For each episode do:

Initialize:

$S = \{\text{list of remaining products to be processed}\}$

Possible\_actions = \{\text{available machines list}\}

for Micro-period = $f, s$ = product:

Choose $u$ from possible\_actions using e-greedy policy.

Take action $u$, calculate $R_p(u, s)$, $S = s \setminus \{s\}$ Update $Q(s, u)$

$S=S'$

Return selected machine and process observations.

End for

End for

### algorithms 1 and 2.

Algorithm 2. Machine agent QL

Initialize:

$Q(s, u) = zeros\_matrix(2^{\text{products}_{\text{number}}, \text{products}_{\text{number}}})$

For each episode do:

$S = \{\text{list of remaining products to be processed}\}$

Possible\_actions = \{\text{products list}\}

While machine is available, micro-period = $f$ and $s \in S$:

Choose $u$ from possible\_actions using e-greedy policy.

Take action $u$, calculate $R_m(u, s)$, $S = S \setminus \{s\}$ Update $Q(s, u)$

$S = S'$ and Possible\_actions = possible actions $\setminus \{u\}$

(continued on next column)

(continued)

Algorithm 2. Machine agent QL

Availability = not\_available(u)

Return machine observation

End while

End for

### 4.2. Second level optimization SLO: Trade-off between buffer use, RES, ESS and TOU prices

In the context of production planning, the SLO focuses on managing the tradeoff between on-peak electricity demand at all macro-periods and the utilization of RES, ESS and buffers. The primary goal is to effectively manage energy consumption from different sources and inventory usage during the planning process. One feasible approach to achieve this goal is to produce a surplus of the required quantity during off-peak macro-periods and store it as reserve stock. This reserve stock can then be utilized to meet the demand during on-peak macro-periods, enabling machines to reduce their working time and energy consumption, resulting in significant energy savings. Since the on-peak demand is defined on a macro-period scale, the SLO problem aims to determine the quantities to be produced and the quantities to be stored at each macro-period. The decision-making process in this problem operates at the macro-period level, taking into account the overall demand and supply requirements. However, the decision-making process in this problem operates at the macro-period level, potentially overlooking the challenge of product sequencing. Nevertheless, the allocation of machines is crucial to ensure smooth operations and minimize energy costs. To address this challenge, a linear optimization model is formulated within the SLO. The primary idea is to separate binary variables from the continuous variables in the objective function and relax production sequence constraints. The binary variables related to machine allocation are assumed to be known. Since the SLO is a linear program, it can determine the best trade-off between energy consumption during peak macro-periods, RES, ESS, and stock use. The obtained SLO problem is presented below.

Relaxed MP1

$$\min \left( \sum_{f \in F} \sum_{t \in T} \left( g_t \cdot e c_{t_f} + r_{t_f} \cdot e r_{t_f} + C^+, r_{t_f}^+ + C^-, r_{t_f}^- + \sum_{p \in P} \sum_{i \in N} C_{i,p,i}^{\text{buffer}} Q_{p,t_f}^i \right) \right) \quad (5a)$$

$$\text{S.T.} \qquad (2a) - (2f) \qquad (5b)$$

$$\sum_{i \in N} t_{m,p}^i \cdot x_{m,p,t_f}^i \le d - \max_{i,j} (s_{m,p}^{i,j}), \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (5c)$$

$$ (2 \text{ h}) \qquad (5d) $$

$$ (3a) - (3j) \qquad (5e) $$

The SLO does not take into account constraints (2i)-(2n) due to its focus on managing the trade-off between on-peak electricity demand macro-periods and RES, ESS, and buffer use. Although capacity constraint (2i) is strongly linked to both binary and continuous variables, a reformulation trick presented in (5c) can separate binary variables while still considering setup times. This involves ensuring that the total production time at each micro-period is less than or equal to the duration of the micro-period minus the maximum setup time on a machine. The transmission of products between processes (constraints (2a)-(2e)) is programmed with machine allocation, which is assumed to be known from the FLO.
$o_m^f$ for machine agents and $o_p^f$ for process agents. For machine agents, the observation includes information about the current product, the current micro-period $f$, and any affected products. For process agents, the observation includes information about available machines, the current micro-period $f$, and remaining products for production. The second type of interaction involves interactions between consecutive processes. At each micro-period, the process agent communicates with its associated machines (communication 1) to check if any quantities were produced during the previous micro-period and are ready to be transferred to the next process at the current micro-period ($om_p^f$). The process agent then passes this information to the next process. In turn, the agent linked to the latter communicates with its associated machines to make decisions. The observation for this type of interaction is represented by $om_p^f$, which includes information about the available products to be transferred to the next process, quantities, and the current micro-period.

### 4.1.7. Agent training

The agents' policy model in the proposed MAS architecture is based on Q-learning, which is a popular reinforcement-learning algorithm. Q-learning learns by building an action-value function that estimates the expected cumulative reward for taking a particular action in a particular state. The Q-value of a state-action pair ($s, u$) is updated based on the reward received after selecting an action $u$ in state $s$, using the update rule given by equation (4a).

$$Q(s, u) = (1 - \alpha)Q(s, u) + \alpha[r + \gamma*\min_u(Q(s', u') - Q(s, u))] \quad (4a)$$

Here, $\alpha$ corresponds to the learning rate that determines the weight given to the new information obtained by the agent, while $\gamma$ is the discount factor that determines the importance given to future rewards. The term $r$ represents the immediate reward received by the agent after taking the action $u$ in state $s$. The term $\min_u(Q(s', u') - Q(s, u))$ represents the estimate of the maximum future reward that can be obtained after transitioning to the next state $s'$ and taking the optimal action $u'$ based on the current Q-values. The objective in this study is to minimize production costs, and the Q-learning algorithm is designed to find the optimal policy that achieves this objective by minimizing the expected cumulative reward over time. The input of the models consists of agent observation and action. Agent interaction and all the proposed Q-learning algorithms are summarized in

<caption>Algorithm 1. Process agent QL</caption>

Initialize:
$Q(s, u) = zeros\_matrix products\_number, machines\_number)$ 
For each episode do:
Initialize:
$S = \{list of remaining products to be processed\}$ 
Possible\_actions = $\{available machines list\}$ 
for Micro-period = $f, s$ = product:
Choose $u$ from possible\_actions using e-greedy policy.
Take action $u$, calculate $R_p(u, s)$, $S = s \setminus \{s\}$ Update $Q(s, u)$
$S = S'$
Return selected machine and process observations.
End for
End for

algorithms 1 and 2.

<caption>Algorithm 2. Machine agent QL</caption>

Initialize:
$Q(s, u) = zeros\_matrix(2products\_number, products\_number)$ 
For each episode do:
$S = \{list of remaining products to be processed\}$ 
Possible\_actions = $\{products list\}$ 
While machine is available, micro-period = $f$ and $s \in S$:
Choose $u$ from possible\_actions using e-greedy policy.
Take action $u$, calculate $R_m(u, s)$, $S = S \setminus \{s\}$ Update $Q(s, u)$
$S = S'$ and Possible\_actions = possible actions $\setminus \{u\}$

(continued on next column)

(continued)

<table><caption>Algorithm 2. Machine agent QL</caption><tr><td>Availability = not\_available(u)</td></tr><tr><td>Return machine observation</td></tr><tr><td>End while</td></tr><tr><td>End for</td></tr></table>

## 4.2. Second level optimization SLO: Trade-off between buffer use, RES, ESS and TOU prices

In the context of production planning, the SLO focuses on managing the tradeoff between on-peak electricity demand at all macro-periods and the utilization of RES, ESS and buffers. The primary goal is to effectively manage energy consumption from different sources and inventory usage during the planning process. One feasible approach to achieve this goal is to produce a surplus of the required quantity during off-peak macro-periods and store it as reserve stock. This reserve stock can then be utilized to meet the demand during on-peak macro-periods, enabling machines to reduce their working time and energy consumption, resulting in significant energy savings. Since the on-peak demand is defined on a macro-period scale, the SLO problem aims to determine the quantities to be produced and the quantities to be stored at each macro-period. The decision-making process in this problem operates at the macro-period level, taking into account the overall demand and supply requirements. However, the decision-making process in this problem operates at the macro-period level, potentially overlooking the challenge of product sequencing. Nevertheless, the allocation of machines is crucial to ensure smooth operations and minimize energy costs. To address this challenge, a linear optimization model is formulated within the SLO. The primary idea is to separate binary variables from the continuous variables in the objective function and relax production sequence constraints. The binary variables related to machine allocation are assumed to be known. Since the SLO is a linear program, it can determine the best trade-off between energy consumption during peak macro-periods, RES, ESS, and stock use. The obtained SLO problem is presented below.

### Relaxed MP1

$$\min \left( \sum_{f \in F} \sum_{t \in T} \left( g_t \cdot e c_{t_f} + r_{t_f} \cdot e r_{t_f} + C^+, r_{t_f}^+ + C^-, r_{t_f}^- + \sum_{p \in P} \sum_{i \in N} C_{i,p,i}^{buffer} Q_{p,t_f}^i \right) \right) \quad (5a)$$

$$\text{S.T.} \qquad (2a) - (2f) \qquad (5b)$$

$$\sum_{i \in N} t_{m,p}^i \cdot x_{m,p,t_f}^i \le d - \max_{i,j} (s_{m,p}^{ij}), \forall m \in M_p, \forall p \in P, \forall t \in T, \forall f \in F \quad (5c)$$

$$ (2 \text{ h}) \qquad (5d) $$

$$ (3a) - (3j) \qquad (5e) $$

The SLO does not take into account constraints (2i)-(2n) due to its focus on managing the trade-off between on-peak electricity demand macro-periods and RES, ESS, and buffer use. Although capacity constraint (2i) is strongly linked to both binary and continuous variables, a reformulation trick presented in (5c) can separate binary variables while still considering setup times. This involves ensuring that the total production time at each micro-period is less than or equal to the duration of the micro-period minus the maximum setup time on a machine. The transmission of products between processes (constraints (2a)-(2e)) is programmed with machine allocation, which is assumed to be known from the FLO.
## 4.3. Summary of resolution steps

To summarize, the proposed algorithm is a two-level optimization approach. At the first level, the integrated lot sizing and production scheduling problem is solved using MAS. The output of the FLO provides the best production schedules and machine affectation, which serves as input for the second level. The SLO problem is focused on managing the tradeoff between on-peak electricity demand macro-periods and RES, ESS, and buffer use. The objective is to effectively balance energy consumption and inventory usage during the planning process. To address this challenge, a deterministic polynomial planning problem is formulated. The obtained SLO problem is solved by a linear program, which seeks the best trade-off between energy consumption during peak macro-periods, RES, ESS, and stock use. Once the second level problem is solved, the best solutions for lot sizes and buffer inventory levels at the macro-period scale are used to update the initial FLO problem. The updated FLO problem is then solved to obtain new lot sizes and production schedules at the micro-period scale.

# 5. Computational results

The proposed model was implemented from scratch using Python 3.9 and executed on a personal computer equipped with a Core™ i7-11850H 2.5 GHz CPU and 32 GB of RAM. The computation times were measured in CPU seconds. For numerical studies, a benchmark experiment was conducted to assess the performance of the proposed framework. Subsequently, computational tests were performed on small, medium, and large problems to evaluate the performance of the RL algorithm.

## 5.1. Input data

For the benchmark experiment, the production parameters were adjusted based on Özdamar and Barbarasoglu's work (1999). They were randomly generated according to the following specifications: The scheduling horizon is a day-ahead horizon divided into $|T| = 4$ macro-periods, each further divided into $|F| = 6$ micro-periods. Each micro-period corresponds to an aggregate of one hour. The FFL production system comprises $|P| = 3$ sequential processes. The two first processes have $|M_1| = |M_2| = 1$ machine each, while the third has $|M_3| = 3$ non-identical parallel machines. The number of final products is set to $|N| = 3$ products. The final demand for products exhibits high variability and is generated from a uniform distribution $U(20, 70)$. The processing times $t_{m,p}^i$ (in minutes) are generated from $U(1, 4)$. Setup times, expressed in minutes, are proportional to the total processing time and are calculated using the formula: $s_{m,p}^{ij} = \frac{S \sum_{j=1,m}^{j,m} t_{m,p}^j * D_{i,t}}{|T| * \max_{p}(|M_p|)}$, where S is generated from $U(0.05, 0.01)$. The setup costs are proportional to the setup times, meaning that the setup costs will increase as the associated times increase. The inventory cost $C_{i,p,t}^{buffer}$ is constant across all processes and products and is set at $2 \epsilon$ per unit. The capacity of buffers $N_p$ is set to 100 units. Tables 4, 5, and 6 depict the generated values for demands, processing times and setup parameters, respectively. Tables 7, 8 and 9 present machines energy consumption, PV energy availability and TOU prices, respectively.

The energy parameters are based on the Duarte et al. (2020) study. The availability of photovoltaic (PV) energy is obtained from a PV power plant located on the roof of the Polytech University of Nantes in France. Daily production of the PV panels corresponds to data collected in April 2019. The on-peak macro-period consists of macro-period $t = 3$. ESS capacity parameters are set as follows: $s_{min} = 0.4 \text{ MWh}$, $s_{max} = 1.5 \text{ MWh}$ and $s_0 = 0.6 \text{ MWh}$. The charging and discharging power limits are both set to $R^+ = R^- = 0.5 \text{ MWh}$. The ESS efficiency are set as follows: $\eta^+ = \eta^- = 1, c^+ = c^- = 25 \epsilon$. The cost of MWh Solar power is estimated from solar panels investment, panel life span and efficiency and it set as $r_{tf} = 50 \epsilon / \text{MWh}$.

<table><caption>**Table 4**
Product demand.</caption><thead><tr><th>time \ products</th><th>i = 1</th><th>i = 2</th><th>i = 3</th></tr></thead><tbody><tr><td>t = 1, f = 6</td><td>40</td><td>41</td><td>33</td></tr><tr><td>t = 2, f = 6</td><td>36</td><td>41</td><td>52</td></tr><tr><td>t = 3, f = 6</td><td>53</td><td>42</td><td>54</td></tr><tr><td>t = 4, f = 6</td><td>29</td><td>65</td><td>30</td></tr></tbody></table>

$E_{i,m,p}^{off}$ is set at 0.005 MWh/unit. The learning parameters associated to agents are set as follows: $\alpha = 0.2, \gamma = 0.5$ and $\epsilon = 0.2$.

## 5.2. Benchmark experiment

### 5.2.1. 1st step: Solve the first level optimization: FLO

The FLO step aims to identify the best sequence and machine allocation using a reinforcement learning (RL) approach. Fig. 9 and Fig. 10 illustrate the training process of the process agents and machine agents, respectively. It is apparent that the cumulative rewards of the two agents

<table><caption>**Table 5**
Processing times (min).</caption><thead><tr><th>Machine \ products</th><th>i = 1</th><th>i = 2</th><th>i = 3</th></tr></thead><tbody><tr><td>p = 1, m = 1</td><td>2</td><td>1.3</td><td>1.35</td></tr><tr><td>p = 2, m = 1</td><td>1.8</td><td>1.3</td><td>1.3</td></tr><tr><td>p = 3, m = 1</td><td>2.5</td><td>1.8</td><td>2</td></tr><tr><td>p = 3, m = 2</td><td>1.7</td><td>2</td><td>1.2</td></tr><tr><td>p = 3, m = 3</td><td>2.2</td><td>1.3</td><td>2.2</td></tr></tbody></table>

<table><caption>**Table 6**
Setup times and costs for two machines demand.</caption><thead><tr><th>Machine</th><th colspan="3">Setup time (min)</th><th colspan="3">Cost (€)</th></tr><tr><th>p = 1, m = 1</th><th>i = 1</th><th>i = 2</th><th>i = 3</th><th>i = 1</th><th>i = 2</th><th>i = 3</th></tr></thead><tbody><tr><td rowspan="3">Machine<br>p = 2, m = 1</td><td>j = 1</td><td>0</td><td>7</td><td>3</td><td>j = 1</td><td>0</td><td>80</td><td>44</td></tr><tr><td>j = 2</td><td>8</td><td>0</td><td>6</td><td>j = 2</td><td>86</td><td>0</td><td>62</td></tr><tr><td>j = 3</td><td>6</td><td>9</td><td>0</td><td>j = 3</td><td>60</td><td>83</td><td>0</td></tr><tr><td colspan="8" style="text-align: center;">Setup time (min)</td></tr><tr><td rowspan="3">Machine<br>p = 2, m = 1</td><td>j = 1</td><td>0</td><td>9</td><td>5</td><td>j = 1</td><td>0</td><td>99</td><td>58</td></tr><tr><td>j = 2</td><td>9</td><td>0</td><td>3</td><td>j = 2</td><td>93</td><td>0</td><td>32</td></tr><tr><td>j = 3</td><td>9</td><td>11</td><td>0</td><td>j = 3</td><td>102</td><td>108</td><td>0</td></tr></tbody></table>

<table><caption>**Table 7**
Machines power consumption $E_{i,m,p}^{on}$ (MW/unit).</caption><thead><tr><th>Machine \ products</th><th>i = 1</th><th>i = 2</th><th>i = 3</th></tr></thead><tbody><tr><td>p = 1, m = 1</td><td>0.02</td><td>0.015</td><td>0.014</td></tr><tr><td>p = 2, m = 1</td><td>0.019</td><td>0.013</td><td>0.014</td></tr><tr><td>p = 3, m = 1</td><td>0.026</td><td>0.021</td><td>0.02</td></tr><tr><td>p = 3, m = 2</td><td>0.017</td><td>0.019</td><td>0.017</td></tr><tr><td>p = 3, m = 3</td><td>0.025</td><td>0.015</td><td>0.015</td></tr></tbody></table>

<table><caption>**Table 8**
PV power availability (MW).</caption><thead><tr><th>f</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th></tr></thead><tbody><tr><td>G<sub>1</sub> (MW)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>G<sub>2</sub> (MW)</td><td>0.1</td><td>0.4</td><td>0.56</td><td>0.634</td><td>0.82</td><td>0.73</td></tr><tr><td>G<sub>3</sub> (MW)</td><td>0.86</td><td>0.71</td><td>0.62</td><td>0.78</td><td>0.43</td><td>0.3</td></tr><tr><td>G<sub>4</sub> (MW)</td><td>0.28</td><td>0.1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table>

<table><caption>**Table 9**
TOU prices (€/MWh).</caption><thead><tr><th>t</th><th>1</th><th>2</th><th>3</th><th>4</th></tr></thead><tbody><tr><td>g<sub>1</sub> (€/MWh)</td><td>70</td><td>70</td><td>130</td><td>70</td></tr></tbody></table>
## 4.3. Summary of resolution steps

To summarize, the proposed algorithm is a two-level optimization approach. At the first level, the integrated lot sizing and production scheduling problem is solved using MAS. The output of the FLO provides the best production schedules and machine affectation, which serves as input for the second level. The SLO problem is focused on managing the tradeoff between on-peak electricity demand macro-periods and RES, ESS, and buffer use. The objective is to effectively balance energy consumption and inventory usage during the planning process. To address this challenge, a deterministic polynomial planning problem is formulated. The obtained SLO problem is solved by a linear program, which seeks the best trade-off between energy consumption during peak macro-periods, RES, ESS, and stock use. Once the second level problem is solved, the best solutions for lot sizes and buffer inventory levels at the macro-period scale are used to update the initial FLO problem. The updated FLO problem is then solved to obtain new lot sizes and production schedules at the micro-period scale.

# 5. Computational results

The proposed model was implemented from scratch using Python 3.9 and executed on a personal computer equipped with a Core™ i7-11850H 2.5 GHz CPU and 32 GB of RAM. The computation times were measured in CPU seconds. For numerical studies, a benchmark experiment was conducted to assess the performance of the proposed framework. Subsequently, computational tests were performed on small, medium, and large problems to evaluate the performance of the RL algorithm.

## 5.1. Input data

For the benchmark experiment, the production parameters were adjusted based on Özdamar and Barbarasoglu's work (1999). They were randomly generated according to the following specifications: The scheduling horizon is a day-ahead horizon divided into $|T| = 4$ macro-periods, each further divided into $|F| = 6$ micro-periods. Each micro-period corresponds to an aggregate of one hour. The FFL production system comprises $|P| = 3$ sequential processes. The two first processes have $|M_1| = |M_2| = 1$ machine each, while the third has $|M_3| = 3$ non-identical parallel machines. The number of final products is set to $|N| = 3$ products. The final demand for products exhibits high variability and is generated from a uniform distribution $U(20, 70)$. The processing times $t_{m,p}^i$ (in minutes) are generated from $U(1, 4)$. Setup times, expressed in minutes, are proportional to the total processing time and are calculated using the formula: $s_{m,p}^{ij} = \frac{S \sum_{j=1,m}^{i=1} j t_{m,p}^j * D_{i,t}}{|T| * \max_{p}(|M_p|)}$, where S is generated from $U(0.05, 0.01)$. The setup costs are proportional to the setup times, meaning that the setup costs will increase as the associated times increase. The inventory cost $C_{i,p,t}^{buffer}$ is constant across all processes and products and is set at $2 \epsilon$ per unit. The capacity of buffers $N_p$ is set to 100 units. Tables 4, 5, and 6 depict the generated values for demands, processing times and setup parameters, respectively. Tables 7, 8 and 9 present machines energy consumption, PV energy availability and TOU prices, respectively.

The energy parameters are based on the Duarte et al. (2020) study. The availability of photovoltaic (PV) energy is obtained from a PV power plant located on the roof of the Polytech University of Nantes in France. Daily production of the PV panels corresponds to data collected in April 2019. The on-peak macro-period consists of macro-period $t = 3$. ESS capacity parameters are set as follows: $s_{min} = 0.4 \text{ MWh}$, $s_{max} = 1.5 \text{ MWh}$ and $s_0 = 0.6 \text{ MWh}$. The charging and discharging power limits are both set to $R^+ = R^- = 0.5 \text{ MWh}$. The ESS efficiency are set as follows: $\eta^+ = \eta^- = 1$, $c^+ = c^- = 25 \epsilon$. The cost of MWh Solar power is estimated from solar panels investment, panel life span and efficiency and it set as $r_{tf} = 50 \epsilon / \text{MWh}$.

<table><caption>**Table 4**
Product demand.</caption><thead><tr><th>time \ products</th><th>i = 1</th><th>i = 2</th><th>i = 3</th></tr></thead><tbody><tr><td>t = 1, f = 6</td><td>40</td><td>41</td><td>33</td></tr><tr><td>t = 2, f = 6</td><td>36</td><td>41</td><td>52</td></tr><tr><td>t = 3, f = 6</td><td>53</td><td>42</td><td>54</td></tr><tr><td>t = 4, f = 6</td><td>29</td><td>65</td><td>30</td></tr></tbody></table>

$E_{i,m,p}^{off}$ is set at 0.005 MWh/unit. The learning parameters associated to agents are set as follows: $\alpha = 0.2$, $\gamma = 0.5$ and $\epsilon = 0.2$.

## 5.2. Benchmark experiment

### 5.2.1. 1st step: Solve the first level optimization: FLO

The FLO step aims to identify the best sequence and machine allocation using a reinforcement learning (RL) approach. Fig. 9 and Fig. 10 illustrate the training process of the process agents and machine agents, respectively. It is apparent that the cumulative rewards of the two agents

<table><caption>**Table 5**
Processing times (min).</caption><thead><tr><th>Machine \ products</th><th>i = 1</th><th>i = 2</th><th>i = 3</th></tr></thead><tbody><tr><td>p = 1, m = 1</td><td>2</td><td>1.3</td><td>1.35</td></tr><tr><td>p = 2, m = 1</td><td>1.8</td><td>1.3</td><td>1.3</td></tr><tr><td>p = 3, m = 1</td><td>2.5</td><td>1.8</td><td>2</td></tr><tr><td>p = 3, m = 2</td><td>1.7</td><td>2</td><td>1.2</td></tr><tr><td>p = 3, m = 3</td><td>2.2</td><td>1.3</td><td>2.2</td></tr></tbody></table>

<table><caption>**Table 6**
Setup times and costs for two machines demand.</caption><thead><tr><th>Machine</th><th colspan="3">Setup time (min)</th><th colspan="3">Cost (€)</th></tr><tr><th>p = 1, m = 1</th><th>i = 1</th><th>i = 2</th><th>i = 3</th><th>i = 1</th><th>i = 2</th><th>i = 3</th></tr></thead><tbody><tr><td rowspan="3">Machine<br>p = 2, m = 1</td><td>j = 1</td><td>0</td><td>7</td><td>3</td><td>j = 1</td><td>0</td><td>80</td><td>44</td></tr><tr><td>j = 2</td><td>8</td><td>0</td><td>6</td><td>j = 2</td><td>86</td><td>0</td><td>62</td></tr><tr><td>j = 3</td><td>6</td><td>9</td><td>0</td><td>j = 3</td><td>60</td><td>83</td><td>0</td></tr><tr><td colspan="8" style="text-align: center;">Setup time (min)</td></tr><tr><td rowspan="3">Machine<br>p = 2, m = 1</td><td>j = 1</td><td>0</td><td>9</td><td>5</td><td>j = 1</td><td>0</td><td>99</td><td>58</td></tr><tr><td>j = 2</td><td>9</td><td>0</td><td>3</td><td>j = 2</td><td>93</td><td>0</td><td>32</td></tr><tr><td>j = 3</td><td>9</td><td>11</td><td>0</td><td>j = 3</td><td>102</td><td>108</td><td>0</td></tr></tbody></table>

<table><caption>**Table 7**
Machines power consumption $E_{i,m,p}^{on}$ (MW/unit).</caption><thead><tr><th>Machine \ products</th><th>i = 1</th><th>i = 2</th><th>i = 3</th></tr></thead><tbody><tr><td>p = 1, m = 1</td><td>0.02</td><td>0.015</td><td>0.014</td></tr><tr><td>p = 2, m = 1</td><td>0.019</td><td>0.013</td><td>0.014</td></tr><tr><td>p = 3, m = 1</td><td>0.026</td><td>0.021</td><td>0.02</td></tr><tr><td>p = 3, m = 2</td><td>0.017</td><td>0.019</td><td>0.017</td></tr><tr><td>p = 3, m = 3</td><td>0.025</td><td>0.015</td><td>0.015</td></tr></tbody></table>

<table><caption>**Table 8**
PV power availability (MW).</caption><thead><tr><th>f</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th></tr></thead><tbody><tr><td>G<sub>1</sub>(MW)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>G<sub>2</sub>(MW)</td><td>0.1</td><td>0.4</td><td>0.56</td><td>0.634</td><td>0.82</td><td>0.73</td></tr><tr><td>G<sub>3</sub>(MW)</td><td>0.86</td><td>0.71</td><td>0.62</td><td>0.78</td><td>0.43</td><td>0.3</td></tr><tr><td>G<sub>4</sub>(MW)</td><td>0.28</td><td>0.1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table>

<table><caption>**Table 9**
TOU prices (€/MWh).</caption><thead><tr><th>t</th><th>1</th><th>2</th><th>3</th><th>4</th></tr></thead><tbody><tr><td>g<sub>i</sub>(€/MWh)</td><td>70</td><td>70</td><td>130</td><td>70</td></tr></tbody></table>
<figure><img src="image_10.png" /><figcaption>Fig. 9. Training reward for machine agents.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 10. Training reward for process agents.</figcaption></figure>

display a downward trend, with occasional increases in value due to the exploitation-exploration strategy related to the QL algorithm. Fig. 11 presents agent’s states and actions that depict the best sequence and machine allocation. It can be seen that only the first process agent is able to initiate an initial action at the first micro-period ($t = 1$ and $f = 1$) and all products are assigned to the machine, M1, of the first process. Subsequently, M1 chooses product $i = 2$ among the set of three products.

Upon completion of the processing of $i = 2$, the aforementioned sets are updated (Fig. 11, (c)). Next, other agents become available to act (i. e., transmit products) at the second micro-period ($t = 1$ and $f = 2$) (Fig. 11, (b)). Therefore, the best product sequence selected by the MAS of process 1 and 2 at the first macro-period is $i = 2/i = 3/i = 1$. This sequence ensures no delays and leads to a lowest setup cost = 256¢. After that, the agent of the third process assigns one product to each machine with the goal of minimizing the setup cost. These machine changeovers have been maintained throughout the entire scheduling horizon to avoid further setup costs.

### 5.2.2. 2nd step: solve the second level optimization

The resolution of the relaxed MP1 aims to ensure the best trade-off between energy consumption during the on-peak macro-period, inventory, RES and ESS utilization. Fig. 12 illustrates the best production scheduling of the first product $i = 1$. It can be observed that the production of $i = 1$ at the first macro-period exceeds the demands in process 1 and 2. Consequently, the excess is stored in the intermediate inventory to be further used in the on-peak macro-period. The energy consumption planning derived from the obtained scheduling scheme is illustrated in Fig. 13. It is obvious that the integration of both onsite RES and an ESS, as well as inventory policies, reduces the grid energy consumption during on-peak macro-period ($t = 3$). As a result, the total inventory cost is set at 461 €, the grid energy cost amounts to 898 €, while the solar power and ESS cost is set at 365 €.

### 5.2.3. 3rd step: resolve the FLO

The final step involves solving the FLO problem once again in order

<figure><img src="image_12.png" /><figcaption>Fig. 12. Best production schedule for item 1.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 11. System sets at different time steps.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Fig. 9. Training reward for machine agents.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 10. Training reward for process agents.</figcaption></figure>

display a downward trend, with occasional increases in value due to the exploitation-exploration strategy related to the QL algorithm. Fig. 11 presents agent's states and actions that depict the best sequence and machine allocation. It can be seen that only the first process agent is able to initiate an initial action at the first micro-period ($t = 1$ and $f = 1$) and all products are assigned to the machine, M1, of the first process. Subsequently, M1 chooses product $i = 2$ among the set of three products.

Upon completion of the processing of $i = 2$, the aforementioned sets are updated (Fig. 11, (c)). Next, other agents become available to act (i. e., transmit products) at the second micro-period ($t = 1$ and $f = 2$) (Fig. 11, (b)). Therefore, the best product sequence selected by the MAS of process 1 and 2 at the first macro-period is $i = 2/i = 3/i = 1$. This sequence ensures no delays and leads to a lowest setup cost = 256¢. After that, the agent of the third process assigns one product to each machine with the goal of minimizing the setup cost. These machine changeovers have been maintained throughout the entire scheduling horizon to avoid further setup costs.

### 5.2.2. 2nd step: solve the second level optimization

The resolution of the relaxed MP1 aims to ensure the best trade-off between energy consumption during the on-peak macro-period, inventory, RES and ESS utilization. Fig. 12 illustrates the best production scheduling of the first product $i = 1$. It can be observed that the production of $i = 1$ at the first macro-period exceeds the demands in process 1 and 2. Consequently, the excess is stored in the intermediate inventory to be further used in the on-peak macro-period. The energy consumption planning derived from the obtained scheduling scheme is illustrated in Fig. 13. It is obvious that the integration of both onsite RES and an ESS, as well as inventory policies, reduces the grid energy consumption during on-peak macro-period ($t = 3$). As a result, the total inventory cost is set at 461 €, the grid energy cost amounts to 898 €, while the solar power and ESS cost is set at 365 €.

### 5.2.3. 3rd step: resolve the FLO

The final step involves solving the FLO problem once again in order

<figure><img src="image_13.png" /><figcaption>Fig. 12. Best production schedule for item 1.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Fig. 11. System sets at different time steps.</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>Fig. 13. Total energy consumption at each macro-period.</figcaption></figure>

to explore scheduling and lot sizing at the micro-period scale. The Gantt chart of the best-obtained scheduling scheme is depicted in Fig. 14. The color-coded boxes provide a visual representation of the produced and buffered quantities across different products through the scheduling horizon. The green boxes correspond to product 1 ($i = 1$). The orange boxes show the quantities produced and buffered of product 2 ($i = 2$), while the blue boxes correspond to product 3 ($i = 3$). It can be seen that the schedule is reasonable in the sense that there is no overlap between different production steps of the same product, and the next production step is started only when the previous production step is completed. The best sequence that leads to a minimal setup cost of 1078 € is presented in Table 10. The proposed approach successfully solves the problem in three primary stages with a CPU time of 3.28 s, resulting in a total cost of 2 802 €.

### 5.2.4. Sensitivity analysis on TOU macro-periods

#### 5.2.4.1. Modification of on-peak macro-periods.
In this experiment, the on-peak-macro-period is switched from the third to the second macro-period ($t = 2$). The other parameters are maintained. Results are shown in Figs. 15-16. As can be seen in Fig. 15, following the same concept, an extra quantity of $i = 1$ is manufactured and stored in the first macro-period to be used in the on-peak macro-period. The energy management scheme shown in Fig. 16 is changed. It can be seen that the ESS is used in the second macro-period. The ESS is discharged in $t = 2$. $f = 1$ and charged in the third macro-period. The manufacturing system relies mainly on alternative sources of energy rather than on conventional energy. Regarding the scheduling problem, the machine assigning

<table><caption>Table 10
Best sequence.</caption><thead><tr><th>Macro-period</th><th>Sequence</th><th>Cost</th></tr></thead><tbody><tr><td>t = 1</td><td>$i = 2/i = 3/i = 1$</td><td>256 €</td></tr><tr><td>t = 2</td><td>$i = 1/i = 3/i = 2$</td><td>293 €</td></tr><tr><td>t = 3</td><td>$i = 2/i = 3/i = 1$</td><td>256 €</td></tr><tr><td>t = 4</td><td>$i = 1/i = 2/i = 3$</td><td>273 €</td></tr></tbody></table>

and production sequence are the same, with respect to the benchmark case. Finally, the total cost, is set to 2 919 € as follows; 1 538 € operational cost, 1 104 € grid energy cost and 276 € solar energy and ESS cost.

#### 5.2.4.2. Modification of time granularity.
In this case, a scheduling horizon of thirty-six hours ahead is simulated, where $|T| = 6$ macro-periods. Each macro-period consists of six micro-periods of one hour. Parameters that depend on time such as demands, solar energy availability, and TOU energy prices are adjusted considering the new planning horizon. The on-peak macro-periods are set to macro-periods $t \in \{3, 5, 6\}$ and the off-peak macro-periods are set to $t \in \{1, 2, 4\}$. In this case, the new time-dependent parameters are shown in Tables 11 and 12. All the other parameters used in the benchmark were kept constant.

Results are shown in Figs. 17-18. As presented in Fig. 17, conventional grid energy is less used during $t \in \{3, 5, 6\}$ compared to other macro-periods. In these on-peak macro-periods, manufacturing facility consumes energy from alternative sources like ESS and PV panels. It can be seen in Fig. 18 that the ESS is discharged in on-peak macro-periods. Finally, the total cost is set to 4482 € as follows; conventional energy cost = 1891 €, solar energy = 389 €, ESS cost = 97 € and operational cost = 2105 €.

### 5.2.5. Cost comparison under different scenarios

The proposed model has been tested under different scenarios to demonstrate the economic performance of our research. We used the proposed approach for the resolution process. In the baseline model, we supposed that there is no RES, no ESS and no inventories ($G_{ij} = 0$MW, $R^+ = 0$MWh and $N_p = 0$). The second scenario carried out the proposed model without considering the RES and the ESS. In the third scenario, the model was tested with the assumption that there were no inventories ($N_p = 0$). As shown in Table 13, it is evident that our model is the most cost-effective, with a total cost equal to 2802 €. Additionally, it can be observed that the conventional electricity cost obtained by our model is approximately 59 % less than the case without RES and ESS. Hence, these results highlight the fact that the integration of RES and ESS can significantly reduce electricity consumption costs without compromising production throughput. Results obtained from the same scenario without an inventory policy emphasize the importance of buffers in reducing energy consumption during on-peak macro-

<figure><img src="image_15.png" /><figcaption>Fig. 14. Best production scheme.</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>Fig. 13. Total energy consumption at each macro-period.</figcaption></figure>

to explore scheduling and lot sizing at the micro-period scale. The Gantt chart of the best-obtained scheduling scheme is depicted in Fig. 14. The color-coded boxes provide a visual representation of the produced and buffered quantities across different products through the scheduling horizon. The green boxes correspond to product 1 ($i$ = 1). The orange boxes show the quantities produced and buffered of product 2 ($i$ = 2), while the blue boxes correspond to product 3 ($i$ = 3). It can be seen that the schedule is reasonable in the sense that there is no overlap between different production steps of the same product, and the next production step is started only when the previous production step is completed. The best sequence that leads to a minimal setup cost of 1078 € is presented in Table 10. The proposed approach successfully solves the problem in three primary stages with a CPU time of 3.28 s, resulting in a total cost of 2 802 €.

### 5.2.4. Sensitivity analysis on TOU macro-periods

#### 5.2.4.1. Modification of on-peak macro-periods.
In this experiment, the on-peak-macro-period is switched from the third to the second macro-period ($t$ = 2). The other parameters are maintained. Results are shown in Figs. 15-16. As can be seen in Fig. 15, following the same concept, an extra quantity of $i$ = 1 is manufactured and stored in the first macro-period to be used in the on-peak macro-period. The energy management scheme shown in Fig. 16 is changed. It can be seen that the ESS is used in the second macro-period. The ESS is discharged in $t$ = 2. $f$ = 1 and charged in the third macro-period. The manufacturing system relies mainly on alternative sources of energy rather than on conventional energy. Regarding the scheduling problem, the machine assigning

<table><caption>Table 10
Best sequence.</caption><thead><tr><th>Macro-period</th><th>Sequence</th><th>Cost</th></tr></thead><tbody><tr><td>t = 1</td><td>i = 2/i = 3/i = 1</td><td>256 €</td></tr><tr><td>t = 2</td><td>i = 1/i = 3/i = 2</td><td>293 €</td></tr><tr><td>t = 3</td><td>i = 2/i = 3/i = 1</td><td>256 €</td></tr><tr><td>t = 4</td><td>i = 1/i = 2/i = 3</td><td>273 €</td></tr></tbody></table>

and production sequence are the same, with respect to the benchmark case. Finally, the total cost, is set to 2 919 € as follows; 1 538 € operational cost, 1 104 € grid energy cost and 276 € solar energy and ESS cost.

#### 5.2.4.2. Modification of time granularity.
In this case, a scheduling horizon of thirty-six hours ahead is simulated, where $|T|$ = 6 macro-periods. Each macro-period consists of six micro-periods of one hour. Parameters that depend on time such as demands, solar energy availability, and TOU energy prices are adjusted considering the new planning horizon. The on-peak macro-periods are set to macro-periods $t \in \{3, 5, 6\}$ and the off-peak macro-periods are set to $t \in \{1, 2, 4\}$. In this case, the new time-dependent parameters are shown in Tables 11 and 12. All the other parameters used in the benchmark were kept constant.

Results are shown in Figs. 17-18. As presented in Fig. 17, conventional grid energy is less used during $t \in \{3, 5, 6\}$ compared to other macro-periods. In these on-peak macro-periods, manufacturing facility consumes energy from alternative sources like ESS and PV panels. It can be seen in Fig. 18 that the ESS is discharged in on-peak macro-periods. Finally, the total cost is set to 4482 € as follows; conventional energy cost = 1891 €, solar energy = 389 €, ESS cost = 97 € and operational cost = 2105 €.

### 5.2.5. Cost comparison under different scenarios

The proposed model has been tested under different scenarios to demonstrate the economic performance of our research. We used the proposed approach for the resolution process. In the baseline model, we supposed that there is no RES, no ESS and no inventories ($G_{ij} = 0$MW, $R^+ = 0$MWh and $N_p = 0$). The second scenario carried out the proposed model without considering the RES and the ESS. In the third scenario, the model was tested with the assumption that there were no inventories ($N_p = 0$). As shown in Table 13, it is evident that our model is the most cost-effective, with a total cost equal to 2802 €. Additionally, it can be observed that the conventional electricity cost obtained by our model is approximately 59 % less than the case without RES and ESS. Hence, these results highlight the fact that the integration of RES and ESS can significantly reduce electricity consumption costs without compromising production throughput. Results obtained from the same scenario without an inventory policy emphasize the importance of buffers in reducing energy consumption during on-peak macro-

<figure><img src="image_16.png" /><figcaption>Fig. 14. Best production scheme.</figcaption></figure>
<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /><figcaption>Fig. 15. Best production schedule for modified peak macro-period.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>Fig. 16. Energy consumption for modified peak macro-period.</figcaption></figure>

<table><caption>Table 11
Demands for the new scheduling horizon.</caption><thead><tr><th>time \ products</th><th>i = 1</th><th>i = 2</th><th>i = 3</th></tr></thead><tbody><tr><td>t = 1, f = 6</td><td>40</td><td>41</td><td>33</td></tr><tr><td>t = 2, f = 6</td><td>36</td><td>41</td><td>52</td></tr><tr><td>t = 3, f = 6</td><td>53</td><td>42</td><td>54</td></tr><tr><td>t = 4, f = 6</td><td>29</td><td>65</td><td>30</td></tr><tr><td>t = 5, f = 6</td><td>45</td><td>31</td><td>27</td></tr><tr><td>t = 6, f = 6</td><td>25</td><td>50</td><td>31</td></tr></tbody></table>

periods. However, it should be noted that inventories alone do not necessarily lead to a reduction in energy consumption during on-peak macro-periods. Interestingly, their integration with other components

<table><caption>Table 12
Energy availability for the new scheduling horizon.</caption><thead><tr><th>f</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th></tr></thead><tbody><tr><td>G<sub>1r</sub> (MW)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>G<sub>2r</sub> (MW)</td><td>0</td><td>0</td><td>0</td><td>0.2</td><td>0.37</td><td>0.67</td></tr><tr><td>G<sub>3r</sub> (MW)</td><td>0.76</td><td>0.61</td><td>0.52</td><td>0.68</td><td>0.33</td><td>0.2</td></tr><tr><td>G<sub>4r</sub> (MW)</td><td>0.18</td><td>0.1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>G<sub>5</sub> (MW)</td><td>0</td><td>0.2</td><td>0.4</td><td>0.434</td><td>0.72</td><td>0.43</td></tr><tr><td>G<sub>6r</sub> (MW)</td><td>0.84</td><td>0.45</td><td>0.77</td><td>0.51</td><td>0.27</td><td>0.19</td></tr></tbody></table>

<figure><img src="image_21.png" /><figcaption>Fig. 17. Energy consumption for the new scheduling horizon.</figcaption></figure>

such as RES and ESS leads to more optimal power consumption and cost savings.
<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /><figcaption>Fig. 15. Best production schedule for modified peak macro-period.</figcaption></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /><figcaption>Fig. 16. Energy consumption for modified peak macro-period.</figcaption></figure>

<table><caption>Table 11
Demands for the new scheduling horizon.</caption><thead><tr><th>time \ products</th><th>i = 1</th><th>i = 2</th><th>i = 3</th></tr></thead><tbody><tr><td>t = 1, f = 6</td><td>40</td><td>41</td><td>33</td></tr><tr><td>t = 2, f = 6</td><td>36</td><td>41</td><td>52</td></tr><tr><td>t = 3, f = 6</td><td>53</td><td>42</td><td>54</td></tr><tr><td>t = 4, f = 6</td><td>29</td><td>65</td><td>30</td></tr><tr><td>t = 5, f = 6</td><td>45</td><td>31</td><td>27</td></tr><tr><td>t = 6, f = 6</td><td>25</td><td>50</td><td>31</td></tr></tbody></table>

periods. However, it should be noted that inventories alone do not necessarily lead to a reduction in energy consumption during on-peak macro-periods. Interestingly, their integration with other components

<table><caption>Table 12
Energy availability for the new scheduling horizon.</caption><thead><tr><th>f</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th></tr></thead><tbody><tr><td>G<sub>1r</sub> (MW)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>G<sub>2r</sub> (MW)</td><td>0</td><td>0</td><td>0</td><td>0.2</td><td>0.37</td><td>0.67</td></tr><tr><td>G<sub>3r</sub> (MW)</td><td>0.76</td><td>0.61</td><td>0.52</td><td>0.68</td><td>0.33</td><td>0.2</td></tr><tr><td>G<sub>4r</sub> (MW)</td><td>0.18</td><td>0.1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>G<sub>5</sub> (MW)</td><td>0</td><td>0.2</td><td>0.4</td><td>0.434</td><td>0.72</td><td>0.43</td></tr><tr><td>G<sub>6r</sub> (MW)</td><td>0.84</td><td>0.45</td><td>0.77</td><td>0.51</td><td>0.27</td><td>0.19</td></tr></tbody></table>

<figure><img src="image_22.png" /><figcaption>Fig. 17. Energy consumption for the new scheduling horizon.</figcaption></figure>

such as RES and ESS leads to more optimal power consumption and cost savings.
<figure><img src="image_22.png" /><figcaption>Fig. 18. Energy consumption for the new scheduling horizon.</figcaption></figure>

<table><caption>**Table 13**
Comparison of costs among different scenarios.</caption><thead><tr><th>Model</th><th>Conventional electricity charge (€)</th><th>Inventory charge (€)</th><th>RES & ESS charge(€)</th><th>Total cost (€)</th></tr></thead><tbody><tr><td>1-Baseline</td><td>2198</td><td>0</td><td>0</td><td>3276</td></tr><tr><td>2-No RES & no ESS</td><td>1649</td><td>461</td><td>0</td><td>3188</td></tr><tr><td>3-No inventories</td><td>1580</td><td>0</td><td>365</td><td>3023</td></tr><tr><td>4-Proposed model</td><td>898</td><td>461</td><td>365</td><td>2802</td></tr></tbody></table>

## 5.3. Performance evaluation of the reinforcement learning approach

Based on Özdamar and Barbarasoglu (1999), we conducted computational experiments on three groups of instances. To this end, small, medium, and large-scale ILSPS instances were generated randomly to validate the effectiveness of our RL algorithm. The small-scale instances (S1 to S5) comprise four products, two processes, and two parallel machines per process. The medium instances (M1 to M5) consist of eight products, three processes, and three parallel machines within each process, while the large-scale instances (L1 to L5) consist of twelve products, four processes and four parallel machines per process. It is important to note that in this analysis, all instances were assumed to have a uniform planning horizon of $|T| = 1$ and $|F| = 6$. For each group of instances, demands and machine processing times exhibit variability from one instance to another. These values were randomly generated according to the methodology outlined before.

### 5.3.1. Training process analysis

The training process and cumulative rewards for process agents and machine agents are shown in Figs. 19, 20, 21 and 22, for one medium instance (MI) and one large instance (LI) respectively. These figures shed light on the evolution and performance of the MAS architecture during the training process. In the assignment problem, the cumulative reward curves start to converge around 100 episodes for the medium instance and around 400 episodes for the large instance. This convergence state indicates that the process agents have learned an optimal policy and consistently achieved high rewards. It also indicates that the process agents have reached a stable level of performance. In addition, the downward trend in energy consumption values in Figs. 19 and 21 demonstrates that process agents are able to reduce energy consumption throughout the training process. Nevertheless, some fluctuations in the energy consumption curve can be observed after the convergence of the model. Such a phenomenon may arise due to the exploitation-exploration trade-off of the process agent. However, the maximum energy levels reported in Figs. 19 and 21 are 67.08 MWh for the medium

<figure><img src="image_23.png" /><figcaption>Fig. 19. MI Training reward for process agents.</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>Fig. 20. MI Training reward for machine agents.</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>Fig. 21. LI Training reward for process agents.</figcaption></figure>

instance and 96.23 MWh for the large instance. These values represent the worst-case scenarios encountered by process agents, yet they still demonstrate improved energy consumption compared to initial or random policies. Additionally, it is worth mentioning that the best energy consumption achieved for the medium instance reaches 59.81
<figure><img src="image_23.png" /><figcaption>Fig. 18. Energy consumption for the new scheduling horizon.</figcaption></figure>

<table><caption>**Table 13**
Comparison of costs among different scenarios.</caption><thead><tr><th>Model</th><th>Conventional electricity charge (€)</th><th>Inventory charge (€)</th><th>RES & ESS charge(€)</th><th>Total cost (€)</th></tr></thead><tbody><tr><td>1-Baseline</td><td>2198</td><td>0</td><td>0</td><td>3276</td></tr><tr><td>2-No RES & no ESS</td><td>1649</td><td>461</td><td>0</td><td>3188</td></tr><tr><td>3-No inventories</td><td>1580</td><td>0</td><td>365</td><td>3023</td></tr><tr><td>4-Proposed model</td><td>898</td><td>461</td><td>365</td><td>2802</td></tr></tbody></table>

### 5.3. Performance evaluation of the reinforcement learning approach

Based on Özdamar and Barbarasoglu (1999), we conducted computational experiments on three groups of instances. To this end, small, medium, and large-scale ILSPS instances were generated randomly to validate the effectiveness of our RL algorithm. The small-scale instances (S1 to S5) comprise four products, two processes, and two parallel machines per process. The medium instances (M1 to M5) consist of eight products, three processes, and three parallel machines within each process, while the large-scale instances (L1 to L5) consist of twelve products, four processes and four parallel machines per process. It is important to note that in this analysis, all instances were assumed to have a uniform planning horizon of $|T| = 1$ and $|F| = 6$. For each group of instances, demands and machine processing times exhibit variability from one instance to another. These values were randomly generated according to the methodology outlined before.

#### 5.3.1. Training process analysis

The training process and cumulative rewards for process agents and machine agents are shown in Figs. 19, 20, 21 and 22, for one medium instance (MI) and one large instance (LI) respectively. These figures shed light on the evolution and performance of the MAS architecture during the training process. In the assignment problem, the cumulative reward curves start to converge around 100 episodes for the medium instance and around 400 episodes for the large instance. This convergence state indicates that the process agents have learned an optimal policy and consistently achieved high rewards. It also indicates that the process agents have reached a stable level of performance. In addition, the downward trend in energy consumption values in Figs. 19 and 21 demonstrates that process agents are able to reduce energy consumption throughout the training process. Nevertheless, some fluctuations in the energy consumption curve can be observed after the convergence of the model. Such a phenomenon may arise due to the exploitation-exploration trade-off of the process agent. However, the maximum energy levels reported in Figs. 19 and 21 are 67.08 MWh for the medium

<figure><img src="image_24.png" /><figcaption>Fig. 19. MI Training reward for process agents.</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>Fig. 20. MI Training reward for machine agents.</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>Fig. 21. LI Training reward for process agents.</figcaption></figure>

instance and 96.23 MWh for the large instance. These values represent the worst-case scenarios encountered by process agents, yet they still demonstrate improved energy consumption compared to initial or random policies. Additionally, it is worth mentioning that the best energy consumption achieved for the medium instance reaches 59.81
<figure><img src="image_26.png" /><figcaption>Fig. 22. LI Training reward for machine agents.</figcaption></figure>

MWh after 300 episodes.

Regarding sequencing problems, each machine agent aims to determine the best product sequencing to minimize setup costs. The cumulative reward of the machine agents represents the setup costs. In Figs. 20 and 22, we can observe a downward trend in the setup costs, indicating that the machine agents have been successfully reducing these costs over time. The cumulative reward curves start to converge around 100 episodes for the MI and around 200 episodes for the LI. This trend reflects that agents have learned to improve their product sequencing strategy, leading to better setup processes and ultimately lower costs. The worst case scenarios found by machine agents within one episode are set at 1298 € and for the MI and 1896 € for the LI. On the other hand, the best scenarios are set at 922 € for the MI and 1607 € for the LI.

To conclude, MAS agents have been trained effectively, which exhibit a downward trend in energy consumption and setup cost as well as convergence of the cumulative reward curve. The agents have learned an optimal policy, resulting in stable performance and consistently improved scheduling decisions.

### 5.3.2. Computational results of generated instances

Further experiments were conducted using the instances presented in section 5.3. In this subsection, the rewards derived from the solutions obtained by our approach were compared with those generated by CPLEX for both assignment and sequencing problems. In the next simulation, CPLEX v.12.10 was used limited in 1000 s.

Table 14 presents the outcomes of these instances for the assignment problem. Notably, the cumulative reward achieved in the assignment problem matches the best energy consumption for all small instances, resulting in a relative error (RE) of 0 % for each instance. It is worth mentioning that smaller instances have fewer variables, constraints and possible actions, which makes it easier for the process agent to quickly explore different actions and learn the optimal policy. For medium instances, the RE varies from 0.61 % to 1.56 %, with an average RE of 1.01 %. These results indicate that the achieved energy consumption is relatively close to the best energy consumption obtained by CPLEX. However, for large instances, the RE varies from 1.5 % to 4.6 % with an average RE of 2.8 %. This implies a slightly higher deviation from energy consumption obtained by the automatic solver compared to smaller instances. The exponential expansion of the search space for potential solutions in larger instances makes it more challenging to explore and to determine good solutions.

Results, associated with the sequencing problem, are shown in Table 15. They largely follow the observations made for the assignment problem. The cumulative rewards obtained by the machine agents match the same solution obtained by CPLEX for all small instances,

<table><caption>Table 14
Process agent results on different instances.</caption><thead><tr><th>Instances<br>Problem<br>Size</th><th>Instance<br>reference</th><th>The best cumulative<br>reward of the<br>assignment problem<br>given by the RL<br>approach</th><th>The cumulative<br>reward of the<br>assignment<br>problem calculated<br>by CPLEX</th><th>cost<br>RE</th></tr></thead><tbody><tr><td rowspan="5">Small<br>(4,2,2)</td><td>S1</td><td>29</td><td>29</td><td>0.0 %</td></tr><tr><td>S2</td><td>21.233</td><td>21.233</td><td>0.0 %</td></tr><tr><td>S3</td><td>22.043</td><td>22.043</td><td>0.0 %</td></tr><tr><td>S4</td><td>21.047</td><td>21.047</td><td>0.0 %</td></tr><tr><td>S5</td><td>20.205</td><td>20.205</td><td>0.0 %</td></tr><tr><td rowspan="5">Average</td><td>M1</td><td>22.705</td><td>22.705</td><td>0.0 %</td></tr><tr><td>M2</td><td>57.836</td><td>57.296</td><td>0.94 %</td></tr><tr><td>M3</td><td>56.699</td><td>55.823</td><td>1.56 %</td></tr><tr><td>M4</td><td>58.494</td><td>57.869</td><td>1.08 %</td></tr><tr><td>M5</td><td>61.014</td><td>60.64</td><td>0.61 %</td></tr><tr><td rowspan="5">Medium<br>(8,3,3)</td><td>L1</td><td>58.7706</td><td>58.1</td><td>1.01 %</td></tr><tr><td>L2</td><td>91.15</td><td>87.14</td><td>4.6 %</td></tr><tr><td>L3</td><td>88.631</td><td>85.24</td><td>3.97 %</td></tr><tr><td>L4</td><td>93.31</td><td>91.6</td><td>1.86 %</td></tr><tr><td>L5</td><td>94.49</td><td>92.36</td><td>2.3 %</td></tr><tr><td rowspan="5">Average</td><td>L1</td><td>90.41</td><td>89.05</td><td>1.5 %</td></tr><tr><td>L2</td><td>91.59</td><td>89.07</td><td>2.8 %</td></tr></tbody></table>

<table><caption>Table 15
Machine agent results on different instances.</caption><thead><tr><th>Instances<br>Problem<br>Size</th><th>Instance<br>reference</th><th>The best cumulative<br>reward of the<br>sequencing problem<br>given by the RL<br>approach</th><th>The cumulative<br>reward of the<br>assignment<br>problem calculated<br>by CPLEX</th><th>cost<br>RE</th></tr></thead><tbody><tr><td rowspan="5">Small<br>(4,2,2)</td><td>S1</td><td>329</td><td>329</td><td>0.0 %</td></tr><tr><td>S2</td><td>347</td><td>347</td><td>0.0 %</td></tr><tr><td>S3</td><td>321</td><td>321</td><td>0.0 %</td></tr><tr><td>S4</td><td>272</td><td>272</td><td>0.0 %</td></tr><tr><td>S5</td><td>325</td><td>325</td><td>0.0 %</td></tr><tr><td rowspan="5">Average</td><td>M1</td><td>318.8</td><td>318.8</td><td>0.0 %</td></tr><tr><td>M2</td><td>1173</td><td>1173</td><td>0.0 %</td></tr><tr><td>M3</td><td>922</td><td>920</td><td>0.2 %</td></tr><tr><td>M4</td><td>1133</td><td>1125</td><td>0.7 %</td></tr><tr><td>M5</td><td>1198</td><td>1185</td><td>1.1 %</td></tr><tr><td rowspan="5">Average</td><td>L1</td><td>1029</td><td>1024</td><td>0.4 %</td></tr><tr><td>L2</td><td>1091</td><td>1085.4</td><td>0.48 %</td></tr><tr><td>L3</td><td>1802</td><td>1781</td><td>1.2 %</td></tr><tr><td>L4</td><td>1607</td><td>1581</td><td>1.6 %</td></tr><tr><td>L5</td><td>1701</td><td>1666</td><td>2.1 %</td></tr><tr><td rowspan="5">Large<br>(12,4,4)</td><td>L1</td><td>1792</td><td>1753</td><td>2.2 %</td></tr><tr><td>L2</td><td>1744</td><td>1711.6</td><td>1.88 %</td></tr></tbody></table>

resulting in a relative error of 0 % for each instance. For medium instances, the relative errors vary from 0.2 % to 1.1 %, with an average equal to 0.48 %. This indicates that the achieved sequencing problem rewards are relatively close to the best solution, with only a small deviation. However, for large instances, the relative errors increase, varying from 1.2 % to 2.3 %, with an average value equal to 1.88 %. These larger instances exhibit a slightly higher deviation from the CPLEX solution compared to the smaller ones. Overall, the analysis of both the performance of the machine agents and the process agents indicates that the proposed method performs well, and the achieved rewards values being relatively close to the best solutions. The results align with the observations made previously regarding the impact of problem size, complexity, and search space.

### 5.3.3. Comparison of results with FIFO and genetic algorithm (GA) approaches

In this subsection, we kept the instances previously used (S1 to S5, M1 to M5 and L1 to L5) to conduct a comparative analysis of the RL approach against the heuristic rule FIFO and genetic algorithm (GA)
<figure><img src="image_27.png" /><figcaption>Fig. 22. LI Training reward for machine agents.</figcaption></figure>

MWh after 300 episodes.

Regarding sequencing problems, each machine agent aims to determine the best product sequencing to minimize setup costs. The cumulative reward of the machine agents represents the setup costs. In Figs. 20 and 22, we can observe a downward trend in the setup costs, indicating that the machine agents have been successfully reducing these costs over time. The cumulative reward curves start to converge around 100 episodes for the MI and around 200 episodes for the LI. This trend reflects that agents have learned to improve their product sequencing strategy, leading to better setup processes and ultimately lower costs. The worst case scenarios found by machine agents within one episode are set at 1298 € and for the MI and 1896 € for the LI. On the other hand, the best scenarios are set at 922 € for the MI and 1607 € for the LI.

To conclude, MAS agents have been trained effectively, which exhibit a downward trend in energy consumption and setup cost as well as convergence of the cumulative reward curve. The agents have learned an optimal policy, resulting in stable performance and consistently improved scheduling decisions.

### 5.3.2. Computational results of generated instances

Further experiments were conducted using the instances presented in section 5.3. In this subsection, the rewards derived from the solutions obtained by our approach were compared with those generated by CPLEX for both assignment and sequencing problems. In the next simulation, CPLEX v.12.10 was used limited in 1000 s.

Table 14 presents the outcomes of these instances for the assignment problem. Notably, the cumulative reward achieved in the assignment problem matches the best energy consumption for all small instances, resulting in a relative error (RE) of 0 % for each instance. It is worth mentioning that smaller instances have fewer variables, constraints and possible actions, which makes it easier for the process agent to quickly explore different actions and learn the optimal policy. For medium instances, the RE varies from 0.61 % to 1.56 %, with an average RE of 1.01 %. These results indicate that the achieved energy consumption is relatively close to the best energy consumption obtained by CPLEX. However, for large instances, the RE varies from 1.5 % to 4.6 % with an average RE of 2.8 %. This implies a slightly higher deviation from energy consumption obtained by the automatic solver compared to smaller instances. The exponential expansion of the search space for potential solutions in larger instances makes it more challenging to explore and to determine good solutions.

Results, associated with the sequencing problem, are shown in Table 15. They largely follow the observations made for the assignment problem. The cumulative rewards obtained by the machine agents match the same solution obtained by CPLEX for all small instances,

<table><caption>Table 14
Process agent results on different instances.</caption><thead><tr><th>Instances<br>Problem<br>Size</th><th>Instance<br>reference</th><th>The best cumulative<br>reward of the<br>assignment problem<br>given by the RL<br>approach</th><th>The cumulative<br>reward of the<br>assignment<br>problem calculated<br>by CPLEX</th><th>cost<br>RE</th></tr></thead><tbody><tr><td rowspan="5">Small<br>(4,2,2)</td><td>S1</td><td>29</td><td>29</td><td>0.0 %</td></tr><tr><td>S2</td><td>21.233</td><td>21.233</td><td>0.0 %</td></tr><tr><td>S3</td><td>22.043</td><td>22.043</td><td>0.0 %</td></tr><tr><td>S4</td><td>21.047</td><td>21.047</td><td>0.0 %</td></tr><tr><td>S5</td><td>20.205</td><td>20.205</td><td>0.0 %</td></tr><tr><td rowspan="5">Average</td><td>M1</td><td>22.705</td><td>22.705</td><td>0.0 %</td></tr><tr><td>M2</td><td>57.836</td><td>57.296</td><td>0.94 %</td></tr><tr><td>M3</td><td>56.699</td><td>55.823</td><td>1.56 %</td></tr><tr><td>M4</td><td>58.494</td><td>57.869</td><td>1.08 %</td></tr><tr><td>M5</td><td>61.014</td><td>60.64</td><td>0.61 %</td></tr><tr><td rowspan="5">Medium<br>(8,3,3)</td><td>L1</td><td>58.7706</td><td>58.1</td><td>1.01 %</td></tr><tr><td>L2</td><td>91.15</td><td>87.14</td><td>4.6 %</td></tr><tr><td>L3</td><td>88.631</td><td>85.24</td><td>3.97 %</td></tr><tr><td>L4</td><td>93.31</td><td>91.6</td><td>1.86 %</td></tr><tr><td>L5</td><td>94.49</td><td>92.36</td><td>2.3 %</td></tr><tr><td rowspan="5">Average</td><td>L1</td><td>90.41</td><td>89.05</td><td>1.5 %</td></tr><tr><td>L2</td><td>91.59</td><td>89.07</td><td>2.8 %</td></tr></tbody></table>

<table><caption>Table 15
Machine agent results on different instances.</caption><thead><tr><th>Instances<br>Problem<br>Size</th><th>Instance<br>reference</th><th>The best cumulative<br>reward of the<br>sequencing problem<br>given by the RL<br>approach</th><th>The cumulative<br>reward of the<br>assignment<br>problem calculated<br>by CPLEX</th><th>cost<br>RE</th></tr></thead><tbody><tr><td rowspan="5">Small<br>(4,2,2)</td><td>S1</td><td>329</td><td>329</td><td>0.0 %</td></tr><tr><td>S2</td><td>347</td><td>347</td><td>0.0 %</td></tr><tr><td>S3</td><td>321</td><td>321</td><td>0.0 %</td></tr><tr><td>S4</td><td>272</td><td>272</td><td>0.0 %</td></tr><tr><td>S5</td><td>325</td><td>325</td><td>0.0 %</td></tr><tr><td rowspan="5">Average</td><td>M1</td><td>318.8</td><td>318.8</td><td>0.0 %</td></tr><tr><td>M2</td><td>1173</td><td>1173</td><td>0.0 %</td></tr><tr><td>M3</td><td>922</td><td>920</td><td>0.2 %</td></tr><tr><td>M4</td><td>1133</td><td>1125</td><td>0.7 %</td></tr><tr><td>M5</td><td>1198</td><td>1185</td><td>1.1 %</td></tr><tr><td rowspan="5">Average</td><td>L1</td><td>1029</td><td>1024</td><td>0.4 %</td></tr><tr><td>L2</td><td>1091</td><td>1085.4</td><td>0.48 %</td></tr><tr><td>L3</td><td>1802</td><td>1781</td><td>1.2 %</td></tr><tr><td>L4</td><td>1607</td><td>1581</td><td>1.6 %</td></tr><tr><td>L5</td><td>1701</td><td>1666</td><td>2.1 %</td></tr><tr><td rowspan="5">Large<br>(12,4,4)</td><td>L1</td><td>1792</td><td>1753</td><td>2.2 %</td></tr><tr><td>L2</td><td>1744</td><td>1711.6</td><td>1.88 %</td></tr></tbody></table>

resulting in a relative error of 0 % for each instance. For medium instances, the relative errors vary from 0.2 % to 1.1 %, with an average equal to 0.48 %. This indicates that the achieved sequencing problem rewards are relatively close to the best solution, with only a small deviation. However, for large instances, the relative errors increase, varying from 1.2 % to 2.3 %, with an average value equal to 1.88 %. These larger instances exhibit a slightly higher deviation from the CPLEX solution compared to the smaller ones. Overall, the analysis of both the performance of the machine agents and the process agents indicates that the proposed method performs well, and the achieved rewards values being relatively close to the best solutions. The results align with the observations made previously regarding the impact of problem size, complexity, and search space.

### 5.3.3. Comparison of results with FIFO and genetic algorithm (GA) approaches

In this subsection, we kept the instances previously used (S1 to S5, M1 to M5 and L1 to L5) to conduct a comparative analysis of the RL approach against the heuristic rule FIFO and genetic algorithm (GA)
approaches. This evaluation encompasses considerations of costs, computation time (CPU) and performance measured by the obtained relative error. The heuristic FIFO, a widely used simple rule, prioritizes products based on their arrival order. In the first micro-period of the scheduling horizon, FIFO assigns products to the primary process machines. This assignment is predicated on the machines' minimal energy consumption and their current availability. Subsequently, for each designed machine, FIFO assigns the first product in line. This precise selection process ensures that the earliest arriving products are processed first. As the workflow progresses to the subsequent process, FIFO allocates the foremost available product to a designated machine. This process goes on for the remaining micro-periods and processes, ensuring that products are processed in the order they arrive and machine are allocated on the basis of their energy consumption. Applying FIFO in our context leads to feasible solutions with partially optimized energy consumption and non-optimized product sequencing. The GA is a popular metaheuristic technique known for its exploration capabilities. With regard to GA application, the method proposed by Valledor et al. (2018) was adapted to our setting. In GA, two populations are defined; the first one is responsible for machine assignment and the second one is responsible for sequencing problem. GA starts with two initial populations, performing selection, crossover and mutation between individuals and returns a feasible solution. The parameters of the GA are set as follows: iteration number = 200, population size = 30, mutation probability = 0.2 and crossover probability = 0.8.

Table 16 presents a comparison of costs obtained by the three methods for the same problem instances. CPLEX_1 cost in columns represents the cost obtained by CPLEX starting from scratch. CPLEX_2 reports the cost obtained by CPLEX starting from the initial solution generated by our proposed reinforcement learning technique. The objective of this strategy is to ascertain whether optimality can be achieved within a time constraint. If the execution time is below 1000 s, the corresponding instance is solved to optimality.

The results demonstrate that the RL approach consistently outperforms FIFO in terms of cost minimization across all problem sizes. In comparison to GA, the RL approach generally achieves better overall results, with smaller total costs on most instances. Only on two instances, the GA outperforms the RL approach in terms of total cost. Nevertheless, the RL approach achieves results close to the CPLEX_1 solutions, with negligible relative errors for small and medium instances. Even for large instances, the RL approach maintains its competitiveness, although with a slightly increased relative error. One-way analysis of variance (ANOVA) was conducted on each type of instances (Small, Medium, Large) using R software, version 4.3.1 (R Core Team, 2021). The cost mean plots together with LSD (Least Significant

Difference) 95 % confidence intervals associated with the various approaches are reported in Fig. 23 according to the type of instances. As shown in the figure, the RL approach shows significant improvements over heuristic rule FIFO and performs competitively against GA. The RL approach consistently achieves lower costs and produces results close to solutions obtained by CPLEX_1, indicating its effectiveness in solving ILSPS problems. The learning nature of agents in the RL approach contributes to this performance.

Regarding computational time evaluation criteria, the average execution times corresponding to the different methods applied to the varying size problems are shown in Table 17. For small instances, both the CPLEX scenarios reach the optimum in all instances. For instances M3 and M5 from medium instances, using the initial solution of the proposed RL approach, CPLEX_2 can find a better solution than CPLEX. However, these solutions are not optimal since their execution times exceed 1000 s as shown in Table 17. For large instances, due to the increasing complexity of these instances, CPLEX_2 also fails to prove optimality within 1000 s.

Regarding other approaches, for all problem instances, the FIFO method takes at most 0.23 s to find a local solution. While FIFO may provide faster execution times, it is not able to lead to the best-quality solutions. On the other hand, GA involves a more sophisticated algorithm that aims to explore the search space and find better solutions. This exploration process typically requires more computational effort. The resulting execution time appears to be very long, to respond to the changes in the FFL environment. However, the RL approach did not exceed 8 s to make a decision while the automatic solver reached a low bound solution within 1000 s. The RL approach has the potential to explore the search space more comprehensively and find solutions that are closer to the CPLEX solver. It is able to find higher-quality solutions than the GA in less time.

Finally, it is important to note that the RL approach can adapt immediately to the environment changes and consequently deal with the uncertainty in the ILSPS problem. It offers a combination of efficiency, solution quality, and adaptability, making it a promising approach for optimizing ILSPS processes.

## 6. Managerial insights

This paper proposes a novel approach for production scheduling that has practical implications and offers valuable insights for different stakeholders. First, the proposed approach can help operations managers select the most energy-efficient production scheduling strategy without compromising production throughput. This dynamic sustainable production-scheduling model that takes energy costs into account

<table><caption>**Table 16**
Experimental results.</caption><thead><tr><th>Instances<br>Problem Size</th><th>Instance reference</th><th>RL approach cost (€)</th><th>GA cost (€)</th><th>FIFO cost (€)</th><th>CPLEX_1 cost (€)</th><th>CPLEX_2 cost (€)</th><th>MAS RE compared with CPLEX_1</th></tr></thead><tbody><tr><td rowspan="5">Small<br>(4,2,2)</td><td>S1</td><td>2939</td><td>2947</td><td>3667</td><td>2939</td><td>2939</td><td>0.0 %</td></tr><tr><td>S2</td><td>2258</td><td>2316</td><td>2914</td><td>2258</td><td>2258</td><td>0.0 %</td></tr><tr><td>S3</td><td>2305</td><td>2342</td><td>3298</td><td>2305</td><td>2305</td><td>0.0 %</td></tr><tr><td>S4</td><td>2166</td><td>2201</td><td>2921</td><td>2166</td><td>2166</td><td>0.0 %</td></tr><tr><td>S5</td><td>2143</td><td>2159</td><td>3153</td><td>2143</td><td>2143</td><td>0.0 %</td></tr><tr><td rowspan="5">Average</td><td></td><td>2362</td><td>2393</td><td>3191</td><td>2362</td><td>2362</td><td>0.0 %</td></tr><tr><td>M1</td><td>6378</td><td>6351</td><td>7117</td><td>6330</td><td>6330</td><td>0.7 %</td></tr><tr><td>M2</td><td>6025</td><td>6113</td><td>7415</td><td>5944</td><td>5944</td><td>1.3 %</td></tr><tr><td>M3</td><td>6397</td><td>6429</td><td>7381</td><td>6333</td><td>6297</td><td>1.01 %</td></tr><tr><td>M4</td><td>6689</td><td>6740</td><td>7479</td><td>6642</td><td>6642</td><td>0.7 %</td></tr><tr><td rowspan="5">Medium<br>(8,3,3)</td><td></td><td>6412</td><td>6571</td><td>6987</td><td>6359</td><td>6329</td><td>0.83 %</td></tr><tr><td></td><td>6380</td><td>6402</td><td>72,767</td><td>6322</td><td>6308</td><td>0.9 %</td></tr><tr><td>L1</td><td>10,005</td><td>10,846</td><td>11,763</td><td>9624</td><td>6924</td><td>3.9 %</td></tr><tr><td>L2</td><td>9678</td><td>9874</td><td>11,475</td><td>9338</td><td>9338</td><td>3.6 %</td></tr><tr><td>L3</td><td>10,004</td><td>9931</td><td>11,468</td><td>9825</td><td>9825</td><td>1.8 %</td></tr><tr><td rowspan="3">Large (12,4,4)</td><td></td><td>10,322</td><td>10,428</td><td>12,389</td><td>10,089</td><td>10,089</td><td>2.3 %</td></tr><tr><td>L4</td><td>9929</td><td>10,246</td><td>11,982</td><td>9767</td><td>9767</td><td>1.6 %</td></tr><tr><td>L5</td><td>9958</td><td>10,265</td><td>11,835</td><td>9619</td><td>9619</td><td>2.64 %</td></tr></tbody></table>
approaches. This evaluation encompasses considerations of costs, computation time (CPU) and performance measured by the obtained relative error. The heuristic FIFO, a widely used simple rule, prioritizes products based on their arrival order. In the first micro-period of the scheduling horizon, FIFO assigns products to the primary process machines. This assignment is predicated on the machines' minimal energy consumption and their current availability. Subsequently, for each designed machine, FIFO assigns the first product in line. This precise selection process ensures that the earliest arriving products are processed first. As the workflow progresses to the subsequent process, FIFO allocates the foremost available product to a designated machine. This process goes on for the remaining micro-periods and processes, ensuring that products are processed in the order they arrive and machine are allocated on the basis of their energy consumption. Applying FIFO in our context leads to feasible solutions with partially optimized energy consumption and non-optimized product sequencing. The GA is a popular metaheuristic technique known for its exploration capabilities. With regard to GA application, the method proposed by Valledor et al. (2018) was adapted to our setting. In GA, two populations are defined; the first one is responsible for machine assignment and the second one is responsible for sequencing problem. GA starts with two initial populations, performing selection, crossover and mutation between individuals and returns a feasible solution. The parameters of the GA are set as follows: iteration number = 200, population size = 30, mutation probability = 0.2 and crossover probability = 0.8.

Table 16 presents a comparison of costs obtained by the three methods for the same problem instances. CPLEX_1 cost in columns represents the cost obtained by CPLEX starting from scratch. CPLEX_2 reports the cost obtained by CPLEX starting from the initial solution generated by our proposed reinforcement learning technique. The objective of this strategy is to ascertain whether optimality can be achieved within a time constraint. If the execution time is below 1000 s, the corresponding instance is solved to optimality.

The results demonstrate that the RL approach consistently outperforms FIFO in terms of cost minimization across all problem sizes. In comparison to GA, the RL approach generally achieves better overall results, with smaller total costs on most instances. Only on two instances, the GA outperforms the RL approach in terms of total cost. Nevertheless, the RL approach achieves results close to the CPLEX_1 solutions, with negligible relative errors for small and medium instances. Even for large instances, the RL approach maintains its competitiveness, although with a slightly increased relative error. One-way analysis of variance (ANOVA) was conducted on each type of instances (Small, Medium, Large) using R software, version 4.3.1 (R Core Team, 2021). The cost mean plots together with LSD (Least Significant

Difference) 95 % confidence intervals associated with the various approaches are reported in Fig. 23 according to the type of instances. As shown in the figure, the RL approach shows significant improvements over heuristic rule FIFO and performs competitively against GA. The RL approach consistently achieves lower costs and produces results close to solutions obtained by CPLEX_1, indicating its effectiveness in solving ILSPS problems. The learning nature of agents in the RL approach contributes to this performance.

Regarding computational time evaluation criteria, the average execution times corresponding to the different methods applied to the varying size problems are shown in Table 17. For small instances, both the CPLEX scenarios reach the optimum in all instances. For instances M3 and M5 from medium instances, using the initial solution of the proposed RL approach, CPLEX_2 can find a better solution than CPLEX. However, these solutions are not optimal since their execution times exceed 1000 s as shown in Table 17. For large instances, due to the increasing complexity of these instances, CPLEX_2 also fails to prove optimality within 1000 s.

Regarding other approaches, for all problem instances, the FIFO method takes at most 0.23 s to find a local solution. While FIFO may provide faster execution times, it is not able to lead to the best-quality solutions. On the other hand, GA involves a more sophisticated algorithm that aims to explore the search space and find better solutions. This exploration process typically requires more computational effort. The resulting execution time appears to be very long, to respond to the changes in the FFL environment. However, the RL approach did not exceed 8 s to make a decision while the automatic solver reached a low bound solution within 1000 s. The RL approach has the potential to explore the search space more comprehensively and find solutions that are closer to the CPLEX solver. It is able to find higher-quality solutions than the GA in less time.

Finally, it is important to note that the RL approach can adapt immediately to the environment changes and consequently deal with the uncertainty in the ILSPS problem. It offers a combination of efficiency, solution quality, and adaptability, making it a promising approach for optimizing ILSPS processes.

## 6. Managerial insights

This paper proposes a novel approach for production scheduling that has practical implications and offers valuable insights for different stakeholders. First, the proposed approach can help operations managers select the most energy-efficient production scheduling strategy without compromising production throughput. This dynamic sustainable production-scheduling model that takes energy costs into account

<table><caption>**Table 16**
Experimental results.</caption><thead><tr><th>Instances<br>Problem Size</th><th>Instance reference</th><th>RL approach cost (€)</th><th>GA cost (€)</th><th>FIFO cost (€)</th><th>CPLEX_1 cost (€)</th><th>CPLEX_2 cost (€)</th><th>MAS RE compared with CPLEX_1</th></tr></thead><tbody><tr><td rowspan="5">Small<br>(4,2,2)</td><td>S1</td><td>2939</td><td>2947</td><td>3667</td><td>2939</td><td>2939</td><td>0.0 %</td></tr><tr><td>S2</td><td>2258</td><td>2316</td><td>2914</td><td>2258</td><td>2258</td><td>0.0 %</td></tr><tr><td>S3</td><td>2305</td><td>2342</td><td>3298</td><td>2305</td><td>2305</td><td>0.0 %</td></tr><tr><td>S4</td><td>2166</td><td>2201</td><td>2921</td><td>2166</td><td>2166</td><td>0.0 %</td></tr><tr><td>S5</td><td>2143</td><td>2159</td><td>3153</td><td>2143</td><td>2143</td><td>0.0 %</td></tr><tr><td rowspan="5">Average</td><td></td><td>2362</td><td>2393</td><td>3191</td><td>2362</td><td>2362</td><td>0.0 %</td></tr><tr><td>M1</td><td>6378</td><td>6351</td><td>7117</td><td>6330</td><td>6330</td><td>0.7 %</td></tr><tr><td>M2</td><td>6025</td><td>6113</td><td>7415</td><td>5944</td><td>5944</td><td>1.3 %</td></tr><tr><td>M3</td><td>6397</td><td>6429</td><td>7381</td><td>6333</td><td>6297</td><td>1.01 %</td></tr><tr><td>M4</td><td>6689</td><td>6740</td><td>7479</td><td>6642</td><td>6642</td><td>0.7 %</td></tr><tr><td rowspan="5">Medium<br>(8,3,3)</td><td></td><td>6412</td><td>6571</td><td>6987</td><td>6359</td><td>6329</td><td>0.83 %</td></tr><tr><td></td><td>6380</td><td>6402</td><td>72,767</td><td>6322</td><td>6308</td><td>0.9 %</td></tr><tr><td>L1</td><td>10,005</td><td>10,846</td><td>11,763</td><td>9624</td><td>6924</td><td>3.9 %</td></tr><tr><td>L2</td><td>9678</td><td>9874</td><td>11,475</td><td>9338</td><td>9338</td><td>3.6 %</td></tr><tr><td>L3</td><td>10,004</td><td>9931</td><td>11,468</td><td>9825</td><td>9825</td><td>1.8 %</td></tr><tr><td rowspan="3">Large (12,4,4)</td><td></td><td>10,322</td><td>10,428</td><td>12,389</td><td>10,089</td><td>10,089</td><td>2.3 %</td></tr><tr><td>L4</td><td>9929</td><td>10,246</td><td>11,982</td><td>9767</td><td>9767</td><td>1.6 %</td></tr><tr><td>L5</td><td>9958</td><td>10,265</td><td>11,835</td><td>9619</td><td>9619</td><td>2.64 %</td></tr></tbody></table>
<figure><img src="image_27.png" /><figcaption>Fig. 23. Mean plots of costs (€) for the different approaches with LSD 95% confidence intervals.</figcaption></figure>

<table><caption>**Table 17**
Average execution times.</caption><thead><tr><th></th><th>RL execution time</th><th>FIFO execution time</th><th>GA execution time</th><th>CPLEX_1</th><th>CPLEX_2</th></tr></thead><tbody><tr><td>Small instances</td><td>2.3 s</td><td>0.14 s</td><td>4.73</td><td>856 s</td><td>135 s</td></tr><tr><td>Medium instances</td><td>4.8 s</td><td>0.18 s</td><td>13.52 s</td><td>+1000 s</td><td>+1000 s</td></tr><tr><td>Large instances</td><td>7.6 s</td><td>0.23 s</td><td>26.47 s</td><td>+1000 s</td><td>+1000 s</td></tr></tbody></table>

enables industries to optimize their operations and ultimately enhance their financial performance. Moreover, the integration of on-site renewable energy sources and energy storage systems provides additional advantages by reducing reliance on conventional grid power and lowering greenhouse gas emissions. Second, this study promotes governmental policies for sustainable development, by supporting in particular the adoption of renewable energy technologies through policy initiatives and regulatory frameworks. The findings highlight the potential benefits of integrating renewable energy sources into production processes, thereby contributing to the overall sustainability of the industry. Finally, based on the computational results, the proposed model and its implementation including a reinforcement learning strategy demonstrate high computational efficiency together as well as the ability to provide best solutions with minimal execution time. This suggests that such an approach could be integrated into either commercial or open-source software tools in order to offer more comprehensive and efficient solutions.

## 7. Conclusion

This study addresses the integrated lot sizing and production scheduling problem in a flexible flow line while considering energy policies. A novel mixed-integer mathematical model is formulated to solve this complex problem. The main objective of such a model is to optimize the overall cost, which includes setup costs, inventory costs, ESS utilization costs, RES costs, and conventional energy consumption costs. To address the complexity of the problem, a multi-level approach based on a dynamic Multi-Agent System architecture is developed. The MAS architecture is designed to simultaneously solve the lot sizing and production scheduling problem by incorporating multiple agents that

interact and make cooperative decisions. The MAS utilizes the Q-learning algorithm to obtain good solutions within a reasonable timeframe. Numerical experiments were conducted to evaluate the performance of the proposed model and the multi-level resolution method. The results demonstrate that the mathematical model is accurate and effective, enabling a good tradeoff between production and energy performance. Moreover, the comparative analysis highlights the robustness and effectiveness of the MAS architecture in solving Integrated Lot Sizing and Production Scheduling (ILSPS) problems. This architecture outperforms other methods in terms of cost reduction and energy consumption optimization. Future work in this area is needed to account for the uncertainty in electricity prices and product demand in this first model. This can be achieved by enhancing the MAS to handle uncertain outcomes through different techniques such as stochastic modeling, scenario analysis, or robust optimization.

## Funding

The first author is a PhD student whose thesis benefits from funding by the French Ministry of Agriculture and Food Sovereignty (MASA) and the Pays-de-la-Loire Region as part of the 2021–2026 State-Region Plan Contract (CPER).

## 8. Declarations

**Ethics approval:** not applicable.

**Consent to participate:** not applicable.

**Consent for publication:** not applicable.
<figure><img src="image_28.png" /><figcaption>Fig. 23. Mean plots of costs (€) for the different approaches with LSD 95% confidence intervals.</figcaption></figure>

<table><caption>**Table 17**
Average execution times.</caption><thead><tr><th></th><th>RL execution time</th><th>FIFO execution time</th><th>GA execution time</th><th>CPLEX_1</th><th>CPLEX_2</th></tr></thead><tbody><tr><td>Small instances</td><td>2.3 s</td><td>0.14 s</td><td>4.73</td><td>856 s</td><td>135 s</td></tr><tr><td>Medium instances</td><td>4.8 s</td><td>0.18 s</td><td>13.52 s</td><td>+1000 s</td><td>+1000 s</td></tr><tr><td>Large instances</td><td>7.6 s</td><td>0.23 s</td><td>26.47 s</td><td>+1000 s</td><td>+1000 s</td></tr></tbody></table>

enables industries to optimize their operations and ultimately enhance their financial performance. Moreover, the integration of on-site renewable energy sources and energy storage systems provides additional advantages by reducing reliance on conventional grid power and lowering greenhouse gas emissions. Second, this study promotes governmental policies for sustainable development, by supporting in particular the adoption of renewable energy technologies through policy initiatives and regulatory frameworks. The findings highlight the potential benefits of integrating renewable energy sources into production processes, thereby contributing to the overall sustainability of the industry. Finally, based on the computational results, the proposed model and its implementation including a reinforcement learning strategy demonstrate high computational efficiency together as well as the ability to provide best solutions with minimal execution time. This suggests that such an approach could be integrated into either commercial or open-source software tools in order to offer more comprehensive and efficient solutions.

## 7. Conclusion

This study addresses the integrated lot sizing and production scheduling problem in a flexible flow line while considering energy policies. A novel mixed-integer mathematical model is formulated to solve this complex problem. The main objective of such a model is to optimize the overall cost, which includes setup costs, inventory costs, ESS utilization costs, RES costs, and conventional energy consumption costs. To address the complexity of the problem, a multi-level approach based on a dynamic Multi-Agent System architecture is developed. The MAS architecture is designed to simultaneously solve the lot sizing and production scheduling problem by incorporating multiple agents that

interact and make cooperative decisions. The MAS utilizes the Q-learning algorithm to obtain good solutions within a reasonable timeframe. Numerical experiments were conducted to evaluate the performance of the proposed model and the multi-level resolution method. The results demonstrate that the mathematical model is accurate and effective, enabling a good tradeoff between production and energy performance. Moreover, the comparative analysis highlights the robustness and effectiveness of the MAS architecture in solving Integrated Lot Sizing and Production Scheduling (ILSPS) problems. This architecture outperforms other methods in terms of cost reduction and energy consumption optimization. Future work in this area is needed to account for the uncertainty in electricity prices and product demand in this first model. This can be achieved by enhancing the MAS to handle uncertain outcomes through different techniques such as stochastic modeling, scenario analysis, or robust optimization.

## Funding

The first author is a PhD student whose thesis benefits from funding by the French Ministry of Agriculture and Food Sovereignty (MASA) and the Pays-de-la-Loire Region as part of the 2021–2026 State-Region Plan Contract (CPER).

## 8. Declarations

**Ethics approval:** not applicable.

**Consent to participate:** not applicable.

**Consent for publication:** not applicable.