LineFlow: A Framework to Learn Active Control of Production Lines
Authors: Kai Müller
Martin Wenzel
Tobias Windisch
Venue: ICML 2025 poster (2025)
Keywords: Machine Learning for Manufacturing
Active production line control
Reinforcement Learning
Production line simulation
Primary area: applications->everything_else
TL;DR: This paper presents LineFlow, an open-source extensible Python framework for simulating production lines and optimizing them using reinforcement learning.

Abstract: Many production lines require active control mechanisms, such as adaptive routing, worker reallocation, and rescheduling, to maintain optimal performance. However, designing these control systems is challenging for various reasons, and while reinforcement learning (RL) has shown promise in addressing these challenges, a standardized and general framework is still lacking. In this work, we introduce LineFlow, an extensible, open-source Python framework for simulating production lines of arbitrary complexity and training RL agents to control them. To demonstrate the capabilities and to validate the underlying theoretical assumptions of LineFlow, we formulate core subproblems of active line control in ways that facilitate mathematical analysis. For each problem, we provide optimal solutions for comparison. We benchmark state-of-the-art RL algorithms and show that the learned policies approach optimal performance in well-understood scenarios. However, for more complex, industrial-scale production lines, RL still faces significant challenges, highlighting the need for further research in areas such as reward shaping, curriculum learning, and hierarchical control.

LineFlow: A Framework to Learn Active Control of Production Lines
Kai Müller! Martin Wenzel
Tobias Windisch"
Abstract
take the interdependence of the individual work steps into account a computationally hard but well understood prob- lem called assembly line balancing problem (Padrón et al., 2009; Scholl & Becker, 2006). Solution strategies are pow- ered by good stochastic models for interlinked production lines, like in (Bierbooms, 2012; Liberopoulos et al., 2006), that help computing the equilibrium state of the throughput for given layout. However, in practice, optimal layouts and theoretic knowledge on the equilibrium of the through- put does not guarantee that the production line unfold its full theoretical potential. This may be due to situations like changing processing conditions, adverse coincidences in the stochasticity of the processes, machine failures. These circumstances can suddenly lead to full buffers, jams, and shifting bottlenecks (Roser et al., 2003; Mahmoodi et al., 2022). Thus, reacting in a dynamic and optimal way at runtime is key to keep the performance of the production line its best. At present, this either a completely manual task accomplished by line workers on-site or by line control systems, that actively adjust line parameters in real-time (see Figure 1). Possible interventions alternative stations (Huang et al., 2019), reallocating work- ers PPérez-Wheelon et al., 2022; Dolgui et al., 2018). ()18) or scheduling maintenance (Geurtsen et al., 2023). Classical
Many production lines require active control mechanisms, such as adaptive routing, worker reallocation, and rescheduling, maintain opti- mal performance. However, designing these con- trol systems is challenging for various reasons, and while reinforcement learning (RI.) has shown promise in addressing these challenges, a stan- dardized and general framework is still lacking. In this work, we introduce LineFlow, an extensi- ble, open-source Python framework for simulat- ing production lines of arbitrary complexity and training RL agents to control them. To demon- strate the capabilities and to validate the under- lying theoretical assumptions of LineFlow, we formulate core subproblems of active line control in ways that facilitate mathematical analysis. For each problem, provide optimal solutions for comparison. We benchmark state-of-the-ard RL algorithms and show that the learned policies ap- proach optimal performance in well-understood scenarios. However, for more complex, industrial- scale production lines, RL. still faces significant challenges, highlighting the need for further re- search in areas such reward shaping, curricu- lum learning, and hierarchical control.
rerouting tasks to
1. Introduction
At its core, manufacturing is about transforming raw ma- terials into finished goods, often large scale. In most production systems, the necessary process steps are carried out by work stations which each component has to traverse sequentially. Typically, the work steps necessary have to be applied in fixed order, creating an interdependence be- tween stations in the sense that performance issues of one station directly affect stations down- and upstream.
Send parts Sink2
Buffer fills
Processing times
π
Figure Active line control based real-time data.
active line control systems typically combine rule-based strategies and mathematical models to optimize schedul- ing and resource allocation, focusing mostly on bottleneck identification,
In theory, finding optimal layouts for production lines that
University of Applied Sciences Kempten, Kempten, Ger- many. Correspondence to: Tobias Windisch <tobias, windishh@@ss kempten.de>.
for instance (Roser et al., 2014; 2017; Li et al., 2009; Lai et al., 2021) and references therein. Find- ing optimal control strategies is complex task that not only requires a detailed understanding of the production line dynamics but also involves inteiintnssive trial-and-error session on the real system and simulations. While effective
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LincFlow: A Framework to Learn Active Control of Production Lines
in predictable scenarios, classical approaches often struggle in dynamic and complex environments with high uncertain- ties, which has motivated the use of machine learning for tasks like predicting bottlenecks (Subramaniyan et al., 2021) and forecasting demand (Gonçalves et al., 2021).
by domain expertise, aim for LineFlow to accelrrate re- search learning active line control in the future.
1.1. Related Work
There has been a lot of research where RL methods are studied to control given manufacturing technology, like for welding (Jin et al., 2019; Masinelli et al., 2020; Zou & Lan, 2020) or molding (Szarski & Chauhan, 2023; Guo et al., 2019) just to name a few. We refer to (Nian et al., 2020) and references therein for an overview. In contrast, our work treats the industrial technology happening inside the processes as a black box and solely considers the processing time of the stations and their statistical interplay. This means, we neither model physical relations nor monitor physical parameters of the processes, which is fruitful application field for RL on its own (Viharos & Jakab, 2021).
Since active production line control is, at its core, a se- quence of observing a state and taking action, it seems like natural next step to learn line control in an end-to-end fashion through interactions with a simulation using rein- forcement learning (RL). RL algorithms have demonstrated the ability to learn complex relationships between states and actions (Vinyals et al., 2019; Silver et al., 2016; Wu et al., 2023; Jumper et al., 2021). Significant research has been conducted on learning control policies for production lines (see Section 1.1); however, progress is hindered because existing studies each rely their own ad hoc domain- specific simulations, making benchmarking, reproducibility, and the assessment of RL progress in active line control difficult. One reason for this is the absence of a theoretical or practical framework to train and evaluate RL agents in various production settings in a standardized way.
A large body of research directly related to our work has been done for production scenarios where tasks of varying length have to be scheduled on heterogeneous machines with different processing capabilities resources. Such scheduling problems are tackled with RL-based approaches in many works, like in (Shi et al., 2020; Shiue et al., 2018; Kim & Lee, 1998; Tortorelli et al., 2022; Ali & Tirel, 2023; Overbeck et al., 202 1). A comprehensive overview is pro- vided in (Kuhnle et al., 2021). Other research focuses on machine interactions, as in (Wang et al., 2016), where RL is used to control layout consisting of two stations and one buffer to maximize throughput or in (Loffredo et al., 2024; 2023), where energy costs are minimized by switch- ing stations dynamically to standby mode. In (Geurtsen et al., 2023), the scheduling of predictive maintenance of a production line using RL has been studied.
In this work, we address exactly this research gap by intro- ducing LineFlow', a free and extensible Python package specifically tailored to simulate production lines in order to train RL agents for active line control at scale (Section 2). LineFlow not only is a fine-graded and fast discrete-event simulation of production lines, it also comes with an imple- mentation of conventional action- and state-spaces as well as a flexible score (Section 2.3) inspired by classical per- formance indicators that allows to train agents for various production settings (Section 3). To evaluate its theoretic un- derpinnings, we study well understood and realistic produc- tion scenarios by computing their mathematical optimum (Section 4) and then comparing it with the performance of control policies interacting with LineFlow (Section 5). To showcase the current limits, we also analyze very com- plex production setting mentioned above with unclear optimum and train RL agents via curriculum learning (Sec- tion 4.4). As there exist strong privacy concerns among manufacturers when it comes to releasing production line layouts, publicly available datasets holding performance information and non-trivial layouts are rare. Here, we hope that LineFlow will close this gap by allowing manufac- turers to provide synthetic non-confidential digital twins preserving their key runtime challenges to the public. In Section D. we model a real-world production line based on publicly available data in LineFlow. Our research is purely application-diie -rivend by the needs of end-users in manufactur- ing and opens up a wide range of challenges for machine learning researchers to investigate. Since implementing ac- tive control systems is currently highly manual task guided
2. Simulating Production Lines
Production lines can be characterized in many ways, like by the types of their processes or parts, their production vol- ume, or how their stations are arranged (Kang et al., 2020; Liberopoulos et al., 2006). Here, we focus on production lines that produce discrete items. This section introduces general principles and mechanisms of such production sys- tems and explains how these reflected in LineFlow.
2.1. Objects of Production Lines
The main objective of production lines studied in this work is to produce discrete items called parts. Typically, the pro- duction of single part requires a specific set of work steps that need to be applied in a certain order involving the pro- duction and assembly of a series of sub-components. Each work step is carried out in a specific station. One of the ob- jectives of LineFlow is to model the statistical dynamics of production line by considering the individual processing
'https:/ /// thub. com hss -as--empte//// /wwmm//nnnfooooo
2

---

LincFlow: A Framework to Learn Active Control of Production Lines
Buffers
neck can vary, whereas the buffer capacities cannot. We refer to (Bierbooms, 2012) for detailed analysis of the interplay of stations and buffers. Bottlenecks in production lines do not only reduce throughput by slowing down the overall part flow, but they can also lead to quality issues and even scrap. One prominent example arises in processes with strict timing constraints between consecutive steps. In adhesive bonding, for instance, components must be joined within certain time window after the adhesive is applied. If downstream bottleneck causes delays and this thresh- old is exceeded, the bond can no longer be guaranteed to meet quality standards, and the affected part must be dis- carded (Chen et al., 2005). Such timing-sensitive constraints make bottleneck prevention critical not only for efficiency, but also for product quality and material waste reduction.
Carricns
Stulion
Figure 2. A production line visualized with LineFlow.
times of the stations and their interplay. To be more precise, the processing times are assumed to be exponentially dis- tributed like in (Bierbooms, 2012) as T = T'+ Ex where T 2 0 is the minimal processing time possible and where Expg denotes the exponential distribution with mean S. We distinguish different types of stations, among them are sources that set up components, processes that apply a piece of work on a single component, assemblies that join two or more components into one, and sinks that remove fully build part from the line (see Figure 2). Components are transported by carriery from station to station via buffers, which work on frst-in-first-ou principle. Buffers can only hold a predefined number of carriers, called their capacity. Time-intense work steps may be distributed over identical parallel stations, while switches handle the routing of car- riers. The arrangement of stations and buffers is typically called the layout of the production line.
To circumvent mitigate such harmful runtime effects, we essentially study three corrective actions and their inter- play: Changing worker assignments to speed up bottleneck stations, changing the distribution of components, and in- cluding waiting times to prevent scrap. On the one hand, increasing the waiting time an important counter activity to prevent scrap, but there is thin line between preventing scrap and slowing down the bottleneck and, in turn, the whole line (see Section 4.1). Switches, on the other hand, allow to change the routing and distribution of components to keep the bottleneck loaded constantly. For instance, if the processing time of Sinkl starts to increase, the switch could change the distribution ratio between the sinks (see Sec- tion 4.2). Finally, reassigning workers from one station to another directly impacts their processing time and can help to handle peak loads but can create a bottleneck elsewhere (see Section 4.3). A more involved scenario where all these actions have to be combined effectively is studied in Sec- tion 4.4. Even if a bottleneck is correctly detected, choosing an optimal action sequence to mitigate is challenging since actions take time to show results. For example, reassigning a worker requires task completion and relocation first. While this reduces the bottleneck's 's mean processing time, high variance makes changes hard to detect. In general, even with the right actions, high data variability delays recog- nizing improvements. Compounding this issue, production lines often operate near equilibrium states, where minor dis- ruptions can cascade into buffer overflows, bottlenecks, or system-wiee jams. Moreover, the combination of different actions may have unintended negative consequences on the performance. For instance, if SinkI is the current bottleneck, then assigning more workers to Sink/ and sending more components to Sink2 additionally, may result in a worse overall performance. To apply optimal actions, line control systems rely on real-time data from various production line objects, which is often noisy, incomplete, and lagged (see Section 3.2 and Section A.7).
2.2. Active Line Control
The overall performance of a production line depends sen- sitively on the individual performances of its stations performance issues of any station are propagated down- and upstream by their linking buffers. For instance, consider the production line shown in Figure 2: If the processing time of Process is higher than the processing time of Assembly, the buffer in between fills up and once its maximal capac- ity is reached, the previous station is blocked as cannot push finished components to the buffer. Such jam can be propagated backwards through the line and may even block the source. Similarly, subsequent stations like the two sinks with faster processing times are not served with components and have to wait till new components arrive. In the equilibrium, the performance of the full system depends essentially on the bottleneck station, which is in simple cases just the station having the highest processing time. Buffers constitute an important parameter of production lines as they can dampen fluctuations in processing times at the stations. However, buffers and their capacity are fixed parameters of the layout, whereas stochasticity in the processing times of the processes can result in a shifting runtime bottleneck (Roser et al., 2003). Thus, the bottle-
II the literature typically denoted as Exp.
3

---

LincFlow: A Framework to Learn Active Control of Production Lines
2.3. Performance Measurements
3. RL for Active Line Control
There are several ways to measure the performance of pro- duction lines. A commonly used metric is the Overall Equipment Effectiveness, short OEE, which quantifies the effectiveness of equipment and machinery used (Nakajima, 1988). Assume we have given a control policy for pro- duction line decomposed into stations PI...., .k- Denote by naa (t, i) and mo (t, i) the cumulated number of OK and not OK (NOK) parts produced station P; at time t. Then, following (Nakajima, 1988), the OEE for P. is essentially +. n (f) where Ti is the min- imum of the processing time of P.. By construction, we have nok (t, i) T, < t for all t and thus the maximal possi- ble OEE is 1. Analogously, the OEE of a production line can be defined. The OEE definition is mainly focusing on productivity and quality, but more economic view requires to also take costs of the individual stations and components into account, for instance as some stations require more energy for their processing, like in (Loffredo et al., 2024), or because only some components are scraped, not the full part. Thus, we assume that applying work at station P; comes with a cost C,, which may be material costs for joining a component or costs for energy as in (Loffredo et al., 2024). The value of the final part is considered to be the sum of all costs c = Σi=1 Cg. At simulation time l, the production has produced an aggregated value of
In this section, we describe how RL can be used to learn policies for active line control. Specifically, we consider active line control as an episodic and partially-obseredd Markoy decision process. Let S be set of states and A set of actions. Transition probabilities P dictate how likely transition from a state E S to a state s' E S is when choosing action E AA at 8, namely P(s'(s, a). When se- lecting a at s. a reward R(s, a) is observed. Additionally given a desired length T E N, this gives a Markov deci- sion process (S, A, T.) where the goal is to find pol- icy S + A to maximize Er,, E R(SE, (88-1)) where (st)ten is chosen under the interplay of P and æ. If the states s E S cannot be observed directly by the pol- icy but only features ʃ E F. instead controlled by another conditional probability density function O, ie.. the likeli- hood of observing f at state s is O(f]s), then the tuple (S, A, P, R, O) defines a partially--bseerve Markov de- cision process where policy F + A needs to be found maximizing E,,o, Σn R(s,, (fe-1) Note that the definitions given here are streamlined versions of the com- mon way of defining Markov decision processes, where the process typically ends in set of defined terminal states and additionally, a discount fascount.actorr 7 E [0, 1] is given to trade-off' rewards in early and late states.
defined as (t)
L.
While the formulation above assumes a discrete-time deci- sion process, we emphasize that the underlying dynamics of the production line still can be continuous time and only the agent interaction-i.e. observation and action execu- tion- restricted to fixed intervals. This modeling choice reflects real-world production control, where decisions like worker reallocation or routing changes are typically made at regular intervals, such as every few seconds or at shift-level granularity. This approach offers a practical compromise between physical realism and learning efficiency, and is con- sistent with prior work in industrial reinforcement learning. Section 3.3 provides details how this is realized in LineFlow.
C+(1) To
Σ a mak i1)
where na (t) E N denotes the number of parts produced at all sinks combined and Tc denotes the minimal possible time the line needs to produce a single part. One can think of Equation| I as the value the production line generates: If C. > O, the line generates revenue, if (t) < 0, it generates costs. If no scrap can occur, maximizing C₂ is equivalent to maximize the number of produced parts.
3.1. Episodes and Rewards
As described in Section 2.3, we consider constant operation times Tsim of the manufacturing setting to be optimized. In principle, control policy can interact with a production line at any time. We, however, assume that the policy can only interact with the production line at fixed and equidistant time points each Tstep apart, thus we get T = Tam. tions in an episode. This approach introduces the possibility that the model may miss certain state changes occurring between observation intervals, such as a very brief buffer overflow. However, from a RL point of view, this has mul- tiple advantages: First, the trajectories all become of fixed length and second, the policy does not need the time scale.
Generally, one wants to find a control policy π that max- imizes Cx(() and simultaneonsly minimizes the time t to reach the maximum. It then depends on the exact usecase to weight a smaller t needed with a higher C(I) reached by. To circumvent this distinction in our work and to cre- ate a general benchmarking environment, one either has to fix the available time Tsim and to maximize the output Cz (T.im), or to fix the desired output Õ and to minimize the time needed by a policy π to reach it. In our benchmark studies, we decided to fix Teim as this typically originates from high-vor volume production where the output needs to be maximized in a given work shift of length Tsim-
TTeep many interac-
4

---

LincFlow: A Framework to Learn Active Control of Production Lines
Allowing arbitrary interactions of policy, particularly at the early stage of the training, can lead to many thousands of interactions within only few time units. Although in LineFlow, Tatep and Teim are considered to be constants coming from the situation to be optimized, both can be var- ied sequentially in a curriculum learning fashion (Narvekar et al., 2020). The goal of an episode is to find a policy ı that maximizes Ct (Tsmm). To allow temporal difference learning (see (Sutton & Barto, 2020, Chapter 6)), decompose CN into discrete quantities for E ;.....TTmmm by defining
(a) WT and WTJ
(b) 1A3,10
(c) PD;
Figure 3. Representatives of the three atomic production line chal- lenges analyzed in this case study.
3.3. Implementation Details
R(s, (8t-1)) Cr (Tstop (1+ - (Tstop 1).
LineFlow is designed to address key challenges in opti- mizing active line controls while meeting the needs of RL researchers. The discrete event simulation, handling station interactions and stochasticity, is built on SimPy (SimPy, 2025). Its object-oriented structure enables easy customiza- tion of stations and rapid setup of complex production sce- narios. A visualization module based on pygame provides insights into layout dynamics and agent interactions. Fully implemented in Python, LineFlow integrates seamlessly with data science tools like pandas (Wes McKinney, 2010) and numpy (Harris et al., 2020). RL interaction follows the gymnasium API (Towers et al., 2024), allowing training with baslle-bsslines3 (Raffin et al., 2021) or skrl (Serrano- Muñoz et al., 2023). Enviromments support vectorization and parallelization accelerate training. Further details are provided in Section A.I, with complete example in Sec- tion A, illustrating the layout in Figure 2. Although agent interaction follows the discrete-time process described in Section 3, the underlying production dynamics in LineFlow is simulated in continuous time using a discrete-event en- gine. The interaction frequency of the agent can be adjusted via a parameter, allowing the discrete-time control ap- proximate continuous-ting behavior with high fidelity.
That way, we get Ei R(1) Crr(Tsmm) as desired.
3.2. States, Observations, and Actions
Due to the stochastic nature of production processes and sensor inaccuracies, the true system state of a production line is not directly observable. Instead, it must be inferred from incomplete, noisy, and delayed observations. Key features relevant for performance estimation include buffer fill levels, station processing times, production rates, station modes, and routing information from switches (Roser al., 2003; 2014). Whenever available, features in LineFlow are constrained by known upper and lower bounds. For example, fill levels are normalized to [o, IJ. while production rates or processing times are positive but unbounded.
The action space in LineFlow primarily consists of dis- crete control decisions, such as assigning workers or routing components across stations. Additionally, stations can be switched on or off, similar to (Loffredo et al., 2023). Work- ers are drawn from predefined pools associated with specific station groups Each worker is modeled as an individual and independent dimension in the action space, where the possible values correspond to the stations the worker can be assigned to. While this introduces symmetries in the action space when workers are indistinguishable, it enables fine-grat -reiiia modeling of worker attributes-suce as skill levels or expertise-wil when they are not. Furthermore, rep- resenting workers as explicit objects within the production line ensures that side constraints, such as a fixed number of available workers, are respected at all times. Switches typically feature two discrete actions: one controlling in- coming buffers and another for outgoing buffers. Beyond the core actions, LineFlow also supports modifications to the physical production environment, such as actively adding or removing carriers from the line. While not included in our case studies, these actions provide additional flexibility for modeling dynamic settings. A list of all observable and actionable dimensions is in Section A.7.
4. Case Studies
In this section, we introduce and study realistic production scenarios where active control is required. The scenarios have been selected for two reasons: First, their theoretical optimum can be computed statistically allowing us quan- tify whether RL algorithms learn optimal policies. Second, they appear as subproblems in many production scenarios.
4.1. Optimal Waiting Time WT and WTJ
In this scenario, the optimal waiting time between parts produced by a source station Se has to be found (see also Figure 3a). The source Se serves together with another source Sy an assembly station A, which joins the compo- nents from both sources to produce the final product and sends to sink S. Components from source Sc have a fixed
5

---

LincFlow: A Framework to Learn Active Control of Production Lines
in a way that the maximal possible reward remains con- stant (see Section B.1 for details). More precisely, we fix constant 0.5 < R < 1.0 and construct f such that expected maximal parts produced is R .N, where N is the expected number of parts of WT without jump. A visualization of the processing time of A for multiple simulations is in Figure 4. Although, by design, the expected maximal number of parts produced by the line is R- _N, this value cannot be reached by control agent. The reason is that at time Trrigger if the processing time of the assembly jumps to higher level, the new processing time of A can first be observed once the first part has been produced. Thus, from Torigeer to Tiiggerr + f-TT+ Expg, the source Se runs with a too low waiting time and possibly sends components to A which are going to expire. The same happens once the processing time of A jumps the lower level. Here, the source may run at higher waiting time causing A to wait. To quantify the optimum reachable by agents learning from observations, we evaluate the performance of an agent learning the means of the processing times online (see Section B.
1000
13:00
3000
12.0
2004
10.0
4022
Figure 4. The jumps in the processing time of the assembly A in WTJ for different simulations of length 4000.
expiration time TAc called assembly condition: If the time from their setup at Sc to the time the process starts A is larger than TAC, then A has to dispose this component and has to wait for the next one. The removal of scrap part not only generates costs, it also blocks the assembly for a certain amount of time as the defective component has to be removed first. Such a situation is typical in many joining processes, like gluing applications, where adhesives can dry out if not processed in time (Chen et al., 2005). If the waiting time is chosen too small, Sc produces to many components which cannot be handled by A in time and thus drives scrap costs. On the other hand, if the waiting time is chosen to high, AA has to wait for components which delays the production of final parts. The goal is to balance the wait- ing time of Sc to maintain a continuous component supply at A. Assuming costs eм and eş for components produced at SM and Sc respectively and that only components of Sc can potentially be scraped, the performance of the line as described in Section 2.3 within a given time frame / is
4.2. Optimal Part Distributions PDs
Our next case study is subproblem of scheduling prob- lems in manufacturing and is already extensively researched in literature (see Section 1.1). Here, components have to be distributed onto & many parallel processes Pi..... Pi: with processing time distributions Ti,..., Ts by a single switch in an optimal way. Moreover, another switch needs to fetch the outputs of the processes and sends them to sink (see Figure 3c). We assume that T; T, + Exps,-1, where T; and S; are constants for all E [A:]- To compute the optimum, we assume for simplicity that get and put times from the connecting buffers are already contained in Tj Within given time frame Taim. the expected maxi- mal number of parts Mf produced by process i is obtained when P. is served with parts constantly without time gaps: Tim (1-S))-z;: When we assume that the processing times of the source and the sink negligible compared to the processing times of the processes P.. The maximal number of parts ' produced by the production line within Taim is the sum of the maximal parts produced by all processes, i.e., N = Σ Nr as all processes can work in parallel. This allows us to compute the maximum number of parts the layout can produce theoretically, see Section B.2. Clearly, the greedy policy deployed at both switches which pushes components on the buffer with lowest fill and fetches components from the buffer with highest fill loads the pro- cesses in optimal way (see also Section B.2).
Tc. ((CM + cs) (t) - cs- mookk (t, A)). Taim
(t) =
We generally assume that the processing times without po- tential waiting times are such that A is the bottleneck of the line. Thus, the maximum number of parts produced depends on the time A needs to get one carriers and one component, to assemble them, and to push the final product to the buffer. The optimal waiting time essentially fills the between the times A and Sc need to handle and process their parts. We give an explicit equation for the optimal waiting time and the maximum number of parts in Section B.
EM = 3(7]]
Taim
To force a dynamic adaption we now introduce the related scenario Waiting time jump (WTJ). Here, we uniformly sample length Jump and trigger time T Tuigger such that [Tuiggerr: Triggerr + +T jumpl C [0, T- Then, the processing time of A at simulation time t from
T + Exps: f- Exps,
if &f [Tuigger: Triigerr + T Tumpp if E [Trigger: T Trigger + T jumpl
4.3. Optimal Worker Assignments WAR,N
As f and Tjump can be different in each episode, the maxi- mal possible reward can vary, too. To ease the comparison of agents over multiple runs, we construct f for given T Jump
In many production scenarios, processes involve manual work which can be completed faster by multiple workers
6

---

LincFlow: A Framework to Learn Active Control of Production Lines
collaborating. In this case study, the goal is to find the right distribution of limited number of workers over mul- tiple stations each having a varying processing time. We assume that the required processing time of a station, where n workers are assigned to, is sampled from the distribution Tr.s.n ;= T (n) + Exps-r where T, S E Ry0 are con- stants and where pe(n)) = exp(-c (-c vi) is a performance coefficient (see also Section B.3). In our benchmarks, we c 0.3, which means that an additional worker reduces the processing time by approximately 74%. Now, consider layout of l sequential stations -I...., Aę with constants Ti...., Tk and Si...., Sy. Moreover, assume the line has a pool of N workers to distribute. As the stations are sequen- tially arranged and connected via buffers (see Figure 3b), the slowest process determines the speed of the overall pro- duction line. This means, partition of N into k integer summands, i..., n1 + +ng N with n4 No needs to be found such that the maximum of all processing times is minimal. Clearly, the assignment process is bound to certain constraints, like delayed start of processes due to the waiting for assigned workers because of traversal times. We refer to Section 3.2 for more details about how the worker assignment is modeled in LineFlow. Let S [N]* be the set of partitions of N into many k non-negative integer numbers. More formally, the following max-min integer optimization problem needs to be solved:
GARER
Figure 5. Scenario CL with 8 assemblies.
filling process between the source, switch, and assembly stations. In Section B.4, we present a profitable heuristic found by extensive testing meaning the number of produced parts outweighs the cost of scrap.
5. Benchmarks
5.1. Layout of Experiments
First, we want to emphasize that our benchmark is in- tended to serve as a proof of principle how evaluation can be done with LineFlow and to show that optimal control can be learned. For that, only select handful of al- gorithms and hyperparameters to consider, although many more would be feasible as well. Concretely, we use the following algorithms in our experiments: PPO (Schulman et al., 2017) and its recurrent version (Pleines et al., 2022), A2C (Mnih et al., 2016), and TRPO (Schulman et al., 2015), all of which can deal with multi-dimensional observations of mixed types and multi-dimensional categorical action spaces. We used the implementations provided by the s stable-base lines (Raffin et al., 2021) package and evaluated multiple hyperparameters with three different ran- dom seeds for each case study. More details on the hyperpa- rameters used can be found in Section C. The performance of the agents was measured in online evaluations, reporting the mean reward of episodes a separate evaluation en- vironment 5 times (see also Figure 6 for the evolution of the reward over global steps). When not stated differently, the deterministic version of a trained policy is evaluated, that is, the action with the highest probability is selected. In all experiments, the agents are trained and evaluated on five environments stacked into a vectorized environment.
min max S[Tr..S..n.] T Mea...
(2)
[E[R]
where in our case, the expected value of the processing time (T(TT,S.. nu] = T, Po po(n,)+-S T, The maximum number of parts that can be produced is then essentially given by -Tat after removing some initial offsets Tort due to the ramp-up of the production line. Mathematically, solving Equation (2) is computationally hard not alone because it min-max problem but also because involves integer partitions. Learning
optimal distribution in an interac- tive environment is even more challenging, as first good estimates for T; and St need to be learned as well as for Pe- In Section B.3, we formulate Equation 2 as mixed-integer optimization problem and compare the empirical results obtained with LineFlow with the theoretical optinum.
4.4. Complex Line CL
5.2. Results for WT, WTJ, WA, and PT)
Finally, we combined all studied problems into a problem we call complex line, short CL (see Figure 5). Here the agent has to distribute workers and components over l se- quential assembly stations such that an assembly condition Tac in WT for the components is kept. Interestingly. simply combining the individual and optimal solutions of the individual subproblems does not lead to an optimal solu- tion for CL. A straightforward distribution approach with a fixed distribution of workers, parts and a fixed waiting time at the source can cause excessive scrap or a slow buffer-
All benchmarks we set Tsim = 4000. For WT and WTJ, we set TA 20 + Exp2, Tsc = 5 5 ExPo.s. and use a con- stant get time T, = The trigger time Tirizerr is uniformly sampled from the interval [500, 1500] whereas Tjump is uni- formly sampled from [1600, 2000). The assembly condition is set to TaC = 35. In PDs, the time distributions are T, = 10 (i + 1) + ExPo.1 for E [Rt.. In WA2 N. we set N = 3 & and use c = 0.3 for the performance coefficient. Moreover, we set T: = (16 + 4).
7

---

LincFlow: A Framework to Learn Active Control of Production Lines
PCC
18110
Figure Evaluation reward over global steps for 3D3.
Optiool
WT
1500 (156
1573 (3
LS7. (175113
154-
155.2115
11
WAL
13.3 (M86))
9486.1 16.4 (362-))
STAT (L96 0
376 17. (260
357.1
WA,
WAr.s
2005 22.02208.000
19LS (166))
2:1 (2908)
207. (2964
33688:21
578
(O7T-9)
379.6
PD,
[01.) (961.))
062.1 1015 (66-2)
1558::::11 (1002))
PD.
0513 USA (7)5.0)
Figure 8. Upper figure shows a comparison of recurrent PPO trained with (blue) and without (red) curriculum using three differ- ent seeds. Lower figure shows a comparison of the stacked PPO trained using curriculum with stack of 40 (red) and 100 (blue).
Table 1. Overview of rewards obtained by the best hyperparam- eter combination. Number in brackets denotes maximal reward obtained (see Section B for details). Note that optimal policies are shaped by stochasticity as well and trained RL agents may outperform the expected mean performance in individual episodes.
parameters have been used, hardly an algorithm reached a strictly positive reward. Since, at the start of the training the number of scrap parts heavily outweighs the number of produced parts the reward starts significantly negative. Interestingly the agents all converged to reward of zero. An investigation revealed that agents learn to dead-lock the line, leading to no produced parts but also, no scrap parts (see Figure 7). From there, the same train data is gener- ated during all subsequent rollouts leading to poor update steps of the agent. In our experiments, no algorithm tested managed to escape from these dead-lock situation when trained from scratch. Thus, we applied a curriculum which slightly increases the scrap costs of component scraped at one of the assemblies. The intention behind is that first, the agents learn the general mechanics of the production line and which actions are necessary to bring parts to the final station. Then, they have to get more efficient by reducing the number of scrap parts. We increase the scrap weight by a constant factor until & is reached (in our experiments 0.006 for the scenario having k 3) once the reward is above 100 in five subsequent evaluations. As optimizing CL requires to memorize past actions, consider only two algorithms here: A stacked PPO (Schulman et al., 2017) re- ceiving the past ic observations (with k = 40 and A = 100) and a recurrent PPO (Heess et al., 2015) as in (Pleines et al., 2023). First, we can see that using an curriculum is effective: Agents obtained through a curriculum obtain a large positive reward right from the beginning and they manage to keep it also when the scrap costs are increased. Figure 8 shows comparison of the recurrent PPO trained with and without the curriculum for different random seeds and learning rates. In addition, we see that the stacked version of PPO does not manage to reach the performance of the recurrent version, even when trained 10 times longer.
The performance of the best hyperparameter combination for each algorithm, averaged over multiple seeds, is shown in Table 1. More details of the rewards obtained during online evaluation of all hyperparameter sets is in Section C. Scenario WT is feasible for all algorithms, but only re- current PPO approaches the optimal value for WTJ. The actor-critic method A2C outperforms policy-based methods in WA by achieving a better performance with fewer steps. However, in part distribution, actor-critic methods fail to learn good controls while policy-based approaches succeed.
5.3. Results for CL
Informed decisions for CL require to keep past actions in memory. For instance, sending a component to station at give step requires to send a part to station i in the one of the subsequent step in order to keep the part flow on the main track. We found that, no matter which hyper-
Figure 7. An agent trained with the PPO algorithm causing a block- ing state of the line: All components send to Ao causing Ai to wait. As A is blocked, the buffer between Ao and A₁ gets full, which in turn blocks Ag and consequently the full linc.
8

---

LincFlow: A Framework to Learn Active Control of Production Lines
When implementing baseline for CL, it is essential to consider the waiting time of the source as well the dis- tribution of components and workers. Our heuristics priori- tized the buffer with the lowest fill level fills. while buffers feeding into later assembly stages receive higher priority. We then conducted a grid search over various waiting times and worker distributions. The best heuristic we identified achieved a reward of 254.6 = 1.1. Overall, even the best- performing RL algorithms in our CL benchmark still fall short of the manually implemented heuristic.
Acknowledgments
The project LineFlow is funded by the Bavarian state min- istry of research. TW is funded by the Hightech Agenda Bavaria. We thank our research partners from Robert Bosch GmbH and DMG Mori AG for helpful discussions, particu- larly Martin Roth, Dominik Böhnlein, Markus Guggemoos, and Thomas Stark. Many thanks to Matthias Burkhardt, Cindy Buhl, Kilian Führer, Andreas Fritz, Fabian Hueber, Lea Müller, and Edgar Wolf for their helpful suggestions.
Impact Statement
6. Discussion
This work introduces a reinforcement learning framework for active control of production lines, enabling optimized operations with improved efficiency. By addressing the challenges of real-time makiiig in dynamic man- ufacturing environments, our method supports sustainable industrial practices by using existing resources of manufac- turing lines in an optimal way. This research aligns with global efforts toward smarter and greener production sys- tems, benefiting industries, workers, and society.
In this work, we explore rich problem class from manufac- turing and make accessible for RL research by introducing novel framework LineFlow. We demonstrated that RL al- gorithms can learn effective control policies for various production line challenges, achieving optimal performance in well-understood scenarios. Moreover, we showed that for more complex, dynamic production settings, traditional RL approaches struggle without additional techniques such as curriculum learning and memory-base policies. We found that learning effective control strategies from interactions alone is difficult as the ramp-up phase of production plays crucial role in training stability and poor early decisions can lead to deadlocks and stalled learning. Thus, more complex settings require structured learning approaches, including curriculum learning and hicrarchical control.
References
Ali, A. M. and Tirel, L. Action masked deep reinforcement learning for controlling industrial assembly lines. In 2023 IEEE World Al loT' Congress (AlloT), pp- 0797-0803, 2023. doi: 1.1100/AAIOTS812220000.. 212..2211222..1144422
Researchers and engineers can now use LineFlow to imple- ment their specific production settings and systematically generate simulation data. It enables both the testing of custom policies and the training of RL agents. The behav- ior of trained agents can be used for analysis, but these policies can also be deployed in the real-world. LineFlow enables research in diverse directions. It supports the study of non-stationary problems, such as machine breakdowns or processing time drifts, which are common in real-world manufacturing. Additionally, improving reward structures for curriculum learning and advancing algorithms could sig- nificantly reduce training times and enhance performance in complex production settings. Another promising avenue is transfer learning, which could facilitate knowledge transfer across related tasks for more efficient and effective solutions. Beyond RL, LineFlow serves data generator, enabling analysis of production line dynamics, bottleneck prediction, and maintenance forecasting through supervised learning. To advance RL research in production control, a unified training and evaluation framework is essential for tracking and improving the state of the art. LineFlow fills this gap and drives further research into optimizing agent performance in complex, dynamic manufacturing environments.
Beal, L., Hill, D., Martin, R., and Hedengren, J. Gekko optimization suite. Processes, 6(8):106, 2018. doi: 10. 3390/pr6080106.
Bierbooms, R. Performance analysis of production lines: Discrete and continuous flow models. PhD Thesis, Tech- nische Universiteit Eindhoven, 2012.
Chen, X.,, Schoenau, G., and Zhang. W. Modeling and con- trol of dispensing processes for surface mount technology. IEEF/ASME Transactions on Mechatronics, 10:326-334, 2005. doi: 10.11097MECH.2205...
Dolgui. A., Kovalev, S., Kovalyov, M. Y., Ma- lyutin, S., and Soukhal, A. Optimal workforce assignment to operations of a paced assembly line. European Journal of Operational Research, 264 (1):200-211, 2018. doi: 10.10166j.ejor22017.066 017. hww.ttpss/www.cceeendeeeeeeeeeee sciiecee/articeee//////777222212
Geurtsen, M., Adan, I., and Atan, Z. Deep reinforcement learning for optimal planning of assembly line main- tenance. Journal of Manufacturing Systems, 69:170- 188, 2023. doi: http:///bbbooo////..o.o.////.... 011. URL sriiece//rttcdee///////276222222
com/
9

---

Loffredo, A., May, M. C., Matta, A., and Lanza, G. Reinforcement learning for sustainability enhancement production lines. Journal of Intelligent Manufac- turing, 377 dec 2024. doi: 10.1007/ 018845-023-022588 LincFlow: A Framework to Learn Active Control of Production Lines
Gonçalves, J. N., Cortez, P., Carvalho, M. S., and Frazão, N. M. A multivariate approach for multi-step demand forecasting in assembly industries: Empirical evidence from an automotive supply chain. Decision Support Sys- tems, 142:1 13452, 2021, doi: 10.10016i..dss220001144434
URL
https://ww..cceenediiieee science/ ccceene//rttice///////3333222222
Kim. G. H. and Lee, C. G. Genetic reinforcement learning approach to the heterogeneous machine scheduling prob- lem. IEEE Transactions on Robotics and Automation, 14 (6):879-88) 893, 1998. doi: 10.1109/70.73677
URL https:
scie
Guo, F., Zhou, X., Liu, J., Zhang, Y., Li, D., and Zhou, H. A reinforcement learning decision model for online process parameters optimization from offline data in injection molding. Applied Soft Computing, 85:105828, 2019. doi: 0.10166j.asoc.211..0333
Kuhnle, A., Kaiser, J.-P., Theiss, F., Stricker, N., and Lanza, G. Designing an adaptive production control system using reinforcement learning. Journal of In- telligent Manufacturing, +2(3):855-877 2021. doi: 10. 10.1007/s10845-02000166
URL https://www.scienceeet pellence/artiilleei///tt/e//////
Lai, X., Shui, H., Ding, D., and Ni, J. Data- driven dynamic bottleneck detection in complex man- ufacturing systems. Journal of Manufacturing Sys- tems, 60:662-675, 2021. doi: j00y0166 016. URL https: hwtp://wwwwww... www.wwwwwwwedddeeeecccoom 2021000020772222225511
Harris, C. R., Millman, K. J., van der Walt, S. J., Gom- mers, R., Virtanen, P., Coumnpeaa, D., Wieser, E., Taylor, J.. Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, F. Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, E. Array programming with NumPy. Nature, 585(7825)3357-33 2020. doi: 020-2699-2
Ni, Li, L.,, Chang, Q., J., and Biller, S. Real time production improvement through bottleneck control. International Journal of production research, 47(21):6145-6158 2009,
Heess, N., Hunt, J. J., Lillicrap, T. P., and Silver, D. Memory- based control with recurrent neural networks, 2015. URL https: /ttes///arxiia aaxiv.iiv org/ abs/ 1512. 04455.
Liberopoulos, G., Papadopoulos, C. T., Tan, B. s., Smith, J. M., and Gershwin, S. B. (eds.). Stochastic Mod- eling of Manufacturing Systems: Advances Design, Performance Evaluation, and Control Issues. Springer, 2006. ISBN 978-3-540-26579-5 978-3-540-29057-5.
Huang. B., Wang, W., Ren, S., Zhong, Y., and Jiang, A proactive task dispatching method based future bottleneck prediction for the smart factory. International Journal of Computer Integrated Manufacturing, 32(3): 278-293,1 2019.
doi: 10. 10.1000733-540-29055-55 URL https: https://link. //link. hawinngerr.oom//// spprhnger..cmm//.00o000//3----- 1101//00000000000000000-00-----0
2019.
Loffredo, A., May, M. C., Schäfer, L., Matta, A., and Lanza, G. Reinforcement learning for energy- efficient control of parallel and identical machines. CIRP Journal of Manufacturing Science and Technol- ogy, 44:91-103, 2023. doi: 10.1016/j.cimj.2222...3 007. URL htt.t:ttp////.m htt.ttss.:/ww..cccceee http://wwwwwwccecceeeeeeeee..... ccckene//zrticlleppp/////151111
Jin, Z., Li, H., and Gao, H. An intelligent weld trol strategy based on reinforcement learning approach. The International Journal of Advanced Manufacturing Technology, 100(9):2163-217) 2019. doi: 10.1007/ 500170-018-2864
Jumper, J. M., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Židek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals. O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D. Highly accurate protein structure prediction with alphafold. Nature, 596:583
Mahmoodi, E., Fathi, M., and Ghobakhloo, M. The impact of industry 4.0 bottleneck analysis in pro- duction and manufacturing: Current trends and future perspectives. Computers & Industrial Engineering, 174:108801, 2022. doi: 10.10166j.cee22222.00883 108001
589, 2021.
Kang, Z. Catal, C., and Tekinerdogan, B. Machine learning applications in production lines: A systematic literature review. Computers Industrial Engineering, 149: 106773, 2020. doi: 10.1016//.cie
URL
ottence/rricce//////000322222221
10

---

LincFlow: A Framework to Learn Active Control of Production Lines
Masinelli, G., Le- Le-Quang. T., Zanoli, S., Wasmer, K., and Shevchik, S. A. Adaptive laser welding control: A rein- forcement learning approach. IEEE Access, 8: 8:103803- 103814, 2020. doi: 0AC09//ACCEESS222229999900
Pleines, M., Pallasch, M., Zimmer, F,, and Preuss, M. Mem- ory gym: Partially observable challenges memory- based agents. In International Conference on Learning Representations, 2023. URL https:://///tt oownreviee.e.eee. net/ het/forumm? 2 d= HcdddC6DDr
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukeuoglu, K. Asyn- chronous methods for deep reinforcement learning. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learn- ing, volume 48 of Proceedings of Machine Learning Re- search, pp. 1928-199 1937, New York, New York, USA, 20- 22 Jun 2016. PMLR. URL https: /ptpp//ecceengs..... mlr. press v htt///nnn.l...mm hhtml.
Pérez-Wheelock, R. M., Ou, W., Yenradee, P., and Huynh, V.-N. A demand-driven model for reallocating workers in assembly lines. IEEE Access, 11883000-83320) 2022. doi: 10.1109AACCESSS2222..144444
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines:: Reliable rein- forcement learning implementations.. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http: html.
/ //jmmr.org/// wttp./////ppppr////////-------
Nakajima, S. Introduction to TPM: Total Productive Main- tenance. Preventative Maintenance Series. Productivity Press, 1988. ISBN 9780915299232. URL https:// books . gwwgle.oooooooooooooooooooo.. de Xi-oosssidxxKK222333eeU
Risdal, M., Prasanth, RumiGhosh, soundar, W., S., and Cukierski, W. Bosch production line perfor- mance. https: https://kaaglee com/ http://ctpptiiooooooee pesch-production line wawpettiioo-nee---efffrtmaaaaate 2016. Kaggle.
Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E., and Stone, P. Curriculum learning for reinforce- ment learning domains: A framework and survey, Jour- nal of Machine Learning Research, 21((81):1-50, 2020. URL http:// http:://jmrr.org///eeeeee http:/apprrs//2122--0ee html.
Roser, C.. Nakano, M.. and Tanaka, M. Shifting bottle- neck detection. In Proceedings of the Winter Simulation Conference. IEEE, 2003.
Roser, C.,, Lorentzen, K., and Deuse, J. Reliable shop floor bottleneck detection for flow lines through process and inventory observations. Procedia CIRP, 19:63-68, 2014. doi: 10. 10.1016/1ppoci:20144.5000
Nian, R., Liu, J., and Huang. B. A review reinforcement learning: Introduction and ap- plications in industrial process control. Com- puters & Chemical Engineering, 139:10 2020. doi: 0010016f.cmmpphemeneg2222.0. URL https: /tttps://www.. www.eciiiiciiiiiiicccccccec com/ http://aacccepppsssssseee
URL poieene//rrticllepp/////22222121
science/
2nd CIRP Robust Manufacturing Conference (RoMac 2014).
Roser, C., Lorentzen, K., Lenze, D., Deuse, J., Klenner, F., Richter, R., Schmitt, J., and Willats, P. Bottleneck prediction using the active period method in combina- tion with buffer inventories. In Lödding, H., Riedel, R., Thoben, K.-D., von Cieminski, G., and Kiritsis, D. (eds.), Advances in Production Management Systems. The Path to Intelligent, Collaborative and Sustainable Manufac- turing, pp. 374-381, Cham, 2017. Springer International Publishing. ISBN 1778--331--6626-11
Overbeck, L., Hugues, A., May, M. C., Kuhnle, Reinforcement learning
A., and Lanza, G. based production control of semi-automated man- ufacturing systems. Procedia CIRP. 103:170- 175, 2021. doi: 10. 10.1006//.pmmci122211.0000 URL setianeearrricll///////222222111
9th CIRP Global Web Conference- - Sustainable, resilient, and agile manufacturing and service operations Lessons from COVID-19.
Scholl, A. and Becker, C. State-of -fate-of-the-..
toffthe-ar art exact and heuristic solution procedures for simple assembly line balancing. European Journal of Operational Research, 168(3):666- 693, 2006.
Padrón, M. de los A. Irizarry, M., Resto, P., and Mcjía, H. P. A methodology for cost-oriented assembly line bal- ancing problems. Journal of Manufacturing Technology Management, 1 147-1165, 2009.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, vol- une 37 of Proceedings of Machine Learning Research, pp. 1889 1897, Lille, France, 07-09 Jul 2015. PMLR.
Pleines, M., Pallasch, M., Zimmer, F., and Preuss, M. Gener- alization, mayhems and limits in recurrent proximal pol- icy optimization, 2022. URL https: //tpx//xivvvo/////////..a org/ abs/ 2205.

---

LincFlow: A Framework to Learn Active Control of Production Lines
URL Ut/httsss//prroedddii wtt.mm.....ss////// schulman15.html
Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., Cola,
G. D., Deleu, T.,, Goulão, M., Kallinteris, A., Krimmel,
M., KG, A., Perez-Vicente, R., Pierré, A., Schulhoff, S.. Tai, J. J.. Tan, H., and Younis, O. G. Gymnasium: A standard interface for reinforcement learning envi- ronments, 2024. URL https:
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
17032.
Serrano-Muñoz, A., Chrysostomou, D., Bøgh, S., and Arana-Arexolaleiba, N. skrl: Modular and flexible li- brary for reinforcement learning. Journal of Machine Learning Research, 24(254):--9, 2023. URL http: //jmlr.org wwtprr/..grppp////////pa http:/apprrss//e http:////////22/011 htmm.
Viharos, Z. J. and Jakab, R. Reinforcement learning for sta- tistical process control in manufacturing. Measurement, 182, 2021. doi: 10.1066f..eesuuccmmt221011110000
URL
Shi, D., Fan, W., Xiao, Y., Lin, T., and Xing, C. Intelli- gent scheduling of discrete automated production line via deep reinforcement learning. International Journal of Production Research, 58(1 (0((11):36223338) 2020. doi: 10. 10.1080/00207543.2222 202.4433..0.11777 1717008.
Vinyals, O., Babuscckin, I., Czarnecki, W. M., Mathieu, M..
Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang. A., Sifre, L., Cai, T., Agapiou, P., Jaderberg. M., Vezhnevets, A. S., Leblond, R., Pohlen, T., Dalibard, V., Budden, D., Sulsky, Y., Molloy, J., Paine. T. L., Gulcehre, C., Wang, Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T., Kavukcuoglu, K., Hassabis, D., Apps, C., and Silver, D. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782): 350-354, 2019. doi: (0.1038//415566019-9
Shiue, Y.-R., Lee, K.-C., and Su, C.-T. Real-time scheduling for a smart factory using a reinforcement learning approach. Computers & Industrial Engineer- ing, 125:604-611 614, 2018. doi: 10.10166j.cie.0118.05 039. URL htwps:ttpsooomm wtw.:///wwwwwiiiieeeeeeee...... com/ sci attcc//articcee///i////333333383
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schritwieseer, J., Antonoglou, I., Panneershelvam, V., Lanetot, M., al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 2016.
Wang, X., Wang, H., and Qi, C. Multi-agent reinforce- ment learning based maintenance policy for a resource constrained flow line system. Journal of Intelligent Manufacturing, 27(2):325-335 2016. doi: 10.1007/ $10845-013-086445
SimPy, T. Simpy 4.1 webpage. 2025. URL https: s simpy. r wwwpy.eeeatthdooss..o wttps/s//eo//////ee//e//e/
Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp. 56 61, 2010. doi: 10.25080/Majora Mat.aaa9oaa-2211220000
Subramaniyan, M., Skoogh, A., Bokrantz, J., Sheikh, M. A., Thürer, M., and Chang, Q. Artificial intelligence for throughput bottleneck stalssis-ttate-of-th and future directions. Journal of Manufacturing Systems, 60:734- 751, 2021.
Wu, P., Escontrela, A., Hafner, D., Abbeel, P., and Goldberg. K. Daydreamer: World models for phys- ical robot learning. In Liu, K., Kulic, D., and Ich- nowski, J. (eds.), Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pp. 2226-2246 PMLR, 14-18 Dec ml mlr. press/
Sutton, R. S. and Barto, A. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learn- ing. The MIT Press, second edition edition, 2020. ISBN 778-0-222-0322--6
2023. URL htttt::/// http::::/ppooocediii
Szarski, M. and Chauhan, S. Instant flow distribution network optimization in liquid composite molding us- ing deep reinforcement learning. Journal of Intelligent Manufacturing, 34(1):197-218, 2023. doi: 10.1007/ s10845-022-01990-5
v205 / wwtp///w...../...oo//////oo.. html.
Zou, Y. and Lan, R. An end-to-end calibration method for welding robot laser vision systems with deep rein- forcement learning. IEEE Transactions on Instrumen- tation and Measurement, 69(7):4270-422 2020. doi: 10. 11100000.2209..99922292
Tortorelli, A., Imran, M., Delli Priscoli, F., and Liberati, F. A parallel deep reinforcement learning framework for controlling industrial assembly lines. Electronics, 11(4), 2022. doi: 10.3390eelctromiccs //www, mdpi. com 2wwmm/007---2 2027999992/17//11111
URL ht tps:
12

---

LincFlow: A Framework to Learn Active Control of Production Lines
A. An Example Use Case in LineFlow
To demonstrate how LineFlow can used to model and simulate production lines, and let an agent interact with it, we provide step-by-step implementation of a simple production line.
A.1. Implementing the Layout
In LineFlow, a production line represented by class that extends the Line class. Inside the Line. build method, we define and connect the various stations, setting their processing times and capacities. Below is an example of small production line, which consists of two source stations, one supplying the main component and the other providing an additional component required for assembly. These components are fed into assembly station, which combines them into single unit. The assembled product then passes through process station. After processing, a switch directs the product to one of two possible output paths, each leading to sink station that collects the finished items.
from lineflow simulation import Line, Source, Sink, Assembly, Process, Switch
class ShowCase (Line);
def
source_mas maicee__miin_ Source
'Sourcel', processing_time=55,
carrier_capactie
source_comp - (
'Scurce2',
processing_time time==,
part_specs= specs= ( "awsaaaaaasccondedaaala condition"; 100:], part
assembly Assembly ('Assembly', processing__tmoo44 Nok_part_orzoo___mme=
process Process ('Process', processing_time-ll
switch Switch (witchh"SSwtehh processing__iime-]]
sink_ 1
Sink ( ('Sink1', bittesssing_time-//0
sink 2 S'nnk'2,,, #rocessing_timm=110
capacity=2, transition_times Ceeembly.connet_ttee (ccomneeetooniintttssssrccaaaaan (scurce_main, capacity=3, transition_time-5
process. prrcesss.conecttto.. www.cnnnnntninnnniuuuuu (assembly, capacity-2, transition_tiie=2)
input (process, capacity=2, transition_timee:
sink c.c.nectt_ttteee patpccttt__inniii (switch, capacity=3, consittion_time-2
sink 2. loonnect_to_iie input (switch, capacity=3, ipansition_tim-22
A.2. Interaction with Agents
In LineFlow, agents interact with the production line by receiving observations from the environment and taking actions to optimize the system's performance. The state space includes features such as buffer fill levels, processing times, and the number of workers assigned to station. Based on these observations, the agent decisions, worker assignments, waiting times at sources. Agents of course be trained, but also defined as a function. The following agent dynamically adjusts the output buffer selection at the switch based on the minimum buffer fill level.
adjust parameters like switch routing
def agent (state, env) : fills ot.. return
value in [1, 211)
"Switch"; ("index_buffer__on fills. ()
"Scurcel": ("vaiting_time". 5),
"Source2"; ("waiting_tine". 2),
13

---

LincFlow: A Framework to Learn Active Control of Production Lines
To observe the simulation in action, we enable visualization by setting realt ime=True. This ensures that the system updates in real-time, allowing for clear understanding of how products move through the production line. The simulation speed can be adjusted using the factor parameter, which controls the scaling of time units. For instance, setting factor=0. 1 speeds up the process by a factor of ten, making it easier to analyze system behavior over extended periods
1ine - ShowCase (realtime- True, facter-0.1) line. run (simulat anplllatiinnnndiieee agent-agent, visual 11 ize True)
A.3. Analysing the Data
The user get all the simulation data in form of pandas dataframe, consisting of the states of all lineob jects using line. get get.observations (). LineFloy also provides preinstall analysis for basic performance indicators like
kine.get.o. www.nn.pptttttt produced () and line. get n scrapppccpppppp pertp () A sample visualization is shown in Figure 9,
Sourcel
Saurcel
Saurcez
Source2 walting time
working
working
10
wwitin
waitng
40
railng
Saing
Assembly mode
Assembly n. Scrap parts scrrappparts
Assembly processing time
Assembllyprocessing
Process,
maddeesseddddeeeeee mode
warking
waiting
faiigg
10
fallng
100
200
300
400
0
100
200
300
400
100
200
300
400
100
200
300
400
Process cereessspccessig.... Lime
Switch_mocd
Switch index buffer
Switch inndeinee_eee__ www.uuufffeooooooee
working
0.05
1.0
wailing
0.00
6.5
1.6
talling
0.05
0.0
100
200
300
100
200
Sink1_mon mode
Sinkl n. __pptss_____________________ t_pprts_producceeee produced
Sink2 mode
Sink2, 2_port_srts___..p produced
__parts_producee
working
walting
walthn
faiing
feing
100
200
300
400
100
200
300
400
100
200
300
100
200
300
400
Buffer Source2 to Assembly fill
Bufter Sourcel, to Assembly
Butfer Assembly to Pracess, fill
Buffer Process Switch
1.0
0.4
0.4
0.0
100
200
300
100
100
200
200
300
Figure 9. Selection of features extracted from the line displayed Figure visualized over time.
A.4. Training RI Agents
To train reinforcement learning (RL) agents, we implemented the rineSimulation (gym.Env) class, which takes a Line object and simulation duration as initialization parameters. This environment integrates seamlessly with stable stable_basel basebbeelllee i (Raffin et al., 2021), allowing RL models to interact with the simulation by specifying it via the env parameter. The following example demonstrates how to train an RL agent using PPO:
from stable baselines? import PPО
line ShowCase
env LineSimulation (line, simulation simulation_tiie=40000) time=4000) model PPO ("Mlppolicy"" env, verbose-1) model. learn
A.5. General Overview
LineFlow is implemented in an object-oriented fashion. The central object is the Line object, which is decomposed of multiple Lineobjects (Section A.6). A Lineobject can either be stationary, like an assembly, or movable,
14

---

LincFlow: A Framework to Learn Active Control of Production Lines
like carrier. Each Li neObjjcct has ObjectSttaee representing parameters of the system a parameter. Each ObjectSttate consists of atomic states where each can be actionable non-actionable and observable non-observable (see Section A.7). The states of all objects in line are accumulated in the LineState which provides centralized interface for Agents interacting with the Line during simulation.
A.6. Different Types of line objects available in LineFlow.
Typical properties when instantiating stationary objects are T and S that define their processing time T = + Exps as well as rework probability, which models rework by letting process run multiple times in row. Any stationary object must be connected to other objects. This is typically done via one or multiple Buffer arbitrarily many incoming and outgoing buffers where an As Assembly ly must have exactly one incoming and one outgoing for the main carrier and arbitrarily many incoming for components that need to be joined with carrier from the main track.
fer objects. For instance, Swi tcc can have
Name
Type
Description
Process
stationary
station that simulates a processing step on the part. The processing step be set to repeat due simulated human error, which doubles the processing time. This leads to doubled processing time.
Process
Sink
stationary Removes components from a carrier. Carriers arrived here are marked as OK. Empty carrier either removed or returned to a Magaz ne or Source using a separate out-buffer depending on the layout. This way, the station can be used in both linear and circular lines.
Sink
Source
stationary Places parts onto carriers. Can set individual properties, called PartSpec, to every part set up, like the assembly condition TAc. Car- riers are either created, taken from a Magaz ine, or fetched separate incoming buffer.
Assembly
stationary
Station for simulating assembly activities on the line. Individual parts and components are delivered with individual carriers, assembled with a simulated processing time and forwarded to the downstream station. Can be connected WorkerPool that can assign Worker objects modifying its processing time.
WorkerPool
stationary
Holds a predefined set of Worker objects and is attached to fixed number of stations. Multiple pools can coexist for a production line allowing modelling different skills or experience workers.
Magazine
stationary
Magazine station is used to manage the carriers. The total number of carriers available to the line can be controlled via this station. The capacity of the carriers, ie. the possible number of components that can be added at the assembly station, also determined by this station. If the number of carriers is of interest, the source can place the parts directly carriers and magazine is required.
Magazin
Switch
stationary
The Switch distributes carriers to different stations, enabling parallel structures within the line.
Switch
Buffer
stationary
The Buffer transports carriers from one station to another. Time needed push and get carriers to and from can be specified well as its capacity and time carrier needs to traverse the buffer.
Carrier
movable
set up at a Source station or a Magazine and holds predefined number of Part objects.
Magazio 22
15

---

LincFlow: A Framework to Learn Active Control of Production Lines
Part
movable
Single unit which is initially created a Source. Holds PartSpec each station handling it can access and individually adapt
Magazin 2
Worker
movable
Belongs to WorkerPool and can be assigned to a station. Traversal time can be configured.
Al
SAURT
Table 2: Different types of LineOb sineobjectss ject
A.7. Different Types of States
This section provides an overview over the clementary states implemented LineFlow. Every state associated at least one LineObject ject and the mapping of selected objects states given in Table 5. In general, state can be actionable or them to other values within their value range via runtime. Per default, every
non-actionable meaning that agents observable, meaning that they part of the observational space agents have access However, to facilitate ablation studies simulate sensor failures, any state can be selectively masked as non-observable. Many states are consequences of events taking place at the line, like how many carriers are on a buffer, and cannot be influenced directly. All states kept updated constantly and represent at any time the current situation. Some states, however, are lagged in the sense that their update happens once an certain event is triggered. For instance, the state holding the processing time of a station updated once the process over with the value of the last process.
The states can be categorized into two main categories: discrete and numeric. Discrete States the side, handle categorical items such as station modes. Internally, the values of discrete state are labeled encoded to integer numbers. A subclass are count states that represent integer numbers, like the number of scrap parts at station. Numeric states, on the other side, hold continuous numerical values, like processing times or buffer fill levels. Their value range may be restricted by upper and lower tolerances. Table provides a list of states implemented in LineFlow. All states are mapped to the respective states of gymnasium (Towers et al., 2024). At any simulation step 6, the states all LineObjects are fetched and put into an agent, which returns new values for the list of actionable features to be updated accordingly.
Name
Type
Actionable
Description
Mode
discrete
non-actionable
The mode station is currently in. Is either working, fail ling, or waiting.
Processing time
numeric
non-actionable
The processing time of the last process.
Waiting time
numeric
actionable
Value used to wait till two parts produced.
Station-Assigament
discrete
actionable
Discrete value denoting to which station a worker as- signed to. Must be one of the predefined stations attached to the worker pool the worker belongs to.
#Workers
count
non-actionable
Number of workers assigned to a station. Aggregated fea- ture computed from the station-asee assoonnmssggmmn all workers
#OKs and #NOKs
count
non-actionable
Number of OK and NOK parts the station has produced so far.
Buffer-Indices
discrete
actionable
The incoming and outgoing index of stationary object, like Switch, it should get and push components to.
Buffer-Fill
numeric
non-actionable
Relative number of carriers on a buffer.
Table 3: Examples of states.
16

---

LincFlow: A Framework to Learn Active Control of Production Lines
Processing
Current mode
Number of workers
Nok
Waiting time
In and out buffer
time
Assembly
O, N-A
O, N-A
O A
O, N-A
Process
O, N-A
0,
O,
0, N-A
Source
N-A
N-A
O.N
0,A
Switch
N-A
N-A
0, A
Table Default mapping selected stationary objects states. Letters and stand observable and actionable respectively, where N marks non-observuble and non-actionable.
A.8. Visualization
Once layout is implemented in LineFlow, object of the respective Line object allows set visuall 1 ze=True during simulation run, which renders the current state in pygame. Figure 10. As layouts implemented in custom line classes, users have control over how these lines are instantiated and can use the power of the programming language of python, i.c., parametrizing the number of sinks or assemblies via constructor arguments or using for-loops to set them up.
Figure 10. Some layouts implemented in LineFlow.
B. Optimality Proofs for Case Studies
In this section, we give proofs for optimal control policies for the scenarios WT and WTJ in Section B.1, PD Section B.2, and WA in Section B.3. In Section B.4, state heuristic for CL yielding near-optimal reward.
B.1. Waiting Time
Let Tsg and Ta be the process time distributions Se and A respectively and denote by Tp and Tg the time distributions to push and get component and from buffer respectively. First, state formula for the optimal waiting time at Sc- Essentially, the waiting time has to fill the gap between the times A and Sc need to handle and process their parts, which are TA + 2 Tg + Tp and Ts + Tp- Thus, the optimal waiting time can be globally determined as the expected value of their difference
T E + Ts
Particularly, it suffices in WT to learn static waiting time. See also Figure 11 for how the waiting time at Se affects the reward. To calculate the maximum number of parts that can be produced, assembly to start processing well the time the last part produced in Tei needs to be finished by the sink. Let Tsc Tsx TA. and Ts be the process time distributions of Sc. SM. A, and the sink S respectively. Moreover, let T TsM A and TA-s be the times part needs to traverse on the buffers between Se and A, SM and A, and A and the sink S. The assembly can start its processing once a part from SM and So has arrived, that is
have consider the time that it take for the
T=A == max (Tso + TT Tse A, TsN + Tp + T
17

---

LincFlow: A Framework to Learn Active Control of Production Lines
where T and To denote the times to push and get component and from buffer respectively. As the first part also needs transfer the sink S and has be processed it, the expected value of the maximal parts produced from 0 to sim is:
T sim
T+A
TA-S Tg Ts
(4)
TA + 2 Tg Tp
160
140
120
100
Produced parts
Espected part macimum
Optmal waiting time TM
80
10
15
25
Waiting time
Figure 11. Visualisation of the setting in WT with optimal waiting time Tw 18.5 as in Equation (3) and the maximal number of expected parts as computed in Equation (4).
Next, we explain the construction of and T jump of WT, with jumping processing time of the assembly A. Following Equation (4) and ignoring the ramp-up and jumps the assembly station, the expected number of parts produced Taim
Taim
T Tsm
E + 2Tg + Tp)
T S
with E = R[2T, Tpl. When the assembly processing with f + Exps instead of T + Exps for period of T Jump, the number of expected parts is:
Tam T Tjmp + Tjump T+S+E ST+
Clearly, if > 1, the expected number of produced parts must be smaller than N. Thus, we want to construct f from a sampled Juump such that the expected number of produced parts R N for a fixed constant < R < 1. Then, for a for a sampled Jump. we look for f such that the expected parts satisfiss:
T jump
T Tjump
sim
ST+S+E R.N
T
T+S+E
is hard to see that this is the case for
1 T Tjump" (T + S F.) 1 f = T (R 1R-11 1) Tem + Tjump
E
To verify that the maximal number of produced parts is in fact R N, we run simulations in LineFlow using varying values of R and fixed waiting time of () at the source to keep the assembly constantly loaded. As this would lead to scrap, which in turn blocks the assembly time for certain amount of time, we additionally set a very large assembly condition TAC- Figure 12 shows the empirical frequency of the number of parts produced for WTJ for different R values as well as R N with N' the expected value of WT showing that and Tjump are constructed as desired.
18

---

LincFlow: A Framework to Learn Active Control of Production Lines
140
Produced parts
130
120
110
0.75
R
Figure 12, Number of parts produced for WTJ and WT with large assembly condition Tac and waiting at Se for varying R.
To get an optimal policy for WTJ, we estimate E[TA 2 Tse from Equation (3) by regressing on [T with the processing times reported from A. Essentially, we take rolling mean of the last processing times observations from A. Clearly, the larger I, the better the estimate for [B[TA] in time period where the mean does not jump. However, the larger I the worse the new estimate the waiting time adjusts to a new level. By varying and testing the agent for WTJ (see Figure 13), we found that gives the best reward. This reward is used as optimal value Section 5.
120
115
Roward
8
110
3
Look-back
Figure 13. Reward for agent for varying look-back l.
B.2. Part Distribution
In Section 4.2, we showed that the maximal number of parts that can be produced in PD2 is
T Tsim
E[ =
(5)
(1 + S.) T.
Particularly, the fraction of all parts produced by P; is
N N
P E
(6)
T
and consequently, the optimal distribution policy for the switch also needs to deliver that number to P: to reach the maximal number of parts. Put differently, the optimal policy sends part with probability p: to P.. We have implemented in LineFlow an agent that greedily controls the switches: A carrier is pushed on a buffer having lowest fill and fetched from a buffer with highest fill. This greedy policy have been evaluated for E {3, 4, 5) and its part distribution compared to the optimal distribution (PI....,Pk) from Equation (6). The result is shown in Figure 14.
19

---

LincFlow: A Framework to Learn Active Control of Production Lines
1
RECRESED SEEE
(a) WAa.9
(b) WA4.12
(c) WA5,15
Figure Performance of possible monotone partitions for = (16 41-444440 4), and 0.3 for a simulation length of 2000 in the worker distribution example.
Part distributions gready
Part distth. iff
P4
distributions
grezdy
P2
greedy
P2
oum
P2
sptimumpp
0.15
Figure Part distributions of the greedy policy compared with optimal distribution for E (3, 4,5}.
B.3. Worker Assignments
In this section, we formulate
To
min max E EETT, sTr (n (e.....)JJNN..... LE(A]
as given in Equation (2) as integer optimization problem. Recall that ETTr..s..ul = T; (p.(n:) + S:). We equivalently
reformulate the minimization problem by introducing a real-valued auxiliary variable m as follows:
min 171 R2 P1 7m
subject to
m 2 T, (pa(ns) S.) Vi E [k]
Σ n4 =
n1 E Nzo
EiEl [k]
m Rzo
In our benchmarks, we used T; = (16+i.4) and == 0.3 for varying & and N = 3- k. We implemented this optimization problem in gekko (Beal et al., 2018). Solving Equation (7) yields (2, 3, 4) for k = 3, (2, 3, 4) for k: = 4, and 22, 2, 4, 4) for 1: = 5. To evaluate the correctness of LineFlow, we enumerated and evaluated all monotonic worker assignment, that n, E ny for i < j, for E {3, 4, 5) for multiple runs with simulation length 2000 (see Figure 15). We found the optimal assignment obtained empirically matches the exact optimum obtained by using Equation
20

---

LincFlow: A Framework to Learn Active Control of Production Lines
Name
Value
Simulation time Tsim
4,000
Step size stop
Rollout steps
1,000
Discount factor
0.99
Number of environments (vectorized)
5
Number of stacked observations
Advantage normalization (only PPO)
False
Batch size in update
1,000
Number of epochs in update
5
Clip range (only PPO and A2C)
0.2
Maximal gradient norm (only PPO)
0.5
Policy sstable-baselins3)
MipPolicy (MipLstmPolln licy for recurrent PPO)
Table 7. General hyperparameters used for all algorithms.
B.4. Complex Line
We encountered several challenges when designing an effective control policy for the complex line. Due to the small buffer capacities between assembly stages, the line is prone to blockages, leading to scrap parts and a significant drop in reward. Maintaining a steady production flow proved to be crucial. To resolve potential jams as efficiently as possible, we implemented switch distribution policy that prioritizes later assembly stations. While this approach helped mitigate blockages, it also introduced new challenges, such as imbalanced utilization of early-stage buffers, which could lead to delays part availability. Regarding worker distribution, our analysis showed that keeping a fixed distribution throughout the episode consistently outperformed any of our redistribution strategies. We attribute this effect to the time required for worker redistribution, which likely disrupts the system's stability. Additionally, frequent reassignments may introduce inefficiencies due to the time needed for workers to relocate and adapt to new tasks. We also explored adaptive worker allocation strategies that dynamically reassign workers based on buffer fill levels. However, these approaches often led to oscillatory behavior, where workers continuously moved between stations without improving throughput. This suggests that overhead of frequent redistribution outweighs its potential benefits in our setup.
C. Benchmarking Details
In our experiments, we used stable-baselines3 (Version 2.3.2). In total, 22 models have been trained and evaluated for every scenario, each model using three different random seeds resulting in 66 trained models for each scenario. The runtimes vary: Roughly 14 hours for PD2 are required to train a single model, whereas only 7 hours for WTJ and 2 hours for WT. To train a stacked and recurrent version of PPO for scenario CL NVIDIA H100 GPUs were used. A single stacked PPO model trains roughly 20 hours, where the recurrent models train for almost days.
21

---

LincFlow: A Framework to Learn Active Control of Production Lines
C.I. Waiting Time
no pooa
(a) WT
(b) WTJ
Figure 16. Best performance of algorithms on evaluation environments for WT and WTJ.
1970
(a) WT
(b) WTJ
Figure/ 17. Reward steps for WT and WTJ.
C.2. Part Distribution
no
Nuu und
feveerte call
(a) A =
(b) 1:
(c) k = 5
Figure Best performance algorithms on evaluation environments for PD
(a) l:
(b)
(c)
Figure 19, Reward over steps for PD:.
22

---

LincFlow: A Framework to Learn Active Control of Production Lines
C.3. Worker Assignment
Me masa my
no on
(a) k
(b) l:
(c) k =
Figure 20. Best performance of algorithms on evaluation environments for WAR,3k.
(a) l: = 3
(b) k = 4
(c) k =
Figure 21. Reward over steps for WA
D. Validation on Real Production Data
In this section, we evaluate the sim-to-real gap of LineFlow using the publicly available production line dataset from (Risdal et al., 2016). In this dataset, each row corresponds to produced part, and the columns record timestamped feature activations at various stations. Based on this information, we reverse-engineered the production layout, which consists of two parallel pre-assembly lines with 12 stations each (see Figure 23), both feeding into shared subsequent line. For our analysis, we focused on the pre-assembly line responsible for the majority of component production.
We first analyzed the distribution of processing times at each station and found that they closely follow exponential distributions, which supports the modeling assumptions used LineFlow (see Figure 22). The limited resolution of the provided timestamps required a pooling of the processing times of successive parts. Specifically, because individual job durations could not be precisely resolved, applied a rolling, non-overlapping window over the production sequence and computed the average processing time across 100 consecutive parts. This smoothing technique allowed us to approximate the underlying processing time distribution while mitigating the effects of timestamp granularity.
16
2.1
Figure 22, Histograms of the averaged processing times in a log scale together with a fitted exponential distribution in red.
Next, we implemented the reconstructed layout in LineFlow and with the aim to compare the number of parts produced in the simulation with those observed in the real system. Since the dataset does not provide information about traversal
23

---

LincFlow: A Framework to Learn Active Control of Production Lines
times the number of slots in the connecting buffer, we made the simplifying assumption that traversal times negligible compared station processing times. Additionally, assumed that the switches before and after the stations distribute parts in round-robin fashion. The results showed close match (see Figure 23): The number of parts produced by our simulation matches the number of parts produced by the real system, providing empirical evidence that it accurately models real-world production dynamics.
29
2000
15000 378600
Figure 23. The implemented layout in Line (Ieft) and its simulated output compared with the output of the real production line (right).
24


Reviews:
[{"review_id": "f6cSt2mUi0", "reviewer": "Reviewer_A1PN", "summary": "This paper proposes LineFlow, an environment construction framework for production lines, which provides a generalized framework for research in the field of production lines. Additionally, it constructs several typical and complex scenarios to evaluate the performance of different reinforcement learning algorithms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very helpful comments and suggestions that helped us improve our\nmanuscript in various places.\n\n### Replies to strengths and weaknesses\n\n> The authors only consider the processing time of stations and their statistical interplay, assuming this time follows an exponential distribution. This design may oversimplify real-world scenarios.\n\nBefore replying, we want to emphasize that both restrictions are not limitations of LineFlow in\ngeneral, but of the specific experiments we performed. In fact, LineFlow allows users to set an individual\nand specific statistical distribution for each process. Details will be released in the documentation of LineFlow.\nThe fact that we used the exponential distribution to model the processing time is because this is\ncommon in many statistical analyses of production lines, such as in [2]. We validated our assumption with\nthe dataset [1] and found that the exponential distribution constitutes a reasonable assumption here\n(see also our reply to reviewer `9j5R` for more details on the analysis).\n\nRegarding the action space: The actions possible depend on the objects used in the layout and, as correctly identified by the\nreviewer, also scale correspondingly. However, we see that for production lines, the action space\nis not directly something that can be explicitly *designed* but is more a consequence of the\nconcrete layout, which we considered as fixed. We apologize that this was not addressed well enough\nin the initial version of our manuscript, and we will expand Section 3.2 in the revised version.\n\nConsequently, we are of the opinion that the curse of dimensionality is less a problem of LineFlow\nin particular, but more an inherent challenge of *RL for active line control*. As correctly\nidentified by the reviewer, this is also what we see in our experiments: Finding optimal control\npolicies for larger layouts requires significantly more training time. We think this is a key challenge\nfor further RL research that can be addressed using LineFlow.\n\n### Replies to specific questions\n\n> Do OK parts mean parts that have been finished and NOK mean not OK?\n\nWe apologize that we have not elaborated on these abbreviations in more detail, and we will provide a\nprecise definition in the revised manuscript. In general, OK denotes parts successfully built into a\nfinal product, while NOK parts denote parts that have been discarded at some station of the line.\n\nRegarding to keep the actions feasible: In fact, one key benefit of LineFlow is that the action\nspace is a consequence of the layout to be optimized, and it is impossible for a given policy to\nviolate constraints induced on the action space. This is due to the fact that we designed LineFlow\nin a way to model elementary objects of assembly lines to match their real-world behavior. In the\nconcrete case of WA, every workers is modeled as a specific objects that can be assigned\nindependently to at most stations. We agree with the reviewer that the action space of the worker\nassignment problem could be presented in more detail, and we will improve this in the revised\nversion.\n\n> How is the running time of each step in the three cases and in CL?\n\nThe times per step are (k=3, 4, 5):\n- WT/WTJ: 0.000052s/ 0.000044s\n- PD: 0.00011s, 0.00013s, 0.00016s\n- WA: 0.00016, 0.00018s, 0.0002s\n- CL: 0.00008s, 0.00010s, 0.00012s\n\n> Why can the best rewards in WT, WTJ, WA, and PD match the optimal rewards while falling far behind the \"simple\" heuristic baselines in CL?\n\nWe think the key reason is that the CL scenario combines multiple control challenges — worker\nassignment, component distribution, waiting time tuning, and timing-sensitive scrap avoidance — into a\nsingle high-dimensional task that requires long-term memory and coordinated action sequences. While\nthe *atomic* tasks each isolate one of these dimensions, CL requires simultaneous control across all\nof them, making it significantly harder to learn from scratch.\n\nIn contrast, the heuristic baseline for CL incorporates prior domain knowledge and is hand-tuned to\nbalance part flow and buffer states in a globally consistent way. This prior knowledge allows the\nheuristic to avoid deadlocks and high scrap rates early on, while RL agents typically suffer from\npoor initial exploration, leading to unproductive or even blocking behaviors during early training\n(as shown in Figure 6). \n\nWe will revise the discussion in Section 5.3 to clarify this gap and to better contextualize the\nstrengths and current limitations of the RL approach in complex tasks.\n\n### References:\n- [1] Meg Risdal et. al., Bosch Production Line Performance. https://kaggle.com/competitions/bosch-production-line-performance, 2016.\n  Kaggle.\n\n- [2] Bierbooms, R. (2012). *Performance Analysis of Production Lines: Discrete and Continuous Flow Models*. PhD Thesis, Technische Universiteit Eindhoven, Eindhoven. ISBN: 9789053355435.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. This work gives the production line design in detail.", "theoretical_claims": "N/A. This paper does not include theoretical claims. The proofs are provided for the optimality of the computation in 3 cases.", "experimental_designs_or_analyses": "Yes. I checked the experiments setting and the results, and did not find main flaws.", "relation_to_broader_scientific_literature": "This work may contribute to RL for combinatorial optimization, especially for scheduling  or VRP, which are classical NP-hard problems.", "essential_references_not_discussed": "Some recent related works concerning the production line problem have been involved.", "other_strengths_and_weaknesses": "**Strengths:**\n\nRegardless of the domain knowledge related to production lines, this paper is relatively easy to follow. It thoroughly considers several typical cases in active line control, introduces fundamental elements, constructs the LineFlow framework, and encapsulates it in the Gym interface, which is convenient for RL researchers. The LineFlow framework allows for easy customization of new scenarios and offers flexible interfaces.\n\n\n\n**Weakness:**\n\n- Some statements are slightly unclear (see questions). \n- The authors only consider the *processing time* of stations and their *statistical interplay*, assuming this time follows an exponential distribution. This design may oversimplify real-world scenarios.\n- The action space design is only briefly discussed. Training an RL agent in a large action space is challenging. If the agent must distribute parts or workers to different stations at scale, this will lead to a complex combinatorial optimization problem and face the notorious curse of dimensionality. Consequently, LineFlow’s running efficiency may be hindered. I also noted that the training time for a single model is non-negligible (approximately 14 hours for PD$_5$ and 6 days with an H100 GPU in CL for recurrent PPO). Thus, how to reduce the action space and how LineFlow addresses this issue should be clearly explained.", "other_comments_or_suggestions": "**Suggestions:**\n\nThe main text includes many specific layout diagrams, while the introduction to the LineFlow framework itself is minimal. It might be better to include the component diagram of LineFlow (Figure 9 in Appendix A.5) in the main text instead.\n\n\n\n**Grammars:**\n\nCheck the grammar: \"We refer to Section A.1 for more details about the worker assignment is implemented in LineFlow.\"", "questions_for_authors": "1. Do OK parts mean parts that have been finished and NOK mean not OK?\n2. What are the action spaces, and how to ensure the actions are feasible? In production line problems, actions should satisfy certain constraints. For example, in Sec. 4.3 Eq.(2), worker assignment is inherently an integer partition optimization problem.\n3. How is the running time of each step in the three cases and in CL?\n4. Is there any possibility of other distributions for the processing time?\n5. Why can the best rewards in WT, WTJ, WA and PD match the optimal rewards while falling far behind the \"simple\" heuristic baselines in CL?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4sXAXJQk2O", "reviewer": "Reviewer_Hikp", "summary": "The paper introduces LineFlow, a reinforcement learning-based framework for optimizing production line reallocation, rescheduling, and routing. The authors develop LineFlow as an extensible Python package that facilitates large-scale RL training and simulation. They benchmark multiple RL algorithms, including PPO, TRPO, and A2C, demonstrating that RL-based policies can approximate optimal solutions in structured scenarios. The study presents three key case studies—optimal waiting time, part distribution, and worker assignment—alongside a more complex case that integrates these factors. The work is application-driven, offering insights for both manufacturing and RL-based decision-making.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the insightful feedback and\nsuggestions. \n\n### Justification of discrete-time MDP\n\nA major concern of the reviewer was why we used a discrete-time MDP (DTMDP) to model active line control\nproblems instead of a continuous-time MDP (CTMDP), and we thank the reviewer for this insightful and\nimportant comment. \nWe want to highlight that in LineFlow, the simulation runs in continuous time and only the\ninteraction with the agents is discretized. \nWe acknowledge that production lines typically operate in continuous time, and a CTMDP formulation\ncould, in principle, more naturally capture the queuing dynamics. However, we opted for a\ndiscrete-time MDP for the following reasons: \n\n- **Control Intervals in Production Systems**: In real-world manufacturing, decision-making often\n  happens at fixed intervals (e.g., every few seconds or minutes), corresponding to worker task\n  assignments, batch processing schedules, or supervisory interventions. Our DTMDP formulation\n  aligns with these real-world control policies.\n- **Discretization of Continuous Events**: While production events (e.g., job completions) occur\n  asynchronously, they can be approximated using a sufficiently fine-grained discrete-time model\n  without significant loss of fidelity. Similar approaches have been used in prior RL research for\n  industrial systems. This is accomplished by the parameter `step_size` in the LineFlow\n  implementation.\n- **Practicality in RL Training**: Many reinforcement learning algorithms, particularly deep RL\n  methods like PPO, are naturally formulated in discrete time. \n  Extending them to continuous-time settings requires additional modifications with unclear practical benefits and more complexity on the agent's side.\n\nWe appreciate this perspective and will expand our discussion in Section 3.1 of the revised\nmanuscript to clarify the trade-offs between discrete-time and continuous-time modeling. Moreover,\nwe also would like to point the reviewer to our reply to `9j5R` about *Real-World validation*, where\none also can see how well the discrete MDP approximate a real-world system.\n\n### RL vs traditional optimization methods\n\nThe reviewer wrote:\n\n> It seems that the paper uses another approach to compute optimal layout. The complexity of that algorithm, and its performance against RL is not provided.\n\nand \n\n> It does not convincingly argue for the necessity of RL over traditional optimization methods.\n\nIt seems there may be a misunderstanding at the reviewers side. If by *optimal layout* the reviewer means *optimal policy*, the optimal policies for the case studies are derived by solving the mathematical optimization problems in Section 4, as detailed in Section C.\n\nIf by *optimal layout* the reviewer refers to layout optimization, then we have to emphasize that our work is\n**not** focused on this. For this topic, established methods like assembly line balancing\nalready exist. Instead, we aim to find a policy $\\pi$ maximizing\n$\\mathbb{E}[C_{\\pi}(T_{\\mathrm{sim}})]$ under the MDP dynamics. \nWe are not aware of any standard approach adressing this task, and we welcome pointers to\nrelevant literature (please, also see our reply to reviewer `R3X2` on the **Comparison with\nindustry-standard approaches**).\n\n### Justification of reward function\n\nThe reviewer wrote:\n\n> The formulation of rewards and the optimization objective (discounted vs. average vs. total reward) lacks motivation, raising doubts about real-world alignment.\n\nWe thank the reviewer for the comment and apologize for any lack of clarity. In\nSection 2.3, we elaborate how our reward structure is based on Overall Equipment\nEffectiveness (OEE), a **widely-used** manufacturing metric, ensuring alignment\nwith industry objectives and providing an interpretable signal for policy\nlearning. We further chose to optimize total reward over a fixed time horizon (e.g., an\n8-hour shift) to reflect typical planning cycles, distributing it temporally to\nfacilitate learning.\n\nWe acknowledge that a more explicit discussion of these design decisions could help\nstrengthen the paper and will revise Section 2.3 accordingly.", "claims_and_evidence": "The submission provides empirical support for its claims, but key justifications are lacking. Given the queuing nature of production lines, the choice of a discrete-time MDP over a continuous-time MDP is not adequately justified. Similarly, the reward function lacks a clear rationale. These gaps raise doubts about the practical justifiability of the results.", "methods_and_evaluation_criteria": "Same as above.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. It is unclear why the problem is modeled as a discrete-time MDP rather than a continuous-time MDP, given the queuing dynamics of production lines.  \n2. The paper does not justify the chosen reward function, leaving it uncertain whether it accurately reflects real-world optimization objectives.  The rationale for framing the control objective (maximizing Cost while minimizing time to reach the maximum) within a discounted optimization framework needs further explanation.  \n3. It seems that paper uses another approach to compute optimal layout. The complexity of that algorithm, and its performance against RL is not provided.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper introduces a Python package for simulating production lines and training RL agents.\n- It evaluates multiple RL algorithms (PPO, TRPO, A2C) across diverse production scenarios.\n\nWeaknesses:\n\n- It lacks justification for using a discrete-time MDP over a continuous-time MDP.\n- The formulation of rewards and the optimization objective (discounted vs. average vs. total reward) lacks motivation, raising doubts about real-world alignment.\n- The paper focuses on practical applications without offering broader theoretical insights for similar problems or presenting novel challenges for the broader ML community.\n- It does not convincingly argue for the necessity of RL over traditional optimization methods.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Given this production line setting, can probability distributions over time delays be ignored? Specifically, why is the system modeled as a discrete-time MDP?\n2. If the goal is to \"find a control policy π that maximizes Cπ(t) while simultaneously minimizing the time t to reach the maximum,\" how is this reduced to bounded-time reward maximization? What is the exact optimization problem being solved? Is it an average-reward (throughput maximization) problem, or a reachability-like problem where the objective is to achieve the maximum value as quickly as possible?\n3. What algorithm is used to compute the optimal reward? What is its complexity, and how does it compare to RL?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RKZNW0Ku6R", "reviewer": "Reviewer_R3X2", "summary": "The paper introduces an open-source Python framework for simulating production lines and training RL agents to control them. The authors demonstrate the framework capabilities through several core subproblems of active line control with corresponding mathematical analyses. Results show that while RL policies approach optimal performance in simpler scenarios, complex industrial-scale production lines still present significant challenges, pointing to the need for further research.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer emphasizing that we provide \"a comprehensive,\nwell-designed framework\" addressing \"practical production line control problems\"\nand \"provide a good comparison of optimal solutions, heuristic approaches, and\nagents' scores\". We would like to thank the reviewer for the insightful feedback and\nsuggestions.  \n\n### Real-world validation with actual production data\n\nThe same concern was raised by reviewer `9j5R`, and we want to point to the\nreply we gave there.\nParticularly, we extract processing and transition times from a real world production data set and\nshow a comparison of real and simulated line performance. We will include these results in the\nappendix of the final version of our paper.\n\n### Reward function design challenges\n\nWe thank the reviewer for highlighting the need for a more complete discussion on reward function\ndesign. Common goals include maximizing parts built or revenue, with the widely used *OEE* score\narising as a special case of our target score (equation 1). In general, LineFlow does not depend on\na fixed reward function and the user can freely select any target goal to be optimized in the layout\nat hand.  To address this further, we will add a section detailing special cases of $C_\\pi$ and\ntheir alignment with typical optimization objectives.\n\n### Comparison with industry-standard approaches\n\nWe are not completely sure what the reveiwer means with *industry-standard approaches* and in that\nparticular context, *comparison*. If the reviewer refers to standard-approaches for *finding control\npolicies* which need to be compared by their reached reward, we think a comparison is implicitly\ngiven already as we compare the control policies found by RL with the theoretically *best possible\npolicy*. Thus, any other policy, whether standard or not, must be inferior to the RL solution. In\npractice, to the best of our knowledge, there exists no *standard approach* to find a good runtime\ncontrol policy for a concrete layout, although there exist proprietary *software tools* that help to\nanalyse production line and given control strategies. If, on the other hand, the reviewer means\ncomparing the *data-efficiency*, this is a more involved question to answer, as this requires\nsetting more rules on how much a priori information an approach is allowed to use. This is currently\nbeyond the scope of our paper but definititely an intereseting future research question.\n\nWe hope that adresses the question well enough and if not we hope the reviewer can elaborate the\nquestion a bit more. Please also see the answer regarding real world validation for `9j5r`.\n\n### Exploration of scalability to larger production systems\n\nIn general, the scope of our paper was to introduce LineFlow and show that optimal policies can be\nlearned in controllable scenarios. The scalability to larger production systems requires a more sophisticated application of RL algorithms than currently done in our work, and we want to leave this open for future research.\n\nReviewer `9j5R` also asked about larger production systems and the curse of dimensionality, and\nwe would like to refer to our answer *Multi-agent RL* given to `9j5R`. We hope that also\nprovides clarification for this point raised. \n\n### Heterogeneous machine types\n\nYes, LineFlow can handle heterogeneous machine types, and our experiments reflect this (see also\nAppendix B). We included scenarios where machines have varying processing times or assembly\nprocesses—key characteristics of heterogeneous machine types. \n\n### Goodness-of-fit for ICML\n\nWe appreciate the reviewer’s acknowledgment of the solid engineering effort behind LineFlow and the\nbridging of reinforcement learning with production line optimization. At the same time, we take the\nconcern regarding the lack of algorithmic innovation seriously and are grateful for the honest and\nconstructive feedback. We also want to apologize if the use of the phrase *theoretical foundation* may\nhave given the impression that the paper aims to introduce novel theoretical results. Our intention\nwas to refer to the theoretical validation of the benchmark tasks—by computing ground-truth\noptima—rather than to suggest the development of new RL theory. We recognize that this wording may\nhave caused confusion and will revise the manuscript to clarify our intent and better reflect the\nnature of our contributions.\n\nWe acknowledge that our work primarily focuses on applying reinforcement learning to an industrial\nproblem rather than introducing a completely new RL algorithm. However, we believe that our\ncontributions align well with ICML’s scope as the methodological contributions and real-world impact\nof our work are valuable to the ICML community. Many impactful papers at ICML focus on novel\napplications, domain-specific adaptations, or applying RL to new challenges, and we believe our work\nfits within this tradition.", "claims_and_evidence": "* LineFlow effectively simulates production lines: The framework comprehensively models various elements of production lines including machines, buffers, and workers with sufficient flexibility and realism.\n* RL agents can learn near-optimal policies for well-defined subproblems: Experiments demonstrate that RL algorithms approach optimal solutions for specific subproblems like optimal waiting time, part distribution, and worker assignment.\n* Complex production line control requires additional techniques: For complex scenarios, basic RL approaches fail without curriculum learning, as shown in their \"complex line\" experiments.", "methods_and_evaluation_criteria": "The methods and evaluation seems appropriate for the domain:\n* The authors properly formulate production line control problems as MDPs, suitable for RL approaches.\n* RL algorithms are benchmarked against theoretical optimal solutions and heuristic approaches, providing good comparisons.\nSome limitations include relatively limited comparison with industry standard approaches and insufficient exploration of scalability to larger production systems.", "theoretical_claims": "The paper lacks novel theoretical contributions. The mathematical formulations presented as \"theoretical foundations\" are standard applications of existing optimization principles to specific production settings. These serve more as validation benchmarks than theoretical advances in either RL or production optimization.", "experimental_designs_or_analyses": "The experiments are competently designed but primarily demonstrate the application of known techniques to a specific domain rather than revealing fundamental new insights about RL or optimization algorithms.", "relation_to_broader_scientific_literature": "The work bridges production line optimization and RL but does not significantly advance either field. It represents a solid engineering effort rather than the kind of theoretical or algorithmic innovation expected at ICML. It might be an ideal paper for IEEE or other venues though!", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "Strengths:\n* Well written paper\n* Addresses practical production line control problems\n* Provides a comprehensive, well-designed framework\n* Effective progression from simple to complex scenarios\n* Thoughtful application of curriculum learning\nWeaknesses:\n* Limited real-world validation with actual production data\n* Incomplete discussion of reward function design challenges\n* Limited comparison with industry-standard approaches", "other_comments_or_suggestions": "No further comments.", "questions_for_authors": "I might have missed this, but does LineFlow handle heterogeneous machine types as stated in the related work? Do your experiments simulation reflect that?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KINDOWTAAL", "reviewer": "Reviewer_9j5R", "summary": "Reinforcement learning (RL) has demonstrated potential in optimizing production line control. However, a standardized and general framework remains absent. To address this, the authors present LineFlow, an open-source, extensible Python framework for simulating production lines of arbitrary complexity and training RL agents to manage them.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are excited that the reviewer\nfinds that \"Despite the importance of active line control across various industries, no\nwell-grounded simulation framework has been available for training RL agents in production line\nsettings. Thus, this new framework has significant impact\".  We thank the reviewer for the suggestions.\n\n## Real-World validation\nA key concern raised by the reviewer was whether LineFlow is capable of accurately simulating real-world applications.\nWe would like to clarify that the case studies presented in the\npaper — WT (waiting time), WTJ (waiting time with jumps), PD (part distribution), and WA (worker\nassignment) — are not synthetic toy problems but are, in fact, directly motivated by real-world\nchallenges faced by our industrial partners.\nWhile we abstracted and isolated these problems to enable mathematical analysis and controlled\nbenchmarking, they reflect real production phenomena observed in practice. They\nare furthermore reflected in the literature, as  the part/assembly line feeding problems [2, 3, 4, 5, 6].\n\nNevertheless, we agree with the reviewer that a real-world validation can strengthen the impact of\nour work. Generally, there exist strong privacy concerns among manufacturers when it comes to\nreleasing production line layouts. Simply releasing the cycle time of a line alone can give a\ncompetitor information about production efficiency, capacity, and competitive advantage.\nConsequently, publicly available datasets holding performance information and non-trivial layouts\nare rare. Here, we hope that LineFlow will close this gap by allowing manufacturers to provide\nsynthetic non-confidential digital twins preserving their key runtime challenges to the public.\n\nTo further validate the sim-to-real gap of LineFlow, we conducted an evaluation using the publicly\navailable Bosch production line dataset from [1]. We first analyzed the processing time\ndistributions and confirmed that they closely follow exponential distributions, supporting the\nmodeling assumptions used in LineFlow. Due to the insufficient resolution of the provided\ntimestamps, this analysis was non-trivial.\n\nhttps://imgur.com/a/icml-rebuttle-images-LLTUpZF\n\nWe then reverse-engineered the production layout (having 13 stations) from the dataset and\nimplemented a corresponding simulation in LineFlow. The comparison between the number of parts\nproduced in the simulation and the number observed in the real-world data showed a very close match:\nLineFlow produces $3909$ parts in a time frame $4000$ parts have been produced for the real dataset,\nproviding strong empirical evidence that LineFlow accurately models real-world production dynamics.\nWe will include this validation in an additional section in the appendix of the revised manuscript.\n\nRegarding industry implementations for active line control, we can only refer to our conversations with industry partners. These discussions revealed that active line control in production is often managed directly on the production line by personnel or through simple heuristics like the ones we implemented for our scenarios. Our results show that these heuristics perform well, but RL agents achieve comparable performance, demonstrating their viability as an alternative approach.\n\n## Multi-agent RL\nThe reviewer is right that multi-agent RL is a good tool for addressing the curse of dimensionality of\nlarge-scale assembly lines. In fact, we declared this as one of the future works.\n\n### References:\n- [1] Meg Risdal et al., Bosch Production Line Performance. https://kaggle.com/competitions/bosch-production-line-performance, 2016. Kaggle.\n- [2] Daria Battini et al., The Integrated Assembly Line Balancing and Parts Feeding Problem with Ergonomics  Considerations, IFAC-PapersOnLine, Volume 49, Issue 12, 2016, Pages 191-196,\n  ISSN 2405-8963, https://doi.org/10.1016/j.ifacol.2016.07.594.\n- [3] Schmid, N. A., and Limère, V. (2019). A classification of tactical assembly line feeding problems. International Journal of Production Research, 57(24), 7586–7609. https://doi.org/10.1080/00207543.2019.1581957\n- [4] Linn I. Sennott et al.,  \nOptimal Dynamic Assignment of a Flexible Worker on an Open Production Line with\nSpecialists, European Journal of Operational Research, Volume 170, Issue 2, 2006, Pages 541-566, ISSN 0377-2217, https://doi.org/10.1016/j.ejor.2004.06.030.\n- [5] Chutima, P. and Chimrakhang, J. (2021), \"Multi-objective worker allocation optimisation in a multiple U-line system\", Assembly Automation, Vol. 41 No. 4, pp. 466-476. https://doi.org/10.1108/AA-12-2020-0198\n- [6] Ritt, M. et al., The assembly line worker assignment and balancing problem with stochastic worker availability. International Journal of Production Research, 54(3), 907–922. https://doi.org/10.1080/00207543.2015.1108534", "claims_and_evidence": "The primary contribution of this submission is a Python framework for simulating production lines with RL agents, and this is supported by the code provided in the supplementary material.", "methods_and_evaluation_criteria": "This paper does not introduce a new method but demonstrates various scenarios implemented using LineFlow.", "theoretical_claims": "Table 1 presents a problematic result where the optimal reward for the WT case is lower than the reward achieved by RL algorithms, despite the provided optimality proof.", "experimental_designs_or_analyses": "Although the authors design diverse case studies, this paper aims to introduce a new framework for simulating production lines. Therefore, it should include real-world case studies built using this framework.", "relation_to_broader_scientific_literature": "The environment in the proposed framework is well-designed, drawing from research on active control of production lines.", "essential_references_not_discussed": "I believe the optimality proof in Appendix C should be supported with relevant references.", "other_strengths_and_weaknesses": "Despite the importance of active line control across various industries, no well-grounded simulation framework has been available for training RL agents in production line settings. Thus, this new framework has significant impact. However, since the experiments are limited to a few case studies, I am concerned about its generalizability to real-world applications.", "other_comments_or_suggestions": "Can the authors use this framework to simulate real-world applications?", "questions_for_authors": "With the growing complexity of production lines involving numerous entities, a multi-agent approach is sometimes necessary. Is this framework extensible to multi-agent RL?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper has received an overall mixed set of reviews, with two positive recommendations and two more critical assessments. After carefully considering the reviews and the authors' rebuttal, I believe most of the major concerns have been adequately addressed.

Reviewer Hikp raised questions regarding some unclear elements, which seem to have been largely addressed.
Reviewer R3X2's main critique was that the paper "applies standard optimization principles to specific production settings", which is a fair point when considering the novelty threshold for publication. However, the reviewer also acknowledged several strengths in the work (well-written, comprehensive framework, appropriate methodology). 

In summary, the quality of the paper is generally appreciated by the reviewers and while not highly novel, this paper brings a useful framework for research in the field. I recommend a weak accept.
Decision: Accept (poster)