# Volumetric Techniques for Product Routing and Loading Optimisation in Industry 4.0: A Review
## 4 Volumetrics
### Deep Reinforcement Learning  
Several studies have ventured into the application of Deep Reinforcement Learning (DRL) in the context of the 3D-BPP, aiming to efficiently pack objects with varying shapes and orientations into a confined space. These studies propose unique contributions and insights into the world of DRL and its applicability within this domain. While most studies do not explicitly mention Industry 4.0, they indirectly align with its principles, showcasing how advanced technologies, including Machine Learning (ML) and robotics, can enhance logistics processes. A study published by [PERSON] et al. [42] employs DRL to optimise the bin-packing process. The approach involves the creation of an intelligent agent that learns to make optimal decisions when packing items into a three-dimensional container. A distinctive feature of the authors methodology is the integration of heuristics derived from human experiences and physical rules with DRL, resulting in an improved packing policy. In a similar vein, [PERSON] et al., 2021 [36] also delve into 3D bin packing using DRL. However, their particular focus lies on the online variant of this problem. Building upon this foundation, [PERSON] et al., 2022 [43] further extend the application of DRL to address the 3D-BPP, emphasising the online version. Their approach introduces novel elements such as a stacking tree structure, a decoupled packing policy learning framework, and the incorporation of a reward function. In a recent study conducted by [PERSON] and [PERSON] [44], DRL was introduced to address the 3D-BPP, with an emphasis on efficiently packing objects with diverse shapes. Their approach entails the amalgamation of DRL with the Soft Actor-Critic (SAC) algorithm and the utilisation of a task simulator to facilitate efficient training. Moreover, they incorporate dynamic adjustments to task difficulty and devise a reward function, both of which contribute to enhancing the learning process. Notably, the trained model exhibits a 70% success rate for moderately challenging tasks. In a parallel endeavour, [PERSON] et al. [45] opt for the Advantage Actor-Critic (A2C) algorithm to address the 3D-BPP within a robotic environment. Their approach emphasises maximising space utilisation while respecting constraints. Furthermore, in the research conducted by [PERSON] and [PERSON] [34], DRL is used to address the 3D-BPP within the context of port logistics. In their study, the authors model the problem as a Markov Decision Process (MDP) and also employ the A2C algorithm, albeit supplemented with Generalised Advantage Estimation (GAE). Their research is centred on the efficient use of space while adhering to specific constraints associated with port logistics. The methodologies presented by [PERSON] et al. and [PERSON] et al. [35; 46] align with the fundamental tenets of Industry 4.0, emphasising the advancement of algorithms and automation to enhance the efficiency of manufacturing and logistics operations. By applying DRL to the Online 3D-BPP, [PERSON] et al. [35] aim to leverage an improved Actor-Critic algorithm and combine DRL with the Monte Carlo Tree Search (MCTS) algorithm. In addition, they introduce a packing configuration tree model to address the challenge of action space discretisation. Similarly, [PERSON] et al. [46] tackle the 3D-BPP within the context of e-commerce warehousing. Their approach is based on the deployment of a Deep Q-Network (DQN) algorithm within the domain of DRL, optimising various aspects, and encompassing the selection of box types, the sequence of loading, and the orientation of the boxes. In two recent studies [11; 47], researchers direct their focus towards tackling the 3D Container Loading Problem (3D-CLP) with DRL. Whereas [PERSON] and [PERSON] [47] propose the application of DRL to a specific instance of the 3D-BPP using the Proximal Policy Optimisation (PPO) algorithm, and conduct experiments involving hyper-parameter adjustments, [PERSON] et al.