SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \mathcal {T} _ {a} V _ {1} - \mathcal {T} _ {a} V _ {2} \right\| _ {\infty} \\ = \max  _ {s} | \mathbb {E} _ {a \sim \pi} [ r (s, a) + \gamma \sup  _ {P _ {s, a} ^ {m} \in \mathcal {P} _ {s, a} ^ {m}} \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m}} [ V _ {1} (s ^ {\prime}) ] - r (s, a) - \gamma \sup  _ {P _ {s, a} ^ {m} \in \mathcal {P} _ {s, a} ^ {m}} \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m}} [ V _ {2} (s ^ {\prime}) ] ] | \\ = \gamma \max  _ {s} | \mathbb {E} _ {a \sim \pi} [ \sup  _ {P _ {s, a} ^ {m} \in \mathcal {P} _ {s, a} ^ {m}} \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m}} [ V _ {1} (s ^ {\prime}) ] - \sup  _ {P _ {s, a} ^ {m} \in \mathcal {P} _ {s, a} ^ {m}} \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m}} [ V _ {2} (s ^ {\prime}) ] ] | \\ \leq \gamma \max  _ {s} \mathbb {E} _ {a \sim \pi} | \sup  _ {P _ {s, a} ^ {m} \in \mathcal {P} _ {s, a} ^ {m}} \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m}} \left[ V _ {1} (s ^ {\prime}) \right] - \sup  _ {P _ {s, a} ^ {m} \in \mathcal {P} _ {s, a} ^ {m}} \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m}} \left[ V _ {2} (s ^ {\prime}) \right] | \\ = \gamma \max _ {s} \mathbb {E} _ {a \sim \pi} | \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m, 1}} [ V _ {1} (s ^ {\prime}) ] - \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m, 2}} [ V _ {2} (s ^ {\prime}) ] | \\ \leq \gamma \max _ {s} \mathbb {E} _ {a \sim \pi} | \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m, 1}} [ V _ {1} (s ^ {\prime}) - V _ {2} (s ^ {\prime}) ] | \\ \leq \gamma \max  _ {s} \mathbb {E} _ {a \sim \pi} \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {m, 1}} \left| V _ {1} \left(s ^ {\prime}\right) - V _ {2} \left(s ^ {\prime}\right) \right| \\ \leq \gamma \max  _ {s} \mathbb {E} _ {a \sim \pi} \max  _ {s} | V _ {1} (s ^ {\prime}) - V _ {2} (s ^ {\prime}) | \\ = \gamma \max  _ {s} \mathbb {E} _ {a \sim \pi} | | V _ {1} - V _ {2} | | _ {\infty} \\ = \gamma \| V _ {1} - V _ {2} \| _ {\infty}, \tag {14} \\ \end{array}
$$
where  $P_{s,a}^{m,1} = \arg \sup_{P_{s,a}^m}\mathbb{E}_{s\sim P_{s,a}^m}V_1(s)$  and  $P_{s,a}^{m,2} = \arg \sup_{P_{s,a}^m}\mathbb{E}_{s\sim P_{s,a}^m}V_2(s)$ .
Let us assume the existence of two different fixed points  $V_{1}^{*}$  and  $V_{2}^{*}$  for the adjustable Bellman operator. As both of these are fixed points, the following equality holds:
$$
\left| \left| V _ {1} ^ {*} - V _ {2} ^ {*} \right| \right| _ {\infty} = \left| \left| \mathcal {T} _ {a} V _ {1} ^ {*} - \mathcal {T} _ {a} V _ {2} ^ {*} \right| \right| _ {\infty} \tag {15}
$$
According to the proof in 14, the following inequality holds:
$$
\left| \left| \mathcal {T} _ {a} V _ {1} ^ {*} - \mathcal {T} _ {a} V _ {2} ^ {*} \right| \right| _ {\infty} \leq \gamma \left| \left| V _ {1} ^ {*} - V _ {2} ^ {*} \right| \right| _ {\infty} \tag {16}
$$
Thus, for both Equation 15 and Equation 16 to strictly hold true simultaneously, it must be the case that  $V_{1}^{*}$  is equal to  $V_{2}^{*}$ .
Definition 1. Let  $f:(0,\infty)\to \mathbb{R}$  be a convex function with  $f(1) = 1$ . Let  $P$  and  $Q$  be two probability distributions on a measurable space. Then, the  $f$ -divergence is defined as
$$
D _ {f} (P | | Q) = \mathbb {E} _ {Q} [ f (\frac {\mathrm {d} Q}{\mathrm {d} P}) ], \tag {17}
$$
where  $\frac{\mathrm{d}P}{\mathrm{d}Q}$  is a Radon-Nikodym derivative.
Proposition 1. The policy evaluation of AR2L can be formulated as a constrained optimization problem. The objective is to maximize the value function over the uncertainty set  $\mathcal{P}^m$ , subject to the constraint defined in  $\mathcal{P}^m$ . By representing the TV distance using the  $f$ -divergence (Shapiro, 2017), the adjustable robust Bellman operator  $\mathcal{T}_a$  can be equivalently written as
$$
\begin{array}{l} \mathcal {T} _ {a} V _ {a} ^ {\pi} (s) = \mathbb {E} _ {a \sim \pi} [ r (s, a) + \frac {\gamma}{1 + \alpha} \inf  _ {\lambda , \mu_ {1}, \mu_ {2}, \mu} \left(\mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {o}} \left[ V _ {a} ^ {\pi} \left(s ^ {\prime}\right) - \mu_ {1} \left(s ^ {\prime}\right) \right] _ {+} + \right. \tag {18} \\ \alpha \mathbb {E} _ {s ^ {\prime} \sim P _ {s, a} ^ {w}} \left[ V _ {a} ^ {\pi} (s ^ {\prime}) - \mu_ {2} (s ^ {\prime}) \right] _ {+} + \mu + \lambda \rho (1 + \alpha)) ], \\ \end{array}
$$
Proof. The  $f$ -divergence can be utilized to express the constraint defined in the uncertainty set  $\mathcal{P}^m$  as follows:
$$
\begin{array}{l} D _ {T V} \left(P _ {s, a} \left(s ^ {\prime}\right) \mid \mid P _ {s, a} ^ {o} \left(s ^ {\prime}\right)\right) + \alpha D _ {T V} \left(P _ {s, a} \left(s ^ {\prime}\right) \mid \mid P _ {s, a} ^ {w} \left(s ^ {\prime}\right)\right) \\ = \frac {1}{2} \int_ {\mathcal {S}} | \mathrm {d} P _ {s, a} \left(s ^ {\prime}\right) - \mathrm {d} P _ {s, a} ^ {o} \left(s ^ {\prime}\right) | + \frac {\alpha}{2} \int_ {\mathcal {S}} | \mathrm {d} P _ {s, a} \left(s ^ {\prime}\right) - \mathrm {d} P _ {s, a} ^ {w} \left(s ^ {\prime}\right) | \\ = \frac {1}{2} \int_ {\mathcal {S}} \left| \frac {\mathrm {d} P _ {s , a} \left(s ^ {\prime}\right)}{\mathrm {d} P _ {s , a} ^ {o} \left(s ^ {\prime}\right)} - 1 \right| \mathrm {d} P _ {s, a} ^ {o} + \frac {\alpha}{2} \int_ {\mathcal {S}} \left| \frac {\mathrm {d} P _ {s , a} \left(s ^ {\prime}\right)}{\mathrm {d} P _ {s , a} ^ {w} \left(s ^ {\prime}\right)} - 1 \right| \mathrm {d} P _ {s, a} ^ {w} \tag {19} \\ = \frac {1}{2} \int_ {\mathcal {S}} | \zeta^ {o} (s ^ {\prime}) - 1 | \mathrm {d} P _ {s, a} ^ {o} + \frac {\alpha}{2} \int_ {\mathcal {S}} | \zeta^ {w} (s ^ {\prime}) - 1 | \mathrm {d} P _ {s, a} ^ {w} \leq \rho^ {\prime}, \\ \end{array}
$$
where  $\zeta : \mathcal{S} \to \mathbb{R}_+$  is the measurable function.  $\zeta^o(s) = \frac{\mathrm{d}P(s)}{\mathrm{d}P^o(s)}$  measures the difference between two dynamics at each sample  $s \in \mathcal{S}$ , similarly for  $\zeta^w$ .
In the policy evaluation step of the adjustable robust Bellman operator, the goal is to obtain a mixture dynamics  $P_{s,a}^{m}$  that maximizes the adjustable value function while satisfying the constraint defined in Equation 19. We hereby formulate this procedure as a constrained optimization problem, written as:
$$
\begin{array}{l} \max  _ {\zeta^ {o}, \zeta^ {w}} \int_ {\mathcal {S}} V (s) \zeta^ {o} (s) d P ^ {o} (s) + \alpha \int_ {\mathcal {S}} V (s) \zeta^ {w} (s) d P ^ {w} (s) \\ \text {s . t .} \int_ {\mathcal {S}} \phi \left(\zeta^ {o} (s)\right) \mathrm {d} P ^ {o} + \int_ {\mathcal {S}} \phi \left(\zeta^ {w} (s)\right) \mathrm {d} P ^ {w} \leq (1 + \alpha) \rho \tag {20} \\ \int_ {\mathcal {S}} \zeta^ {o} (s) \mathrm {d} P ^ {o} (s) = 1; \quad \alpha \int_ {\mathcal {S}} \zeta^ {w} (s) \mathrm {d} P ^ {w} (s) = \alpha \\ \zeta^ {o} (s) \mathrm {d} P ^ {o} (s) - \alpha \frac {1}{\alpha} \zeta^ {w} (s) \mathrm {d} P ^ {w} (s) = 0, \forall s \in \mathcal {S}, \\ \end{array}
$$
where the value function  $V: \mathcal{S} \to \mathbb{R}$  is also a measurable function. We use  $P^o$  and  $P^w$  to denote  $P_{s,a}^o$  and  $P_{s,a}^w$  for brevity,  $\rho = \frac{\rho'}{1 + \alpha}$  is the normalized radius constant, and  $\phi(x(s)) = \frac{1}{2} |x(s) - 1|$  is
a convex and lower semicontinuous function. We thus can use the Lagrangian multiplier method to solve the optimization problem in 20. The Lagrangian function is written as:
$$
\begin{array}{l} L \left(\zeta^ {o}, \zeta^ {w}, \lambda , \mu_ {1}, \mu_ {2}, \mu_ {3} (s)\right) = \int_ {\mathcal {S}} \left[ \left(V (s) - \mu_ {1} - \mu_ {3} (s)\right) \zeta^ {o} - \lambda \phi \left(\zeta^ {o}\right) \right] d P ^ {o} \tag {21} \\ + \alpha \int_ {\mathcal {S}} \left[ (V (s) - \mu_ {2} + \frac {1}{2} \mu_ {3} (s)) \zeta^ {w} - \lambda \phi (\zeta^ {w}) \right] \mathrm {d} P ^ {w} + \lambda \rho (1 + \alpha) + \mu_ {1} + \mu_ {2} \alpha \\ \end{array}
$$
The Lagrangian dual of problem 20 is
$$
\inf  _ {\lambda \geq 0, \mu_ {1}, \mu_ {2}, \mu_ {3} (s)} \sup  _ {\zeta^ {o}, \zeta^ {w}} L \left(\zeta^ {o}, \zeta^ {w}, \lambda , \mu_ {1}, \mu_ {2}, \mu_ {3} (s)\right) \tag {22}
$$
Since the measurable function space  $L(\mathcal{S},\Sigma (\mathcal{S}),P)$  is decomposable, the supremum in 22 can be taken inside the integral (Shapiro, 2017), that is
$$
\begin{array}{l} \sup _ {\zeta^ {o}, \zeta^ {w}} L \left(\zeta^ {o}, \zeta^ {w}, \lambda , \mu_ {1}, \mu_ {2}, \mu_ {3} (s)\right) = \int_ {\mathcal {S}} \sup _ {\zeta^ {o}} [ (V (s) - \mu_ {1} - \mu_ {3} (s)) \zeta^ {o} - \lambda \phi (\zeta^ {o}) ] d P ^ {o} \\ + \alpha \int_ {\mathcal {S}} \sup  _ {\zeta^ {w}} [ (V (s) - \mu_ {2} + \frac {1}{2} \mu_ {3} (s)) \zeta^ {w} - \lambda \phi (\zeta^ {w}) ] d P ^ {w} + \lambda \rho (1 + \alpha) + \mu_ {1} + \mu_ {2} \alpha \tag {23} \\ \end{array}
$$
We can observe that the two terms inside the integral are identical to the definition of the conjugate function  $\phi^{*}(y) = \sup_{x\in \mathbb{R}_{+}}\{xy - \phi (x)\}$ . And  $\phi (x)$ , as a convex, semicontinuous function, is defined as  $\frac{1}{2} |x(s) - 1|$ , resulting in  $\phi^{*}(y) = \max \{y, - \frac{1}{2}\}$ . The dual problem can be transformed as:
$$
\begin{array}{l} \inf  _ {\lambda \geq 0, \mu_ {1}, \mu_ {2}, \mu_ {3} (s)} L (\lambda , \mu_ {1}, \mu_ {2}, \mu_ {3} (s)) = \inf  _ {\lambda \geq 0, \mu_ {1}, \mu_ {2}, \mu_ {3} (s)} \int_ {\mathcal {S}} \max  \{V (s) - \mu_ {1} - \mu_ {3} (s), - \frac {1}{2} \lambda \} d P ^ {o} \\ + \alpha \int_ {\mathcal {S}} \max  \left\{V (s) - \mu_ {2} + \frac {1}{\alpha} \mu_ {3} (s), - \frac {1}{2} \lambda \right\} \mathrm {d} P ^ {w} + \lambda \rho (1 + \alpha) + \mu_ {1} + \mu_ {2} \alpha \tag {24} \\ \end{array}
$$
We use  $[x]_{+}$  to represent  $\max \{x,0\}$  for brevity. It is then transformed as
$$
\begin{array}{l} \inf  _ {\lambda \geq 0, \mu_ {1}, \mu_ {2}, \mu_ {3} (s)} L (\lambda , \mu_ {1}, \mu_ {2}, \mu_ {3} (s)) = \inf  _ {\lambda \geq 0, \mu_ {1}, \mu_ {2}, \mu_ {3} (s)} \int_ {\mathcal {S}} [ V (s) - \mu_ {1} - \mu_ {3} (s) + \frac {1}{2} \lambda ] _ {+} \mathrm {d} P ^ {o} \\ + \alpha \int_ {\mathcal {S}} [ V (s) - \mu_ {2} + \frac {1}{\alpha} \mu_ {3} (s) + \frac {1}{2} \lambda ] _ {+} \mathrm {d} P ^ {w} + \lambda \rho (1 + \alpha) + \mu_ {1} - \frac {\lambda}{2} + (\mu_ {2} - \frac {\lambda}{2}) \alpha \tag {25} \\ \end{array}
$$
Next, we set  $\mu_1'(s) = \mu_1 + \mu_3(s) - \frac{1}{2}\lambda$  and  $\mu_2'(s) = \mu_2 - \frac{1}{\alpha}\mu_3(s) - \frac{1}{2}\lambda$ . And we observe that  $\forall s \in S, \mu_1'(s) + \alpha \mu_2'(s) = \mu_1 - \frac{\lambda}{2} + (\mu_2 - \frac{\lambda}{2})\alpha$ . We thus set  $\mu = \mu_1'(s) + \alpha \mu_2'(s)$ .
$$
\begin{array}{l} \inf  _ {\lambda \geq 0, \mu_ {1}, \mu_ {2}, \mu_ {3} (s)} L (\lambda , \mu_ {1}, \mu_ {2}, \mu_ {3} (s)) \\ = \inf  _ {\lambda \geq 0, \mu_ {1}, \mu_ {2}, \mu_ {3} (s)} \int_ {S} [ V (s) - \mu_ {1} ^ {\prime} (s) ] _ {+} \mathrm {d} P ^ {o} + \alpha \int_ {S} [ V (s) - \mu_ {2} ^ {\prime} (s) ] _ {+} \mathrm {d} P ^ {w} + \lambda \rho (1 + \alpha) + \mu \tag {26} \\ \end{array}
$$
In addition, based on the analysis of the conjugate function  $\phi^{*}(y)$  in (Panaganti et al., 2022), we can derive the following two inequalities:
$$
\frac {V (s) - \mu_ {1} - \mu_ {3} (s)}{\lambda} \leq \frac {1}{2}; \quad \frac {V (s) - \mu_ {2} + \frac {1}{2} \mu_ {3} (s)}{\lambda} \leq \frac {1}{2} \tag {27}
$$
We thus can obtain  $\lambda = \max_s\{V_a^\pi (s) - \mu_1(s),V_a^\pi (s) - \mu_2(s),0\}$ . Building on the proofs above, we can define the adjustable robust Bellman operator as shown in Equation 18.
A.3 Pseudocode of Algorithms
The pseudocodes for both the exact AR2L algorithm and the approximate AR2L algorithm are presented in Algorithm 1 and Algorithm 2, respectively. Note that in practice, the Kullback-Leibler (KL) divergence is adopted to constrain the deviation between distributions in the exact AR2L algorithm.
Algorithm 1 Exact AR2L Algorithm
Initialize: packing policy  $\pi_{\mathrm{pack}}$  , permutation-based attacker  $\pi_{\mathrm{perm}}$  , mixture-dynamics model  $\pi_{\mathrm{mix}}$  and their corresponding value functions  $V^{\pi_{\mathrm{pack}}}$ $V^{\pi_{\mathrm{perm}}}$ $V^{\pi_{\mathrm{mix}}}$    
Input: stationary item distribution  $p_b$    
Parameter: robustness weight  $\alpha$  , number of observable items  $N_B$    
Output: packing policy  $\pi_{\mathrm{pack}}$  .   
1: for  $i = 0$  to max_iteration do   
2: # train the permutation-based attacker   
3: Reset the environment and observe  $\mathbf{C}_0,\mathbf{B}_0\sim p_b$    
4: for  $t = 0$  to max_step do   
5: Get permuted item sequence:  $\mathbf{B}_t^\prime = \pi_{\mathrm{perm}}(\mathbf{C}_t,\mathbf{B}_t)$    
6: Get location for  $b_{t,1}^{\prime}:l_{t} = \pi_{\mathrm{pack}}(\mathbf{C}_{t},\mathbf{B}_{t}^{\prime})$    
7: Pack  $b_{t,1}^{\prime}$  into the bin, and observe  $\mathbf{C}_{t + 1},\mathbf{B}_{t + 1}\sim p_b$  , reward  $r_t^{\mathrm{pack}}$  , termination  $d_t$  .   
8: if  $d_{t} = =$  True then   
9: Update  $\pi_{\mathrm{perm}}$  on episode samples to maximize.  $-\sum r_t^{\mathrm{pack}}$  .   
10: end if   
11: end for   
12: # train the mixture-dynamics model and the packing policy   
13: Reset the environment and observe  $\mathbf{C}_0,\mathbf{B}_0\sim p_b$  .   
14: for  $t = 0$  to max_step do   
15: Get permuted item sequence:  $\mathbf{B}_t'' = \pi_{\mathrm{mix}}(\mathbf{C}_t,\mathbf{B}_t)$    
16: Get location for  $b_{t,1}^{\prime}:l_{t} = \pi_{\mathrm{pack}}(\mathbf{C}_{t},\mathbf{B}_{t}^{\prime})$    
17: Pack  $b_{t,1}^{\prime}$  into the bin, and observe  $\mathbf{C}_{t + 1},\mathbf{B}_{t + 1}\sim p_b$  , reward  $r_t^{\mathrm{pack}}$  , termination  $d_t$  .   
18: if  $d_t = =$  True then   
19: Update  $\pi_{\mathrm{mix}},V^{\pi_{\mathrm{mix}}}$  on episode samples to maximize  $\sum r_t^{\mathrm{pack}} - (D_KL(\pi_{\mathrm{mix}}||\mathbb{1}_{\{x = b_{t + 1},1\}}) + \alpha D_KL(\pi_{\mathrm{mix}}||\pi_{\mathrm{perm}}))$  .   
20: Update  $\pi_{\mathrm{pack}},V^{\pi_{\mathrm{pack}}}$  on episode samples to maximize  $\sum r_t^{\mathrm{pack}}$  .   
21: end if   
22: end for   
23: end for
A.4 Implementation Details
To implement a packing policy that can work in both the discretized grid world and the continuous solution space, we use the packing configuration tree (PCT)(Zhao et al., 2022a) to represent the state, and employ a transformer(Vaswani et al., 2017) network as the packing policy. PCT is utilized to represent the bin configuration and available empty spaces for the most recently packed item, while the versatile transformer network is capable of handling the variable number of nodes in PCT. We adopt the same network configuration as reported in PCT for the packing policy, with the addition of positional encoding to encode the position information of each item in the observed item sequence.
Figure 9: In the  $xoy$  plane, circles represent the potential positions generated by the IPs heuristics.
In addition, those available spaces are generated using a heuristic method that takes into account the packing constraints and preferences for the item to be packed. Specifically, in the discrete setting, we utilize the empty maximal spaces (EMSs) (Ha et al., 2017) heuristics to generate potential positions, and subsequently apply the same
constraints used in PCT to check the stability of these positions, thereby identifying the feasible positions  $\mathbf{L}$ . In the continuous setting, we propose a new heuristic methods called Intersection Points (IPs) to generate potential positions. Given the 2D packed items in the  $xoy$  plane, the intersection points are generated from the intersection of the sides of cuboid-shaped items and the bin, as shown in Figure 9. To extend this method from 2D to 3D cases, we allow candidate positions to be obtained from the intersection points of the sides in the top view.
Algorithm 2 Approximate AR2L Algorithm
Initialize: packing policy  $\pi_{\mathrm{pack}}$  , permutation-based attacker  $\pi_{\mathrm{perm}}$  , and their corresponding value   
functions  $V_{a}^{\pi_{\mathrm{pack}}}$ $V^{\pi_{\mathrm{perm}}}$  .   
Input: stationary item distribution  $p_b$    
Parameter: robustness weight  $\alpha$  , number of observable items  $N_B$    
Output: packing policy  $\pi_{\mathrm{pack}}$  .   
1: for  $i = 0$  to max_iteration do   
2: # train the permutation-based attacker   
3: Reset the environment and observe  $\mathbf{C}_0,\mathbf{B}_0\sim p_b$    
4: for  $t = 0$  to max_step do   
5: Get permuted item sequence:  $\mathbf{B}_t^\prime = \pi_{\mathrm{perm}}(\mathbf{C}_t,\mathbf{B}_t)$    
6: Get location for  $b_{t,1}^{\prime}:l_{t} = \pi_{\mathrm{pack}}(\mathbf{C}_{t},\mathbf{B}_{t}^{\prime})$  .   
7: Pack  $b_{t,1}^{\prime}$  into the bin, and observe  $\mathbf{C}_{t + 1},\mathbf{B}_{t + 1}\sim p_b$  , reward  $r_t^{\mathrm{pack}}$  termination  $d_t$  .   
8: if  $d_{t} = =$  True then   
9: Update  $\pi_{\mathrm{perm}}$  on episode samples to maximize.  $-\sum r_t^{\mathrm{pack}}$  .   
10: end if   
11: end for   
12: # train the packing policy   
13: Reset the environment and observe  $\mathbf{C}_0,\mathbf{B}_0\sim p_b$  .   
14: for  $t = 0$  to max_step do   
15: Get location for  $b_{t,1}:l_t = \pi_{\mathrm{pack}}(\mathbf{C}_t,\mathbf{B}_t)$  .   
16: Pack  $b_{t,1}$  into the bin, and observe  $\mathbf{C}_{t + 1},\mathbf{B}_{t + 1}\sim p_b$  , reward  $r_t^{\mathrm{pack}}$  termination  $d_t$  .   
17: if  $d_t = =$  True then   
18: Permute  $\mathbf{B}_t,t\in [0,T]$  to  $\mathbf{B}_t^\prime ,t\in [0,T]$  using  $\pi_{\mathrm{perm}}$  .   
19: Use 18 to compute target values for adjustable Bellman values on   
 $\{{\bf C}_t,{\bf B}_t,{\bf B}_t^\prime ,l_t,r_t^{\mathrm{pack}},d_t\} ,t\in [0,T]$  .   
20: Update  $\pi_{\mathrm{pack}},V^{\pi_{\mathrm{pack}}}~$  on episode samples to maximize  $\sum r_t^{\mathrm{pack}}$  .   
21: end if   
22: end for   
23: end for
In our implementations of both the permutation-based attacker and the mixture-dynamics model, we utilize the transformer (Vaswani et al., 2017) network to model these policies. This choice is motivated by the variable number of observed boxes in different real packing systems, as well as the dynamic changes in the number of packed items during each episode. The transformer is capable of capturing long-range dependencies in spatial data and sequential data. Specifically, we first use two independent element-wise fully-connected (FC) layers to project heterogenous elements of the state into homogenous features. Then, the scaled dot-product attention is relied to calculate the weighted sum of features based on the relevance between given input features. Next, we use the skip-connection operation and the element-wise feed-forward FC layer to obtain the final features  $\pmb{x} = \{\pmb{x}_C,\pmb{x}_B\} \in \mathbb{R}^{d_x\times N}$ , where  $\pmb{x}_C\in \mathbb{R}^{d_x\times N_C}$  and  $\pmb{x}_B\in \mathbb{R}^{d_x\times N_B}$  denote features of  $\mathbf{C}_t$  and  $\mathbf{B}_t$  respectively,  $d_{x}$  is the feature size, and  $N = N_{C} + N_{B}$  denotes the variable feature number. Finally, the probability distribution over  $\mathbf{B}_t$  is described as
$$
\pi_ {\text {p e r m}} \left(\mathbf {B} _ {t} \mid \mathbf {C} _ {t}, \mathbf {B} _ {t}\right) = \operatorname {s o f t m a x} \left(c _ {\text {t e m p}} \cdot \tanh  (y)\right), \quad y = \frac {\bar {x} ^ {T} \boldsymbol {x} _ {B}}{\sqrt {d _ {x}}} \in \mathbb {R} ^ {N _ {B}}, \tag {28}
$$
where  $\bar{x} = \frac{1}{N}\sum_{i=1}^{N}x_{i}$  and  $c_{\mathrm{temp}}$  is the temperature constant.
As the number of packed items  $\mathbf{C}$  and the number of available feasible positions  $\mathbf{L}$  both vary at different time steps and have irregular shapes, we combine this data into one batch by filling  $\mathbf{C}$  and  $\mathbf{L}$  to fixed lengths of 80 and 120 in the discrete setting, and 100 and 120 in the continuous setting, respectively, using dummy elements. Each valid element in  $\mathbf{C}$  is a vector that specifies the size and position of an item that has already been packed. Each vector in  $\mathbf{B}$  represents the sizes of an item that has yet to be packed. Each vector in  $\mathbf{L}$  contains the position of a feasible empty space that can accommodate a new item. To permute the observed item sequence, we feed both  $\mathbf{C}$  and  $\mathbf{B}$  to a transformer network that serves as the attacker. In this transformer network, the sizes of the two element-wise fully connected (FC) layers are fixed at 64. Additionally, there is one attention
layer with one head attention, and the sizes of the query, key, and value vectors are set to 64. The size of the element-wise feed-forward FC layer is also set to 64. In our experiment, we utilize the PPO (Schulman et al., 2017) algorithm to effectively train the packing policy, permutation-based attacker, and mixture-dynamics model. PPO employs multiple parallel processes (64 in this case) to interact with their respective environments and collect data. The rollout length for each iteration is set to 30, and the learning rate is set to 0.0003. To ensure a fair comparison, we keep the same hyperparameter settings mentioned above for each of the methods that we have reproduced.
A.5 Related Work
Heuristic Methods. In heuristic methods proposed for solving 3D-BPP, the score function is often designed to rank item placements based on expert knowledge, which can represent the packing requirements and preferences to some extent. Ha et al. (2017) relied on the deep-bottom-left (DBL) (Karabulut and Inceoglu, 2005) rule to sort empty maximal spaces (EMSs) (Parreño et al., 2008) for the current item. The best-match-first strategy (Li and Zhang, 2015) gives priority to feasible placements that have the smallest margin for the observed item. Hu et al. (2017) introduced the least surface area heuristic, which selects the maximal empty space and orientation that result in the smallest surface area. The Heightmap-Minimization method (Wang and Hauser, 2019) favors items placement that result in the smallest occupied volume as observed from the loading direction. The maximize-accessible-convex-space method, as described by Hu et al. (2020), aims to optimize the empty space available for future, potentially large items. Despite their simplicity and effectiveness, these methods find difficulties in handling complex packing preferences and adapting to diverse scenarios for solving online 3D-BPP. This is because their packing strategies rely heavily on expert experience and may not be flexible enough to accommodate different situations.
DRL-based Methods. To further develop highly effective policies, DRL-based methods are broadly proposed by formulating the online 3D-BPP as a sequential decision-making problem. In this formulation, the state usually includes the bin configuration, which contains information about both the container and the packed items, as well as the observed item on the conveyor. To learn a DRL-based policy, the container is discretized to represent the bin configuration, and a convolutional neural network (CNN) is served as a policy (Hu et al., 2017; Verma et al., 2020; Zhao et al., 2021, 2022b; Yang et al., 2021; Song et al., 2023). However, these learning-based methods only work in a grid world with limited spatial discretization accuracy, which reduces their practical applicability. To deploy DRL-based methods on solving online 3D-BPP with continuous solution space, Zhao et al. (2022a) utilized the Packing Configuration Tree (PCT) to represent the bin configuration and potential available spaces. They then employed a graph attention network (Veličković et al., 2017) to encode the PCT and observed boxes, to encode the PCT and observed boxes, and to select a location from available spaces generated for the most preceding item. However, these methods often focus on optimizing average performance and do not account for worst-case scenarios.