Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a novel deep-RL based method for the online 3D bin packing problem. Here, cuboidal items arrive in random order and need to be packed as tightly as possible. The key idea of this paper is to enable to robot to "backtrack" suboptimal packing actions by temporarily unpacking part of the stack into a buffer region, and thus build more optimal packing configurations. The method is evaluated on extensive simulation experiments and a real world demonstrator.

strengths_and_weaknesses: ## Strengths

The paper is extremely well-written and easy to understand, despite the technical intricacies of the work. The method is technically sound, offering a great mix of explicit modelling and learning. The experiments are thoughtfully conceived, carried out and analyzed. The method demonstrates performance improvements over the state of the art and purely heuristic methods, mainly owed to the ability to perform backtracking through unpacking. Overall the paper was a very enjoyable and insightful read.

## Weaknesses

I have two concerns regarding the evaluation of the work.

### Efficiency in terms of number of actions

The paper fully focuses on the cube utilization, i.e. reducing wasted space. While this is obviously a key metric, in industrial applications efficiency and units per hour (UPH) are equally important factors -- and obviously there is a trade-off between cube utilization and efficiency. I'm missing an analysis on how many unpacking actions are required to reach the improved cube utilization. This should be addressed by introducing an additional metric such as the average number of packing+unpacking actions. 

### Unfair baseline comparison?

Directly related to the previous question is whether the baselines are evaluated on a fair basis. If I understand correctly, no baseline (except for PUN-random-unpacking) actually does unpacking. If I interpret Figure 7 correctly, PUN-packing-only (Uti. = ~0.67) significantly underperforms the best performing baseline Zhao et al. (Uti. = 0.834). This immediately raises the question how much better Zhao et al. would perform if it was extended to use unpacking. For example, how well would Zhao et al. perform with random unpacking? 

### Minor points

- Heuristics (140-148). I would encourage the authors to make the paper self-sufficient and explain the heuristics, at least 1/2 sentence per heuristic, rather than just naming them.
- Figure 2. I was initially confused what "packing mask" and "unpacking mask" meant - I would suggest to explicitly mention "heuristics" in the figure to ensure to make clear that the Heuristics section 140-148 will explain it.
- L154: Typo, -e-8 is not a mathematical expression, better use $10^{-8}$
- L161: What does the t in the sum refer to? Is this a typo?
- Sim-to-real strategy. I don't fully understand how individual items are identified in the real world demo, isn't some sort of instance segmentation both on the converyor as well as on the stack needed to identify individual boxes?
- Limitations. I would suggest to discuss how perception and action errors might affect 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper makes an important contribution to a highly relevant problem. The idea to introduce unpacking and learn an unpacking policy is compelling. However, the impact of unpacking needs to be analyzed more carefully, in particular on the efficiency of the approach (number of actions), and also on whether the baselines comparisons are fair.

issues: - Metrics/evaluation: Measure efficiency in terms of number of (additional unpacking) actions 
- Fair baseline comparison: Is PUN-packing-only significantly worse than Zhao et al? If yes, how much better is Zhao et al with some simple (e.g. random) or clever (?) unpacking strategy?
- Minor issues

(see Strengths and Weaknesses section for details)
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a solution to the 3D Bin Packing problem via the use of Deep Reinforcement Learning. Specifically, a 2 branch architecture is proposed to introduce the unpacking action and to learn how to apply optimally both packing and unpacking actions on boxes off of a conveyor belt in order to fill a bin while leaving as little space empty as possible.
strengths_and_weaknesses: Strengths: Cool method, very well presented, enjoyable to read. Evaluation was quite thorough. 

Weaknesses: While evaluation was thorough, would have liked to have seen simple Imitation Learning approaches discussed, cited and compared against (see below for more details).
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Recommendation is based on not having cited, discussed and compared against at least one simple Imitation Learning approach. Imitation Learning approaches where expert demonstrations have both packing and unpacking actions, should be able to do fairly well on the task which would at least make a strong argument in favor of the efficacy of the approach if it outperformed them. I am willing to change my recommendation if authors address this adequately during rebuttal period. 
issues: Ln. 88: discretize? Or define? 

Ln. 91-92: "The state of the current item n with the size of 91 ln × wn × hn is represented as..." That means that the state of each object never changes then since no object will ever change in width, height and length? If I'm misinterpreting the sentence, please rephrase to clarify.

Ln. 98: nit: tiny typo "defined".

Ln. 110: "a shared feature extraction module": how does this look like? 

Ln. 112: "Either the packing or the unpacking branch consists of...": Did authors mean to say "The packing and unpacking branches each consist of..."?

Ln. 118. "based on the heuristic rules": what are the heuristic rules? Assuming they are explained later in the paper, however please add such note here to point the reader to that fact, e.g. "based on heuristic rules that will be explained in section XXX".

Ln 132-133: "The agent will continue to predict the packing or unpacking action for the item n at the next time step until packing it into the bin" --> how does this avoid possible infinite loops? 

General comment: I'm assuming that based on the items and the capacity of the temporary buffer, there can be no solutions found. Does the method simply stop? Or does a best effort as far it's possible? Or?

Ln. 147: "that on the top". Typo.

Ln. 147-148: a) Is this the first rule for unpacking? If yes, please explain what it's aiming to do. b) Was the second rule omitted? Or mentioned later?

Equations 3 & 4: what is M_i?  

Ln. 251-254: Not sure what's the rationale for reward function 1. Success in packing or unpacking actions alone does not really give any indication about whether that was a good move. Please explain.  

Ln. 64-65: I assume that the no gains observed above 12 was due to the number of items not changing. It's not obvious why this would still hold if number of items on the conveyor belt was very large. Have you increased the number of items and still observed this? If yes, do you hypothesize on why?

General comment: In general, the idea of unpacking and "retrying" in assembly tasks is not new. There is a whole line of work by Zeng et. al on Assembly & Disassembly (Transporter Nets, Form2Fit etc.) that was not cited where authors using Imitation Learning techniques to solve assembly tasks. 

Also, have the authors attempted to solve this problem with Imitation Learning techniques like Behavioral Cloning? Not convinced that simple BC wouldn't have comparable results. Would like to have seen this at least as a baseline.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a deep RL based architecture for the 3D bin packing problem (3D BPP). The policy is learnt in simulation and is able to capture the synergy between picking and packing to achieve a target 3D packing configuration that minimizes the wasted space in the box. The policy takes in as input a heightmap and state representation specifying the item size and outputs pixel-locations for the pack/unpacking actions. The policy learns the synergy between pick and unpacking actions. Apart from outperforming prior learning and heuristic methods in simulation, the policy transfers to the real robot and even everyday objects (like cereal boxes).

strengths_and_weaknesses: Strengths:
The paper is well written. It also has good ablations and experimental comparisons with prior work. I’m not familiar with the SOTA of the field, but the task seems unique enough.
The approach is also demonstrated on the real robot with strong sim2real transfer results.

Weaknesses:
The data generation process could be explained better.
The approach only seems to work on limited object geometries and fixed-size box volumes. It would be great to clarify if the method can be expanded or if it's beyond the scope.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation:  See above.
issues:  See above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The author presented a 3D bin packing algorithm, where unpacking action can be used to correct wrong decisions. They propose a packing-and-unpacking network (PUN) based on Deep Reinforcement Learning (DRL). The architecture is composed of two networks, a first one for the packing action and the second one for the unpacking one. At execution, both network output the value function evaluated at the current state and the action with the max reward is chosen. The networks output also the packing and unpacking position, respectively. The proposed algorithm is tested against 10 benchmarks both in simulation and experiments.
strengths_and_weaknesses: -Strengths
1) The experiments clearly show the effectiveness of the proposed strategy both in simulation and experiments.
2) The idea of having an unpacking option is useful in practice.

-Weaknesses
1) It is a bit unclear how the two networks are trained.
2) The algorithm is not theoretically sound as it is based on several heuristics.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think that overall the Strengths are more than the Weaknesses. In particular, I think that the result section clearly shows the benefit of the method. My only recommendation is to better explain the training process for clarity, and I think that such change is in the reach of the authors.
issues: I think that the clarity of the paper can be improved. In the Method section, I would add a training subsection where the authors could explain better the training process. It seems that the value function associated with the packing policy is a function of how good the unpacking policy is, i.e., the problem cannot be decoupled. Are the authors considering some coupling effect in the training process? It seems that it is a good idea to decompose the problem into subtasks, but could the authors provide a discussion on the pro and cons?
review_title: null

Meta-review: metareview: This paper proposes a novel DRL based approach for the online 3D bin-packing problem. The main idea is inspired by how a human operator would have to use unpacking as an support action to backtrack sub-optimal packing actions. The synergies between packing and unpacking actions are optimized via DRL. The proposed method has several strengths: technical sound, a great mix of explicit modelling and learning, thoughtfully conceived, carried out and analyzed experiments, and demonstration on the real robot with strong sim2real. On the other hand, there are concerns about the used metrics/evaluation, i.e. can there be a trade-off between cube utilization and efficiency? In addition, there is a question on the choice of the baseline for comparisons where the PUN-packing-only baseline significantly underperforms the best performing baseline Zhao et al. (Uti. = 0.834). As another baseline, imitation Learning approaches can also be alternative as the authors have also discussed about the human performance baseline. For clarify, it's required to further explain about the data generation process and the training of the two proposed networks.

---------------

The authors have made a great rebuttal response to address the reviewers' concerns. Additional results and experiments are convincing. The authors should add them to the final version.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)