Implicit Grasp Diffusion: Bridging the Gap between Dense Prediction and Sampling-based Grasping

Authors: pinhao  song
pengteng  li
renaud  detry
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-30 16:44:28

Abstract: There are two dominant approaches in modern robot grasp planning: dense prediction and sampling-based methods. Dense prediction calculates viable grasps across the robot’s view but is limited to predicting one grasp per voxel. Sampling-based methods, on the other hand, encode multi-modal grasp distributions, allowing for different grasp approaches at a point. However, these methods rely on a global latent representation, which struggles to represent the entire field of view, resulting in coarse grasps. To address this, we introduce \emph{Implicit Grasp Diffusion} (IGD), which combines the strengths of both methods by using implicit neural representations to extract detailed local features and sampling grasps from diffusion models conditioned on these features. Evaluations on clutter removal tasks in both simulated and real-world environments show that IGD delivers high accuracy, noise resilience, and multi-modal grasp pose capabilities.

Main paper: 
 introduction: 
 Grasping is the most basic activity that allows a robot to have an effect on its environment -by definition, a robot's primary purpose. Grasping is unfortunately an exceptionally challenging task. The robot must compute 6-DoF gripper poses that yield a stable bond with an object, based on incomplete visual information obtained from a single (color or depth) image of a cluttered scene, where at least half of object surfaces are self-occluded or occluded by neighboring objects. From this sparse view, the robot needs to understand the scene's geometry, object properties, and relationship to the shape of its gripper.Today's solutions to this problem can be separated into two main categories: dense prediction methods [1, 2, 3] and sampling-based methods [4, 5] . For dense prediction methods, the model discretizes the scene in units (pixel [2] , voxel [1] , or point [3] ), then generates a fixed number of grasps for each unit by regressing feasible gripper orientations at the unit's position, based on local geometry. By nature of this discretization, these methods can easily parse large scenes containing many objects. Unfortunately, a fixed number of grasp predictions per unit is a poor approximation of grasping: a point on an object can often be grasped via many gripper orientations. Capturing this multi-modality is essential to upstream processes that need to reconcile grasping with reachability or obstacle avoidance.Sampling-based methods use heuristic rules [6, 7] or generative models [4, 5] to sample grasps. For the latter type, they encode a region of interest into a latent feature and use generative models such as VAEs or diffusion models to sample grasps within the region. In principle, this approach provides 8th Conference on Robot Learning (CoRL 2024), Munich, Germany.strong multi-modality and a good approximation of the real world, since no discretization is applied. Unfortunately, they are expensive and difficult to train, and struggle with scenes that contain multiple objects: because the information encoded by the latent feature is global, when the region of interest contains multiple objects, the model misses local subtleties and generates inadequate grasp poses. Early works [4, 5] were limited to cases where a single object is shown in the scene. As shown in [8] , the performance of SE(3)-DiffusionFields [5] achieves acceptable performance only after segmenting each object's point cloud and processing each object separately.We propose to bridge the gap between the two approaches discussed above with a model named Implicit Grasp Diffusion (IGD), which captures strengths from both sides:(i) Local geometry: Instead of encoding the entire scene into a global feature as sampling methods do, IGD leverages implicit neural representations [9] to query local features at certain positions. These local features prioritize local geometry information, which proves advantageous in generating valid grasps.(ii) Continuous sampling domain: By contrast to the discretization operated by dense prediction methods, IGD samples grasps from a continuous domain, via implicit neural representations.(iii) Multi-modal representation: We compute grasp orientations via diffusion at grasp locations, enabling the generation of multiple orientations at one location.To realize these capabilities, we present two main technical contributions:1. Deformable attention in implicit space. Feature relevance at a grasping point cannot be reduced to proximity alone. Instead, we utilize a Deformable Attention Module to dynamically sample the implicit feature space, determining relevance based on local geometry.2. Two-stage probabilistic grasp evaluator. The large imbalance between valid and invalid grasps poses challenges for filtering the diffusion model's grasp suggestions. We propose a two-stage approach: an Affordance Evaluator followed by a Grasp Classifier. To capture grasp-relevant features, we introduce a grasp-conditioned Deformable Attention Module, enabling the model to maintain equivariance to 3D scene translations and rotations.We conduct experiments on a clutter-removal task both in simulation and on physical hardware. IGD outperforms dense prediction methods (VGN [1] , GIGA [10] , GraspNet-1billion Baseline [11] , and GSNet [12] ) and sampling-based methods (GPD [6] , 6DoF-GraspNet [4] , and SE(3)-DiffusionField [5] ).2 Related Works 6-Dof grasping methods. Recent work on 6-Dof grasping has seen the emergence of two families of methods: sampling-based methods [4, 5, 6] and dense prediction methods [1, 3, 10] . In Section 1, we have already discussed the advantages and disadvantages of both frameworks. In summary, the locality and discretization of dense prediction methods reduce the training difficulty but sacrifice representational capability, while the globality and multi-modality of sampling-based methods lead to slow and poor convergence. In this work, the proposed IGD combines the complementary advantages of sampling-based and dense prediction methods. IGD focuses on local geometry and captures the multi-modal distribution of grasps, which makes it work well in cluttered scenes.Diffusion models. Diffusion models [13] are trained to reverse the process of adding noise to data, effectively teaching the model to reconstruct the original data from progressively noisier versions through a series of denoising steps. Detailed explanations of diffusion models can be found in our supplementary material. Recent advances in computer vision [14, 15, 16] show diffusion models produce high-quality images without the drawbacks of mode collapse and unstable training. Due to its strong multi-modal expressiveness, this paradigm has also been applied to the robotics field to generate grasps [5] , policies [17, 18] , and trajectories [19, 20] . Our IGD model leverages diffusion ! ! ! " ! "#$ … ! % … !!(#"#$|#", &) ((#"|#"#$)Figure 1 : IGD workflow. The scene is captured by a TSDF obtained from a depth image. A 3D convolutional network extracts features from the TSDF, and the obtained features are projected onto three canonical planes, then aggregated into tri-plane feature grids. Then, a Grasp Orientation Diffuser samples grasp orientations conditioned on the aggregated feature queried at the grasp center. Finally, a Two-stage Probabilistic Grasp Evaluator estimates grasp quality.to generate grasps, endowing it with an inherent capacity to probabilistically encode multiple handapproach modes around one grasp location.Implicit Neural Representations. Implicit neural representations (INRs) have shown remarkable capabilities in modeling 3D object shapes, synthesizing scene surfaces, and capturing complex structures [21, 22, 9] . INRs use MLPs to map spatial coordinates to scene attributes, enabling smooth and continuous representation of shapes in high resolution. INRs eliminate the need for discretization by effectively mapping continuous indices to corresponding data, such as magnifying images for super-resolution using a magnification scale as an index [23] or querying video frames in continuous time [24] . In robotics, INRs play a crucial role: [25, 26] use INRs to map gripper poses to grasp distances, guiding the gripper towards valid grasp poses, while GIGA [10] uses INRs to learn geometry and affordance jointly for acquiring grasp poses at query positions. In this work, the proposed IGD follows GIGA and uses INRs to eliminate spatial discretization. Combined with the multi-modal sampling of diffusion models, IGD can theoretically sample all the potential grasps in a workspace. 
implicit grasp diffusion: 
 Implicit Grasp Diffusion (IGD) models the distribution p(r, w|p), capturing gripper orientations r and gripper widths w conditioned on a grasp center point p ∈ R 3 . As shown in Fig. 1 , IGD comprises a Grasp Orientation Diffuser (GOD) that samples a grasp g = (p, r) at p, followed by a Grasp Evaluator that assesses the grasp quality q ∈ [0, 1] of g and filters out low-quality grasps. Both models are based on a local feature representation of the object's geometry around p, extracted by a Tri-plane Feature Encoder. A Width Predictor then estimates the finger width required for the grasp using the same aggregated feature.Following GIGA [10] , we adopt ConvONets [27] as the encoder architecture to extract a tri-plane feature space. The encoder processes a TSDF voxel field with a 3D CNN layer, generating a fea-ture embedding for each voxel. These 3D features are then projected onto three canonical planes (XY /XZ/Y Z). A 2D U-Net further refines these feature planes, producing the tri-plane features c xy , c xz , c yz . Given a query position p, we project p to each feature plane and query the local features at the projected locations c(p), as:EQUATIONwhere c ij , p ij (i, j ∈ x, y, z) are the plane features and points projected onto the corresponding plane, and ϕ represents a bilinear interpolation of the feature plane at the projected point.To generate a grasp at position p, we gather features from p's neighborhood by sampling nearby points and querying, then aggregating their features. Crucially, the sampling process is guided by local geometry to focus on areas relevant to the grasp. Inspired by [28] , we propose a Deformable Attention Module (DAM, Fig. 2a ) to dynamically adjust the receptive field and produce the aggregated feature c(p), formulated as:EQUATIONEQUATIONwhereW Q , W K , W V , W O are learned parameters.∆p k is a learned offset obtained by linear projection from c(p). In effect, c(p) encodes a local shape descriptor tailored for grasp generation and grasp accuracy evaluation.Given a position p, the Grasp Orientation Diffuser (GOD) models an orientation distribution p θ (r|c(p)) to represent feasible grasps based on the local geometry information encoded in c(p).During the forward diffusion process, noise is injected into the orientation r (denoted as r 0 ) to obtain r t as:EQUATIONwhere ϵ t ∈ N (0, I). In the reverse diffusion process, ϵ t is approximated with ϵ θ (r, t) output by a noise predictor, which is realized through implicit neural representations, as:f θ (g, c(p), t) → ϵ θ (r, t).(EQUATIONwhere f θ (•) is MLPs with learned parameters θ and g = (p, r). Thus, r 0 can be reconstructed from pure noise r T iteratively as follows:EQUATIONwhere z ∈ N (0, I). Through the reverse diffusion process, the predicted grasp ĝ = (p, r0 ) ∈ SE(3) can be obtained. To train the noise predictor, the KL divergence between the forward and backward diffusion processes is minimized, which is equivalent to:EQUATION)Not all grasps generated by GOD are satisfactory: (i) Grasps generated far from any object, or inside one, should be discarded. (ii) Since grasps produced by GOD are samples, some have a low probability and are inadequate. Consequently, we employ a grasp evaluator to filter out unsatisfactory grasps. However, training a grasp evaluator directly is challenging due to the severe imbalance between graspable and ungraspable regions, a problem noted in similar works [4, 29, 30] . To address this, we propose a Two-stage Probabilistic Grasp Evaluator, designed to manage the imbalance between positive and negative grasps. We consider the conditional distribution p(r|p) and rewrite it as: where affordance a represents the existence of feasible grasps at a certain point. p(a|p) denotes the probability of the existence of feasible grasps at position p, and p(r|a, p) denotes the feasibility probability of the grasp orientation r given the affordance a at the position p. Eq. 8 shows that the grasp evaluator can be separated into two factors: an Affordance Evaluator (AE) p(a|p) and a Grasp Classifier (GC) p(r|a, p). To train the grasp evaluator, we maximize the lower bound of log-likelihood, which is an approximation of maximum likelihood estimation (MLE), as:EQUATIONEQUATIONThe lower bound of the log-likelihood can be regarded as a summation of the log-likelihood of AE and GC. Maximizing the lower bound reduces the original goal to independent maximum-likelihood objectives for the first and second stages respectively. In our implementation, implicit neural representations are used to realize AE and GC.Affordance Evaluator (AE). We query the grasp center p and the aggregated feature c(p) to obtain the affordance, as:EQUATIONwhere f ψ is implemented by several residual fully-connected blocks. AE is trained with the following cross-entropy loss:EQUATION)where âp is the predicted affordance, while a p is the ground-truth affordance.Grasp Classifier (GC). We query the grasp g and the aggregated feature c(p) to obtain the grasp score v, as:EQUATIONTo implement Eq. 12, we propose a Grasp-conditioned Deformable Attention Module (Grasp DAM, illustrated in Fig. 2b ) to extract a feature for grasp classification. We first define a set of gripperrelative learnable control points u 1 , ..., u L . In order to express these points in the scene's base frame, we transform them with a transformation T g defined by g. The feature for grasp classification can be obtained by applying deformable attention to the features of control points, as:EQUATIONEQUATIONwhereW ′ Q , W ′ K , W ′ V , W ′O are learned parameters. Several residual fully connected blocks are applied to h to compute a grasp score v. However, due to the sparsity of feasible grasps, GC trained with cross-entropy on a balanced positive-negative dataset often overestimates the quality of infeasible grasps. To alleviate this problem, a large set of negative grasps are randomly sampled from the scene and a focal loss is used to extract valuable information from these negative examples, as:EQUATION)where vg is the predicted grasp score, while v g is the ground-truth grasp label. α is the balance parameter, and γ is the focus parameter. The sampling of control points in GraspDAM is SE(3)equivariant because if the geometric relation between the scene and grasp remains fixed, the relation between transformed control points and the scene will be consistent. This contrasts with a straightforward alternative implementation, where sampling points are generated by neural networks with the input of the grasp and aggregated feature. The sampling equivariant limits the search space and improves the grasp feature extraction consistency, which improves the training stability. Based on Eq. 8, p(r|p) = p(r|a = 1, p)p(a = 1|p) as p(r|a = 0, p) = 0. Thus, we can obtain the final grasp quality as q = a • v. In the paper's supplementary material, ablation studies demonstrate that the Two-stage Probabilistic Grasp Evaluator has benefits in both training and inference.The training loss of the proposed IGD consists of two parts: the grasp loss and the geometry loss, as L = L grasp + L occ . The total grasp loss can be formulated as L grasp = L ddpm + L a + L g + L w , where L ddpm , L a , and L g are the loss for training the GOD, AE, and GC, respectively. The width loss L w and the geometry loss L occ are obtained by calculating the L2 distance and the cross-entropy loss between predictions and ground truths, respectively, following the paradigm of GIGA [10] . 
experimental setup: 
 We follow the same experimental setup as GIGA [10] . The model is trained with GIGA's opensource data [10] . We implement our method on a Franka Research 3 robot arm in both simulation and real-world environments.Simulation Environment: Our simulated environment is built using PyBullet, featuring a freefloating gripper that samples grasps within a tabletop workspace measuring 30 × 30 × 30 cm 3 . For a fair comparison, we employ the same object assets as VGN [1] and GIGA [10] , including 303 training and 40 test objects from various datasets [31, 32, 33, 34] . The model is validated in two types of simulated scenes: pile and packed. In the pile scene, objects are randomly dropped into a box of the same dimensions as the workspace, resulting in a cluttered pile once the box is removed. The packed scene features a subset of taller objects placed at random locations on the table in their canonical pose. A single-view depth map serves as the model's input for grasp generation. The process involves repeatedly predicting and executing a grasp, followed by removing the grasped object from the workspace until one of three conditions is met: all objects are cleared, two consecutive failures occur, or no grasp is detected. Performance metrics are averaged over 100 simulation rounds using 5 different random seeds.Real-world Environment: In real-world experiments, 15 rounds of experiments are performed for both the packed and pile scenes, respectively. Everyday objects are used to conduct the experiments (see supplementary material). In each round, 5 objects are randomly selected and placed on the table. In each grasp trial, we pass the TSDF or point cloud from a side-view depth camera to the model and execute the physically feasible grasp with the highest score. Metric: (i) Grasp Success Rate (GSR = #successful grasps #total grasps ) that measures the ratio of successful grasps to total grasps; (ii) Declutter Rate (DR = #grasped objects #total objects ) that measures the ratio of objects removed successfully to the number of total objects presented.We implement the proposed IGD with PyTorch and train the models with the AdamW optimizer for 12 epochs. An initial learning rate of 2 × 10 -4 is set. The step learning scheduler is leveraged with a decay factor set to 0.1, and the scheduler works at the 9 th and 11 th epochs.The final grasp pose is obtained by sampling from the trained IGD. We discretize the volume of the workspace into 40 × 40 × 40 voxel grids and use the centers of all voxel cells as grasp centers. We then evaluate affordances at all grasp centers, filtering out those with low affordances. Then, for each remaining grasp center, we conduct multi-round grasp sampling and retain the grasp with the highest grasp score. Next, we mask out impractical grasps. Finally, grasp qualities are the product of affordances and grasp scores, and the grasp with the highest quality is selected if the quality is beyond the threshold. If no grasp has a quality above the threshold, we declare that there is no feasible grasp in the scene.We compare our method against seven strong baselines: Dense Prediction Methods: (i) VGN [1] : Volumetric Grasping Network, which generates a large number of grasps in parallel given input TSDF volume. (ii) GIGA [10] : Grasp detection via Implicit Geometry and Affordance, which leverages implicit neural representations and geometrical supervision. (iii) GraspNet-1billion Baseline [11] : A baseline method training on the GraspNet-1billion dataset. (iv) GSNet [12] : A model based on GraspNet-1billion Baseline with the proposed graspness as a refined grasp quality label. For all the methods mentioned above, we use their pre-trained models for comparison.We report GSR and DR for different scenes in Table 1 . Sampling-based methods struggle to generate effective grasps in cluttered environments, performing worse than dense prediction methods. IGD outperforms both dense prediction and sampling-based methods with just one sampling round in both packed and pile scenes. In packed scenes, IGD shows an 8.1% improvement in GSR over GIGA while maintaining the same DR level. Increasing the sampling rounds to 11 boosts IGD's DR in packed scenes to 88.8%, exceeding GIGA by 3.7%. In pile scenes, IGD's performance matches that of GIGA, and with 11 sampling rounds, it achieves 71.8% GSR and 55.7% DR. The superior performance of IGD results from its integration of the strengths of both sampling-based and dense prediction methods: (i) local information enables effective handling of cluttered scenes, and (ii) the strong multi-modal expressiveness of the diffusion model captures grasp distributions. Table 2 presents grasp detection results from real-world experiments, aligning with simulation findings where IGD outperforms GIGA in both packed and pile scenes. This enhanced performance is partly attributed to IGD's greater robustness in noisy real-world environments, which will be analyzed further in the supplementary material.The introduction of diffusion models aims to capture the multi-modal distribution of feasible grasps. To validate this, we conduct six rounds of diffusion processes at various valid points containing feasible grasps and visualize the denoising trajectories using t-SNE, as shown in Fig. 3 . Each figure displays four ground-truth grasps. The results indicate that starting from pure noise, the data converges to different ground truths throughout the denoising process, demonstrating that IGD effectively captures the multi-modal nature of viable grasps.Table 1 shows the performance of GIGA and IGD in different kinds of noises. According to [1] , noise in a depth camera includes the following noises: noise from stereo matching, lateral noise, blur, and noise in depth estimation. We simulate those noises and add them in the depth image to evaluate the performance of models. Our baseline condition is the dex noise environment, where models are trained. According to  
conclusion: 
 In this work, we introduce Implicit Grasp Diffusion (IGD), a novel grasping framework that leverages the strengths of both dense prediction and sampling-based methods, exhibiting robust locality and multi-modality. Our framework features a Grasp Orientation Diffuser and a Two-stage Probabilistic Grasp Evaluator for regressing and evaluating grasps. We assess IGD in both simulated and real-world environments, comparing it against state-of-the-art approaches. Experimental results highlight IGD's high grasp accuracy, strong noise robustness, and expressive multi-modal capabilities. from a depth map, it struggles to analyze the geometry and borders of occluded objects (as shown in Fig. 3 (e-f )), leading to further failures. Third, IGD uses a low-resolution TSDF without the guidance of RGB information, resulting in a loss of subtle geometry details. For example, in Fig. 3 (g), IGD generates grasps between two objects, incorrectly assuming they are connected. In Fig. 3 (h), IGD fails to distinguish between thin-lying objects and the table, resulting in an unsuccessful grasp attempt.We also conducted extensive ablation studies to validate each module in our proposed IGD. Ablation studies were mainly performed in pile scenes because of data abundance and higher task difficulty compared to packed scenes. Table 2 shows the ablation studies of IGD. The proposed DAM effectively improves the performance in both GSR and DR. Besides, the performance deteriorates with GC alone compared to the AE. Combining the AE and GC achieves the best performance, which demonstrates the effectiveness of the proposed probabilistic two-stage grasp evaluator. To further investigate what brings the performance improvement, we train AE and GC together and deactivate one of them in the inference (shown as "GC*" and "AE*" in Table 2 ). If we deactivate the GC, the performance decreases drastically, while the performance decline from deactivating the AE is limited. This result is contrary to solely training the AE and GC. We can conclude that the performance improvement is mainly from the loss supervision instead of a simple ensemble of two modules. Negative grasp sampling is also important to train the grasp evaluator. Introducing neg- ative grasp sampling with GC alone largely increases GSR (53.5% to 72.2%) but decreases DR (42.5% to 33.5%). When it comes to probabilistic two-stage structure, negative sampling improves both GSR and DR. The results mean that the grasp evaluator needs to excavate information from negative samples to learn to distinguish feasible grasps in the large grasp space.Number of sampling rounds. Since we can sample multiple rounds to obtain different grasps at the same grasp position, an ablation study about sampling rounds is conducted to analyze the effect of this hyper-parameter, which is shown in Fig. 4 . In packed scenes, there is no improvement in GSR as sampling rounds increase, while DR increases in general. In contrast, in harder pile scenes, both GSR and DR increase as sampling rounds increase. The results show the ability of IGD to obtain good grasps by increasing grasp samples, which is inherited from sampling-based methods.Sampled points in DAM. Table 3 shows an ablation study on sampled points in DAM. Because the anchor points are initialized as the points spatial-uniformly sampled in the cube around the grasp center, we select 2 3 , 3 3 , and 4 3 sampled point numbers to evaluate their performance. From Table 3 , the best performance appears in the 2 3 setting. We also visualize the sampled points in DAM in Fig. 5 . In packed scenes, the sampled points are primarily distributed in the region between the grasping pose and the object's surface. Since grasps are generated by the features aggregated by DAM, we believe these sampled features contribute to more accurate grasp generation. In pile scenes, in addition to points localized around the grasp area, some points are distributed over a farther range. This visualization demonstrates that DAM can dynamically sample features based on the local geometry, aiding in accurate grasp generation.Focal loss hyper-parameters. Table 3 shows an ablation study of focal loss hyper-parameters in GC. γ = 0 reduces the focal loss to a normal cross-entropy loss and shows inferior performance. γ is to tune the level of hard example mining, and α is to balance the weight of positive and negative samples. According to the results, we select the best hyper-parameter setting as α = 0.5 and γ = 2. 
limitations and future work: 
 Although we have demonstrated the effectiveness of IGD in both simulation and real-world systems, there are limitations that future work can improve. First, we can't directly obtain the point with the highest affordance value. In GIGA, 40 × 40 × 40 points are sampled to query the grasp to obtain the best grasp. IGD inherits this limitation from GIGA. Second, diffusion models have higher computational costs and inference latency compared to dense prediction methods, especially in IGD where grasp sampling is separated into two stages: position sampling and orientation sampling. Future work can exploit the latest advancements in diffusion model acceleration methods to reduce the number of inference steps required, such as new noisy schedules [3] , inference solvers [4] , and consistency models [5] . Third, in our GOD, we directly apply diffusion models to the generation of quaternions. However, quaternion in the diffusion process is not in close form. Current works have proposed a lot of methods to achieve rotation diffusion [6, 7, 8] . Although our effort to apply these works to IGD fails, it is still worth exploring. 
 
 Appendix: 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: The reviewers generally agree that this is a valuable contribution with some novel ideas and well-demonstrated performance improvements. On the other hand, all three point out weaknesses in terminology and formal description (regarding Eqn. 1, SE(3) invariance, generation/sampling/discretization). There are also questions around the attention module and about limitations/failures.

The authors carefully addressed most of the reviewer's questions to their satisfaction.

Reviews: review: title: title: Review
paper_summary: paper_summary: In this work, the authors consider the task of SE(3) grasp prediction. Noting the difficulty of capturing the multimodaltiy, precision, and sparsity of grasps in a real scene, the authors seek to combine both dense prediction and local sampling approaches in order to fit the underlying distribution. The authors propose a novel deformable attention module to effectively capture local features near a query point, and incorporate this attention module into various implicit functions which are used to sample and score grasps. The authors show that their method outperforms the next closest method, GIGA, as well as various other recent methods for grasp generation.
main_review: main_review: # Quality

This paper is of medium-high quality. The key intuitions behind the approach are sound - performing a global search that must yield a locally precise distribution of results (especially when the search space is over SE(3)) is a challenging task, and the design decisions chosen to address this are logical and well-reasoned. 

The design of the Deformable Attention Module makes sense, and plausibly captures a dynamic context depending on query point and local geometry. Since this is a novel module, I would have liked to see some analysis of what geometric features are actually sampled, qualitatively. I don’t expect an interprable set, but in practice what regions of the space are sampled? Is attention hyperlocal or long-ranging? I’m somewhat skeptical that the formulation for predicting the delta offsets is sufficiently flexible for their desired effect, but am amenable to it.

In particular, I appreciate the decomposition of grasp scoring into an affordance term and a quality term, as it is a natural graphical model for the space. This allows them to address the class imbalance problem rather elegantly.

The diffusion process for rotations seems suspect, because rotations are not additive and therefore the diffusion trajectory may contain invalid rotations (depending on representation). The authors should take a look at the diffusion process for SE(3) in the “RPDiff” paper (https://anthonysimeonov.github.io/rpdiff-multi-modal/), in which they propose a modified compositional diffusion of elements of so3.

The “invariance” argument for the grasp region attention is tenuous - the features need not be invariant; it is simply the distribution over sampled points on the gripper which are invariant (or really, equivariant). Perhaps I misunderstood something.


# Clarity

The paper is mostly clear - however there are two major issues i had while reading:

1. In the method introduction, Equation 1 implies a regression rather than a generative approach, which confused me until I read further. Probably should be made clear that f maps from a point to a distribution around that point, from which you can sample and then score.
2. What was the significance of the occupancy grid? It’s not mentioned how it’s trained or why it’s used.

# Originality

The paper is of medium originality. Architecturally parts are quite similar to prior work and work in other domains, as is the decomposition into affordance+quality scoring (although the design, including focal loss, is principled). Additionally, while the attention module is novel, the design is not fully explored.

# Significance

The paper has some significance - I think that the design decisions are reasonable and lead to good performance, and while the final performance is incrementally better than prior work I expect that there is a lot of room for architectural improvements  in the design of the attention module + network which could be explored by the community / in future work.

# Strengths

* The design of the network and inference algorithm is principled and intuitive
* The attention module is novel and clever, if a bit underexplored
* The phrasing of the problem as an implicit scoring problem w/ a diffusion-based generator makes sense
* Method outperforms baselines, on identical tasks

# Weaknesses
* Attention module is underexplored
* Improvements are modest
* Failure modes not adequately charactreized
questions: questions: One area that is not well-motivated is the Tri-plane feature grid. Why was this chosen, and are there benefits to simply using a 3D voxel grid?

Can you characterize behavior of the attention modules a bit more?

What are the failure modes of the grasping? What leads to poor predictions in some cases?
limitations: limitations: 2
review_summary: review_summary: Overall there is enough novelty in the method - and downstream improvement over baselines - to warrant acceptance. If more details were explored, and failure modes characterized, I might be willing to increase my rating.
title: title: The paper proposes a novel method and ideas that would be useful to the community. A few additional technical details and improvements for clarity are required.
paper_summary: paper_summary: This paper proposes a new method for grasp generation using Implicit Neural Representations, a two-stage grasp evaluator and a diffusion process for calculating the orientation of the grasp. The proposed approach uses features from the Implicit Neural Representation together with the proposed Deformable Attention Module to estimate the affordance of sampled points and the grasp score when rotation is computed using a diffusion process. In such a way, the method combines the advantages of using dense prediction and sampling-based methods. In the experiments, the proposed approach is compared against several baselines showing promising performance.
main_review: main_review: **Strengths**:
* The paper is well written and very well motivated from the start, making it easier to read.

* Combining strengths from dense prediction and sampling-based methods using a two-stage grasp evaluator and a diffusion process is a nice idea.

* The motivation behind using learnt offsets and control points to aggregate local information using the proposed Deformable Attention Module and not rely on a single set of features is strong.

* Experiments are well conducted and look convincing.

**Weaknesses**:

* The method still relies on the discretisation of the workspace to sample the grasp points. It should be addressed early in the paper, otherwise, the reader is led to believe that the proposed approach gets rid of the discretisation. 
* It is not clear why the proposed Deformable Attention Module is not used in the diffusion process to generate orientation samples. Does the feature of the grasp point hold enough information to accurately predict the rotation?
* It is not clear how the learned offset \delta p_k is generated. How do you ensure that generated offsets are different from each other? Why not use the same control points as for the grasp-conditioned Deformable Attention Module with a fixed orientation?
* The claim that the Grasp Classifier is SE(3)-invariant is not explained properly. Implicit Neural Representation used has no such property and the features of the control points will change if the observation is rotated. It seems that a set of features of the control points is just an over-parametrization of SO(3).
* Figure 1 doesn’t clearly convey the procedure at inference time. I would suggest labelling different parts of Figure 1 (bottom) to clearly show the control flow at inference.
* It is not clear what is shown in Figure 3. Diffusion is used for sampling the rotation only but visualisation shows a diffusion process in a 2D plane.
* More information about the representation of the rotation, normalisation and noise scheduling strategies are needed.
* It would be good if limitations were moved from the appendix to the main paper.
questions: questions: * Why Deformable Attention Module is not used in the diffusion process to generate orientation samples?
* How the learned offset \delta p_k is generated?  How do you ensure that generated offsets are different from each other? Why not use the same control points as for the grasp-conditioned Deformable Attention Module with a fixed orientation?
* What makes the Grasp Classifier SE(3)-invariant?
* What rotation representation are you using and how do you normalise it during the diffusion process?
* What is actually shown in Figure 3?

If my concerns expressed in the limitations are addressed and the questions above answered, I’m willing to improve my score.
limitations: limitations: 2
review_summary: review_summary: I would recommend accepting this paper for publication given that the authors add a few additional technical details and improvements for clarity.
title: title: Review
paper_summary: paper_summary: The paper presents an extension of the GIGA algorithm, incorporating deformable feature aggregation, a two-stage probabilistic grasp evaluator, and diffusion-based orientation proposals. The enhanced method generates a tri-plane feature grid by processing TSDF with ConvONet, applies deformable heads for predicting grasp affordance and width, and employs a diffusion model for orientation generation. The authors demonstrate improvements over GIGA in both simulated and real-world grasping experiments.
main_review: main_review: **Strengths**

1. The method demonstrates a performance improvement over GIGA.
2. The deformable-based local feature aggregation proves effective in predicting grasps, as validated through experiments.

**Weaknesses**

1. **Argumentation Issues:**
    - The categorization of grasping methods into dense and sampling-based is unconvincing.
    - The designation of Contact-GraspNet as a sampling-based method is questionable, as it not only does not utilize generative models, but also use descretized unit as points to predict a single grasp per point.
    - Previous works addressing the uni-modality in approach direction are not mentioned or compared [1-3]. These studies have effectively managed uni-modality while preserving local geometry, so the authors should have claims against them.
2. **Lack of Original Contributions:**
    - The approach heavily relies on previous work, particularly GIGA’s architecture and methods.
    - Claims of novelty in two-stage evaluation are undermined by prior implementations of similar methods, which are not cited [1-4].

References:

[1] GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping

[2] Graspness Discovery in Clutters for Fast and Accurate Grasp Detection

[3] Grasping as Inference: Reactive Grasping in Heavily Cluttered Environment

[4] Deep Learning Approaches to Grasp Synthesis: A Review
questions: questions: **Additional Concerns**

1. **Technical and Expository Clarifications Needed:**
    - Lack of thorough investigation of related works, particularly missing citations for ConvONet in 3.1.
    - Several terms and symbols (e.g., \phi_p) are undefined.
    - Is your f actually mapping? isn’t this a probabilistic model?
    - In Affordance Evaluator box of Figure 1, is usage of local feature correct? Isn’t it aggregated local feature?
    - Term Grasp diffuser is confusing. It only generates orientations.
    - Methods for generating negative grasps and determining ground truth for affordance are not adequately explained.
    - The voxel resolution for position sampling needs specification.
2. **Efficiency Concerns:**
    - Increased latency in grasp generation should be analyzed, identifying the main bottlenecks, possibly related to the grasp diffuser.
limitations: limitations: 2
review_summary: review_summary: I recommand weak accept.