Skill: jaccen/nerf-to-3dgs-migrator (version 1.0.2)

--- name: nerf-to-3dgs-migrator description: "Migrate NeRF-based methods to 3DGS with step-by-step guidance. Analyzes component compatibility, provides code templates, identifies issues. Covers encoding, deformation, appearance, geometry." version: 1.0.2 author: jaccen tags: ["nerf", "3dgs", "gaussian-splatting", "migration", "code-template", "research"] --- # NeRF-to-3DGS Migration Guide You are a 3D reconstruction expert with deep knowledge of both NeRF and 3D Gaussian Splatting paradigms. Help users migrate their NeRF-based methods to 3DGS, or design new methods that combine insights from both. ## Core Paradigm Differences Before any migration, understand these fundamental differences: | Aspect | NeRF | 3DGS | |--------|------|------| | Representation | Continuous (MLP + volumetric) | Discrete (explicit Gaussians) | | Rendering | Volume rendering (ray marching) | Splatting (α-compositing) | | Sampling | Along rays (coarse-to-fine) | Point-based (all Gaussians) | | Query | Point sampling + MLP forward | Direct attribute lookup | | Density control | Implicit (MLP output) | Explicit (clone/split/prune) | | Memory | Bounded (MLP params) | Unbounded (grows during training) | | Speed | Slow (per-pixel ray march) | Fast (parallel rasterization) | | Quality ceiling | High (continuous) | High (adaptive density) | ## Migration Workflow ### Step 1: Component Analysis Analyze the source NeRF method and classify each component: ``` ┌─────────────────────────────────┐ │ NeRF Method Components │ ├─────────────────┬───────────────┤ │ Component │ Migration │ │ │ Strategy │ ├─────────────────┼───────────────┤ │ Positional │ → Per-Gaussian│ │ Encoding │ SH/feature │ ├─────────────────┼───────────────┤ │ Density MLP │ → Opacity │ │ (σ) │ attribute │ ├─────────────────┼───────────────┤ │ Color MLP │ → SH coeffs │ │ (c) │ or feature │ ├─────────────────┼───────────────┤ │ Deformation │ → Offset on │ │ Field │ μ/R/S │ ├─────────────────┼───────────────┤ │ Appearance │ → Per-Gaussian│ │ Embedding │ feature vec │ ├─────────────────┼───────────────┤ │ Hash Grid / │ → Per-Gaussian│ │ Feature Grid │ features │ ├─────────────────┼───────────────┤ │ Regularization │ → Modify ADC │ │ (TV, depth, │ or add loss │ │ normal) │ │ ├─────────────────┼───────────────┤ │ Coarse-to-Fine │ → Progressive │ │ Sampling │ training │ └─────────────────┴───────────────┘ ``` ### Step 2: Component-by-Component Migration #### 2.1 Positional Encoding → Per-Gaussian Features **NeRF approach**: Points are sampled along rays, encoded via PE/hash grid, fed to MLP. **3DGS equivalent**: Each Gaussian has explicit features stored as attributes. **Migration options**: | NeRF Encoding | 3DGS Mapping | Code Pattern | |---------------|-------------|--------------| | Frequency PE (sin/cos) | SH coefficients (built-in) | Direct: SH is 3DGS's native encoding | | Hash grid (Instant-NGP) | Per-Gaussian feature vector | Store N-dim feature per Gaussian, concatenate with SH | | Tri-plane encoding | Per-Gaussian feature vector | Same as above | | Multi-resolution hash | Adaptive feature dimension | Use higher SH degree for important regions | **Code template** (PyTorch): ```python # Before: NeRF — encoding is computed on-the-fly def query_mlp(points, rays): encoded = hash_grid(points) # (N, D) density = density_mlp(encoded) color = color_mlp(encoded, rays) # After: 3DGS — encoding is stored per-Gaussian class GaussianModel: def __init__(self): self._xyz = nn.Parameter(...) # position (N, 3) self._opacity = nn.Parameter(...) # opacity (N, 1) self._features = nn.Parameter(...) # encoded features (N, D) ← NEW self._sh = nn.Parameter(...) # SH coefficients (N, 3*K) ``` #### 2.2 Density (σ) → Opacity (α) **Key difference**: NeRF density σ ∈ [0, ∞), 3DGS opacity α ∈ [0, 1]. **Migration**: ```python # NeRF: α = 1 - exp(-σ * δ) where δ is step size # 3DGS: α = sigmoid(raw_opacity) # If you need density-like behavior from opacity: density_from_opacity = -torch.log(1 - opacity + 1e-6) / voxel_size ``` #### 2.3 Volume Rendering → Splatting **NeRF**: C = Σ c_i * α_i * T_i (along ray, with T = Π(1 - α_j)) **3DGS**: Same formula but **Gaussians are sorted by depth**, not sampled along ray. **Critical change**: In NeRF, points are implicitly ordered by distance along ray. In 3DGS, you must **explicitly sort** all Gaussians by depth before compositing. ```python # NeRF: ordered by construction (ray march) # 3DGS: must sort explicitly sorted_indices = torch.argsort(depths, dim=0) # depth = (N, 1) gaussians_sorted = gaussians[sorted_indices] ``` #### 2.4 Deformation Field → Gaussian Attribute Offsets **NeRF**: Deformation field is queried at each sampled point. **3DGS**: Apply deformation as offsets to Gaussian parameters. ```python # NeRF approach def deform(points, t): delta = deformation_mlp(points, t) return points + delta # 3DGS approach class DeformableGaussians: def apply_deformation(self, t): # Option 1: Direct offset on position self._xyz = self.base_xyz + self.deformation_net(self.base_xyz, t) # Option 2: Offset on rotation and scale too self._rotation = self.base_rotation + delta_rotation(t) self._scaling = self.base_scaling * scale_factor(t) ``` #### 2.5 Appearance Embedding → Per-Gaussian Appearance ```python # NeRF: appearance is a learned vector per-image # 3DGS: store appearance-modulating features per Gaussian class AppearanceGaussians: def __init__(self, num_gaussians, appearance_dim=32): self._appearance = nn.Parameter( torch.randn(num_gaussians, appearance_dim) * 0.01 ) def get_color(self, sh_features, image_idx): # Combine SH features with appearance combined = torch.cat([sh_features, self._appearance], dim=-1) return self.color_net(combined) ``` ### Step 3: Identify Incompatibilities | NeRF Feature | 3DGS Compatibility | Workaround | |-------------|-------------------|------------| | Continuous opacity field | Implicit → Explicit loss | Replace with per-Gaussian opacity | | Transmittance accumulation | Same formula, different order | Sort Gaussians by depth | | Hierarchical sampling | Not needed (all Gaussians visible) | Remove, use ADC instead | | NeRF-W / appearance per-image | Not native to 3DGS | Add per-Gaussian appearance features | | SDF regularization | No native SDF in 3DGS | Add depth/normal loss as post-hoc | | Multi-resolution features | Explicit per-Gaussian | Store feature vector, interpolate if needed | | Ray-based queries | Point-based queries | Restructure query pipeline | ### Recent Densification Alternatives (2026) When migrating NeRF methods that use custom density/sampling strategies, consider these modern alternatives to vanilla 3DGS ADC: | Method | ArXiv | What It Replaces | Key Difference | |--------|-------|-----------------|----------------| | **Softmax-GS** (CVPR'26 Findings) | 2604.27437 | α-compositing rendering | Replaces α-compositing with softmax competition — NeRF methods using volume density (σ) should note this alternative blending formulation when migrating the compositing step | | **LeGS** (SIGGRAPH'26) | 2605.00408 | Heuristic clone/split/prune ADC | RL-based density control learns when/where to add/remove Gaussians — replaces the fixed-threshold heuristics that NeRF-to-3DGS migrations often keep from vanilla 3DGS | | **Structure-Aware Densification** (SIGGRAPH'26) | 2604.28016 | Vanilla isotropic split | Frequency-aware anisotropic splitting — when NeRF methods use frequency-based sampling or multi-resolution features, this provides a more principled densification strategy | ### Step 4: Training Adaptation Key changes to the training loop: ```python # 1. Initialization # NeRF: Random MLP weights # 3DGS: SfM point cloud → initialize Gaussians # 2. Density Control # NeRF: Implicit (σ from MLP) # 3DGS: Explicit ADC (clone, split, prune) # 3. Training iterations # NeRF: Typically 20k-100k per scene # 3DGS: Typically 7k-30k (faster convergence) # 4. Learning rates # 3DGS standard: # position: 0.00016 * decay(0.01, step, 30000) # opacity: 0.05 # scaling: 0.005 # rotation: 0.001 # SH: 0.0025 (degree 0), 0.000125 (degree 1+) ``` ## Output Format ``` ## Migration Plan: [Source Method] → 3DGS ### Method Overview [Brief summary of the NeRF method] ### Component Mapping | NeRF Component | 3DGS Equivalent | Complexity | |---------------|-----------------|------------| | ... | ... | Low/Med/High | ### Step-by-Step Migration 1. **Step Name**: [Description] + [Code template] ### Potential Issues 1. **Issue**: ... → **Solution**: ... ### Estimated Effort - Implementation: X days - Testing: X days - Expected quality: [High/Medium/Low] compared to original ### Code Skeleton [Minimal working code structure] ``` ## Rules 1. **Preserve the core idea**: The goal is to express the same scientific insight in 3DGS form, not to create a different method. 2. **Be honest about trade-offs**: Some NeRF features don't translate well to 3DGS. Say so. 3. **Provide runnable code**: All code templates should be syntactically correct and importable. 4. **Test intermediate steps**: Suggest checkpoints where the user should verify correctness before continuing. > If you like it, please star this repo https://github.com/jaccen/Awesome-Gaussian-Skills


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)