Sushmey/shapenet-model

2D to 3D Reconstruction System

A comprehensive system for reconstructing 3D objects from 2D images using deep learning. This system is designed to work with ShapeNet datasets and provides both single-view and multi-view reconstruction capabilities.

Features

• Multi-format 3D Output: Generates voxel grids, point clouds, and meshes
• Flexible Architecture: Supports both single-view and multi-view reconstruction
• Interactive Visualization: Web-based 3D viewers and comparison tools
• Batch Processing: Process multiple images or entire directories
• Comprehensive Training: Full training pipeline with validation and metrics
• Easy-to-use Interface: Command-line tools and Python API

System Architecture

The system consists of several key components:

1. Data Preprocessing (data_preprocessing.py): Converts 3D models to 2D renderings
2. Model Architecture (model_architecture.py): Neural network implementations
3. Training Pipeline (training_pipeline.py): Training and validation loops
4. Inference System (inference_system.py): Model inference and 3D generation
5. Visualization Tools (visualization_tools.py): Interactive 3D viewers and comparisons
6. Main Interface (main.py): Unified command-line interface

Installation

1. Clone the repository:

   git clone <repository-url>
   cd shapenet
   

2. Create and activate virtual environment:

   python -m venv shapenet_venv
   source shapenet_venv/bin/activate  # On Windows: shapenet_venv\Scripts\activate
   

3. Install dependencies:

   pip install -r requirements.txt
   

Quick Start

1. Preprocess Your Dataset

First, preprocess your ShapeNet dataset to generate 2D renderings:

python main.py preprocess

This will:

• Load 3D models from your ShapeNet dataset
• Generate multiple 2D views of each model
• Create training/validation/test splits
• Save processed data for training

2. Train the Model

Train the reconstruction model:

python main.py train

This will:

• Load the preprocessed data
• Train the neural network
• Save model checkpoints
• Generate training curves

3. Run Inference

Generate 3D models from 2D images:

# Single image
python main.py inference --input path/to/image.jpg --output results/

# Multiple images
python main.py inference --input path/to/images/ --output results/

4. Run Full Pipeline

Execute the complete pipeline:

python main.py pipeline --input path/to/test_image.jpg

Usage Examples

Python API

from main import ReconstructionSystem

# Initialize system
system = ReconstructionSystem()

# Preprocess data
system.preprocess_data()

# Train model
system.train_model()

# Run inference
system.run_inference('input_image.jpg', 'output_dir/')

# Evaluate model
system.evaluate_model()

Direct Module Usage

from inference_system import InferenceEngine
from visualization_tools import ReconstructionVisualizer

# Load trained model
engine = InferenceEngine('checkpoints/best_model.pth')

# Generate 3D from image
results = engine.generate_3d_from_single_view('image.jpg')

# Visualize results
visualizer = ReconstructionVisualizer()
fig = visualizer.create_interactive_plotly_viz(
    voxels=results['voxels'],
    pointcloud=results['pointcloud']
)
fig.show()

Configuration

Create a config.json file to customize the system:

{
    "data_path": "/path/to/shapenet",
    "processed_data_path": "/path/to/processed_data",
    "checkpoint_dir": "/path/to/checkpoints",
    "output_dir": "/path/to/outputs",
    "model_type": "single_view",
    "num_views": 12,
    "image_size": 224,
    "voxel_size": 32,
    "num_points": 2048,
    "batch_size": 4,
    "learning_rate": 0.001,
    "num_epochs": 100
}

Model Types

Single-View Reconstruction

• Takes one 2D image as input
• Faster inference
• Good for real-time applications
• Use model_type: "single_view"

Multi-View Reconstruction

• Takes multiple 2D images as input
• Higher accuracy
• Uses attention mechanism
• Use model_type: "multi_view"

Output Formats

The system generates 3D reconstructions in multiple formats:

1. Voxel Grids: 3D binary or probability grids
2. Point Clouds: 3D point coordinates
3. Meshes: 3D triangular meshes (OBJ format)
4. Interactive Visualizations: Web-based 3D viewers

File Structure

shapenet/
├── data_preprocessing.py      # Data preprocessing pipeline
├── model_architecture.py      # Neural network architectures
├── training_pipeline.py       # Training and validation
├── inference_system.py        # Model inference
├── visualization_tools.py     # Visualization utilities
├── main.py                    # Main interface
├── requirements.txt           # Dependencies
├── config.json               # Configuration file
├── README.md                 # This file
├── PartSym dataset/          # Your ShapeNet dataset
├── processed_data/           # Preprocessed data
├── checkpoints/              # Model checkpoints
└── outputs/                  # Inference results

Performance Tips

1. GPU Usage: The system automatically uses GPU if available
2. Batch Size: Adjust based on your GPU memory
3. Image Size: Larger images provide better quality but slower processing
4. Voxel Resolution: Higher resolution voxels are more detailed but require more memory

Troubleshooting

Common Issues

1. CUDA Out of Memory:

   • Reduce batch size
   • Use smaller image sizes
   • Use CPU instead of GPU

2. Missing Dependencies:

   • Install all requirements: pip install -r requirements.txt
   • Check Python version compatibility

3. Data Loading Errors:

   • Verify dataset path in configuration
   • Check file permissions
   • Ensure OBJ files are valid

Debug Mode

Enable debug logging:

import logging
logging.basicConfig(level=logging.DEBUG)

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Citation

If you use this system in your research, please cite:

@software{shapenet_reconstruction,
  title={2D to 3D Reconstruction System},
  author={Your Name},
  year={2024},
  url={https://github.com/yourusername/shapenet}
}

Acknowledgments

• ShapeNet dataset for 3D models
• PyTorch for deep learning framework
• Trimesh and Open3D for 3D processing
• Plotly for interactive visualizations