alessandrolapizza/digit-force-mlp

DIGIT normal-force MLP head

A lightweight MLP regressor (768 → 256 → 64 → 1) that estimates the normal contact force F_z (in newtons) from a DIGIT vision-based tactile image (the original DIGIT by Meta AI, commercial unit from GelSight; not the newer DIGIT 360). It sits on top of the frozen DINOv2 ViT-B/14 tactile encoder from FeelAnyForce (Shahidzadeh et al., ICRA 2025), which is not retrained. The head reads the encoder's 768-dimensional summary of the whole image (the final CLS token of the vision transformer).

This head is the force estimator of the real-time haptic teleoperation pipeline of the EPFL CREATE Lab semester project Development of Fingertip Haptic Device and Optimal Mapping Method of High Dimensional Tactile Data to Lower Dimensional Haptic Feedback. Code, report and CAD are on GitHub.

Pipeline

DIGIT frame ──► background subtraction ──► resize 224×224 ──► FeelAnyForce DINOv2 ViT-B/14 (frozen) ──► 768-d CLS ──► this MLP ──► F̂_z [N]
               I − I_bg + 127                + normalise

Files

File	Content
mlp_regressor.pth	PyTorch state_dict of the nn.Sequential head (keys 0.*, 3.*, 6.*)
config.json	Architecture, preprocessing constants, backbone reference and training settings

Usage

The backbone code and weights come from FeelAnyForce:

git clone https://github.com/prgumd/FeelAnyForce && cd FeelAnyForce && git checkout 1947bc8
hf download amirsh1376/FeelAnyForce checkpoint_v1.pth.tar --local-dir ckpt               # ~2 GB
hf download alessandrolapizza/digit-force-mlp mlp_regressor.pth config.json --local-dir .   # this head

The code below is the one used in the project's inference server: load_model(), predict() and subtract_bg() copied unchanged from inference_server/model.py. Only FAF_ROOT is set here instead of being imported from the server's config.

import os
import sys
import glob
import json

import numpy as np
import cv2
import torch
import torch.nn as nn
from torchvision import transforms as T
from PIL import Image

FAF_ROOT = "/path/to/FeelAnyForce"   # in the server: from config import FAF_ROOT


# Returns (backbone, mlp, transform, device), on CUDA if available, otherwise MPS, otherwise CPU
def load_model(mlp_path: str):
    if torch.cuda.is_available():
        device = 'cuda'
    elif torch.backends.mps.is_available():
        device = 'mps'
    else:
        device = 'cpu'
    print(f'\n  Device: {device}')

    sys.path.insert(0, FAF_ROOT)
    from args import get_parser
    from composed_model import ComposedModel

    # Frozen FeelAnyForce encoder (its constructor calls .cuda(), bypassed while building it)
    print('  Loading backbone ...')
    ckpt = torch.load(
        os.path.join(FAF_ROOT, 'ckpt', 'checkpoint_v1.pth.tar'),
        map_location='cpu', weights_only=False,
    )
    args = get_parser().parse_args([])
    for k, v in ckpt['config'].items():
        setattr(args, k, v)

    original_cuda = nn.Module.cuda
    nn.Module.cuda = lambda self, *a, **kw: self
    backbone = ComposedModel(args)
    nn.Module.cuda = original_cuda

    backbone.load_state_dict(
        {k.replace('module.', ''): v for k, v in ckpt['state_dict'].items()},
        strict=True,
    )
    backbone.eval()
    backbone.to(device)

    # MLP head 768 -> 256 -> 64 -> 1
    print('  Loading MLP ...')
    mlp = nn.Sequential(
        nn.Linear(768, 256), nn.ReLU(), nn.Dropout(0.2),
        nn.Linear(256, 64),  nn.ReLU(), nn.Dropout(0.2),
        nn.Linear(64, 1),
    )
    mlp.load_state_dict(torch.load(mlp_path, map_location='cpu', weights_only=True))
    mlp.eval()
    mlp.to(device)

    # Resize to 224 and normalise with FeelAnyForce's background-subtracted statistics
    with open(os.path.join(FAF_ROOT, 'mean_and_std.json')) as f:
        stats = json.load(f)
    mean, std = stats['TacForce_train_set.csv']['nobg']
    transform = T.Compose([
        T.Resize((224, 224)),
        T.ToTensor(),
        T.Normalize(mean, std),
    ])

    print('  Model ready.')
    return backbone, mlp, transform, device


# Predicted Fz (>= 0) for one background-subtracted BGR frame
def predict(backbone, mlp, transform, device, frame_bgr: np.ndarray) -> float:
    rgb = frame_bgr[:, :, ::-1].copy()
    pil = Image.fromarray(rgb.astype(np.uint8))
    tensor = transform(pil).unsqueeze(0).to(device)
    with torch.no_grad():
        feats = backbone.get_encoding(tensor)
        fz = mlp(feats).item()
    return max(0.0, fz)


# clip(frame - background + 127) in RGB, returned as BGR
def subtract_bg(frame: np.ndarray, bg: np.ndarray) -> np.ndarray:
    if frame.shape != bg.shape:
        bg = cv2.resize(bg, (frame.shape[1], frame.shape[0]))
    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    bg_r = cv2.cvtColor(bg, cv2.COLOR_BGR2RGB)
    diff = (rgb.astype(np.int16) - bg_r.astype(np.int16) + 127).clip(0, 255).astype(np.uint8)
    return cv2.cvtColor(diff, cv2.COLOR_RGB2BGR)


# Usage: BGR uint8 frames (OpenCV order); bg = mean of frames captured with the gel at rest (the server averages 30)
backbone, mlp, transform, device = load_model("mlp_regressor.pth")
fz = predict(backbone, mlp, transform, device, subtract_bg(frame, bg))

Training

• Data: alessandrolapizza/digit-force-dataset. It holds 2,940 image–force pairs (4 indenter geometries × 35 contact positions × 21 force levels, 0–10 N), collected with a Franka FR3 arm under closed-loop force control. Labels come from a uniaxial load cell.

• Inputs: the background-subtracted images (tactile_nobg/), encoded once by the frozen backbone.

• Recipe:

  • 80/20 split (random_state=42);
  • L1 loss, Adam (lr 1e-3) with ReduceLROnPlateau;
  • early stopping (patience 50).

  The reference training script is scripts/training/train_mlp.py in the GitHub repository.

Results

Metric	Value
Validation MAE (588 samples)	0.227 N (2.3 % of the 0–10 N range), as reported in the project report

License and attribution

The weights are released under CC BY 4.0. They depend on the FeelAnyForce encoder weights (checkpoint_v1, CC BY 4.0, © Shahidzadeh et al.), which are not redistributed here. Please cite both works.

@misc{dezen2026fingertip,
  author       = {De Zen, Alessandro},
  title        = {Development of Fingertip Haptic Device and Optimal Mapping Method of High Dimensional Tactile Data to Lower Dimensional Haptic Feedback},
  year         = {2026},
  howpublished = {Semester project report, CREATE Lab, EPFL},
  url          = {https://go.epfl.ch/Fingertip-Haptic-Device-and-Optimal-Mapping-Method}
}

@inproceedings{shahidzadeh2025feelanyforce,
  author    = {Shahidzadeh, Amir-Hossein and Caddeo, Gabriele M. and Alapati, Koushik and Natale, Lorenzo and Ferm{\"u}ller, Cornelia and Aloimonos, Yiannis},
  title     = {FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors},
  booktitle = {2025 IEEE International Conference on Robotics and Automation (ICRA)},
  pages     = {251--257},
  year      = {2025},
  doi       = {10.1109/ICRA55743.2025.11127723}
}