Title: Semantic-metric Bayesian Risk Fields: Learning Robot Safety From Human Videos With a VLM Prior
Primary area: applications to robotics, autonomy, planning
Keywords: Safety, Risk-aware Planning

Abstract: Humans interpret safety not as a binary signal but as a continuous, context- and spatially-dependent notion of risk. While risk is subjective, humans form rational mental models that guide action selection in dynamic environments. This work seeks to extract an implicit human risk model by introducing a novel, semantically-conditioned and spatially-varying parametrization of risk, supervised directly from safe human demonstration videos and VLM common sense. Notably, we define risk through a Bayesian formulation. The prior is furnished by a pretrained vision-language model, while the likelihood function is a learned ViT that maps pretrained features (e.g., DINOv3), to pixel-aligned risk values. Our pipeline produces pixel-dense risk images that can then be used as value-predictors in robot planning tasks or be projected into 3D for use in conventional trajectory optimization to produce human-like motion. This learned mapping enables generalization to novel objects and contexts, and has the potential to scale to much larger training datasets. In particular, the Bayesian framework that is introduced enables adaptation of our model to additional observations or common sense rules. We demonstrate that our proposed framework produces contextual risk that is aligned with human preferences. Additionally, we illustrate several downstream applications of the model; as a value predictor for a visuomotor policy or in conjunction with a classical trajectory optimization algorithm. Our results suggest that the proposed method is a significant step toward enabling autonomous systems to internalize human-like risk reasoning.

TL;DR: We propose a framework for learning contextual and spatially-dependent risk from safe human demonstrations and common sense rules.