I think this could be just
`plot_object$layers <- Filter(function(x) identical(x$geom$objname, geom), plot_object$layers)`