view: remove refs and graphs from the view column enum

They are never shown independently so should not be exposed as a view column type.