Support deleting refs with swift backend

Complete delete support with swift.