This feels like this and the other function should panic(). This would mean code is messed up. /cc @xiangli-cmu