Support multiple backends and add alloc backend

This is a little hairy, can we break up this into some smaller functions?