Collapse graph related structs

We don't need these to be separately defined structs, so let's collapse
them into a single structure definition.