Keep a separate graph for pure adjacency.