Refactor to use one single instance of callgraph consistently

This completes the refactoring and all the tests pass now.

Closes #46.