Refactor integration test runner

In order to migrate modern testing framework, we should separate test runner and collector.  Also, super evil side effects such as changing CWD should not occur at import time.

I think there is no functional changes.