Upgrading to new pipeline phase 2

This entails: fixing integration tests and fully moving onto the demand driven pipeline.