ingoring the reference set outputs