Can you write an invariance test instead for that? 
(see this pr for more information on how it works https://github.com/scikit-learn/scikit-learn/pull/2460/files#diff-1f2e474d14c659d2a0d166dc2b220ef7R65)