No we dont need matplotlib. actually I figured out THREE.js can accept color strings directly for its input. 
So no need to convert to rgb or any other values. We can remove matplotlib's color converter altogether.
I can do it in master once we have other things got merged in.