Fix line color handling

@jakevdp please confirm that this fixes your problem.

Sorry about this, I am very surprised this did not come up sooner.