Build man before building gems

[ci skip]