Sadly there's no straight answer to this. If you use a flat, knobbly, tough casing mud tyre on dry gravel you'll be slow, but if you use a slick in the mud you'll be even slower as you'll have to walk. Tread pattern, width, and casing all come into play but so to does the tyre profile. A well-rounded tyre will have a smaller, faster contact patch but will be less grippy, though I also find rounder tyres to cover more predictably.
Generally speaking, all else being equal, a slicker tyre will be faster, but speed is about choosing the right tyre for the right conditions more than it is about any single tyre being 'the fastest'.