The ongoing debate between polyester and nylon is a heated one, but unfortunately there is no clear answer to this question. Both polyester and nylon offer unique benefits and suffer from unique drawbacks, so a lot depends on what you are looking for in a tent.
Polyester is both cheap and lightweight, so is one of the most common materials used in tent floor construction.
Nylon is capable of being woven to a thicker degree than polyester, so it can often be more durable and more waterproof.