The sequence numbers in packets sent get incremented by one each time, abnormal behaviour, but characteristic of sequence number collectors, such as rbone and the unpublished tool used to takedown Tsutomo Shimomura's site back on Christmas Day, 1994. Nmap collects the initial sequence numbers received from the target, and looks for a pattern in the way they are incremented. Really old UNIX systems still use a constant increment, while newer and more secure systems use a random increment. Countermeasures Nmap's accuracy relies on collecting a lot of information from the target's stack. Anything that tampers with this information will affect the prediction of the target's OS version.
Firewalls do to a greater or lesser extent.
Most stacks permit tuning some of the tested parameters, but only a small portion of them. Packet filtering firewalls are ineffective. Application gateways running on a firewall prevent nmap from identifying the target, as nmap will be probing the firewall's stack instead. Of course, the best defense is to prevent any scanning from penetrating your network in the first place, and detecting scanning if it takes place within the confines of your network.