My house is very drafty. I've called in various professionals, but none has located the source. Air leaks are located by following smoke trails from a device called a smoke pencil. Check your state energy office or local cooperative extension service for a contractor who conducts blower-door tests. You can also detect drafts on your own.
Many drafts sneak in between the baseboard and flooring.
Stop them by applying a thin bead of clear caulk to the gap under the baseboard. Drafty windows and doors are often caused by cold air blowing through the spaces between the rough openings and the side jambs. Block out the drafts by removing the interior trim, filling the spaces with fiberglass insulation and then sealing them with aerosol foam caulk.