The most likely source of the smell is outside air being drawn down the chimney. As it passes through the chimney and fire box it picks up some of the smokey odor.
Air is more likely to be drawn down the chimney if some other appliance is exhausting air.
For example, a conventional furnace pulls air from the house to feed combustion. If the house is tight the fireplace chimney may be the best source of this air. A range hood or a clothes dryer are other exhaust sources that may contribute to the problem. You can reduce the problem by installing a tight fitting damper at the top end of your fireplace chimney and by providing a source of fresh air to feed your furnace. Ask a chimney technician about the damper and a furnace technician about the outside air intake for the furnace.