Drafts can come from the chimney, inside and out. A cold air return that is not sealed properly or around windows where cold air enters into your home. Make sure you use caulk on all of these potential areas of entry if they are present in your space. If there isn’t an area like this then make sure no one opened up the door while someone was lighting it! That will really cause drafts.