No. Villagers don't take items from any containers - even ones that are their workstations. The only exception is farmer composting plants in his composter and taking the bonemeal it produces. But chests, barrels, smokers, blast furnaces etc are safe.