Chests in your farms are safe. Chest outside the farm are only safe it is not blocking the path of an NPC as they go about their schedule. If a chest or any other structure is blocking their path, they will break it, although I've not tried this with a loaded chest yet.