Down pillows aren’t toxic. Down comes from the fine layer of feathers of birds. Some species of birds have more or less of it. Down’s natural softness and insulation make it a popular pillow filling. It can be pricey, and many people don’t like the ways in which it is sometimes harvested.