No. You can leave soil plugs on the lawn after aeration. The plugs will decompose and filter back into the holes left by your aeration machine, depending on their size. Remember, if you have used a spike aeration machine with small holes, you will need to be on the lookout for compaction.
Often, your lawn mower will break up plugs up the next time you mow your lawn. They will generally work back into the soil within two to three weeks, leaving you with little to no clean-up!
Letting the plugs work back into the soil reduces waste and ensures that nutrients are not lost by aeration; just be mindful that the soil can sometimes cause compaction issues.