Either it bugs you more than it should, or you're not able to give it the attention it deserves. Carry a notepad, a tape recorder, anything. To cut down stress, get everything out of your head and into a system your mind can trust. Decide on the goal. Decide on the next step.
Most stress stems from people failing to decide instantly on the next step.
You must first define your desired outcome for a project, a phone message, or a newspaper clipping. Finally define the context that will foster that action. Then, ascertain the next step toward that goal. Keep your eye on the goal, make your next step, and remain flexible to what arises. Finally, figure out what you need to achieve that next action.