Every time you stumble, start over. Take it one breath at a time. This is the level of the game where you can actually win. Observe each breath with care and precision, taking it one split second on top of another, with fresh resolve piled one on top of the other. In this way, continuous and unbroken awareness will eventually result.
Mindfulness of breathing is a present-time awareness.
When you are doing it properly, you are aware only of what is occurring in the present. You don't look back and you don't look forward. You forget about the last breath, and you don't anticipate the next one. When the inhalation is just beginning, you don't look ahead to the end of that inhalation. You don't skip forward to the exhalation which is to follow.