Roof replacement is a major home improvement project. New roof replacement can cause disruptions, which is why many homeowners temporarily move out during a roof replacement.
There is no rule that says you absolutely must temporarily relocate during a roof replacement. As long as you stay away from falling roof debris likely to fall in your garden, you will be safe. To avoid being disrupted, plan in advance.