You must turn off the water when changing a shower arm or doing other plumbing activities. After turning off the water, remove the old arm, install the new one, and tighten the nuts with your wrench.
When done, turn the water supply back on and let it run for a few minutes. If everything is secure, you’ll know this by feeling if there are any loose nuts or bolts in the wall that could cause a leak later on.