You can do it but you can also leave it on the stem. However, some growers prefer to remove the dried and yellowing leaves for aesthetic purposes and to prevent the spores of fungal diseases from infecting the whole plant through the decaying leaves.