There could be several reasons why the masking fluid ripped the paper. Here are some of my observations and suggestions that you may consider to help you perfect this technique.
Masking fluid ideally needs to be removed as soon as the painted section surrounding the masked area has dried completely. In my own experience, I have kept the masking fluid on the paper for two days only. If you leave it on for longer you run the risk of damaging the paper.
Masking fluid, unfortunately does not have a very long life span. If the masking fluid is too old, it looses its ability to be peel off easily.
If you add too much water to your masking fluid (yes, you can dilute it a bit), it may cause the fluid to soak too deeply into your paper and when you try to remove the masking, it may rip the paper.
Some watercolor papers are too soft and can’t tolerate the strong adhesive masking fluid, which also causes the paper to rip when you remove the masking.
Please always test your brand of masking fluid on a small scrap of paper before you apply it to the bigger sheet of watercolor paper. Some brands of masking fluid work well on some watercolor papers but can be disasterous on other brands.
I also have to mention here about one misconception. It is believed that masking fluid cannot be applied on wet paper. That’s a myth. You can apply masking fluid on wet paper. The wetter the paper, the more diffused edges masking fluid will create. In my school I have a free tutorial on using masking fluid where I also show what effect you can expect when you apply masking fluid on wet paper. Of course you have to remember about points 4 and 5 – some papers just don’t want to cooperate with specific masking fluids. You always have to test your masking with the paper you’re using to make sure they work well together.