Yes you can use a rhinestone file to cut vinyl. However, since rhinestone files are made up of circles the file will still cut circles even if you use vinyl. If you choose a holographic or glitter vinyl material you can create a "Faux" rhinestone transfer which looks a lot like using real rhinestones but without having to purchase all of the extra supplies.
Below are examples of "Faux" rhinestone transfers:
Holographic Vinyl
Glitter Vinyl