If you don't have much on your PC, you might get away with copying a few files to a few floppies. Chances are though, that you need a lot more space. Thankfully there are several options that make it easy. A very popular tool is the Iomega zip drive. It holds 100 MB of data per cartridge and can easily be transported.
Tape drives are relatively cheap, but slow, and the tapes are very sensitive to heat etc.
Another possibility is to back up to another hard drive, either a second drive in your PC or another hard drive in another PC if your PC is connected to a network. A new and increasingly popular way is to back up to a CD burner. You can get a re-writeable CD burner that will let you reuse the same CD for your backups over and over. You can also use the CD burner to make your own personalized music CDs. My personal recommendation is the Hewlett Packard 8100i.