I had a look through the 2.0.x kernel source to make sure that I wasn't mistaken about this. Here's a summary of what goes on. This basically puts the request directly into the disk queue code. Both of these functions are file system dependent, inode accessing code.
Swap files are obviously less efficient.
Again, though, this inefficiency is probably not very significant compared to the time to actually swap the block. As far as the usefulness and performance of swap is concerned, well, that depends heavily on the characteristics of the processes executing on your system.