In December 1999 The Hun of The Hun's Yellow Pages won a lawsuit against a company that was stealing links from these Yellow Pages. The president of the court decided a link-site is a database. Read the lawyer's version here Part from copying being illegal it's not a very decent thing to do either. People work hard to review links and to gather a network of people that submit galleries of a high quality.
Unfair competition is illegal as well and can be fought in court.
In most cases the defendant will have to pay the bill, but a starting-link site doesn't have to means to be able to pay anyway. There are different ways though. The screwthescrewers program is a piece of code that redirects people coming from sites that are known to steal links from other linksites. It can redirect to any place you want. Only traffic coming from sites that steal will be redirected.